fix(conversion): match accounts on a normalized company name (#626)

.changeset/lead-conversion-normalized-account-match.md

@@ -0,0 +1,50 @@
+---
+'hotcrm': patch
+---
+
+Lead conversion now reuses an existing account when the company name differs only in capitalisation or spacing — converting a lead for `ACME  Corp` attaches it to your `Acme Corp` account instead of creating a second one.
+
+The flow used to dedupe on the raw `crm_account.name`, so every case or spacing
+variant produced its own account and the account list slowly filled with
+near-duplicates of the same company. Matching is now **normalize-then-exact**:
+both sides are lower-cased, trimmed and have runs of internal whitespace
+collapsed before comparison. It is deliberately not fuzzy — `Acme Corp` and
+`Acme Corporation` remain two different companies — because ranking candidate
+matches needs a human review step this app does not have.
+
+The comparison happens on two new derived columns, `crm_account.name_normalized`
+and `crm_lead.company_normalized`, maintained by the `account_protection` and
+`lead_duplicate_check` hooks. Both are read-only and hidden: nobody authors
+them, and the display values (`name`, `company`) are untouched, so the account
+created from a lead still carries the company name exactly as it was typed.
+
+Storing the keys is forced rather than preferred, and each alternative was
+measured against 17.0.0-rc.1 (the measurements are pinned in
+`test/account-name-normalized-match.test.ts`, so a platform upgrade that changes
+any of them fails loudly instead of leaving stale reasoning in a comment):
+
+- a flow template cannot fold a string — the automation engine's token resolver
+  understands only `NOW()` / `TODAY()`, so `{LOWER(x)}`, `{TRIM(x)}` and
+  `{x.toLowerCase()}` all resolve to nothing;
+- a formula field has no physical column, so nothing can filter on it;
+- `$regex` is not a case-insensitive equality on SQL at all — it compiles to a
+  substring `LIKE`, which also matches `Not Acme Corp Ltd`, cannot collapse
+  whitespace, and cannot use an index.
+
+`name_normalized` carries a plain index, **not** a unique one. Account-name
+uniqueness already lives, per organization, on `name` (#625); a unique
+normalized column would subsume that constraint and re-open a decision made one
+release earlier, for a guarantee this change does not need. It would also be
+impossible to add to any deployment that already holds both spellings, since
+creating a unique index fails on existing duplicates.
+
+**Upgrading an existing deployment.** Both columns start empty on rows written
+before this version. An account with no key is invisible to the lookup, so
+conversion would create *more* duplicates than before, not fewer; a lead with no
+key stops its conversion outright, because the automation engine refuses to run
+a query whose filter resolved to nothing rather than widening it. A one-time
+backfill (re-save each account and open lead; the hooks derive the keys) is
+documented in `docs/MAINTENANCE.md` §3.3. Fresh installs need nothing: seed
+writes run lifecycle hooks, so every row is stamped as it is created.
+
+Fixes #626.

content/docs/sales/accounts.mdx

@@ -59,7 +59,7 @@ The hierarchy gives you:
 
 ## Built-in rules
 
-- The account **name must be unique within your organization**. You'll get an error if you try to create a second account with a name you already use. Another organization on the same platform is free to have its own "Acme Corp" — the rule is scoped to your data, not the whole platform. Names are compared exactly as typed, so `Acme Corp` and `ACME  Corp` currently count as two different accounts.
+- The account **name must be unique within your organization**. You'll get an error if you try to create a second account with a name you already use. Another organization on the same platform is free to have its own "Acme Corp" — the rule is scoped to your data, not the whole platform. The uniqueness check compares names exactly as typed, so `Acme Corp` and `ACME  Corp` can both exist as separate accounts. Lead conversion is more forgiving: it ignores capitalisation and extra spaces when looking for an account to reuse (see [Leads](./leads)), so converting a lead for `ACME  Corp` attaches it to your existing `Acme Corp` rather than creating a near-duplicate.
 - **Annual revenue must be zero or positive** — no negative numbers allowed.
 - You **cannot delete** an account that has open opportunities or active contracts. Set it to *Inactive* instead (`is_active = false`).
 

content/docs/sales/accounts.zh-Hans.mdx

@@ -59,7 +59,7 @@ Acme Corp Global
 
 ## 内置规则
 
-- 客户**名称在你所在的组织内必须唯一**。如果你尝试用一个已存在的名称再建一个客户，会收到错误。同一平台上的其他组织可以有自己的 "Acme Corp" —— 这条规则只作用于你自己的数据，而不是整个平台。名称按原样精确比较，因此 `Acme Corp` 与 `ACME  Corp` 目前算作两个不同的客户。
+- 客户**名称在你所在的组织内必须唯一**。如果你尝试用一个已存在的名称再建一个客户，会收到错误。同一平台上的其他组织可以有自己的 "Acme Corp" —— 这条规则只作用于你自己的数据，而不是整个平台。唯一性校验按原样精确比较名称，因此 `Acme Corp` 与 `ACME  Corp` 可以作为两个独立的客户同时存在。线索转换则更宽松：它在寻找可复用的客户时会忽略大小写和多余空格（见[线索](./leads)），因此转换公司名为 `ACME  Corp` 的线索时，会挂到你已有的 `Acme Corp` 上，而不是新建一个近似重复的客户。
 - **年营收必须为零或正数**——不允许负数。
 - 你**无法删除**一个有开放商机或活跃合同的客户。改为将其设为*非活跃*（`is_active = false`）。
 

content/docs/sales/accounts.zh-Hant.mdx

@@ -59,7 +59,7 @@ Acme Corp Global
 
 ## 內建規則
 
-- 客戶**名稱在你所屬的組織內必須唯一**。如果你嘗試用一個已存在的名稱再建一個客戶，會收到錯誤。同一平台上的其他組織可以有自己的 "Acme Corp" —— 這條規則只作用於你自己的資料，而不是整個平台。名稱按原樣精確比較，因此 `Acme Corp` 與 `ACME  Corp` 目前算作兩個不同的客戶。
+- 客戶**名稱在你所屬的組織內必須唯一**。如果你嘗試用一個已存在的名稱再建一個客戶，會收到錯誤。同一平台上的其他組織可以有自己的 "Acme Corp" —— 這條規則只作用於你自己的資料，而不是整個平台。唯一性檢查按原樣精確比較名稱，因此 `Acme Corp` 與 `ACME  Corp` 可以作為兩個獨立的客戶同時存在。潛在客戶轉換則更寬鬆：它在尋找可重用的客戶時會忽略大小寫與多餘空格（見[潛在客戶](./leads)），因此轉換公司名為 `ACME  Corp` 的潛在客戶時，會掛到你既有的 `Acme Corp` 上，而不是新建一個近似重複的客戶。
 - **年營收必須為零或正數**——不允許負數。
 - 你**無法刪除**一個有開放商機或活躍合約的客戶。改為將其設為*非活躍*（`is_active = false`）。
 

content/docs/sales/leads.mdx

@@ -94,7 +94,7 @@ When a lead is qualified, click **Convert**. A small form pops up asking:
 
 The system then, in a single step:
 
-1. Creates an **Account** from the lead's company information.
+1. Creates an **Account** from the lead's company information — or reuses the one you already have. The match ignores capitalisation and extra spaces, so a lead for `ACME  Corp` lands on your existing `Acme Corp` instead of creating a second account. It is not a fuzzy match: `Acme Corp` and `Acme Corporation` are still two different companies.
 2. Creates a **Contact** from the lead's person information, attached to that account, marked as primary.
 3. *(Optional)* Creates an **Opportunity** in the *Prospecting* stage with a 90-day close date.
 4. Marks the lead as **Converted** and locks it.

content/docs/sales/leads.zh-Hans.mdx

@@ -94,7 +94,7 @@ description: 在入站咨询成为真正交易之前对其进行捕获、评分
 
 然后系统会一步完成：
 
-1. 根据线索的公司信息创建一个**客户**。
+1. 根据线索的公司信息创建一个**客户**——如果已经有同名客户，则直接复用。匹配时忽略大小写和多余空格，因此公司名为 `ACME  Corp` 的线索会落到你已有的 `Acme Corp` 上，而不是再建一个客户。这不是模糊匹配：`Acme Corp` 与 `Acme Corporation` 仍然是两家不同的公司。
 2. 根据线索的个人信息创建一个**联系人**，附加到该客户上，并标记为主要联系人。
 3. *（可选）* 在*开发期*阶段创建一个**商机**，成交日期设为 90 天后。
 4. 将线索标记为**已转换**并锁定它。

content/docs/sales/leads.zh-Hant.mdx

@@ -94,7 +94,7 @@ description: 在入站詢問成為真正交易之前對其進行擷取、評分
 
 然後系統會一步完成：
 
-1. 根據潛在客戶的公司資訊建立一個**客戶**。
+1. 根據潛在客戶的公司資訊建立一個**客戶**——若已有同名客戶，則直接重用。比對時忽略大小寫與多餘空格，因此公司名為 `ACME  Corp` 的潛在客戶會落到你既有的 `Acme Corp` 上，而不是再建一個客戶。這不是模糊比對：`Acme Corp` 與 `Acme Corporation` 仍然是兩家不同的公司。
 2. 根據潛在客戶的個人資訊建立一個**聯絡人**，附加到該客戶上，並標記為主要聯絡人。
 3. *（可選）* 在*開發期*階段建立一個**商機**，成交日期設為 90 天後。
 4. 將潛在客戶標記為**已轉換**並鎖定它。

docs/MAINTENANCE.md

@@ -172,6 +172,74 @@ orphans (the `__search` class above is excluded until #3955 lands):
 `--allow-destructive` drops columns irreversibly — never run it without the
 backup from step 2, and never against a database whose plan you have not read.
 
+### 3.3 Backfilling a hook-derived column
+
+Some HotCRM columns are **derived**: no one authors them, a lifecycle hook
+computes them from another field on every write. Two of them are match keys the
+lead-conversion flow reads (#626):
+
+| Column | Derived from | Writer |
+| --- | --- | --- |
+| `crm_account.name_normalized` | `crm_account.name` | `account_protection` |
+| `crm_lead.company_normalized` | `crm_lead.company` | `lead_duplicate_check` |
+
+A **fresh install needs nothing here.** Seed writes run lifecycle hooks
+(`skipTriggers` suppresses record-change automation, not hooks — measured in
+#617), so every seeded and every subsequently created row gets its key stamped
+on insert.
+
+An **in-place upgrade does**. A row written before the column existed holds
+`NULL`, and the two objects then fail in two different ways — both measured, and
+worth knowing apart when triaging:
+
+- **An account with no key is invisible to the match.** The conversion finds
+  nothing and creates a *second* account for a company that already has one —
+  silently. This is the failure that makes the backfill non-optional: it is
+  more duplicates than the behaviour the change replaced.
+- **A lead with no key stops the conversion.** The filter value resolves to
+  nothing, and `get_record` refuses to run rather than widen the query:
+  *"refusing to run — 1 filter condition(s) resolved to nothing and were dropped
+  from the query: `{leadRecord.company_normalized}` (at name_normalized)"*. The
+  run is recorded failed and the lead stays unconverted. Loud, and the message
+  names the missing key — re-save that lead and convert again.
+
+> [!NOTE]
+> **This section is a contingency, not a step in any current upgrade.** HotCRM's
+> deployment shape today is **fresh installs only**, which is the whole reason
+> the procedure below is documented rather than automated, and the reason
+> `name_normalized` carries no unique index (see `src/objects/account.object.ts`).
+> Both conclusions are conditional on that premise. If HotCRM ever acquires
+> long-lived installs that upgrade in place, re-read this section and the index
+> decision together — neither is a universal judgement.
+
+The backfill is a **re-save**: write a row's own `name` / `company` back to it
+and the hook derives the key. Nothing else about the row changes, and re-saving
+a row that already has a key is a no-op — so the pass is idempotent and safe to
+repeat or to run over every row rather than hunting for the empty ones.
+
+1. **Take a database backup.** This rewrites every account and lead row.
+2. **Read the rows.** Any path you already use is fine — a Console export, a
+   `GET` against the record API, or a direct read replica query. You need only
+   `id` plus the source field (`name` for accounts, `company` for leads).
+3. **Write each row back to itself**, one `PATCH` per record, against the
+   record endpoint `PATCH {basePath}/data/:object/:id` (`basePath` is
+   `/api/v1`):
+
+   ```bash
+   curl -s -X PATCH "$HOTCRM/api/v1/data/crm_account/$ID" \
+     -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
+     -d '{"name": "Acme Corp"}'          # the row's OWN current name
+   ```
+
+   Do **not** send `name_normalized` itself: it is `readonly`, so an incoming
+   value is stripped, and the hook would overwrite it anyway. Repeat for
+   `crm_lead` with `{"company": "…"}`; converted leads can be skipped
+   (`is_converted = true`) — nothing converts them again.
+4. **Verify.** No row should be left with an empty key, and the end-to-end
+   check is the one that matters: convert a lead whose company differs from an
+   existing account only in case or spacing, and confirm it **reuses** that
+   account instead of creating a second one.
+
 ## 4. Seed-data staleness — the #1 HotCRM pitfall
 
 Stale seed data is the most common cause of "Studio shows a red

src/flows/lead-conversion.flow.ts

@@ -76,10 +76,47 @@ export const LeadConversionFlow: Flow = {
     },
     {
       // Account dedupe: before creating a new account, look for an existing one
-      // with the same company name. Exact-name match (case/whitespace sensitive)
-      // — the standard lightweight dedupe; fuzzy matching is out of scope here.
+      // with the same company name — NORMALIZED (#626). This used to compare
+      // `crm_account.name` against the raw `{leadRecord.company}`, so
+      // "Acme Corp" and "ACME  Corp" produced two accounts.
+      //
+      // Both sides of this comparison are stored, hook-maintained columns, and
+      // that is forced rather than chosen: a flow template cannot normalize
+      // ANYTHING. `service-automation`'s `resolveToken` recognises exactly one
+      // function form — `NOW()` / `TODAY()` — and every bare identifier in the
+      // expression fallback is substituted before evaluation, so no string
+      // method is reachable either: `{LOWER(x)}`, `{TRIM(x)}` and
+      // `{x.toLowerCase()}` all resolve to `undefined`, and an unwrapped
+      // `LOWER({x})` interpolates literally to "LOWER(Acme Corp)". A formula
+      // field is no help either — it has no physical column to filter on. So
+      // the producer canonicalizes (`account_protection`,
+      // `lead_duplicate_check`) and this node does a plain, indexed equality
+      // match. `test/account-name-normalized-match.test.ts` re-measures all of
+      // that rather than trusting this paragraph.
+      //
+      // Normalize-then-EXACT only: lower + trim + collapse internal
+      // whitespace. Fuzzy matching stays out of scope, as before.
+      //
+      // If the lead carries NO `company_normalized`, this node does not fall
+      // back and does not match everything — `get_record` REFUSES TO RUN:
+      //
+      //   get_record: refusing to run — 1 filter condition(s) resolved to
+      //   nothing and were dropped from the query: `{leadRecord.company_
+      //   normalized}` (at name_normalized). An absent condition does not
+      //   narrow a query, it widens it …
+      //
+      // (measured on 17.0.0-rc.1; pinned in the test file). That is the right
+      // failure: the only way to reach it is a lead row written before this
+      // change, which is what the backfill in docs/MAINTENANCE.md §3.3 exists
+      // for, and a conversion that stops with that message is far cheaper to
+      // diagnose than one that quietly creates a duplicate account.
+      //
+      // Deliberately NOT papered over with a second, case-sensitive lookup on
+      // the raw `name`: a missing key means the producer did not run, and a
+      // tolerant consumer path would hide that while restoring the exact bug
+      // this node exists to fix.
       id: 'find_account', type: 'get_record', label: 'Find Existing Account',
-      config: { objectName: 'crm_account', filter: { name: '{leadRecord.company}' }, outputVariable: 'matchedAccount' },
+      config: { objectName: 'crm_account', filter: { name_normalized: '{leadRecord.company_normalized}' }, outputVariable: 'matchedAccount' },
     },
     {
       id: 'decision_account', type: 'decision', label: 'Account Already Exists?',
@@ -89,6 +126,12 @@ export const LeadConversionFlow: Flow = {
       // NEW-account branch. outputVariable is `createdAccount`; the assignment
       // below normalizes both branches onto a single `accountId` id string so
       // downstream nodes don't need to know which path ran.
+      //
+      // `name` carries the lead's company VERBATIM — the display value. The
+      // match key `name_normalized` is deliberately absent: it is readonly and
+      // hook-owned, and `account_protection` derives it from the `name` written
+      // here, so an account created by this node is immediately findable by the
+      // next conversion.
       id: 'create_account', type: 'create_record', label: 'Create Account',
       config: {
         objectName: 'crm_account',