fix(search,core): a word's length decided whether its accents mattered

[lessevv]

Search picks its matching arm by the needle's length — three characters is FTS5's trigram minimum, and anything shorter falls back to a LIKE over the same indexed body. The two arms did not compare alike. FTS5's tokeniser folds case over the whole of Unicode; SQLite's LIKE and LOWER() fold ASCII and stop, and no pragma widens them (PRAGMA case_sensitive_like can only turn folding off, and icu_load_collation answers no such function in this build).

So the length of a word decided whether its accents mattered, and nothing on the screen said why.

Before and after

Over one transaction whose counterparty is MÖRK BAR, through SearchQuery::search():

needle	characters	arm	before	after
örk	3	FTS5 MATCH	[1]	[1]
ör	2	LIKE	[]	[1]

Same two row sets, ÖRK/ÖR — both found before and after, which is the half that always worked and is why nobody caught this.

Non-ASCII beyond Latin-1, measured on the shipped SQLite 3.53.3 with a 50,000-row table:

needle	body LIKE '%n%'	LOWER(body) LIKE LOWER('%n%')	beatrax_fold(body) LIKE beatrax_fold('%n%')
ör	0	0	1
ölk	0	0	1
αθη	0	0	1
моск	0	0	1

The middle column is what the LIKE arm did.

The change

"Ignore case" is now one function. Modules/Core/Public/Support/UnicodeFolding::of() is the whole rule (mb_strtolower, UTF-8); beatrax_fold is that same method registered as a SQLite user function so SQL can call it. LikeNeedle folds both sides of every predicate it builds through it.

The ASCII fold is gone, not matched: LOWER( no longer appears in any SQL under Modules/, and a contract case keeps it that way. startsWithAnyCase lost its suffix because every predicate now folds, so the distinction it named no longer exists.

The palette's PHP-side matching (EntityNameSearch, SearchTokenFilters, DidYouMeanSuggester) reads the same method, so the counterparty section and the goal / pot / category / recurring sections stop disagreeing about ölkanne — the second split the docs recorded.

Where it is bound, and every connection path covered

A user function is per-connection, which is the whole risk of this approach. It is bound in exactly one place — Modules/Core/Internal/Providers/UnicodeFoldingProvider — on ConnectionEstablished, which is the one event every way of getting a connection passes through. Enumerated against Illuminate\Database\DatabaseManager, not guessed:

How a connection appears	Where the event comes from	Covered by
First resolve of a named connection	DatabaseManager::connection()	the listener
A second, inline-config connection	connectUsing() / build()	the listener
An explicit reconnect	DatabaseManager::reconnect()	the listener
A dropped connection re-opened mid-query	Connection::run() → reconnectIfMissingConnection() → the manager's reconnector	the listener
A connection opened before the listener existed (the desktop: NativePHP opens nativephp in its own boot() and the resolved connection is cached)	replayed by the existing AlreadyOpenConnectionsProvider	the listener
One of those the replay must skip — it re-writes PRAGMA journal_mode, illegal inside a transaction	—	the provider's own Application::booted() walk
Queue worker, scheduler, artisan, sync:serve, mobile and desktop shells	each boots the same providers in its own process	the listener

The one PDO nothing reaches is Modules/DevMode/Internal/Sql/IsolatedSelectProcess, which opens the file itself in a child process to run one developer-typed SELECT under a timeout. It already carries the same exemption for PRAGMA busy_timeout; a folded query typed into that panel is refused by name, and the exemption is stated in the guard and the docs.

What happens when it is missing

The query fails. It does not fall back. A fallback to LOWER() would be the ASCII-only comparison this removes — fewer rows, no signal, the original defect hidden behind its own fix. UnicodeFolding::registerOn() refuses at connect time if the PDO cannot carry a user function, and a connection that somehow escaped registration raises SQLite's own no such function: beatrax_fold, naming the connection. Both are loud; neither is an empty result set.

Proof — the two inversions

1. The search test is red without the fix. Reverting LikeNeedle::predicate() to column LIKE ? ESCAPE '\' and undoing the EntityNameSearch fold:

✓ it still reaches an ASCII counterparty at both lengths          <- positive control
⨯ it reaches an accented counterparty ... (örk found, ör did not)
⨯ it reaches a Cyrillic note ...
⨯ it reaches a Greek description ...
⨯ it reaches an accented goal by the reader own lower-case spelling
Tests: 4 failed, 1 passed

The ASCII control stays green, so the four failures are the fold and not the fixture.

2. The guard is red with the registration removed. Commenting out $this->app->register(UnicodeFoldingProvider::class) — the registration actually taken out, not reasoned about:

⨯ answers on the connection the application hands out
⨯ answers on every connection config/database.php configures
⨯ answers on a connection DatabaseManager::reconnect() re-made
⨯ answers on a connection that lost its PDO and re-opened inside the query
⨯ answers on a connection that was already open before the listener existed
⨯ answers inside a job the queue worker ran
⨯ answers in a second process that booted the console kernel
✓ leaves no SQL in the tree that folds case any other way
Tests: 7 failed, 1 passed

Each failure reads SQLSTATE[HY000]: General error: 1 no such function: beatrax_fold (Connection: …) — which is the refusal, demonstrated rather than described.

tests/Contracts/AFoldedQueryCannotRunOnAnUnfoldedConnectionArchTest.php enumerates connections from config/database.php rather than naming them (each redirected at :memory: so it never opens the reader's own ledger), pins the resulting list so a new connection has to be looked at, carries its own negative control on the replay case, and gives the tree walk a positive control so "no offenders" and "read nothing" cannot look the same.

Cost

Folding a column defeats an index on it. Neither arm ever used one — LIKE '%needle%' cannot, and the previous LOWER(name) LIKE 'needle%' could not either. What it adds is one PHP call per row already being read: measured over a 50,000-row table scanned end to end, 24.2 ms plain against 56.2 ms folded. The LIKE arm's scan is bounded by FtsCandidateResolver::LIKE_FALLBACK_CANDIDATE_CAP and is only reached by a needle too short to tokenise.

What still folds independently, named

• Modules/Counterparties/.../CounterpartySlugResolver uses strtolower(self::toAscii(...)). Deliberate and unchanged: it derives a URL slug, where ASCII is the point, not a reader's search needle.
• Modules/Ingestion normalisation lower-cases and strips diacritics per A3-R3. A different rule for a different job — search folds case and keeps accents.
• The two arms' residual difference: the LIKE arm folds through mb_strtolower, the FTS arm through FTS5's own table. They agree across Latin, Greek and Cyrillic. They differ where a lowercase mapping expands — mb_strtolower('İ') is i plus a combining dot, FTS5 leaves İ alone — and neither arm finds istanbul in İSTANBUL. A gap in both, not a disagreement between them, recorded in the docs.

Spec

The spec said nothing about case folding in B9 — the page described two arms and never said how either compares. Since "search ignores case, including accents" is reader-visible behaviour, beatrax-app/spec#112 added B9-R19 and is merged:

Case-insensitive matching MUST fold over the whole of Unicode rather than ASCII alone, and MUST apply the same fold in every search arm, so that a query's length never changes which rows a word reaches. Folding case MUST NOT strip diacritics.

Spec: B9-R19

Docs

• New: .docs/architecture/case-folding-is-one-function.md — the measurement, the function, every connection path, the refusal-over-fallback decision, the residuals, the cost.
• .docs/features/search/architecture.md — "Where the two arms disagree" recorded this as known and deliberately not fixed, and named a UDF as the fix somebody would have to take as a decision about the application's connections. Rewritten as "How the two arms compare".

Gates

pint --test, phpstan analyse (level 10 strict), composer analyse:deps all clean. Tests run by file: the whole Modules/Search suite (178 passed), the new folding test, the new contract guard, BoundaryArchTest, CommentPolicyArchTest, CommentsNameSymbolsThatExistArchTest, OneConnectionHoldsOneBusyTimeoutArchTest, and the Core connection/pragma tests.

[github-actions]

📎 Spec references

This PR cites the following spec identifiers:

• B9-R19 — Case-insensitive matching MUST fold over the whole of Unicode rather than ASCII alone, and MUST apply the same fold in every search arm, …

_{Updated automatically on each push.}

[lessevv]

Both blockers addressed in fix(core): name the exception a connection that cannot fold raises.

php:S112 ×2 — the generic exception. The refusal itself stands; Sonar was objecting to the type, not the behaviour. Modules/Core/Public/Exceptions/UnicodeFoldingUnavailableException is the dedicated class, and it carries public readonly string $connectionName rather than keeping the answer only inside the message — which connection came up short is the one question a caller has. Two named constructors for the two ways to arrive:

Why	Constructor
The PDO is not a Pdo\Sqlite, so it has no createFunction()	notASqlitePdo()
SQLite itself declined the registration	driverRefused()

The first is not hypothetical: new PDO('sqlite:…') returns a base PDO, which is how Laravel builds a connection below PHP 8.4 and how anything opening the file outside the framework does at any version. The messages still name the connection.

new_coverage 76.5%. Your guess was right — the registerOn() throws were bare. Measured rather than assumed: **/tests/** is in sonar.coverage.exclusions, so only six production files count. Clover over the changed set, before/after:

100.0%  15/15  Modules/Core/Public/Support/UnicodeFolding.php
100.0%  10/10  Modules/Core/Internal/Providers/UnicodeFoldingProvider.php
100.0%  13/13  Modules/Core/Public/Exceptions/UnicodeFoldingUnavailableException.php
100.0%  19/19  Modules/Search/Internal/Services/LikeNeedle.php

The four Search files still show a handful of uncovered lines (DidYouMeanSuggester 130/134, EntityNameSearch 123/128, FtsCandidateResolver 93, SearchTokenFilters 110) — I checked each against git diff -U0 origin/main, and none is a line this PR added or changed, so none counts toward new_coverage.

No assertion-free tests. All four newly-covered branches are reached with real objects, not mocks:

• notASqlitePdo — a real new PDO('sqlite::memory:').
• driverRefused — Modules/Core/tests/Support/RefusingSqlitePdo, a genuine Pdo\Sqlite subclass overriding createFunction() to return false. Needed because SQLite returns false only for an arity it rejects, and production always passes 1. (Pdo\Sqlite is non-final and the method is non-final — checked before writing it.)
• The provider's non-SQLite early return — a connection configured driver: mysql holding the PDO the first case proves throws, so a green line is the driver check declining, not the registration quietly succeeding.
• The fold's null arm — select beatrax_fold(NULL), asserting NULL. A NULL folding to '' would make a row with no note match a search for the empty string.

The determinism case asserts too: SQLite refuses a non-deterministic function inside an index expression, so create index … on notes(beatrax_fold(body)) succeeding is the proof the flag was passed.

One thing that surfaced on the way: composer analyse:deps went red on Unknown Symbol: Pdo. The checker's symbol list has PDO, not Pdo, so Pdo::connect() reads as undeclared. Switched to new Pdo\Sqlite(…), which is the same object. Green again.

Gates re-run whole: pint --test, phpstan analyse level 10 strict, composer analyse:deps, plus CommentPolicyArchTest, CommentsNameSymbolsThatExistArchTest, ATestHelperNameIsOwnedByOneFileArchTest, ATestDoubleTheAutoloaderCannotFindArchTest, BoundaryArchTest and the new unit file — 109 passed.

Noted on the flaky coverage shards; I will re-run a shard before believing it if one goes red on a file I did not touch.

[lessevv]

PublicSurfaceArchTest was right — moved to Modules/Core/Internal/Exceptions/, no pin added.

Why Internal and not a pin. The question was whether an outside consumer exists that a fully-qualified-name scan cannot see. It does not, and that is checkable rather than a matter of taste: the only method that throws is UnicodeFolding::registerOn(), and the contract guard added earlier in this branch pins that method to exactly one caller —

expect($scan['hits']['registrations'])
    ->toBe(['Modules/Core/Internal/Providers/UnicodeFoldingProvider.php']);

— so a consumer in another module cannot appear without failing a guard on this same branch. The two things an outside catcher could plausibly want are answered elsewhere anyway: falling back to ASCII folding is the defect this branch removes, and reporting a broken install is Core's own doctor seam, which is in Core. The pinned list only shrinks, and a class written in this PR is the weakest possible case for adding to it.

Two things checked rather than assumed while moving it:

• Internal/Exceptions/ is the convention in ten modules, and 22 other Public/ classes already throw their own module's Internal exception. So a Public method throwing an Internal type is idiomatic here — which settled a shape question I would otherwise have guessed at. I left registerOn() on UnicodeFolding rather than splitting the registration away from the fold it installs. UnicodeFolding itself stays Public and stays consumed: Modules/Search reads of() and sql().
• Both CI reds were this same finding, not one real and one flake. quality shard 3 and coverage shard 3 each reported PublicSurfaceArchTest with 7343 passed / 1 failed. The coverage red was not the known shard flake, so re-running it would have burned a cycle without fixing anything — worth knowing before anyone re-runs it out of habit.

Verification, run at the directory level this time rather than at the guards my diff suggested:

Check	Result
vendor/bin/pest tests/Contracts (whole directory)	1673 passed, exit 0
pint --test	passed
phpstan analyse (real config, level 10 strict)	No errors
composer analyse:deps	no unknown symbols
Modules/Core/tests/Unit/AConnectionThatCannotFoldIsRefusedAsItOpensTest	6 passed

Coverage is unaffected by the move — Internal/ is in sonar.sources and is not coverage-excluded, and the same tests reach it:

100.0%  13/13  Modules/Core/Internal/Exceptions/UnicodeFoldingUnavailableException.php
100.0%  10/10  Modules/Core/Internal/Providers/UnicodeFoldingProvider.php
100.0%  15/15  Modules/Core/Public/Support/UnicodeFolding.php

One process note for the record: I edited the docs page after launching the contracts run, so that batch predated the edit. Re-ran CommentPolicyArchTest and CommentsNameSymbolsThatExistArchTest (the two that read .docs/) on top — 22 passed — rather than letting a green result stand for a tree it had not seen.

Point taken on scope generally. A rule keyed on where a file lives is invisible to a search of the diff, which is exactly why the directory-level run is the right default.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud