[avgbob]

Summary

Focused tracker fix extracted independently from the Gate5 fork and based directly on current upstream main / v1.7.1 (72b3492).

This changes close-contact handling from:

detect candidates
  -> proximity/ghost merge
  -> Hungarian track association

to:

detect candidates
  -> Hungarian track association
  -> conservative proximity/coalescing policy

The reason is simple: proximity suppression cannot know whether two close blobs represent two persistent fingers until association has been performed.

Behavior

The post-association policy is intentionally small:

• two close candidates assigned to two different established tracks survive;
• state 2 (active) and state 3 (lift-pending) both count as established continuity;
• if only one side is established, the established track wins over a close new candidate;
• if neither side has established continuity, the higher pre-penalty raw_w wins, preserving the previous duplicate-rejection behavior;
• the existing strict ghost_dist boundary is preserved;
• the existing per-finger-count ghost-radius scaling is preserved unchanged.

This does not include the later Gate5 birth-grace, sequential-close-birth, or occlusion heuristics. Those remain out of scope.

Why

A physical MSHW0231 pinch capture showed the failure mode: legitimate nearby contacts could both exist as detector candidates, but the pre-association ghost merge destroyed one before Hungarian matching had a chance to associate the pair with two already-established tracks.

The key property this PR pins is therefore:

once two close candidates can be explained as two different established tracks, proximity alone must not delete either contact.

Regression coverage

tracker_coalescing_host_test.c includes the staged real mshw0231-raw.c and directly exercises the policy:

• two established close tracks survive;
• active + lift-pending continuity survives;
• an established track beats a heavier close new candidate;
• ambiguous new candidates still use pre-penalty raw weight;
• exactly ghost_dist remains outside the strict suppression boundary.

tracker_coalescing_source_test.py additionally pins the pipeline order:

raw_hungarian_match()
  -> raw_post_assoc_coalesce()
  -> raw_update_slots()

Both are wired into make -C tests test.

Scope

This PR is independent of #87 and does not include:

• standard-transport CapImg activation;
• installer/profile changes;
• new tracker tuning constants;
• recent-birth grace windows;
• long occlusion recovery;
• Windows classification/merge-group emulation.

The exact clean branch has not yet been physically re-tested after extraction, so I am opening it as a draft rather than extending the prior hardware result beyond what was actually tested.

Windows reverse-engineering provenance

The tracker work is also downstream of the Windows analysis rather than an isolated Linux heuristic exercise:

• Surface MSI mining report — identifies TouchPenProcessor0C19.dll as the software Heat processor bound to MSHW0231&Col02.
• Windows detector alignment — documents the recovered detector thresholds, grid geometry, baseline depth, ghost/velocity radius, edge penalties, and other values compared directly with the Linux tracker.
• Windows golden state machine — anchors the transport/lifecycle side so tracker experiments are performed on the same 0x0c Heat path Windows uses.
• Windows source hashes — provenance for the recovered Microsoft sources behind the analysis.

This PR intentionally extracts only the conservative association-before-coalescing result; the broader Windows-vs-Linux detector reconstruction remains documented in the linked analysis.

Review follow-up

The first review found a real state-1 provisional-contact regression. The branch now preserves the chosen survivor's provisional history when:

• coalescing keeps a candidate assigned to a fresh/empty slot;
• the suppressed candidate owned a state-1 provisional slot; and
• the surviving candidate itself is within the same per-axis Hungarian association range of that provisional slot.

This preserves continuity for the one contact coalescing already chose; it does not create a two-close-new-contact exception.

The host suite now includes the review's reachable full-pipeline reproduction: 30 flat 72x48 baseline frames at raw 220, then six moving frames with disconnected 3x5 strong/weak rectangles (40/80), four cells apart and moving three cells/frame. It also covers winner-at-B/loser-at-A, rejected 0xff assignments, raw-vs-edge-penalized weight, 3+ slot ghost-radius scaling, direct provisional re-home, and an out-of-range negative control.

real_frame_replay_test.c has been migrated to Hungarian -> post-association coalescing, is compiled by the aggregate host gate even without the private corpus, and stale raw_ghost_merge/pre-merge terminology was removed from the driver stage list, pipeline docs, replay diagnostics, and raw-replay fixture docs.

Exact-head CI and physical pinch/contact-birth/motion/lift-re-touch checks remain pending; the PR stays draft until those are available.

[Syax89]

Hi @avgbob, thanks for keeping this separate from #87. I reviewed c116402adf83fad517cf7118b2d037d0e54c7c60 against 72b3492, with two independent reads followed by executable checks against the actual tracker source.

The main idea checks out: in a synthetic full-pipeline pinch sequence, two established contacts converging to four grid cells apart collapse to one on the base revision, while this PR keeps both. That is a useful improvement. I did find two regressions that I would like us to address before merge, plus some test/documentation follow-ups.

1. P2 — Coalescing can discard provisional continuity and indefinitely restart debounce

Location: driver/mshw0231-raw.c:989–997, particularly assigned_slot[loser] = 0xff.

A reachable sequence is:

1. A moving contact has a stronger candidate and a weaker nearby duplicate.
2. Hungarian assigns the weaker candidate to the existing state-1 provisional slot because it is closer to the previous position; the stronger candidate gets an empty slot.
3. Neither slot is established under the state-2/3 rule, so coalescing correctly chooses the stronger candidate by raw weight, but discards the loser's provisional association.
4. raw_update_slots() clears the now-unassigned state-1 slot and starts the winner's empty slot at duration 1.
5. Repeating the pattern alternates provisional slots without ever completing debounce.

This is not a request to preserve two newly born close fingers. The regression is losing the history of the single surviving contact.

Full-pipeline reproduction

The host probe uses the actual mshw0231_raw_consume_samples() → detection → Hungarian → coalescing → slot update → MT path, with the shipped SL4 configuration and default tracker parameters:

• 30 flat baseline frames, 72×48, sample value 220;
• six touch frames containing two disconnected 3×5 rectangles centered four cells apart;
• stronger rectangle value 40, weaker value 80;
• both move three cells per frame; the weaker trails the stronger by four cells;
• stronger center starts at column 20, weaker at 16; rows 18–22, center column ±1;
• time advances 10 jiffies per frame.

Both blobs pass the production gates, with measured raw weights 46680 and 33360. The surviving strong candidate remains within the ordinary three-cell association radius of its preceding provisional position.

Observed comparison:

Touch frame (zero-based)	PR HEAD	Base
0	slot 0, state 1, duration 1; 0 emitted contacts	same
1	slot 1, state 1, duration 1; 0 contacts	slot 0, state 1, duration 2; 0 contacts
2	slot 0, state 1, duration 1; 0 contacts	slot 0, state 2, duration 3; 1 contact
5	slot 1, state 1, duration 1; 0 contacts	slot 0, state 2, duration 6; 1 contact

A separate real-stage-chain probe with weights 6000/5000, separation 300 and movement 200 (grid ×100) reproduces the same loss of provisional continuity. Both the base/head normal and fully ASan/UBSan-instrumented probe builds complete with their expected differential assertions.

The minimal direction seems to be reconciling survivor assignments before slot update: when a winner on an empty slot suppresses a candidate assigned to a state-1 slot, preserve that provisional association if the winner is validly in range. We should keep the intended protection of two established contacts rather than restore destructive pre-association merging. A multi-frame sample-to-MT regression would be valuable here.

2. P2 — The optional real-frame replay target no longer compiles

tests/real_frame_replay_test.c:605 still calls the deleted raw_ghost_merge().

$ make -C tests real_frame_replay_test
real_frame_replay_test.c:605:17: error: implicit declaration of function 'raw_ghost_merge'
make: *** [Makefile:125: real_frame_replay_test] Error 1

The target exits 2 on HEAD, while the same replay source compiles against the exact staged base tracker. local-checks depends on this target, so that entry point is broken too. This happens before corpus detection: the private Windows corpus is not needed to establish the compile failure.

Could we migrate analyse_frame() and its attribution logic to the new associate-then-coalesce ordering? A legacy compatibility shim would make it compile but leave its explanations modeling a different pipeline from production. A compile-only check for this diagnostic would also catch stale helper callers without requiring the external corpus in CI.

Test coverage and smaller follow-ups

The new helper tests are useful as isolated coalescer tests, but their handwritten assigned_slot arrays do not exercise the Hungarian/slot-state interaction. Several established-slot fixtures leave blob_slot_gx/gy at zero, so the actual matcher would not produce those assignments from the same fixture state. That is a coverage boundary, not a separate runtime defect: please keep the unit tests and add reachable stage-chain or full-pipeline cases, particularly the moving provisional-contact sequence above.

Other useful cases are the winner-at-b / loser-at-a branch, rejected 0xff assignments, unequal raw versus penalized weights, and ghost-radius scaling at three or more active slots.

Minor cleanup:

• Add tests/tracker_coalescing_host_test to .gitignore, alongside the other generated host binaries; clean already removes it.
• Update the diagram and parameter description in docs/PIPELINE.md, the stage list near mshw0231_raw_process_samples(), and references in replay comments/fixture documentation that still describe pre-merge before Hungarian or name raw_ghost_merge().

What passed, and what remains unproven

• Default make -C tests test: passed.
• Clean make -C tests SANITIZE=1 test: passed, within the Makefile's existing instrumentation coverage. The raw-stage recipes use separate compile flags, so the additional differential probes explicitly instrumented all their translation units with ASan/UBSan.
• Both kernel modules built against Debian Linux 6.12.111 headers, with BTF generation disabled because the host lacks pahole; no modules were loaded.
• The established-contact convergence positive control passed on HEAD and showed the intended improvement over base.
• No new overflow, lost downstream dependency on blob-array compaction, or change in the per-finger-count radius scaling was established.

I considered the lift-pending/state-3 reacquisition case too, but am not reporting it as a proven phantom-contact bug. State 3 is intentionally still published during the lift window, and a strong close blob can represent either a genuine returning finger or an artifact. A synthetic ambiguity alone is not enough reason to remove that continuity rule. Similarly, capacity stress with artificially pre-filled slots has not established real input-history reachability, so I have not promoted it to another production finding.

These are synthetic host results, not physical qualification or a replay of the unavailable private corpus. After the two regressions are fixed, the exact resulting revision still needs CI execution and hardware checks, including pinch/close contacts, contact birth, motion, lift and re-touch. At review time the GitHub CI run was waiting for approval (action_required), not failing.

Overall: the fix addresses a real weakness in the old ordering, and I would like to retain that improvement. Preserving provisional continuity and updating the replay caller should let us move forward without trading away new-contact recognition or the existing diagnostic tool. Thanks again for the focused contribution.

[avgbob]

Thanks — the state-1 case was a real miss in my first extraction. I reproduced the logic from your sequence and pushed a narrower continuity fix plus the rest of the requested audit cleanup.

Current head: c846dd640c4f8d8c7b05b9c7deb0cf304ba90d6d.

The fix does not exempt two close new contacts from coalescing. Instead, after coalescing has already selected one survivor, if:

• the suppressed candidate owned a state-1 provisional slot,
• the survivor had been assigned to a fresh/state-0 slot, and
• the survivor itself is within the same per-axis Hungarian range of the provisional slot,

then the survivor inherits that provisional slot before the loser is discarded. That preserves debounce/history for the one surviving contact rather than restarting it every frame.

I added the reachable full-pipeline regression using the exact shape from your review: 30 flat 72x48 baseline frames at raw 220, then six frames of disconnected 3x5 rectangles at raw 40/80, four cells apart, moving three cells/frame. The test requires the contact to publish from frame 2 onward and the provisional duration not to restart.

I also added the requested branch cases:

• winner at B / loser at A;
• rejected 0xff assignment;
• unequal raw_w vs edge-penalized w;
• 3+ active-slot ghost-radius scaling;
• direct valid state-1 re-home;
• out-of-range re-home negative control.

The stale diagnostic/tooling pieces are cleaned up too:

• real_frame_replay_test.c now re-runs Hungarian -> post-association coalescing and passes the actual matcher bmd;
• it is now a build prerequisite of the aggregate host gate even when the private corpus is absent;
• .gitignore includes the tracker host binary;
• docs/PIPELINE.md and the source stage list now show Hungarian -> post-association coalescing -> slot update;
• remaining raw-replay fixture/test prose no longer names the deleted raw_ghost_merge()/pre-merge stage.

I left the state-3 and capacity behaviors alone, per your review.

What is still pending is exactly what you called out: CI is action_required, and this exact revision still needs physical pinch/close-contact, contact-birth, motion, lift/re-touch checks before I would call it qualified.

[Syax89]

Thanks — re-verification of c846dd640c4f8d8c7b05b9c7deb0cf304ba90d6d confirms that both original regressions are fixed. The real moving-contact pipeline now retains durations 1–6 and publishes from frame 2; the previous head stayed at duration 1. A separate reachable established-contact convergence sequence preserves two contacts. The replay builds again and uses the actual matcher/coalescer chain.

Two test/diagnostic follow-ups remain:

Medium: the new pipeline fixture leaks 3456 bytes under LSan

tests/tracker_coalescing_host_test.c:264–322 creates a real pipeline device. Processing allocates shid.heatmap_buf (driver/mshw0231-raw.c:1767–1774), but teardown frees only touch_input.

make -C tests clean
make -C tests SANITIZE=1 test

The tracker test fails with LeakSanitizer reporting 3456 bytes in one allocation, aggregate exit 2. A clean fully C-instrumented run reproduces it; the ordinary host suite passes. Please free the fixture-owned buffer using the existing neighboring teardown pattern (e.g. heatmap_pure_test.c:145–149). Production teardown already frees it: this is not a new kernel-driver leak. Disabling leak detection makes the supplementary run pass but is not a fix or a clean sanitizer pass.

Low: replay attribution displays an unscaled ghost radius

tests/real_frame_replay_test.c:403–409 ignores active count. Comparing the exact diagnostic and production functions gives:

Active contacts	Replay diagnostic	Production
0–2	6	6
3	6	4
4–5	6	3

The actual coalescing decision calls production code, so this affects the explanation/attribution rather than the real tracker decision. Please reuse the effective scaling for the displayed diagnostic too.

Further checks: re-home/range/raw-weight/established-preservation/radius mutations are rejected when the known fixture leak is isolated in a generated probe. That probe is separate from the unchanged failing suite. Full-pipeline movement, established convergence, sibling helper cases, explicit replay build, and replay of the locally available 537-frame Windows corpus passed. Both kernel modules compile against Debian 6.12.111 headers with BTF disabled; none loaded.

Coverage note: the committed established-slot cases are mostly handcrafted assignments; retaining a reachable multi-frame convergence regression would strengthen them. Fork/history Gate5/mode3 wording in PIPELINE should not imply this isolated upstream PR provides that installer mode. No speculative state-3/capacity changes are requested. Historical corpus success is not exact-head hardware qualification; physical pinch/birth/motion/lift/re-touch and CI remain pending (workflow action_required). No source edits made.

[avgbob]

@Syax89 I addressed both remaining test/diagnostic findings in the isolated tracker branch. 28bd712 frees the fixture-owned 3456-byte heatmap_buf after the full pipeline regression, matching the neighboring test teardown; 9cc5cc0 updates the replay-only ghost-radius display to the same 1/3/4/5+ active-slot scaling used by production raw_scaled_ghost_dist(). No production tracker change.

Fork CI run 38925926108 passes the normal and ASan/UBSan host gates, whitespace and both kernel builds. No new real-device tracker qualification is implied; PR remains draft pending physical close-contact/pinch/lift/recontact data and your re-review.

[Syax89]

Thanks again for the fixes. Here is the focused follow-up verification.

At exact head 9cc5cc0988cf7e354c0378e1d0a3963bb9610f58, the two previously reported test/diagnostic residuals are resolved: the fixture-owned heatmap buffer is freed, and the replay diagnostic radius agrees with production, including three-contact behavior.

The retained evidence is executable rather than token-only: explicit leak-detection ASan/UBSan passes with the current fixture and fails with the original teardown; the actual diagnostic helper matches the production function in 3120 bounded comparisons; the real establish→converge raster control and moving-birth/rehome control retain the intended behavior. The existing 537-frame corpus replay ran (not SKIP), with four assertions passing. Exact-head CI remains green. These detailed artifacts are in the prior final-check report; this focused follow-up did not repeat the full campaign or change source.

I would resolve the original software review concerns. A committed establish→converge raster regression remains a useful nonblocking coverage follow-up, distinct from a production defect. Physical close-contact/pinch, birth/motion/lift/recontact, lifecycle stability and drop/reset qualification are still pending on this exact pin. Test PR88 separately on its raw profile; no PR87+88 combined build has been qualified and no merge is implied by this note.

[Syax89]

Merged into main via rebase (commit 8ee90fe), together with the docs correction for this PR's raw_mode/raw_input_beta claims and the surrounding qualification tooling. The only conflict was the additive one in tests/Makefile, resolved by keeping both target sets. Host gate green locally (make -C tests test + SANITIZE=1).