Repository navigation
fix(2.3): four defects the host suite could not see, found on the device - #18
Merged
Merged
Conversation
…FILL-01 could not pass
First hardware run of this file, and it failed on the app behaving correctly:
Expected: DictationPhase.listening
Actual: DictationPhase.starting
[Dictation 727ms] attached <- and no frame after it
`_FixtureAudioInput` opened a stream and delivered nothing until a test called
`playToEnd()` -- which happens *after* the assertion that the screen reached
`listening`. When 52c2779 redefined that phase as **audio is arriving** rather
than *the input was asked for*, the host doubles were taught to deliver a frame
and this one was not. It had never run on hardware, so nothing said so.
That is the same gap that let a `RangeError` reach a device two commits earlier,
in the same direction: a double gentler than the hardware is a test that cannot
fail. Both are now on Task 2.3's row in the sprint plan.
Playback therefore starts on `onListen` -- when the pipeline subscribes, which is
when a microphone starts delivering -- and `playToEnd()` waits for it instead of
starting it. `onListen` and not a call inside `startStream` because this is a
**broadcast** controller: anything added before the consumer subscribes is
dropped, and the point of the fixture is that its first chunk and the recogniser's
first frame are the same bytes.
The chunk delay also moved to *before* each add. A capture device fills a buffer
and then delivers it, so the first frame lands one period after the input opens --
~50ms behind `attached` in the device logs. It also stops the first `add` running
synchronously inside `onListen`, which would depend on `dart:async` registering
the subscription before invoking the callback. It does; it is not a property worth
resting a device test on.
And the two waits after the tap are now conditions rather than durations. The
demo iPad takes 727ms from tap to attach (190ms opening the microphone, 536ms
loading the model) against a settle of 10 x 25ms, so the assertion was reached
only because `runAsync` overruns its delay. Task 2.2 measured that load varying
359-530ms across ten runs on one host; a fixed wait tuned to one of them is a test
that passes by luck.
Two corrections to the instructions in this file's own doc comment, both of which
cost a failed run: `flutter test` cannot launch on a wirelessly tethered iOS
device and has no `--publish-port` flag despite the error recommending one, so
the `flutter run` form is now written next to it.
Still unverified on hardware -- this fixes what the first run found; it does not
claim the run now passes.
1166 host tests green, analyze and format clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQvrrfnwcfMr1HkZa1Lw8u
…g the tool was not enough
On the device, with both models installed, Gemma 4 E2B answered a four-field
inquiry by calling `get_local_parts_inventory`, writing a correct grounded plan,
leaving the work order at **0 of 4**, and closing with:
"If you require any further assistance or need to record the work order
fields, please let me know."
That is the model behaving well. It knew the tool was there -- it named the
capability -- and treated it as an offer, because the preamble named one tool and
said MUST about it, while `record_work_order_fields` existed only in its own
schema description. One tool carried an order and the other carried a
description. The turn budget was not the constraint: `defaultMaxTurns` is 4 and
one turn had been spent.
So the preamble now orders both, and the closing clause is aimed at the sentence
above -- an offer to record is not a recording, and the technician never sees the
prose version.
The rule this generalises to, written on the constant: **a tool the app depends
on being called is named in the preamble, not only in its own schema.** A schema
tells a model what a tool *is*; the preamble tells it what the job *requires*.
Duplicating the two tool names as literals is a real cost and it is bounded
rather than accepted: `prompt_compiler.dart` cannot import the tool
implementations (retrieval layer; the dependency would run backwards), so
`prompt_compiler_test.dart` now asserts both spellings against
`GetPartsInventoryTool.toolName` and `RecordWorkOrderFieldsTool.toolName`, and
that each is ordered rather than merely mentioned. Without that, a rename leaves
the model under orders to call a tool the registry has never heard of --
recoverable via `ToolCallGuard`, wasteful, and traceable to a string nobody would
think to grep.
Seven goldens moved, and the diff was read rather than accepted: **18 added
lines, all identical, zero deletions** -- the preamble sentence and nothing else.
No scripted response or tool call changed, which is what it should be.
TC-GOLD-02 carries two measured constants that the extra line moves, and its own
comment records that guessing them was the mistake the first time. Re-measured
rather than re-guessed: 153 -> 155 lines, drift region 46 -> 47.
Mutation: removing the sentence fails **13** tests.
1169 host tests green, analyze and format clean.
**Not yet verified on device** -- this is the fix for what the device run found,
and whether the model now calls the tool is exactly the thing only a device can
say.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQvrrfnwcfMr1HkZa1Lw8u
…mixed two jobs
Diagnosed a brake fault, which recorded four fields. Diagnosed a door fault on
the same screen. It overwrote the two fields it had values for and left the
other two behind:
Fault code E-305 <- this job
Replacement parts BELT-330-DRV <- this job
Technician hours 1.5 <- the BRAKE job
Safety checkpoints lockout/tagout <- the BRAKE job
The technician's second inquiry mentioned neither hours nor lockout. Both stale
values still carried `FormFieldOrigin.agent`, so the panel drew them with the
agent marker -- asserting the model had recorded them *for this fault*.
Nothing was invented. Every value really had been produced by the agent, one
inquiry earlier, which is what made it convincing and what makes it worth
preventing: a work order is signed off.
`WorkOrderFormViewModel.reset()` existed and had zero production callers. The
sprint plan carried that as a harmless residue waiting for a "new job" button.
It was not harmless.
`forNewInquiry()` is the narrower operation the screen actually needed:
* the **agent's** fields go, because they answer the previous question;
* what the **technician typed** stays, for the reason `applyUpdates` will not
overwrite it -- their work is what this whole model is built to rank above the
agent's, and a new question about the same job is not permission to discard it.
Emptying the form outright is still `reset()`, and belongs to a new *job*;
* parked **suggestions** go with the agent's fields. A suggestion is the agent's
reading of the previous inquiry offered against a field the technician holds;
carrying it forward presents stale text as a live offer -- the same defect one
indirection deeper;
* **refusals** clear. The panel's line says "the assistant sent N values this
form has no fields for", which is about the run being reported. A test named
"rejections accumulate across runs" pins that within an inquiry; across two it
is wrong.
Called when a diagnosis **starts**. The agent's first `record_work_order_fields`
call can arrive before its answer does, so clearing on completion would erase the
run that just filled the form.
Bound at both levels, and both mutations kill:
* removing the `beginInquiry()` call from the button fails the screen test;
* keeping the suggestion instead of dropping it fails the suggestion test -- and
the failing test is the one that makes that claim, checked rather than assumed
after the first run reported a different test's name.
Also confirmed on the same device run, and it closes the check the previous
commit opened: **`get_local_parts_inventory` still fires** now that both tools
carry a `MUST` -- "BELT-330-DRV is carried but out of stock at Aisle 1, Shelf C"
in the same run that recorded the fields. The second order did not displace the
first.
1174 host tests green, analyze and format clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQvrrfnwcfMr1HkZa1Lw8u
…for it Two prompts designed to force an ambiguity were run against Gemma 4 E2B on the demo iPad, and both times it asked in **prose** rather than through the tool's `clarification` argument. The second is the sharper result: given a genuinely ambiguous door fault it *both* guessed -- recording `E-305` and `BELT-330-DRV` -- *and* then asked the technician which part they had fitted. The two things the argument exists to prevent, in one turn. So the modal cannot be reached by driving the model, and a test that kept trying prompts until one complied would be measuring the prompt it took to get there. This scripts the question straight into the real form viewmodel instead, which is exactly how a payload delivers one in production -- the `asked` half of a call the screen already parses -- and asserts what only hardware can: * the overlay renders over the real screen, and the recording on the same call still lands (the question is additional to the fields, not instead of them); * an option is hittable, and a tap fills the field it was asked about -- checked on the state *and* on the controller, which are synced separately; * dismissing writes nothing and does not reopen. A technician who does not know which part they fitted must be able to say nothing, and saying nothing must not put a guess in a work order. No models, no `--dart-define`s. **What this does not prove is written into the file**: that the model ever asks. The modal is verified as a UI and unverified as an interaction, and the sprint plan says so in those words. One correction, recorded because the reasoning was published before it was checked. The touch-target assertion was written as 44pt against a claim that `OutlinedButton`'s Material 3 default is 40 and would fail. It measures **48.0**: the `minimumSize` constant is indeed `Size(64, 40)`, but default padding and visual density carry the rendered height the rest of the way. A style setting `minimumSize` to 48 was written to "fix" it and then deleted -- it changed nothing, and **its own mutation survived**, which is this task's standing test for whether a fix is a fix. The host assertion stays at 48 even though no code sets it, and the reason is on the test: the number is load-bearing and arrived by accident, so a future change to density, padding or button type could drop it under the HIG minimum with nothing else complaining. 1175 host tests green, analyze and format clean. The device file is unrun -- `integration_test` needs a device and cannot execute on this host, which is the whole reason it exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQvrrfnwcfMr1HkZa1Lw8u
…ather than remembered Eight manual end-to-end checks for a real device, and the reason each exists. Every expectation is taken from `assets/elevator_manual_seed.json` -- the three manual entries, the five SKUs, their stock levels and shelf locations are tabulated at the top -- so a run that disagrees with this file is a finding rather than a judgement call. `BELT-330-DRV` is seeded at zero stock and gets its own scenario: an offline assistant that knows the shelf is empty is the strongest thing this app does, and the check is to read the *prose* rather than the tool card, because a grounded tool result contradicted by the sentence beside it is the failure worth catching. A section on what the recogniser is and is not good at comes before the voice scenarios, because otherwise a model limitation reads as a bug. It carries the measured pairs -- "took me an hour and a half" verbatim, "BRK-990-XP" as "R K 99 YEARS B" -- and the two rules that follow from them: do not dictate SKUs (the part number comes from the manual, which is the design), and say a fault code digit by digit, since `normalizeSpokenDigits` cannot rescue a digit the recogniser never emitted. The model has no digit tokens at all, so every numeral on screen arrived through normalisation. Three scenarios exist to exercise fixes from the device session that nothing else would show by hand: the first word of a dictated sentence surviving (the primer), the technician's typed value outranking the agent's, and one job not smearing into the next. What is already covered by an `integration_test` file is listed rather than re-checked by hand, with the `--publish-port` caveat for wireless iOS attached to the commands rather than left to be rediscovered. Two things are recorded as limits rather than steps: * **the agent will not ask a clarifying question.** Two prompts built to force an ambiguity both produced prose, and the second guessed *and* asked. Trying to provoke the modal by hand is not a good use of a device session; * **R12-F0 scroll release is the one owed check**, and it is stated at the width the README states it: the panel was *unscrollable* during generation because `jumpTo` disposes the active drag -- fixed and bound on the host under both platforms' physics, never run on hardware. The doc separates that from the snap-back symptom, because they have different causes. Linked from the README beside the on-device acceptance test. Also closes **TC-PROV-SET-04** in the sprint plan: it had run green on device with `LLM before: absent`, which is the criterion's weaker half -- installing the STT set alongside nothing cannot disturb an LLM that is not there. Re-run with Gemma already installed, so the coexistence claim was actually put at risk, and it held. The last item on that list is now 1.11's scroll check. 1175 host tests green, analyze clean. No production code in this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQvrrfnwcfMr1HkZa1Lw8u
Emptying the fault description meant holding backspace. A `suffixIcon` on the field now does it, and appears only when there is something to clear -- an always-present button is a permanent invitation to destroy the field sitting a thumb's width from the microphone, and it would put a control that does nothing on the empty screen a technician first meets. `suffixIcon` rather than another button in the row beside the microphone: that row is already tight at the demo device's width, and this task has one recorded overflow from adding chrome to this column. **The tap routes through `_onInquiryEdited`, and both halves of that matter.** `controller.clear()` is a *programmatic* write, so `onChanged` does not fire -- the asymmetry `_onDictation` already documents, arriving through a different door. Calling `clear()` alone leaves the text gone and **Diagnose still live over an empty inquiry**, because the button's enabled state is computed in `build` from `_inquiry.text` and nothing asked for a rebuild. And clearing **is an edit**, so it takes the field on exactly the terms typing does (R0-F1): during a capture it releases the mirror and then stops the microphone, in that order. The alternatives are worse in both directions -- a clear that left the mirror attached is undone by the next partial, and one that left the microphone open goes on filling a field the technician just emptied. The dictation state is deliberately untouched. `start()` resets the transcript and `_onDictation` re-reads the base from the field, so the next capture already begins from what is there -- which, after this, is nothing. That is asserted rather than assumed: if either stopped being true the cleared words would reappear on the next capture, which a technician would read as the button not working. Four tests, and three mutations to show they are not decoration: * `clear()` alone (no rebuild, no stop) -> fails **3**; * `setState(_inquiry.clear)` -- rebuild kept, take-the-field dropped -> fails **2**; * the button always shown -> fails **2**. 1179 host tests green, analyze and format clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQvrrfnwcfMr1HkZa1Lw8u
The README was **3431 lines across twenty chapters** -- an engineering journal, and a good one, but not something anyone evaluating this repository will read. The depth is the most differentiating thing here and none of it is deleted; it moves behind a front page that can be finished in ten minutes. **README, 260 lines:** what the app is, a worked example of one inquiry end to end, a status table, an architecture diagram, six design decisions *with what each cost*, what the device taught that the host suite could not, how to run it, how it is tested, and an index. **docs/, fifteen files, 3202 lines moved losslessly.** Headings promoted one level; each carries a link back. Nothing was rewritten on the way -- a summary of a measurement is not the measurement, and these chapters are where the numbers and the arguments live. Two claims in the new front page were **wrong when first written and corrected before commit**, both by checking rather than remembering: * "10 device integration tests" -- there are **9**. `e2e_fixtures.dart` is shared scaffolding, not a test, and it got counted by `ls`; * "Verified on an iPad Air M4 (iOS 26.5)" -- true of the inference and frame-budget measurements, **not** of the voice and work-order runs, which were made on an iPad Pro 11 running iOS 17.5. Both devices are now named, with which work each carried. A portfolio README claiming a device that did not run the feature is exactly the kind of unearned claim the rest of this repo spends its comments preventing. The old status paragraph was also stale in a way that mattered: it said *"STT and vision are still fakes, and the demo screen ... is the next task"*, three tasks after STT shipped and the screen was built. `VisionEngine` really is still a fake and the status table says so -- it is the one remaining stub. Cross-references that the split broke are now links rather than "see X below": four of them, found by grepping for the pattern rather than by rereading. A link checker over every relative markdown link in the repository reports **0 broken**, including the `docs/device-test-scenarios.md` reference that the extraction had rewritten to the wrong depth. No production code touched. 1179 host tests green, analyze clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQvrrfnwcfMr1HkZa1Lw8u
karolkulesza
added a commit
that referenced
this pull request
Aug 20, 2026
fix(2.3): four defects the host suite could not see, found on the device
karolkulesza
added a commit
that referenced
this pull request
Aug 22, 2026
fix(2.3): four defects the host suite could not see, found on the device
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four defects found by running Task 2.3 on the demo iPad, after PR #16 was already reviewed, accepted and merged. None of them was reachable from the host suite as it stood, and three share one cause: a test double kinder than the hardware is a test that cannot fail.
Every device check Task 2.3 owed is now closed.
What was wrong
IN VIBRATING; later utterances fineInt16List.sublistViewover a mic frame, which is aUint8Listcarved out of a platform message at whatever offset the allocator returnedWorkOrderFormViewModel.reset()had zero production callersrecord_work_order_fieldswas never called0 of 4MUST; the other existed only in its own schemaThe fixes
8f5bbdf— the fixture input never delivered until asked. TC-VOICE-FILL-01's first hardware run failed on the test, not the app:_FixtureAudioInputdelivered nothing untilplayToEnd(), which runs after the assertion that the screen reachedlistening— a phase redefined as audio is arriving while this fixture was left alone and never run on hardware to notice. Playback now starts ononListen; the waits after the tap became conditions rather than durations, because the device takes 727ms to attach against a 10×25ms settle and only reached the assertion becauserunAsyncoverruns.d458721— order the model, don't describe the tool. Gemma 4 E2B closed its answer with "if you… need to record the work order fields, please let me know." It knew the tool existed and treated it as an offer. The rule now on the constant: a tool the app depends on being called is named in the preamble, not only in its own schema. Seven goldens moved; the diff was read rather than accepted — 18 added lines, all identical, zero deletions, so no scripted response or tool call changed.5ca3a45— a new inquiry drops the agent's fields. The technician's typed values stay, parked suggestions go with the agent's fields, refusals clear. Called when a diagnosis starts: the first tool call can arrive before the answer, so clearing at the end would erase the run that just filled the form.b9ed5f7— TC-UI-CLAR-01 on hardware. Scripts theaskedhalf of a tool payload into the real viewmodel rather than driving the model, because the model will not ask: two prompts built to force an ambiguity both produced prose, and the second guessed (E-305,BELT-330-DRV) and asked — the two things theclarificationargument exists to prevent, in one turn.How the STT fix was found
Two earlier answers were wrong. Both were reasoned from reading the code; neither changed the technician's report. What ended it was instrumenting the one quantity nobody had — when the technician actually started speaking. Onset landed 1.8s after the first audio frame with
dropped 0B, which retired the whole capture path in one line.A host reproduction (real weights,
say-synthesised audio, the runtime called directly) then reproducedIN VIBRATINGexactly, and went on to refute three further hypotheses before any could become fixes three, four and five: leading silence 0–5000ms, room noise at four amplitudes, and a larger model (the 2023-06-26 zipformer answers "HAVE BEEN VIBRATING"). Only speech primes it. The primer is one second taken from a fixture the bundle already ships, so the app grew by 32KB.Both halves are asserted, deliberately: the bare run must lose "cabin" and the primed run must keep it. If the model is ever fixed upstream, the first assertion fails and says to delete the workaround.
Verification
beginInquiry()removed → screen test fails; suggestion kept → suggestion test fails;_peakOfreverted → 23 fail where it previously failed noneKnown limitation, stated rather than implied
The clarification modal is verified as a UI and unverified as an interaction. The path that opens it depends on the model volunteering a
clarification, and this one prefers prose.integration_test/clarification_test.dartcarries that in its own doc comment.🤖 Generated with Claude Code