Repository navigation
fix(company-research): ground pricing model cues and reject camelCase JSON keys - #64
Merged
Merged
Conversation
…ON keys The Airtable live benchmark tripped the fail-closed pricing contract: the embedded plan JSON key `costPerUserPerMonthInCents` was read as a per-unit model cue and title-cased into the meaningless claim `Peruser`, whose linked raw-JSON excerpt could not ground it. - Require a word boundary around the per-unit pricing cue so camelCase machine keys are never treated as readable pricing prose. - Fail closed in extract_pricing: a cue only becomes the model when its linked excerpt actually grounds the derived value (excerpt_supports). - Add a deterministic regression fixture for costPerUserPerMonthInCents (extraction + contract-level brief) plus a control for valid model evidence. The validator is unchanged; unsupported claims are still rejected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The live 20-company keyless benchmark (airtable.com) failed the fail-closed pricing
contract with:
pricing.model: no linked evidence excerpt supports the claim textThe Airtable pricing HTML embeds plan JSON. The camelCase key
costPerUserPerMonthInCentsmatched the per-unit model cue(?:per|/)\s*(?:user|seat|...)(it matched thePerUserinside the key), wastitle-cased into the meaningless claim
Peruser, and its linked excerpt was the rawJSON fragment where the key stays
costPerUserPerMonthInCents— soexcerpt_supportscould not ground it.
Reproduced deterministically and against the live page
(
model_value='Peruser', excerpt = raw JSON,excerpt_supports -> False).Fix (bounded, harness/extraction only)
tests/run-company-research-keyless-e2e.py:(
(?<![A-Za-z])...(?! [A-Za-z])) so a camelCase machine key is never read as readablepricing prose. Real prose (
charged per seat,$20/user/month) still matches.extract_pricing: a cue only becomes the model when its linkedexcerpt actually grounds the derived value (reuses the validator's
excerpt_supports).If ungrounded, the cue is skipped and extraction continues (or falls back), never
emitting an unsupportable label.
With the fix, Airtable yields a grounded model
Per Seat(excerpt: "plans are chargedper seat") instead of
Peruser.Tests
tests/test-company-research-v2-quality-fixtures.py: new deterministic regressionfixture for
costPerUserPerMonthInCents— extraction-level (model != Peruser,excerpt-grounded) and contract-level (two-page brief validates end to end, pricing
known + grounded) — plus a fail-closed body where the JSON key is the only per-unit
text (no pricing) and a control for valid model prose.
AssertionError: Peruser;GREEN after the fix.
Evidence (all on this head)
test-company-research-v2-scenarios.py+test-company-research-v2-quality-fixtures.pypass.tests/test-package.shpasses (schema fixture, scenarios, quality fixtures, distribution, registry links).git diff --checkclean.DISABLE_TELEMETRY=1 python3 tests/run-company-research-keyless-e2e.py --output ...:schema_validationpassed for all 20 companies,quality_failures: none,unsupported_claim_count: 0, airtablepricing.unknown: falsewith modelPer Seat.tests/company_brief_validator.pyis unchanged; no unsupported claims are accepted.