Skip to content

fix(company-research): ground pricing model cues and reject camelCase JSON keys - #64

Merged
replynodes-ai merged 1 commit into
mainfrom
fix/company-research-pricing-model-grounding
Oct 9, 2026
Merged

replynodes-ai merged 1 commit into
mainfrom
fix/company-research-pricing-model-grounding

Conversation

@replynodes-ai

Copy link
Copy Markdown
Member

Problem

The live 20-company keyless benchmark (airtable.com) failed the fail-closed pricing
contract with:

pricing.model: no linked evidence excerpt supports the claim text

The Airtable pricing HTML embeds plan JSON. The camelCase key
costPerUserPerMonthInCents matched the per-unit model cue
(?:per|/)\s*(?:user|seat|...) (it matched the PerUser inside the key), was
title-cased into the meaningless claim Peruser, and its linked excerpt was the raw
JSON fragment where the key stays costPerUserPerMonthInCents — so excerpt_supports
could not ground it.

Reproduced deterministically and against the live page
(model_value='Peruser', excerpt = raw JSON, excerpt_supports -> False).

Fix (bounded, harness/extraction only)

tests/run-company-research-keyless-e2e.py:

  1. Require a word boundary around the per-unit pricing cue
    ((?<![A-Za-z])...(?! [A-Za-z])) so a camelCase machine key is never read as readable
    pricing prose. Real prose (charged per seat, $20/user/month) still matches.
  2. Fail closed in extract_pricing: a cue only becomes the model when its linked
    excerpt actually grounds the derived value (reuses the validator's excerpt_supports).
    If ungrounded, the cue is skipped and extraction continues (or falls back), never
    emitting an unsupportable label.

With the fix, Airtable yields a grounded model Per Seat (excerpt: "plans are charged
per seat") instead of Peruser.

Tests

  • tests/test-company-research-v2-quality-fixtures.py: new deterministic regression
    fixture for costPerUserPerMonthInCents — extraction-level (model != Peruser,
    excerpt-grounded) and contract-level (two-page brief validates end to end, pricing
    known + grounded) — plus a fail-closed body where the JSON key is the only per-unit
    text (no pricing) and a control for valid model prose.
  • Verified RED (reverting the runner change) → AssertionError: Peruser;
    GREEN after the fix.

Evidence (all on this head)

  • Focused: test-company-research-v2-scenarios.py + test-company-research-v2-quality-fixtures.py pass.
  • Package: tests/test-package.sh passes (schema fixture, scenarios, quality fixtures, distribution, registry links).
  • git diff --check clean.
  • Full 20-company keyless E2E DISABLE_TELEMETRY=1 python3 tests/run-company-research-keyless-e2e.py --output ...:
    schema_validation passed for all 20 companies, quality_failures: none,
    unsupported_claim_count: 0, airtable pricing.unknown: false with model Per Seat.

tests/company_brief_validator.py is unchanged; no unsupported claims are accepted.

…ON keys

The Airtable live benchmark tripped the fail-closed pricing contract:
the embedded plan JSON key `costPerUserPerMonthInCents` was read as a
per-unit model cue and title-cased into the meaningless claim `Peruser`,
whose linked raw-JSON excerpt could not ground it.

- Require a word boundary around the per-unit pricing cue so camelCase
  machine keys are never treated as readable pricing prose.
- Fail closed in extract_pricing: a cue only becomes the model when its
  linked excerpt actually grounds the derived value (excerpt_supports).
- Add a deterministic regression fixture for costPerUserPerMonthInCents
  (extraction + contract-level brief) plus a control for valid model
  evidence.

The validator is unchanged; unsupported claims are still rejected.
@replynodes-ai
replynodes-ai merged commit 5487198 into main Oct 9, 2026
2 checks passed
@replynodes-ai
replynodes-ai deleted the fix/company-research-pricing-model-grounding branch October 9, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant