Skip to content

10 KB file-read guard silently degrades application context on real repositories #217

Description

@NahumKorda

Summary

The context phase refuses to read documentation files above 10 KB, then proceeds to generate an application context — and a confidence score — from whatever remains, without reflecting that primary evidence was withheld.

Evidence

The context phase emits a warning per skipped file and continues:

[2/7] Generating application context...
Warning: Could not read <doc>.md: <doc>.md is too large (N bytes > 10000); refusing to read
Warning: Could not read <doc>.md: <doc>.md is too large (N bytes > 10000); refusing to read

On our target this fired for both top-level documentation files — one roughly 3x the limit, the other about 1.6x. The scan then continued and emitted application_context.json with "confidence": 0.85, derived from dependency manifests and framework detection alone.

Why it matters

The two files that describe the application most directly were both excluded, on a target whose main README is around 3x the limit — not an unusual size for one. The application context feeds the Stage 1 prompt and the Stage 2 attacker personas, so degrading it degrades every downstream stage. The emitted confidence gives no hint that inputs were skipped, so a consumer cannot tell a well-grounded context from a partly-guessed one.

The guard in utilities/file_io.py is correct to exist — repository content is attacker-controlled — but a 10 KB ceiling is low for documentation specifically.

Suggested fix

Raise the ceiling for the context stage, or read a bounded prefix (first N KB) instead of refusing outright. Either way, reduce the reported confidence when a source was skipped, so the degradation is visible in the artifact rather than only in stderr.


Observed on one full-pipeline run at production scale: a private TypeScript monorepo (Angular front end, NestJS back end), 1,351 analysis units from 866 source files, run with --verify and no --limit, ~6.5 hours wall clock. The engine was invoked directly (python -m openant scan …, Python 3.13); the Go CLI was not built, so none of this involves the Go↔Python envelope. Stage 1 detection and Stage 2 verification ran on Claude Opus 5; application context, enhancement and reporting on Claude Sonnet 5 — all seven phases routed through OpenRouter (anthropic/claude-opus-5, anthropic/claude-sonnet-5) via a locally-added openrouter provider adapter, on a build based on upstream 2ed78f6. Cost figures are expressed as proportions of the run total; absolute amounts are omitted deliberately, as is the identity of the scanned repository.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions