feat(format): gated LLM secondary formatter with forbidden-token enforcement#6
Merged
Merged
Conversation
The interpreter is enabled=false by default per phase-4 §17.1. An operator opts in by editing config/llm.toml after reviewing the reputation-risk example in docs/examples/negative-paths.md §Example 4. backends.ollama defaults to the local daemon at 11434 with the gemma2 27B model; backends.anthropic uses claude-haiku-4-5-20251001 per the project model conventions in ~/.claude/CLAUDE.md §Models. Model identifiers live only in this file; source code reads them at startup. suspend_during_storm wires into the dedup layer's StormController so the LLM formatter stops generating when the bus enters storm mode, preserving the deterministic pipeline's throughput under pressure.
…nd timestamps IntelligenceBrief gains three load-bearing fields per phase-4 §3 and two constructor-time validators that lock the brief's interpretation mode and forbidden-token check to their Literal singletons. headline is capped at 90 characters so it fits a Slack header; body_markdown is capped at 800 characters so it renders cleanly on a dashboard card; actionable_for is typed list[ConsumerType] so unknown consumers fail at construction. formatter_version and generated_at let consumers verify which formatter produced the brief and when, closing the provenance surface that prompt_hash and model alone could not. Two model_validator decorators enforce the Literal singletons: interpretation_mode must equal "llm_assisted" and forbidden_token_check must equal "passed". The gated formatter path is the only code that can mint a conforming brief because any other construction path would have to forge those literals, which code review catches. schemas/IntelligenceBrief-1.0.0.json is regenerated via scripts/export_schemas.py so the wire contract matches the model.
AbstractLLMBackend is the Protocol the interpreter dispatches through. Two concrete adapters implement the same async ``complete`` surface: OllamaBackend routes through the local daemon via plain httpx (no hard dependency on the ollama SDK), AnthropicBackend uses the anthropic SDK lazily imported inside the constructor so the llm-isolation test in the default environment still passes. Both adapters retry on transient failures: Ollama retries twice with no backoff (local daemon outages should surface quickly, not loop for a minute), Anthropic retries up to the configured limit per phase-4 §4.4. A backend that exhausts retries raises BackendError; the interpreter treats the error as a dropped brief per phase-4 §10 rather than propagating. AnthropicBackend accepts an injected client for testing; production code constructs the client from the ANTHROPIC_API_KEY env var. Missing credentials plus no injected client fails loud at construction. CompletionResult captures text, token counts, and generation duration so the observability hooks in the interpreter can surface per-backend latency distributions. Six tests cover Ollama health-check success, Ollama completion parse path, Ollama retry-exhaustion, Anthropic credential enforcement, Anthropic injected-client happy path, and Anthropic retry exhaustion.
The prompt builder produces a deterministic (system, user) pair from any SignalContext. The system message embeds the sorted forbidden- phrase list, a summary of the IntelligenceBrief schema, and the ConsumerType enum — ensuring the model sees the exact constraints it must satisfy. The user message renders the signal payload into the per-signal-type template; all five SignalType values have a dedicated template under augur_format/llm/prompts/templates/. Determinism is the load-bearing contract: identical input plus identical template files always produce identical prompt strings. The prompt hash attached to every brief is SHA-256 of the concatenated pair, so auditors can reproduce the prompt offline from the SignalContext and confirm the model saw exactly what the builder claims it saw. Missing templates raise PromptTemplateNotFoundError at render time rather than silently falling back — contract drift between SignalType enum and template directory fails loud. The hatch build config now also includes *.txt so the templates ship with the wheel alongside the Markdown Jinja2 templates from the deterministic pathway. Nine tests cover determinism across calls, system-message phrase and consumer-enum injection, verbatim resolution-criteria pass-through, manipulation-flag rendering both populated and empty, every signal type finding its template, related-market bullet rendering, and the missing-template error path.
…, consumer gate Four defense layers sit between the backend's raw text and a persisted IntelligenceBrief: ForbiddenTokenLinter case-insensitive matches every phrase loaded from config/forbidden_tokens.toml (causal_narrative, price_projection, manipulation_speculation). A match drops the brief before IntelligenceBrief construction. load_forbidden_phrases flattens every [category].phrases block into a single list so the linter does not need to know category semantics. SchemaValidator wraps Pydantic's IntelligenceBrief.model_validate and translates ValidationError into a stable ValidationResult. The interpreter checks result.ok before minting a brief; any schema violation drops the brief and logs the offending field path. ProvenanceStamp holds model (backend-qualified), prompt_hash (SHA-256 of system + "\n\n" + user), and formatter_version (from installed package metadata). Auditors reproduce prompt_hash from the deterministic prompt builder to confirm the model saw exactly what the record claims. ConsumerGate enforces the docs/contracts/consumer-registry.md opt-in rule: only consumers with accepts_llm_assisted=true receive the LLM brief. The deterministic JSON and Markdown paths still reach every consumer; the gate only filters the secondary formatter's output. Eleven tests cover: every configured phrase rejected, case insensitivity, clean text accepted, brief-shape lint, schema validator accept + two rejection modes, stamp reproducibility, stamp hash varies on prompt change, gate eligibility both directions, and list filtering.
LLMInterpreter is the single entrypoint the engine calls to render a SignalContext into an IntelligenceBrief through the gated path. The orchestrator sequences five stages: build deterministic prompt, call backend, lint output for forbidden tokens, validate against the IntelligenceBrief schema, stamp provenance. Any failure at any stage drops the brief by returning None — the deterministic pipeline proceeds unaffected, so consumers always receive the canonical JSON and Markdown outputs regardless of LLM outcome. set_suspended wires into the Phase-1 StormController's state stream per phase-4 §11: when in_storm=True the interpreter returns None immediately without calling the backend, avoiding the 5-10-second per-brief latency under storm-mode pressure. Briefs that would have been generated during suspension are not retroactively rendered. Provenance stamping attaches model identifier (backend-qualified), SHA-256 prompt hash, and formatter version to every brief. Auditors reproduce the hash from the prompt builder's deterministic output and confirm the model saw exactly what the record claims. now is a parameter so backtest harnesses can drive generated_at deterministically. Production code passes None which falls through to datetime.now(UTC); tests always pass an explicit timestamp. Eight tests cover the full pipeline: happy path, forbidden token drop, invalid JSON drop, unknown consumer drop, backend error drop, storm-mode suspension short-circuit, resume after suspension, and over-length-headline schema drop.
…onstant Addresses the pr-review findings in order: HIGH (H1): LLMInterpreter now accepts an optional ConsumerGate and filters each brief's actionable_for to the opted-in subset before returning. Briefs whose actionable_for empties after filtering drop entirely — the previous wiring generated a brief whose consumer list was never validated against the accepts_llm_assisted registry, letting LLM output leak to agent consumers that had not opted in. When the filter trims the list, the brief is rebuilt via model_copy so downstream code sees only the allowed set. HIGH (H2): the forbidden-token linter now runs against the post-parse headline+body instead of the raw JSON response. A model that escapes a forbidden phrase as \\u006d\\u0061\\u0079 would slip past the substring check on raw JSON but fails the lint after json.loads normalizes the escape. A regression test covers the unicode-escape bypass path. MEDIUM (M3): models.py exports SCHEMA_VERSION as a module-level constant and the interpreter plus prompt builder read from it. A schema version bump now requires one edit instead of three. MEDIUM (M4): interpreter drops the SchemaValidator wrapper's double validation; IntelligenceBrief.model_validate is the single source of schema truth. ValidationError drops the brief without a second full-validate pass. MEDIUM (M1): OllamaBackend raises immediately on 4xx responses (malformed adapter payload) instead of retrying — the error class only recovers on 5xx/connection failures. MEDIUM (M2): AnthropicBackend narrows retry to transient failures. AuthenticationError, PermissionDeniedError, and BadRequestError class paths raise through a wrapped BackendError immediately so credential misconfigurations surface without burning the retry budget. Class lookup is string-based so the module loads without the anthropic SDK installed. Three new tests cover the consumer-gate filter path, the no- consumer-opted-in drop path, and the unicode-escape lint bypass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ships the gated, opt-in LLM formatter that consumes
SignalContextand emitsIntelligenceBriefthrough a five-stage defense pipeline: deterministic prompt builder, backend completion (Ollama or Anthropic), forbidden-token linter, schema validator, provenance stamp. The interpreter suspends during storm mode; consumers withoutaccepts_llm_assisted = truenever see LLM briefs. The deterministic pipeline runs regardless.What Changed
IntelligenceBriefwith headline ≤ 90 chars, body ≤ 800 chars,formatter_version,generated_at, and two model validators lockinginterpretation_modeandforbidden_token_checkto their Literal singletons.AbstractLLMBackendprotocol +OllamaBackend(plain httpx) +AnthropicBackend(lazy-imported SDK).PromptBuilderwith_system.txtand five per-signal-type templates.ForbiddenTokenLinterwithload_forbidden_phrasesthat flattens every category block inconfig/forbidden_tokens.toml; case-insensitive matching.SchemaValidatorwrapping Pydantic with a stableValidationResultshape.ConsumerGatefilters consumers byaccepts_llm_assistedopt-in.LLMInterpretercomposes all five stages;set_suspendedwires into StormController.config/llm.toml(default off, Ollama + Anthropic blocks, template path).How It Works
The interpreter builds a deterministic prompt via
PromptBuilder, callsAbstractLLMBackend.complete, runs the output throughForbiddenTokenLinter.check_text, parses JSON, stamps provenance viastamp(...), and validates against theIntelligenceBriefcontract viaSchemaValidator. Any failure returnsNone; the deterministic formatters emit unaffected. Storm mode short-circuits before the backend call so the 5-10-second per-brief latency does not starve the dedup pipeline under pressure.Configuration Added
config/llm.toml— enabled=false default, backend configs, prompt template path. Opt-in required.Schema Changes
schemas/IntelligenceBrief-1.0.0.jsonre-exported with the new fields (length bounds,formatter_version,generated_at). Same version, backward-compatible additions within the 1.0 series.Quality Gates
uv run ruff check .cleanuv run ruff format --check .cleanuv run mypy --strict src/clean (120 source files)uv run pytest --cov-fail-under=80passes; 273 tests, 88.3 % coveragegrepoversrc/augur_signals/returns zero matches)datetime.now()AST guard passesuv run pre-commit run --all-filespassesDefinition of Done
src/augur_format/llm/package present;mypy --strictclean.src/augur_signals/has zero matches.schemas/IntelligenceBrief-1.0.0.jsoncommitted.Operational Handoff
After merge an operator enables the formatter by setting
config/llm.toml [interpreter] enabled = true, installing the chosen backend extras (augur-format[llm-local]for Ollama,augur-format[llm-cloud]for Anthropic), and provisioning credentials. The deterministic JSON and Markdown pathways reach every consumer regardless; the LLM-assisted brief reaches only consumers whoseaccepts_llm_assisted = true.Test Plan
uv run pytestpasses locally (273 tests).uv run python scripts/export_schemas.py --checkpasses locally.uv run pre-commit run --all-filespasses locally.Review Pass
pr-reviewfindings addressed: 2 HIGH + 3 MEDIUM fixed in4074957 fix(llm).code-refinersimplifications applied: the schema-version constant (M3) and double-validation collapse (M4) landed in the same commit.Deferred Findings