Skip to content

feat: LLM strategy with strict cost caps and fail-closed guardrails - #39

Open
vishwakt wants to merge 5 commits into
mainfrom
feat/llm-strategy
Open

vishwakt wants to merge 5 commits into
mainfrom
feat/llm-strategy

Conversation

@vishwakt

@vishwakt vishwakt commented Jul 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

Parked for now (no spare paper account to run it) — kept current with main and complete for the talk.

LLM strategy

  • New llm strategy: schema-validated Claude API signals behind the same Strategy ABC as the 7 technical strategies; fails closed to HOLD on any error (API failure, refusal, validation, missing key)
  • Three independent cost-containment layers: capped 10-symbol cohort (disjoint from the general watchlist), in-strategy daily call budget (LLM_MAX_DAILY_CALLS, default 40), no LLM re-invocation in notification enrichment
  • Runtime model selection via SSM ai_model (default claude-opus-4-8); structured LLM_METRIC log lines for CloudWatch Logs Insights
  • DailyScan timeout raised to 900s; disabled by default — enabling is a one-line config change

Signal log (signals.db)

  • Every strategy decision, HOLDs included, with price at decision time, an inputs tag, and a context snapshot of non-reproducible data (LLM model id + indicators shown)
  • Separate SQLite file synced only by the daily scan — stays out of the per-minute trades.db version churn
  • Fail-open: a log write failure never interrupts the scan
  • This is the unbiased decision record (no survivorship bias) that any future model-driven arbitration gets measured against

Test plan

  • 104 tests pass (24 new across LLM fail-closed paths, cost caps, scan-plan enforcement, signal log roundtrip/filters, engine fail-open)
  • ruff check + format clean; sam validate --lint passes

…sed guardrails

- New 'llm' strategy: sends recent OHLCV + indicators to the Anthropic API,
  maps schema-validated structured output to a Signal; any error (API
  failure, refusal, validation, missing key) fails closed to HOLD
- Cost containment, three independent layers:
  1. scheduler plan: llm runs only on the capped scheduler.llm_symbols
     cohort (never the general watchlist, entries and exits alike)
  2. daily call budget circuit breaker inside the strategy
     (LLM_MAX_DAILY_CALLS, default 40)
  3. notification enrichment never re-invokes the llm strategy
- config.json: 10-symbol llm cohort carved out of the general watchlist
  (disjoint sets for clean per-strategy attribution); llm NOT enabled by
  default — add 'llm' to scheduler.strategies to activate
- Model selectable at runtime via SSM ai_model (default claude-opus-4-8);
  API key via SSM SecureString anthropic_api_key
- Structured LLM_METRIC log lines (action, confidence, latency, tokens)
  for CloudWatch Logs Insights analysis
- 16 new tests; full suite 91 passed
Every generate_signal() result — HOLDs included — is appended to a
signals table with the price at decision time, an inputs tag, and a
context snapshot of non-reproducible data (LLM model id + the indicator
values it was shown). Executed trades and risk rejections alone are a
survivorship-biased view; this is the unbiased decision record any
future model-driven arbitration must be measured against.

Kept in its own SQLite file synced to S3 only by the daily scan, so it
stays out of the per-minute trades.db version churn. Fail-open: a log
write failure never interrupts the scan.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant