Skip to content

Align live scoring with backtest, add live forward test, revise paper on market timing - #2

Merged
zariffromlatif merged 6 commits into
mainfrom
claude/laughing-gates-gmc4fm
Oct 16, 2026
Merged

zariffromlatif merged 6 commits into
mainfrom
claude/laughing-gates-gmc4fm

Conversation

@zariffromlatif

@zariffromlatif zariffromlatif commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

What changed

Commit 1: backend fixes

  • One shared adverse-language list. It now lives in scoring/service.py, and live signals, the backtest and the stress-test deck all use it. Live signals score with the backtest formula, benign name matches are capped below shippable, and signal type comes from content.
  • Shared pipeline service. app/services/pipeline/service.py handles observation → evidence → signals for both the API and the workers.
  • Real Celery jobs replace the stubs: hourly EDGAR + GDELT ingestion, weekly summaries, delivery retry with backoff (re-running every gate), dead-lettering, health alerts and retention archiving.
  • Operator-to-client assignments. New table, migration 0002, and admin endpoints POST/DELETE /auth/assignments.
  • Cleanup: datetime.utcnow() replaced, unused OpenAI settings removed, ruff pinned and clean.

Commit 2: pre-registered live forward test (simulations/forward_test/)

  • A 20-name watchlist, committed before any data was collected; its hash is recorded on every run.
  • A daily GitHub Action (.github/workflows/forward-test.yml) scores live GDELT news and SEC filings with the frozen backtest scorer. It commits append-only, timestamped logs to the forward-test-log branch.
  • summarize.py reports alerts per name per week, human-labeled precision, and the 5-session price return before and after each flag.

Commit 3: working paper Revision 2

  • simulations/backtest/market_reaction_check.py compares each first flag with the first corpus headline reporting the stock already falling. Result: 0 of 9 first flags came before the market reaction, and 8 of 9 were that headline. Tests pin this.
  • The paper now presents the system as triage, not anticipation. It adds Clopper–Pearson confidence intervals (recall 66–100%, control false-positive rate 0–71%), a hindsight-selection section and known false-positive classes (ticker words such as COIN, substring adverse terms, EDGAR metadata only). It fixes the repo URL and test count and removes the commercial framing.

Commit 4: README and deck

  • README rewritten with a demo GIF (docs/assets/demo.gif, built from real command output), a map of what the project demonstrates, the market-timing column in the results table, and a limitations section.
  • The stress-test deck's track-record text carries the same timing caveat as the paper.

Why this was needed
The production signal path didn't use the backtested formula, most jobs were stubs, and CI lint failed. Separately, the published "lead time" was measured to formal resolution dates, which overstated the result. This PR measures timing against the market reaction and starts an out-of-sample record.

Validation

  • Relevant tests pass (pytest -q aetherius/tests): 116 passed (was 76).
  • SVB-2023 backtest still passes: all three events' metrics are bit-identical, and simulations/validation/ is unchanged.
  • Lint passes (ruff check aetherius simulations scripts).
  • Docs updated: README, backend README, working paper and its README, forward-test README.

Additional checks:

  • End-to-end run on a local Postgres covering migrations, assignment scoping, signal creation and the retry/dead-letter/archive sweeps.
  • Live forward-test smoke run against SEC EDGAR (4 real bank 8-Ks logged; a rerun added 0 rows).
  • The workflow's minimal dependency list was verified in a clean virtualenv.

Scope

  • Ingestion, entity mapping, scoring, or signals (aetherius/app/services/...)
  • Delivery / templates / review workflow
  • Backtests (simulations/backtest/...)
  • Docs / governance / landing page only

Safety Checklist

  • No secrets committed (.env, API keys, tokens)
  • No generated artifacts committed (simulations/artifacts/, runtime logs). Forward-test logs go to their own branch, not main.
  • No client-specific / private data included
  • Marketing / README claims still match implementation reality: the README and paper now state the market-timing result and the limitations.

Notes for Reviewers

  • Migration impact: a new table, operator_client_assignments (migration 0002).
  • Backward compatibility:
    • Delivering a brief now works for an admin or an analyst assigned to that client.
    • POST /observations now also returns signals_created.
    • ingest_dummy is replaced by ingest_watchlist_sources.
  • Operational risks:
    • The forward-test workflow needs contents: write, which it declares. It creates and pushes forward-test-log on its first run. Set the repo variable SEC_USER_AGENT to a real contact address.
    • The Cloudflare Pages check failure isn't caused by this PR; see the earlier comment.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C

…ng, lint fixes

- Move adverse-language taxonomy and severity helpers into scoring service as
  the single source of truth for live signals, backtest harness and deck.
- Live signals now score with the backtest formula (real source confidence,
  freshness and adverse severity), cap benign name matches below shippable,
  infer signal type from content, and carry specific headlines/watch-next text.
- Observations (API and scheduled ingestion) auto-raise signals for shippable
  evidence via a shared pipeline service.
- Replace stub Celery tasks: hourly EDGAR/GDELT ingestion, weekly summary
  drafting, delivery retry with backoff (re-runs all gates), dead-lettering,
  health alerts, and non-destructive retention archiving.
- Add operator_client_assignments table, migration and admin endpoints;
  client-scoped routes and delivery now check assignments.
- Replace deprecated datetime.utcnow(); drop unused OpenAI settings.
- Pin ruff, configure lint rules, fix all findings; lint scripts in CI.
- Backtest harness keeps the committed commit label on identical reruns.
- Fix README clone URL; add tests (76 -> 102).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Copilot AI balanced review requested due to automatic review settings October 16, 2026 05:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copy link
Copy Markdown
Owner Author

The Cloudflare Pages check is failing, but this PR doesn't cause it. The repo CI (test: ruff + pytest) passes on this head.

  • The Cloudflare check started and finished in the same second (05:58:47Z), so it failed before any build ran. That points to the Pages project's own settings.
  • This PR changes no site or landing-page files. ec04b46 on main removed the legacy landing page, which the Pages project probably still tries to build.
  • There's no fix to port into this PR, since the change lives in Cloudflare, not the repo. Either disconnect the aetherius Pages project from this repo, or point its build directory at something that still exists. I don't have access to re-run this check.

Generated by Claude Code

Daily GitHub Action scores live GDELT news and SEC filings for a
pre-registered 20-name watchlist with the backtest's own scorer and commits
append-only, timestamped logs to the forward-test-log branch. A summarizer
reports alert burden, human-labeled precision and 5-session price context
before/after each flag. Adds markdown to requirements (used by
render_working_paper.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Adds simulations/backtest/market_reaction_check.py, which compares each
first flag with the first corpus headline reporting the stock already
falling. Result, pinned by tests: 0 of 9 first flags precede the market
reaction; 8 of 9 are that headline. The paper now frames the system as
fast, auditable triage rather than anticipation, adds exact binomial
confidence intervals, hindsight-selection and known false-positive
sections, fixes the repo URL and stale test count, removes commercial
framing, and describes the pre-registered forward test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Adds docs/assets/demo.gif (real test, backtest and timing-check output
plus the generated deck), a skills map, the market-timing column and
confidence intervals in the results table, and a limitations section.
The stress-test deck's track-record text now carries the same timing
caveat as the paper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Deploying aetherius with  Cloudflare Pages  Cloudflare Pages

Latest commit: 05759e5
Status: ✅  Deploy successful!
Preview URL: https://24237da8.aetherius-7cp.pages.dev
Branch Preview URL: https://claude-laughing-gates-gmc4fm.aetherius-7cp.pages.dev

View logs

@zariffromlatif zariffromlatif changed the title Align live scoring with backtest, implement worker jobs, add client scoping, fix lint Align live scoring with backtest, add live forward test, revise paper on market timing Oct 16, 2026
site/ is a self-contained one-page project site (no build step) with the
honest results, limitations, forward-test description and demo GIF, plus a
1200x630 social preview image. Cloudflare Pages was configured to publish
landing_page/, which ec04b46 removed; pointing its build output directory
at site/ fixes the failing check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
@zariffromlatif
zariffromlatif merged commit e887d72 into main Oct 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants