Repository navigation
Align live scoring with backtest, add live forward test, revise paper on market timing - #2
Merged
Merged
Conversation
…ng, lint fixes - Move adverse-language taxonomy and severity helpers into scoring service as the single source of truth for live signals, backtest harness and deck. - Live signals now score with the backtest formula (real source confidence, freshness and adverse severity), cap benign name matches below shippable, infer signal type from content, and carry specific headlines/watch-next text. - Observations (API and scheduled ingestion) auto-raise signals for shippable evidence via a shared pipeline service. - Replace stub Celery tasks: hourly EDGAR/GDELT ingestion, weekly summary drafting, delivery retry with backoff (re-runs all gates), dead-lettering, health alerts, and non-destructive retention archiving. - Add operator_client_assignments table, migration and admin endpoints; client-scoped routes and delivery now check assignments. - Replace deprecated datetime.utcnow(); drop unused OpenAI settings. - Pin ruff, configure lint rules, fix all findings; lint scripts in CI. - Backtest harness keeps the committed commit label on identical reruns. - Fix README clone URL; add tests (76 -> 102). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Owner
Author
|
The Cloudflare Pages check is failing, but this PR doesn't cause it. The repo CI (
Generated by Claude Code |
Daily GitHub Action scores live GDELT news and SEC filings for a pre-registered 20-name watchlist with the backtest's own scorer and commits append-only, timestamped logs to the forward-test-log branch. A summarizer reports alert burden, human-labeled precision and 5-session price context before/after each flag. Adds markdown to requirements (used by render_working_paper.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Adds simulations/backtest/market_reaction_check.py, which compares each first flag with the first corpus headline reporting the stock already falling. Result, pinned by tests: 0 of 9 first flags precede the market reaction; 8 of 9 are that headline. The paper now frames the system as fast, auditable triage rather than anticipation, adds exact binomial confidence intervals, hindsight-selection and known false-positive sections, fixes the repo URL and stale test count, removes commercial framing, and describes the pre-registered forward test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Adds docs/assets/demo.gif (real test, backtest and timing-check output plus the generated deck), a skills map, the market-timing column and confidence intervals in the results table, and a limitations section. The stress-test deck's track-record text now carries the same timing caveat as the paper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Deploying aetherius with
|
| Latest commit: |
05759e5
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://24237da8.aetherius-7cp.pages.dev |
| Branch Preview URL: | https://claude-laughing-gates-gmc4fm.aetherius-7cp.pages.dev |
site/ is a self-contained one-page project site (no build step) with the honest results, limitations, forward-test description and demo GIF, plus a 1200x630 social preview image. Cloudflare Pages was configured to publish landing_page/, which ec04b46 removed; pointing its build output directory at site/ fixes the failing check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What changed
Commit 1: backend fixes
scoring/service.py, and live signals, the backtest and the stress-test deck all use it. Live signals score with the backtest formula, benign name matches are capped below shippable, and signal type comes from content.app/services/pipeline/service.pyhandles observation → evidence → signals for both the API and the workers.0002, and admin endpointsPOST/DELETE /auth/assignments.datetime.utcnow()replaced, unused OpenAI settings removed, ruff pinned and clean.Commit 2: pre-registered live forward test (
simulations/forward_test/).github/workflows/forward-test.yml) scores live GDELT news and SEC filings with the frozen backtest scorer. It commits append-only, timestamped logs to theforward-test-logbranch.summarize.pyreports alerts per name per week, human-labeled precision, and the 5-session price return before and after each flag.Commit 3: working paper Revision 2
simulations/backtest/market_reaction_check.pycompares each first flag with the first corpus headline reporting the stock already falling. Result: 0 of 9 first flags came before the market reaction, and 8 of 9 were that headline. Tests pin this.COIN, substring adverse terms, EDGAR metadata only). It fixes the repo URL and test count and removes the commercial framing.Commit 4: README and deck
docs/assets/demo.gif, built from real command output), a map of what the project demonstrates, the market-timing column in the results table, and a limitations section.Why this was needed
The production signal path didn't use the backtested formula, most jobs were stubs, and CI lint failed. Separately, the published "lead time" was measured to formal resolution dates, which overstated the result. This PR measures timing against the market reaction and starts an out-of-sample record.
Validation
pytest -q aetherius/tests): 116 passed (was 76).simulations/validation/is unchanged.ruff check aetherius simulations scripts).Additional checks:
Scope
aetherius/app/services/...)simulations/backtest/...)Safety Checklist
.env, API keys, tokens)simulations/artifacts/, runtime logs). Forward-test logs go to their own branch, notmain.Notes for Reviewers
operator_client_assignments(migration0002).POST /observationsnow also returnssignals_created.ingest_dummyis replaced byingest_watchlist_sources.contents: write, which it declares. It creates and pushesforward-test-logon its first run. Set the repo variableSEC_USER_AGENTto a real contact address.🤖 Generated with Claude Code
https://claude.ai/code/session_01HTqzkhjyuhZZun8Az1tW6C