diff --git a/CHANGELOG.md b/CHANGELOG.md index a21b7a9..4aad192 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added + +- **Windows templates for the marketdata `series` domain**: `docs/examples/windows/run-series.cmd` + (the `--build-tradingview` wrapper, chained syncs, no in-file retry) and + `series-routine.md` (the instructions a Claude Code Desktop local routine on the producer + box runs, with its allow rules and the backfill). `WINDOWS_SCHEDULING.md` gains the + section; `SYNCING.md` notes that `_raw/tradingview/` rides under the existing `_raw` + exclusion. Docs only; the producer itself is in marketdata 0.3.0. + ### Removed — BREAKING - **The databento provider is gone, and with it every remaining trace of price data.** diff --git a/docs/SYNCING.md b/docs/SYNCING.md index 90ce509..a284220 100644 --- a/docs/SYNCING.md +++ b/docs/SYNCING.md @@ -167,7 +167,7 @@ This is the part that matters, and on a real store it is most of the bytes. | `bars/` | **yes** | the data — `bars///_.parquet` | | `metadata/` | **yes** | contract specs | | `manifest.json` | **YES** | the bar store's ONLY index. Not the COT store's legacy file — carry it, and carry it LAST | -| `_raw/` | **NO** | databento's append-only PAID raw store, producer-internal | +| `_raw/` | **NO** | databento's append-only PAID raw store, and the TradingView routine's raw JSON (`_raw/tradingview/`), both producer-internal | | `_cache/`, `citpy/`, `vintage/` | n/a | the bar store has none of these | On one real store the `_cache/` and `_raw/` exclusions dropped the payload from 270 MB to diff --git a/docs/WINDOWS_SCHEDULING.md b/docs/WINDOWS_SCHEDULING.md index a36bb2d..b08c171 100644 --- a/docs/WINDOWS_SCHEDULING.md +++ b/docs/WINDOWS_SCHEDULING.md @@ -94,6 +94,26 @@ before enabling it: - **Schedule it daily, not weekly.** Nearly every request returns 304, so a daily run costs almost nothing while catching holiday-shifted and backlog releases with no schedule logic. +`run-series.cmd` — the **series** domain (marketdata 0.3.0: the FOMO share, new 52-week +highs and lows, the Cboe put/call ratios, from TradingView). Copy +[`docs/examples/windows/run-series.cmd`](examples/windows/run-series.cmd). **This one is not +a Task Scheduler task.** TradingView has no data API; the series reach the box through a +claude.ai connector, which only a Claude session can call, so the producer is a Claude Code +Desktop **local routine** whose instructions are the template +[`series-routine.md`](examples/windows/series-routine.md), also copied into `` (fill its +markers, then set the allow rules it lists in your code folder's `.claude\\settings.json`). The +routine pulls each symbol through the connector, writes the results verbatim under +`MARKETDATA_STORE\_raw\tradingview\`, and runs this wrapper, which is +`marketdata-update --build-tradingview` (local files only, every guard in code, refuses +stale or disagreeing input with nothing written) followed by the two replica syncs. Two +routines a night, weekdays, about 18:30 and 19:45 ET: after the equities task and its +retries, before the 20:55 futures window, because this wrapper and that task both end by +mirroring the same two replicas. The second routine is the retry. The verifier cannot see a +Desktop routine, so it checks the store instead (a freshness row on `series/tradingview/`) +and says in its GUARD PROOFS list what it cannot check. Full design: marketdata +`docs/design/breadth-domain-scoping.md` and cot-analyzer +`docs/design/tradingview-breadth-scoping.md`. + ## Creating the tasks diff --git a/docs/examples/windows/run-series.cmd b/docs/examples/windows/run-series.cmd new file mode 100644 index 0000000..d5a4a01 --- /dev/null +++ b/docs/examples/windows/run-series.cmd @@ -0,0 +1,76 @@ +@echo off +REM marketdata SERIES build wrapper: stage 2 of the TradingView breadth producer. +REM Copy this file into your scheduler folder and overwrite the three markers below. +REM Do NOT put angle brackets in a .cmd file: cmd reads them as redirection and +REM the file fails with "The syntax of the command is incorrect" even on comment +REM lines, which is why these are plain-text markers you replace. +REM REPLACE_WITH_MARKETDATA_STORE_PATH = your BAR store e.g. C:\Users\you\marketdata_store +REM REPLACE_WITH_VENV_PATH = your venv e.g. C:\Users\you\code\marketdata\.venv +REM REPLACE_WITH_SCHEDULER_DIR = this folder e.g. C:\Users\you\cotdata\scheduler +REM +REM WHAT RUNS THIS, AND WHY IT IS NOT A TASK SCHEDULER TASK +REM ------------------------------------------------------------------------ +REM The series domain (the FOMO share, new 52-week highs and lows, the put/call +REM ratios) comes from TradingView, which has no data API. The feed reaches this +REM box through a claude.ai connector, so stage 1 is a Claude Code Desktop LOCAL +REM ROUTINE: Claude calls the connector once per registry series symbol and writes +REM each result VERBATIM under MARKETDATA_STORE\_raw\tradingview\. The routine's +REM last step is this file. See series-routine.md beside it for the routine's +REM exact instructions and the allow rules it runs under. +REM +REM Stage 2 is the one command below. marketdata-update --build-tradingview reads +REM ONLY those local files: it validates them (the connector's JSON shape, the +REM registry range per kind, strictly increasing stamps, the registry anchors), +REM refuses any bar that disagrees with a bar the store already holds (the store +REM is never rewritten by a build), appends only what the store lacks, and +REM refuses as STALE, exit 1 and nothing written, when the newest bar is older +REM than the latest weekday whose 16:30 ET close has passed. +REM +REM WHY THERE IS NO RETRY LOOP IN HERE +REM ------------------------------------------------------------------------ +REM run-equities.cmd retries because its fetch hits Yahoo. This build hits +REM nothing: a refusal is either a bad raw file, which a retry cannot fix, or a +REM stale one, which only a later connector pull fixes. So the retry is the +REM SECOND ROUTINE, scheduled later the same evening, and on a good night it +REM finds the store current, writes nothing, and exits 0. +REM +REM WHEN IT RUNS, AND WHY THOSE TIMES +REM ------------------------------------------------------------------------ +REM Two routines, weekdays: about 18:30 and 19:45 ET. Both sit AFTER the 17:30 +REM equities task and its in-file retries (done by 17:50) and BEFORE the 20:55 +REM futures task, whose repeating trigger fires every 15 minutes for five hours +REM and syncs both replicas at whichever repeat captures. This wrapper ends by +REM calling the same two sync scripts, and two mirror passes running +REM concurrently against the same replicas is a race nobody wants to debug, so +REM the series routines stay out of the futures window entirely. +REM +REM `if errorlevel 1` tests >= 1 and needs no expansion, so it is safe here. +REM `|| exit /b %ERRORLEVEL%` would NOT be: cmd expands %ERRORLEVEL% when it parses +REM the line, which is BEFORE the command on that line has run, so it would return +REM the previous command's code. On its own line, after the command, it is correct. +setlocal +set "MARKETDATA_STORE=REPLACE_WITH_MARKETDATA_STORE_PATH" +set "MDEXE=REPLACE_WITH_VENV_PATH\Scripts\marketdata-update.exe" + +REM Unscoped: every registry series symbol. A symbol with no raw files is a +REM refusal, not a skip, because the routine pulls every one of them every night +REM and a missing one means it did not. +"%MDEXE%" --build-tradingview +if errorlevel 1 exit /b %ERRORLEVEL% + +REM --------------------------------------------------------------------------- +REM Chained replica syncs, same discipline and same order as run-prices.cmd: the +REM Mac sync first, the VPS push second, so the Mac replica is current even on a +REM day the VPS is unreachable. Both scripts mirror BOTH stores, so the COT and +REM futures passes here are cheap no-op re-scans. Reached on an "already current" +REM build too (exit 0, nothing written): a no-op mirror is cheap, and skipping it +REM would need the wrapper to tell the two exit-0 cases apart. +REM +REM The raw JSON under _raw\tradingview never rides along: both scripts exclude +REM _raw by name at any depth, for databento's paid raw store. Keep the directory +REM name exactly _raw or both exclusions silently stop applying. +call "REPLACE_WITH_SCHEDULER_DIR\sync-store.cmd" +if errorlevel 1 exit /b %ERRORLEVEL% + +call "REPLACE_WITH_SCHEDULER_DIR\push-to-server.cmd" +exit /b %ERRORLEVEL% diff --git a/docs/examples/windows/series-routine.md b/docs/examples/windows/series-routine.md new file mode 100644 index 0000000..2d08815 --- /dev/null +++ b/docs/examples/windows/series-routine.md @@ -0,0 +1,117 @@ +# The series routine: what Claude does on the producer box each evening + +This file is the whole instruction set for the Claude Code Desktop **local routines** that +produce marketdata's `series` domain (the FOMO share, new 52-week highs and lows, the Cboe +put/call ratios). Copy it into your scheduler folder beside `run-series.cmd`, fill the three +markers (`REPLACE_WITH_CODE_ROOT`, e.g. `C:\Users\you\code`; `REPLACE_WITH_MARKETDATA_STORE_PATH`; +`REPLACE_WITH_SCHEDULER_DIR`), and point each routine at it with a one-line instruction: + +> Follow REPLACE_WITH_SCHEDULER_DIR\series-routine.md exactly. + +Why a routine and not a task: TradingView has no data API. The series reach the box through a +claude.ai connector, which only a Claude session can call. The routine is the transport; every +check is in `marketdata-update --build-tradingview`, which `run-series.cmd` runs (see that +file's header). Design: marketdata `docs/design/breadth-domain-scoping.md`, cot-analyzer +`docs/design/tradingview-breadth-scoping.md`. + +## Routine settings + +| field | value | +|---|---| +| Type | Local | +| Folder | `REPLACE_WITH_CODE_ROOT` | +| Schedule | Weekdays. Two routines: one about 18:30 ET, one about 19:45 ET (the second is the retry). Both after the 17:30 equities task and its retries, both before the 20:55 futures window, whose repeats sync the same replicas this wrapper syncs. | +| Instructions | the one line above | +| Permission mode | the default. The allow rules below remove the two prompts the probe showed (the bars call, the file write). Anything else the model tries still prompts, and a prompt stalls the run, which is the point. | + +Allow rules, in `REPLACE_WITH_CODE_ROOT\.claude\settings.json` (project level; the docs say +user-level `~/.claude/settings.json` rules apply to routines too, so mirror them there if a +prompt still appears). The connector's server name in a rule is how the session prints the +tool: run one interactive call first and copy the name from `/permissions` or from the +prompt text. Two spellings are listed because the docs show `mcp__claude_ai___` +while sessions have printed `mcp____`; an unmatched rule is harmless. + +```json +{ + "permissions": { + "allow": [ + "ToolSearch", + "mcp__REPLACE_WITH_CONNECTOR_ID__mcp-tv-get-ohlcv", + "mcp__claude_ai_TradingView__mcp-tv-get-ohlcv", + "Edit(marketdata_store/_raw/tradingview/**)", + "Edit(/c/Users/you/code/marketdata_store/_raw/tradingview/**)", + "Bash(cmd /c \"C:/Users/you/code/cotdata/scheduler/run-series.cmd\")" + ] + } +} +``` + +`Edit` rather than `Write`: the docs state a path rule written for `Write` is accepted and +never consulted. Paths in rules are POSIX form even on Windows. The Bash rule matches the +whole command text, so step 4 below must be typed exactly as the rule has it. + +## Steps + +Do these in order, and nothing else. + +1. **Load the TradingView tool schema.** The connector's tools are deferred; one schema-load + call for `get_ohlcv` (the tool named `mcp-tv-get-ohlcv`) is required before it can be + called. +2. **One bars call per symbol** in the table below: `symbol` as listed, `interval` `1D`, + `count` `10`. If a call fails, skip that symbol and continue; the build will refuse it by + name and the later routine will retry. +3. **Write each tool result verbatim** to + `REPLACE_WITH_MARKETDATA_STORE_PATH\_raw\tradingview\\.json`, where + `YYYY-MM-DD` is today's date in US Eastern time. Verbatim means the complete JSON object + exactly as the tool returned it: no reformatting, no summary, no added or dropped keys, no + rounding. Create the folders if they do not exist. Do not print the result back. +4. **Run the wrapper**, exactly this command and nothing else: + `cmd /c "REPLACE_WITH_SCHEDULER_DIR_FORWARD/run-series.cmd"` + (forward slashes; the allow rule matches this text). +5. **Report** the wrapper's exit code and the lines the build printed, one per symbol. Then + stop. + +Never alter a number. Never write under any other path. Never run any other command. Do not +retry a refused build: a refusal is a bad file (a human's to look at) or a stale one (the next +routine's to fix). + +## The symbols + +From marketdata's `registry.yaml`, classes `Market Breadth` and `Options Sentiment`. The +registry is the authority; if it and this table differ, the registry wins, and this table is +due an edit. + +| internal (folder name) | `symbol` for the call | +|---|---| +| `NASDAQ_FOMO_5D` | `INDEX:NCFD` | +| `SPX_FOMO_5D` | `INDEX:S5FD` | +| `NASDAQ_PCT_ABOVE_20D` | `INDEX:NCTW` | +| `NASDAQ_PCT_ABOVE_200D` | `INDEX:NCTH` | +| `SPX_PCT_ABOVE_200D` | `INDEX:S5TH` | +| `NASDAQ_NH52W` | `INDEX:HIGQ` | +| `NASDAQ_NL52W` | `INDEX:LOWQ` | +| `NYSE_NH52W` | `INDEX:HIGN` | +| `NYSE_NL52W` | `INDEX:LOWN` | +| `CBOE_PCC` | `USI:PCC` | +| `CBOE_PCCE` | `USI:PCCE` | + +## What a good night looks like + +The first routine reports exit code 0 and one `+1 bar(s)` line per symbol (or `already +current` on a symbol the vendor has not updated yet). The second reports exit code 0 and +`already current` for every symbol. A `REFUSED` line names the file and the bar; leave the +file where it is and read the message, because the build never rewrites a stored bar and the +disagreement is either a vendor restatement or a transcription slip, and only a person can say +which. Next morning `verify-scheduling.ps1` reports the series freshness row green. + +## Backfill, once, and not through the routine + +A first fill of history is not a `count=5000` pull through the routine: writing five thousand +bars verbatim through the model is slow, expensive and, with no stored bars to overlap, has +nothing to catch a slip. Use TradingView's own chart export instead: open each symbol on a 1D +chart, Export chart data, CSV. That file is exact and needs no transcription. Convert it with +marketdata's CSV import (`marketdata-update --tradingview-csv --symbols `, +which writes a raw file in the connector's shape so the same build and the same guards apply), +then run `marketdata-update --build-tradingview --expect-session none` once, then let the +evening routine take over. The registry anchors are checked on that build: for +`NASDAQ_FOMO_5D` the 2026-07-29 close must read 48.05.