Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,15 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- **Windows templates for the marketdata `series` domain**: `docs/examples/windows/run-series.cmd`
(the `--build-tradingview` wrapper, chained syncs, no in-file retry) and
`series-routine.md` (the instructions a Claude Code Desktop local routine on the producer
box runs, with its allow rules and the backfill). `WINDOWS_SCHEDULING.md` gains the
section; `SYNCING.md` notes that `_raw/tradingview/` rides under the existing `_raw`
exclusion. Docs only; the producer itself is in marketdata 0.3.0.

### Removed — BREAKING

- **The databento provider is gone, and with it every remaining trace of price data.**
Expand Down
2 changes: 1 addition & 1 deletion docs/SYNCING.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ This is the part that matters, and on a real store it is most of the bytes.
| `bars/` | **yes** | the data — `bars/<domain>/<source>/<symbol>_<tier>.parquet` |
| `metadata/` | **yes** | contract specs |
| `manifest.json` | **YES** | the bar store's ONLY index. Not the COT store's legacy file — carry it, and carry it LAST |
| `_raw/` | **NO** | databento's append-only PAID raw store, producer-internal |
| `_raw/` | **NO** | databento's append-only PAID raw store, and the TradingView routine's raw JSON (`_raw/tradingview/`), both producer-internal |
| `_cache/`, `citpy/`, `vintage/` | n/a | the bar store has none of these |

On one real store the `_cache/` and `_raw/` exclusions dropped the payload from 270 MB to
Expand Down
20 changes: 20 additions & 0 deletions docs/WINDOWS_SCHEDULING.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,26 @@ before enabling it:
- **Schedule it daily, not weekly.** Nearly every request returns 304, so a daily run costs
almost nothing while catching holiday-shifted and backlog releases with no schedule logic.

`run-series.cmd` — the **series** domain (marketdata 0.3.0: the FOMO share, new 52-week
highs and lows, the Cboe put/call ratios, from TradingView). Copy
[`docs/examples/windows/run-series.cmd`](examples/windows/run-series.cmd). **This one is not
a Task Scheduler task.** TradingView has no data API; the series reach the box through a
claude.ai connector, which only a Claude session can call, so the producer is a Claude Code
Desktop **local routine** whose instructions are the template
[`series-routine.md`](examples/windows/series-routine.md), also copied into `<DIR>` (fill its
markers, then set the allow rules it lists in your code folder's `.claude\\settings.json`). The
routine pulls each symbol through the connector, writes the results verbatim under
`MARKETDATA_STORE\_raw\tradingview\`, and runs this wrapper, which is
`marketdata-update --build-tradingview` (local files only, every guard in code, refuses
stale or disagreeing input with nothing written) followed by the two replica syncs. Two
routines a night, weekdays, about 18:30 and 19:45 ET: after the equities task and its
retries, before the 20:55 futures window, because this wrapper and that task both end by
mirroring the same two replicas. The second routine is the retry. The verifier cannot see a
Desktop routine, so it checks the store instead (a freshness row on `series/tradingview/`)
and says in its GUARD PROOFS list what it cannot check. Full design: marketdata
`docs/design/breadth-domain-scoping.md` and cot-analyzer
`docs/design/tradingview-breadth-scoping.md`.



## Creating the tasks
Expand Down
76 changes: 76 additions & 0 deletions docs/examples/windows/run-series.cmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
@echo off
REM marketdata SERIES build wrapper: stage 2 of the TradingView breadth producer.
REM Copy this file into your scheduler folder and overwrite the three markers below.
REM Do NOT put angle brackets in a .cmd file: cmd reads them as redirection and
REM the file fails with "The syntax of the command is incorrect" even on comment
REM lines, which is why these are plain-text markers you replace.
REM REPLACE_WITH_MARKETDATA_STORE_PATH = your BAR store e.g. C:\Users\you\marketdata_store
REM REPLACE_WITH_VENV_PATH = your venv e.g. C:\Users\you\code\marketdata\.venv
REM REPLACE_WITH_SCHEDULER_DIR = this folder e.g. C:\Users\you\cotdata\scheduler
REM
REM WHAT RUNS THIS, AND WHY IT IS NOT A TASK SCHEDULER TASK
REM ------------------------------------------------------------------------
REM The series domain (the FOMO share, new 52-week highs and lows, the put/call
REM ratios) comes from TradingView, which has no data API. The feed reaches this
REM box through a claude.ai connector, so stage 1 is a Claude Code Desktop LOCAL
REM ROUTINE: Claude calls the connector once per registry series symbol and writes
REM each result VERBATIM under MARKETDATA_STORE\_raw\tradingview\. The routine's
REM last step is this file. See series-routine.md beside it for the routine's
REM exact instructions and the allow rules it runs under.
REM
REM Stage 2 is the one command below. marketdata-update --build-tradingview reads
REM ONLY those local files: it validates them (the connector's JSON shape, the
REM registry range per kind, strictly increasing stamps, the registry anchors),
REM refuses any bar that disagrees with a bar the store already holds (the store
REM is never rewritten by a build), appends only what the store lacks, and
REM refuses as STALE, exit 1 and nothing written, when the newest bar is older
REM than the latest weekday whose 16:30 ET close has passed.
REM
REM WHY THERE IS NO RETRY LOOP IN HERE
REM ------------------------------------------------------------------------
REM run-equities.cmd retries because its fetch hits Yahoo. This build hits
REM nothing: a refusal is either a bad raw file, which a retry cannot fix, or a
REM stale one, which only a later connector pull fixes. So the retry is the
REM SECOND ROUTINE, scheduled later the same evening, and on a good night it
REM finds the store current, writes nothing, and exits 0.
REM
REM WHEN IT RUNS, AND WHY THOSE TIMES
REM ------------------------------------------------------------------------
REM Two routines, weekdays: about 18:30 and 19:45 ET. Both sit AFTER the 17:30
REM equities task and its in-file retries (done by 17:50) and BEFORE the 20:55
REM futures task, whose repeating trigger fires every 15 minutes for five hours
REM and syncs both replicas at whichever repeat captures. This wrapper ends by
REM calling the same two sync scripts, and two mirror passes running
REM concurrently against the same replicas is a race nobody wants to debug, so
REM the series routines stay out of the futures window entirely.
REM
REM `if errorlevel 1` tests >= 1 and needs no expansion, so it is safe here.
REM `|| exit /b %ERRORLEVEL%` would NOT be: cmd expands %ERRORLEVEL% when it parses
REM the line, which is BEFORE the command on that line has run, so it would return
REM the previous command's code. On its own line, after the command, it is correct.
setlocal
set "MARKETDATA_STORE=REPLACE_WITH_MARKETDATA_STORE_PATH"
set "MDEXE=REPLACE_WITH_VENV_PATH\Scripts\marketdata-update.exe"

REM Unscoped: every registry series symbol. A symbol with no raw files is a
REM refusal, not a skip, because the routine pulls every one of them every night
REM and a missing one means it did not.
"%MDEXE%" --build-tradingview
if errorlevel 1 exit /b %ERRORLEVEL%

REM ---------------------------------------------------------------------------
REM Chained replica syncs, same discipline and same order as run-prices.cmd: the
REM Mac sync first, the VPS push second, so the Mac replica is current even on a
REM day the VPS is unreachable. Both scripts mirror BOTH stores, so the COT and
REM futures passes here are cheap no-op re-scans. Reached on an "already current"
REM build too (exit 0, nothing written): a no-op mirror is cheap, and skipping it
REM would need the wrapper to tell the two exit-0 cases apart.
REM
REM The raw JSON under _raw\tradingview never rides along: both scripts exclude
REM _raw by name at any depth, for databento's paid raw store. Keep the directory
REM name exactly _raw or both exclusions silently stop applying.
call "REPLACE_WITH_SCHEDULER_DIR\sync-store.cmd"
if errorlevel 1 exit /b %ERRORLEVEL%

call "REPLACE_WITH_SCHEDULER_DIR\push-to-server.cmd"
exit /b %ERRORLEVEL%
117 changes: 117 additions & 0 deletions docs/examples/windows/series-routine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# The series routine: what Claude does on the producer box each evening

This file is the whole instruction set for the Claude Code Desktop **local routines** that
produce marketdata's `series` domain (the FOMO share, new 52-week highs and lows, the Cboe
put/call ratios). Copy it into your scheduler folder beside `run-series.cmd`, fill the three
markers (`REPLACE_WITH_CODE_ROOT`, e.g. `C:\Users\you\code`; `REPLACE_WITH_MARKETDATA_STORE_PATH`;
`REPLACE_WITH_SCHEDULER_DIR`), and point each routine at it with a one-line instruction:

> Follow REPLACE_WITH_SCHEDULER_DIR\series-routine.md exactly.

Why a routine and not a task: TradingView has no data API. The series reach the box through a
claude.ai connector, which only a Claude session can call. The routine is the transport; every
check is in `marketdata-update --build-tradingview`, which `run-series.cmd` runs (see that
file's header). Design: marketdata `docs/design/breadth-domain-scoping.md`, cot-analyzer
`docs/design/tradingview-breadth-scoping.md`.

## Routine settings

| field | value |
|---|---|
| Type | Local |
| Folder | `REPLACE_WITH_CODE_ROOT` |
| Schedule | Weekdays. Two routines: one about 18:30 ET, one about 19:45 ET (the second is the retry). Both after the 17:30 equities task and its retries, both before the 20:55 futures window, whose repeats sync the same replicas this wrapper syncs. |
| Instructions | the one line above |
| Permission mode | the default. The allow rules below remove the two prompts the probe showed (the bars call, the file write). Anything else the model tries still prompts, and a prompt stalls the run, which is the point. |

Allow rules, in `REPLACE_WITH_CODE_ROOT\.claude\settings.json` (project level; the docs say
user-level `~/.claude/settings.json` rules apply to routines too, so mirror them there if a
prompt still appears). The connector's server name in a rule is how the session prints the
tool: run one interactive call first and copy the name from `/permissions` or from the
prompt text. Two spellings are listed because the docs show `mcp__claude_ai_<server>__<tool>`
while sessions have printed `mcp__<connector-id>__<tool>`; an unmatched rule is harmless.

```json
{
"permissions": {
"allow": [
"ToolSearch",
"mcp__REPLACE_WITH_CONNECTOR_ID__mcp-tv-get-ohlcv",
"mcp__claude_ai_TradingView__mcp-tv-get-ohlcv",
"Edit(marketdata_store/_raw/tradingview/**)",
"Edit(/c/Users/you/code/marketdata_store/_raw/tradingview/**)",
"Bash(cmd /c \"C:/Users/you/code/cotdata/scheduler/run-series.cmd\")"
]
}
}
```

`Edit` rather than `Write`: the docs state a path rule written for `Write` is accepted and
never consulted. Paths in rules are POSIX form even on Windows. The Bash rule matches the
whole command text, so step 4 below must be typed exactly as the rule has it.

## Steps

Do these in order, and nothing else.

1. **Load the TradingView tool schema.** The connector's tools are deferred; one schema-load
call for `get_ohlcv` (the tool named `mcp-tv-get-ohlcv`) is required before it can be
called.
2. **One bars call per symbol** in the table below: `symbol` as listed, `interval` `1D`,
`count` `10`. If a call fails, skip that symbol and continue; the build will refuse it by
name and the later routine will retry.
3. **Write each tool result verbatim** to
`REPLACE_WITH_MARKETDATA_STORE_PATH\_raw\tradingview\<INTERNAL>\<YYYY-MM-DD>.json`, where
`YYYY-MM-DD` is today's date in US Eastern time. Verbatim means the complete JSON object
exactly as the tool returned it: no reformatting, no summary, no added or dropped keys, no
rounding. Create the folders if they do not exist. Do not print the result back.
4. **Run the wrapper**, exactly this command and nothing else:
`cmd /c "REPLACE_WITH_SCHEDULER_DIR_FORWARD/run-series.cmd"`
(forward slashes; the allow rule matches this text).
5. **Report** the wrapper's exit code and the lines the build printed, one per symbol. Then
stop.

Never alter a number. Never write under any other path. Never run any other command. Do not
retry a refused build: a refusal is a bad file (a human's to look at) or a stale one (the next
routine's to fix).

## The symbols

From marketdata's `registry.yaml`, classes `Market Breadth` and `Options Sentiment`. The
registry is the authority; if it and this table differ, the registry wins, and this table is
due an edit.

| internal (folder name) | `symbol` for the call |
|---|---|
| `NASDAQ_FOMO_5D` | `INDEX:NCFD` |
| `SPX_FOMO_5D` | `INDEX:S5FD` |
| `NASDAQ_PCT_ABOVE_20D` | `INDEX:NCTW` |
| `NASDAQ_PCT_ABOVE_200D` | `INDEX:NCTH` |
| `SPX_PCT_ABOVE_200D` | `INDEX:S5TH` |
| `NASDAQ_NH52W` | `INDEX:HIGQ` |
| `NASDAQ_NL52W` | `INDEX:LOWQ` |
| `NYSE_NH52W` | `INDEX:HIGN` |
| `NYSE_NL52W` | `INDEX:LOWN` |
| `CBOE_PCC` | `USI:PCC` |
| `CBOE_PCCE` | `USI:PCCE` |

## What a good night looks like

The first routine reports exit code 0 and one `+1 bar(s)` line per symbol (or `already
current` on a symbol the vendor has not updated yet). The second reports exit code 0 and
`already current` for every symbol. A `REFUSED` line names the file and the bar; leave the
file where it is and read the message, because the build never rewrites a stored bar and the
disagreement is either a vendor restatement or a transcription slip, and only a person can say
which. Next morning `verify-scheduling.ps1` reports the series freshness row green.

## Backfill, once, and not through the routine

A first fill of history is not a `count=5000` pull through the routine: writing five thousand
bars verbatim through the model is slow, expensive and, with no stored bars to overlap, has
nothing to catch a slip. Use TradingView's own chart export instead: open each symbol on a 1D
chart, Export chart data, CSV. That file is exact and needs no transcription. Convert it with
marketdata's CSV import (`marketdata-update --tradingview-csv <file> --symbols <INTERNAL>`,
which writes a raw file in the connector's shape so the same build and the same guards apply),
then run `marketdata-update --build-tradingview --expect-session none` once, then let the
evening routine take over. The registry anchors are checked on that build: for
`NASDAQ_FOMO_5D` the 2026-07-29 close must read 48.05.
Loading