Repository navigation
cotmetrics 0.15.0: FlowRoles, speculator and retail series, weekly flow z - #52
Merged
Merged
Conversation
… state vocabulary (0.15.0) The Disaggregated and TFF category frames carried levels only: net, percent of OI, the range index, a level z-score and momentum. The gold flow-state study (docs/analysis/2026-09-26-cot-flow-states-gold.md) and its 42-market replication showed that the week-over-week change per cohort, scaled by that cohort's own trailing 52-week standard deviation of weekly changes, is the picture a single week's bar chart cannot give, and that nothing in the package computed it. This adds it as a column family beside the existing ones, computed inside get_category_data, so cot-analyzer can draw it without computing anything. cotmetrics.flows is pure and store-free and takes the OUTPUT of build_category_frame, inheriting the Swap_/Swap__ resolution, the dtype coercion and present_categories. Per present category it emits dNet, dLong, dShort, "Flow Z 52w" and "Flow Thin"; the window rides in the column name because it is fixed and is NOT the page lookback (params.yaml sets Custom to 8 weeks on OJ and 216 on PA, and a flow scaled by those would be a different statistic per market). The z subtracts no mean and uses no epsilon: the rolling mean of a first difference is ~0 by construction, and indicators.calculate_z_score, whose own test pins that a constant series yields 0, would label an empty window QUIET where the honest answer is no reading. NaN is kept under min_periods and on a zero-sd window. weekly_change masks a diff across an index hole longer than 8 days and across a change of contract population, read from the CFTC_Contract_Market_Code_Quotes column that build_category_frame now carries as "Source Code": cotdata rewrites the plain code column to the primary code on predecessor rows, so the Quotes twin is the only per-row record of which contract a row came from. Measured on the live store the seam mask fires on exactly LBR 2023-02-21, 02-28, 03-14 and RTY 2008-09-23, 2017-08-15, the gap mask on 6N 2006-07-11, and nowhere else; those rows printed 2 to 4 sigma "flows" that were population switches. The counterparty composite row (the sum of the cohorts on the other side of the speculators' flow) is keyed per (report, symbol) from cotmetrics.flow_roles, a generated data module, because the grouping the gold study used, Producer/Merchant plus Swap Dealers, is a precious-metals fact: elsewhere the swap dealers carry the index book and sit on neither side. The roles come from the same-week price correlation of each cohort's flow, measured on the ratio-adjusted marketdata series (docs/analysis/2026-09-26-cot-cohort-roles-propadj.md, in the companion docs PR), and only the 21 markets whose counterparty set is stable on rolling 156-week windows under a criterion fixed before the run carry a measured entry; the other 21 fall back to the per-report default with the source recorded as such. The opinion triple that feeds the state is the per-report default everywhere so the eight state names keep one meaning. state_eligible is False on TFF equities, rates and crypto (ADR-0005, still Proposed), so those markets get z cells and no state. scripts/analysis/gen_flow_roles.py regenerates the module from the two CSVs and refuses when the CSV's stability flag disagrees with the recorded criterion. flow_signs and flow_state are the gold script's classifier restated in the fomo_zones style: object labels, None wherever an input is NA (the first 26 weeks of every market are None, never QUIET, which both reproducers got wrong), the eight Disaggregated names verbatim, a sign string on TFF. It is a caption vocabulary and not a signal: the pre-registered crucible test of VALUE_ACCUM failed the gauntlet on all three pillars on 2026-09-26 (#50). Nothing here attaches a forward return, nothing in signals, conditions, models, movers or synthesis reads these columns, and no SETUP_* constant is minted. Held out on purpose: dGross, straddle and derisk (fixed multipliers designed on gold and untested anywhere), percent-of-OI flows, TFF state names. get_category_data concatenates the flow frame after build_category_frame and before the price merge, re-attaching attrs (pandas drops them on concat, the same trap the merge already handles). No new lru_cache key, no signature change; the frame gains 33 columns per market and every pre-existing column is unchanged (verified on Gold against main). The store-free suite gains 28 tests: the exact zero-sum identity on both reports, the 26-row warm-up, NaN not 0 on a constant category, both masks, the Swap_/Swap__ parity, the window fixed against lookbacks of 8 and 216, the roles coverage invariant, ADR-0005 eligibility, the composite equal to the sum of its members, and the concat path through get_category_data with a monkeypatched get_cot. Parity against the 42-market universe parquet: max difference exactly 0.0 on 40 markets; LBR, RTY and 6N differ only on their masked rows and the 52 rows after. cot-analyzer's own suite (897 tests) passes against this tree unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Decision 6 of docs/design/cot-flows.md, taken on the day's evidence. On every Disaggregated market Producer/Merchant is the counterparty in at least two thirds of the rolling 156-week windows, while on the nine physicals that fail the stability criterion (CC, CL, GF, HE, HO, NG, RB, SB, ZW) the swap dealers swing between neutral, speculator and counterparty because they carry the index book. A fallback should be the conservative core, and every cohort is still drawn as its own row, so the default composite is Producer/Merchant with swap in the neutral set. The precious-metals grouping (Producer/Merchant plus Swap Dealers) survives only as a measured entry where the swap dealers do absorb flow: GC, PA, PL, SI and, with Other Reportable, HG, SB, SI and HO. Regenerated from the same CSVs; the 21 measured entries are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Sep 26, 2026
…fallback The nine physicals that fail the stability criterion fall back to Producer/Merchant with the swap dealers neutral, since Producer/Merchant is the counterparty in at least two thirds of the rolling windows on every physical and the swap dealers there carry the index book. GC keeps its measured entry; LBR's measured set equals the new default. Shipped in #52. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… set For cot-analyzer PR 3 (docs/design/cot-flows.md), so the view draws the level markers and boxes the divergent weeks without computing either itself. - build_flow_frame emits, per category whose range index the frame carries, "<prefix><header> Flow From Level" (the prior row's range index at the page lookback) and "<prefix><header> Flow Level Mark" (Int64: 1 for an active flow leaving the top of the range, -1 the bottom, 0 otherwise, NA where either input is missing). The one part of the flow family that follows the lookback, on purpose, and its names carry the header the way index_col does. - level_marks(z, from_level): strict |z| > FLOW_ACTIVE_Z and strict 20 / 80 (FLOW_LEVEL_LOW / HIGH, the Cowork mockup's cutoffs, unsearched). - DIVERGENT_FLOW_STATES: the twelve all-active mixed-sign labels, six Disaggregated names and six TFF sign strings. - The fixed-window test now compares every column except the level pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Series level with a different index than z was reindexed onto z's index and came back all NA. Two Series must now share an index, and anything paired with an array must match its length. The docstring records that the level inherits the range index's straddle of a contract-code seam. Tests now assert marks of both directions exist and that a seam week writes its level with no mark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… all-NA" This reverts commit 8d2bbb1.
…nt-state set" This reverts commit 39eba10.
Scoped by the "Scope, restated" block of the cross-universe handoff: PR 1 and PR 2 only. The two reverts above take out the PR 3 material (prior-week level columns at 20/80, the divergent-state set). - flow_rows / MERGED_ROWS: on GC, SI, PL and PA, Producer/Merchant and Swap Dealers draw as one "Commercials" row (from the cohort-collapse table); everywhere else each category is its own row. build_flow_frame emits the merged row's dNet, legs, z and thin flag, its range index at the page lookback, and attrs["flow_rows"]. On the real store the Commercials dNet equals Legacy Commercial's net change exactly on Gold and Silver (1,058 weeks). - level_marks(index): 1 at or above 90, -1 at or below 10, the same week's index, as the mockup script draws them (FLOW_LEVEL_LOW / HIGH, unsearched). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review of the running panel: at the page's tuned lookback (26 weeks on gold) a cohort sits in its top or bottom decile for months, and the markers came in runs of fifteen that hid the cells. - Every row carries "Level 156w", its net's range index over a fixed three years (min 52 weeks), the prototype's window; like the z window it is fixed and named with it, and the page lookback never reaches it. - level_entries replaces level_marks: 1 on the week the level enters its top decile, -1 its bottom, inclusive at 90 / 10. On the live store that is 0 to 10 marks per row over 104 weeks on Gold, Corn, S&P 500 and Euro. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…low_z "Scope, restated v2" (cross-universe handoff) withdraws the multi-row heatmap, the state vocabulary and the markers, and keeps one role series per market. This commit takes the branch to that surface; the net diff against main is now only what v2 asks of cotmetrics. - flow_roles.py, generated by gen_flow_roles.py from the propadj cohort-collapse table (threshold 0.15, gross-share floor 0.05, measured 2026-09-26): FlowRoles per (report, symbol) with speculator, counterparty, neutral, inert and retail behaviour; per-report default Managed Money / Leveraged Funds for markets the table does not cover; ZT has no speculator. The propadj table, not the cache-close one v2 names, on the user's decision. - flows.py: weekly_change (gap and contract-seam masks), flow_z (fixed 52w, min 26, no mean, NaN never 0), speculator_net, retail_net, and speculator_frame (net, range index at the page lookback, dNet, flow z, retail net, roles in attrs). - CotIndexer.get_speculator_data(name, lookback): that frame for /analysis. get_category_data is main's again. - Removed: per-category flow columns, the counterparty composite, flow signs and state names, merged rows, level marks, the 3-year level. Legacy back-fill before 2006 is not built (not in the requested scope). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mspinola
changed the title
mspinola
added a commit
that referenced
this pull request
Sep 26, 2026
The display is one role series per market: the speculator's index and one flow-z strip on the existing /analysis Positioning Index panel, built as cotmetrics #52 and cot-analyzer #142. Section 3 now describes what is built and where it departs from the v2 block (propadj roles table, page lookback, retail as a label, no 20/80 bands, no Legacy back-fill yet); the withdrawn heatmap, state strip, Flow view and board are a short record, full specs in this file's history. Decisions, hazards, render checks and the don'ts follow. Section 2 measurements unchanged. Carries Cowork's superseding note in substance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Sep 27, 2026
"Scope, restated v2" (Cowork, 2026-09-26 evening) goes to the top verbatim, with a short status note under it: built as cotmetrics #52 and cot-analyzer #142, #140 and #141 closed, departures recorded in docs/design/cot-flows.md section 3. The v1 block, which v2 supersedes and which had been written over in the main checkout's copy, follows it verbatim as the record (recovered from the session that read it), per the append-only rule for handoffs. The rest of the file is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The cotmetrics half of "Scope, restated v2" (top of
docs/handoffs/2026-09-26-cot-flow-states-cross-universe.md, and the superseding note at the top ofdocs/design/cot-flows.md): one role series per market. The branch's history carries the withdrawn designs (per-cohort flow columns, state vocabulary, markers); the last commit takes it to v2, and the net diff against main is only this:cotmetrics.flow_roles, generated byscripts/analysis/gen_flow_roles.pyfrom the propadj cohort-collapse table (same-week price correlation, threshold 0.15, gross-share floor 0.05, measured 2026-09-26): aFlowRolesentry per (report, symbol) naming the speculator, counterparty, neutral and inert cohorts and how retail behaves, with a per-report default (Managed Money / Leveraged Funds) for markets the table does not cover. ZT is the one market with no speculator. The table is the propadj one rather than the cache-close one v2 names; that was the user's call, because the cache close is additively back-adjusted and truncated HO, RB and OJ.cotmetrics.flows:weekly_change(NaN across a missed report week and a contract-code switch, viaSource Codefrombuild_category_frame),flow_z(fixed 52-week window, min 26, no mean subtracted, NaN never 0),speculator_net,retail_net, andspeculator_frame.CotIndexer.get_speculator_data(name, lookback): the speculator net, its range index at the page lookback, its flow z and the retail net, for the Positioning Index panel on /analysis (mspinola/cot-analyzer PR to follow).get_category_datais unchanged from main.Not built: the Legacy back-fill of the speculator series before 2006 (not in the requested scope).
Verification: store-free suite 517 passed, 3 skipped (main 498); ruff clean;
gen_flow_roles.py --checkconfirms the committed table is current. On the live store: Gold's speculator is Managed Money (retail spec-like), Corn's Managed Money (retail neutral), Euro's Asset Manager + Leveraged Funds (retail spec-like); Gold's speculator net equals its Managed Money net exactly.Merge after #51 so the cited docs resolve.
🤖 Generated with Claude Code