Skip to content

refactor(vintage): drop the flow decomposition, which was a less capable copy #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mspinola merged 1 commit into from
Aug 2, 2026
Merged

refactor(vintage): drop the flow decomposition, which was a less capable copy #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,36 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Removed
- **`vintage_flow.decompose` and the `cotdata-vintage flow` subcommand.** Flow
decomposition was duplicated in `crowdmon.futures.flow`, and measurement showed the two
were not alternatives but **one function**: this copy was that one at `tolerance=1.0`
with the gap rule off, agreeing on **100.000000% of 135,835 transitions (2006-2026, 27
markets) with zero mismatches**, and `d_long`/`d_short`/`d_net` identical on every row.
The copy here was strictly the less capable one: it could not decline to label a
genuinely two-sided week (no `mixed` state) and it differenced across a 294-day absence
as though it were a week. The general implementation stays in `crowdmon`, which is where
the module spec puts the positioning engine. The dedup could not go the other way,
because `crowdmon/tests/test_boundaries.py` forbids cotdata from importing its own
consumer. Measurement in `crowdmon/docs/design/amendments-2026-08-02.md` §B29, asserted
by `crowdmon/tests/test_flow_equivalence.py`.

**No deprecation path, and the reason is checkable rather than asserted.**
`decompose` was never in `cotdata.__all__`, has never appeared in this changelog under
any version, and is in no **published** release: PyPI carries 0.1.0, which predates the
entire vintage subsystem, and the 0.2.0 note below states in terms that tagging is not
publishing. Its only caller in the workspace was the CLI subcommand removed with it.

**Not removed:** `zero_sum_check`, `from_vintage` and `from_current_store` stay. The
zero-sum identity is a statement about cotdata's own parse and is consumed by
`crowdmon.futures.cot_adapter` on every load. `min_frac_oi`, the optional dead zone, has
no equivalent in the surviving implementation and is gone; it defaulted to off and
nothing set it.
- **`cotdata-vintage flow` is replaced by `cotdata-vintage zero-sum`**, which keeps the
half of that command that was cotdata's own: source selection plus the identity check,
now also reporting the rounding-tolerance column and any non-weekly intervals. A command
named `flow` that no longer decomposes flow would be a worse outcome than a renamed one.

## [0.2.0] - 2026-08-02

The vintage release. `cotdata` gains an as-published provenance layer beside the
Expand Down
34 changes: 33 additions & 1 deletion docs/design/cot_vintage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,27 @@ Two smaller things fell out:

## 7. Flow decomposition, and what it found in the canonical schema

Module spec §6.4 built as `vintage_flow.py`. Weekly ΔLong versus ΔShort per
> **The decomposition has been REMOVED from this package (2026-08-02). What it found has
> not.** `crowdmon.futures.flow.decompose` was built independently on 2026-08-01, and the
> two were carried side by side as defensible alternatives until somebody measured them:
> this one was that one **at `tolerance=1.0` with the gap rule off**, agreeing on
> **100.000000% of 135,835 transitions with zero mismatches** and identical `d_long`,
> `d_short` and `d_net` on every row. One function, with the copy here hard-wired to the
> corner where nothing is ever `mixed` and no interval is ever refused. The measurement is
> in `crowdmon/docs/design/amendments-2026-08-02.md` §B29 and is asserted by
> `crowdmon/tests/test_flow_equivalence.py`.
>
> **This section is kept rather than deleted** because what it records is a result about
> *the canonical schema*, obtained by running a decomposition over it. The zero-sum sweep,
> the spreading finding and the `validate()` finding below all stand and none of them
> depend on where the classifier lives. `zero_sum_check` stays here; it is a statement
> about cotdata's own parse.
>
> The design points immediately below are preserved as the record of how the classifier
> was arrived at. **They describe `crowdmon.futures.flow` now**, with one exception noted
> inline.

Module spec §6.4 was built as `vintage_flow.py`. Weekly ΔLong versus ΔShort per
market/category, labelled `new_longs` / `short_covering` / `new_shorts` /
`long_liquidation`. No prices, no contract master, no multiplier: every input is a column
the canonical schema already stores, which is what makes it a clean smoke test of that
Expand All @@ -374,6 +394,18 @@ Two design points that are not in the spec because they only appear against real
to overfit. An optional `min_frac_oi` dead zone adds a `quiet` state; it defaults to 0.0
because any other value is a judgement in the same class as the fragility weights and
belongs in a caller's config where it can be swept.

> **This is the one point that did not carry over, and the argument turned out to be
> half right.** Parameter-free was a real virtue and it cost the ability to say a week
> was genuinely two-sided: dominant-leg always commits, so a category that added 30,000
> longs and 28,000 shorts is called `new_longs`. The surviving implementation takes a
> dominance `tolerance` instead and reports a `mixed` state, which is the modal outcome
> at **60% of weeks** on the liquid panel, so the four-state table describes a minority
> of the data. The objection that a parameter is something to overfit was answered by
> sweeping it rather than by avoiding it: the tolerance moves 28.74% of labels across
> 0.15 to 0.40, and **not one of those changes moves a week from one pure state to a
> different pure state**. It gates whether the classifier commits, never which direction
> it commits to. `min_frac_oi` itself was never set by anything and is simply gone.
- **`oi_corroborates`.** Futures are closed and zero-sum, so contracts exist only because
somebody opened them: fresh positioning should coincide with rising open interest and
exits with falling. Where it does not, the label is describing a transfer of an existing
Expand Down
71 changes: 39 additions & 32 deletions src/cotdata/vintage_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,41 +270,51 @@ def _cmd_asof(args) -> int:
return 0


def _cmd_flow(args) -> int:
def _cmd_zero_sum(args) -> int:
"""The schema check that was the first half of `flow`.

`flow` also printed a decomposition, which has moved to `crowdmon.futures.flow` (it was
the same function there, hard-wired to one corner of its parameters). What is left is
cotdata's own claim about its own parse, so the subcommand is named for that instead of
keeping a name that would now be a lie.
"""
from . import vintage_flow

if args.source == "current":
canonical = vintage_flow.from_current_store(args.market)
print(f"vintage flow: {args.market} from the CURRENT-STATE store. NOT "
f"point-in-time. Revisions are already applied, so this is for looking at "
f"history and validating the schema, never for evaluating a rule.")
print(f"zero-sum: {args.market} from the CURRENT-STATE store. NOT point-in-time. "
f"Revisions are already applied, so this is for looking at history and "
f"validating the schema, never for evaluating a rule.")
else:
canonical = vintage_flow.from_vintage(as_of=args.as_of, market_code=args.market)
if canonical.empty:
print("vintage flow: no vintage observations yet. The vintage series begins "
"at first capture; use --source current for history.")
print("zero-sum: no vintage observations yet. The vintage series begins at "
"first capture; use --source current for history.")
return 0

z = vintage_flow.zero_sum_check(canonical)
unbalanced = int((~z["balanced"]).sum())
print(f" zero-sum: {len(z) - unbalanced}/{len(z)} weeks balanced"
outside = int((~z["within_tolerance"]).sum())
print(f" {len(z) - unbalanced}/{len(z)} weeks balanced exactly"
+ (f" *** {unbalanced} UNBALANCED ***" if unbalanced else ""))

fl = vintage_flow.decompose(canonical, min_frac_oi=args.min_frac_oi)
if args.category:
fl = fl[fl["category"] == args.category]
if fl.empty:
print(" no weeks to decompose.")
return 0
off = fl[fl["days_elapsed"] != 7]
print(f" {len(z) - outside}/{len(z)} within CFTC's own rounding tolerance"
+ (f" *** {outside} OUTSIDE ***" if outside else ""))

# Not a weekly interval is a real finding about the series, so it is reported here
# rather than lost with the decomposition: COT was FORTNIGHTLY before 1992-10-13, and
# holidays shift the rest. A consumer differencing these rows must handle it.
dates = z["report_date"].sort_values().drop_duplicates()
gaps = dates.diff().dt.days.dropna()
off = gaps[gaps != 7]
if len(off):
print(f" {len(off)} of {len(fl)} intervals are not 7 days (COT was FORTNIGHTLY "
f"before 1992-10-13, and holidays shift the rest). Those rows are not a "
f"weekly change and are not comparable to one.")
print(f"\n{fl['state'].value_counts().to_string()}\n")
cols = ["report_date", "category", "d_long", "d_short", "d_net", "d_oi",
"days_elapsed", "state", "oi_corroborates"]
print(fl.tail(args.last)[cols].to_string(index=False))
print(f" {len(off)} of {len(gaps)} intervals are not 7 days "
f"(min {int(off.min())}, max {int(off.max())}). Those are not weekly "
f"changes and are not comparable to one.")

cols = ["report_date", "long_total", "short_total", "spread_total",
"open_interest", "imbalance", "oi_gap", "balanced"]
print()
print(z.tail(args.last)[cols].to_string(index=False))
return 0


Expand Down Expand Up @@ -352,20 +362,17 @@ def main(argv=None) -> int:
a.add_argument("--market", default=None)
a.set_defaults(func=_cmd_asof)

fw = sub.add_parser("flow", help="Weekly flow decomposition per market/category.")
fw.add_argument("--market", required=True, help="CFTC market code, e.g. 088691.")
fw.add_argument("--source", choices=("vintage", "current"), default="vintage",
zs = sub.add_parser("zero-sum",
help="Check the long/short/open-interest identity per market-week.")
zs.add_argument("--market", required=True, help="CFTC market code, e.g. 088691.")
zs.add_argument("--source", choices=("vintage", "current"), default="vintage",
help="'vintage' is point-in-time and starts at first capture; "
"'current' reads the existing store back to 1986 but has "
"revisions already applied, so it is NOT point-in-time.")
fw.add_argument("--as-of", default=None, dest="as_of",
zs.add_argument("--as-of", default=None, dest="as_of",
help="Point-in-time timestamp (vintage source only).")
fw.add_argument("--category", default=None, help="e.g. noncommercial.")
fw.add_argument("--min-frac-oi", type=float, default=0.0, dest="min_frac_oi",
help="Dead zone as a fraction of prior open interest; weeks under it "
"on both legs are 'quiet'. Default 0.0 (no dead zone).")
fw.add_argument("--last", type=int, default=20, help="Rows to print (default 20).")
fw.set_defaults(func=_cmd_flow)
zs.add_argument("--last", type=int, default=20, help="Rows to print (default 20).")
zs.set_defaults(func=_cmd_zero_sum)

args = p.parse_args(argv)
config.store_root() # fail fast if COTDATA_STORE unset
Expand Down
Loading
Loading