diff --git a/.env.example b/.env.example index 0cadf74..4d0cf9a 100644 --- a/.env.example +++ b/.env.example @@ -1,2 +1,5 @@ # QuantCockpit uses local-only defaults. Add optional local overrides here. # QUANTCOCKPIT_DB_PATH=./quantcockpit.duckdb +# QUANTCOCKPIT_DEMO_AS_OF=2026-07-20T18:00:00Z + +# v0.2 has no AI provider integration and requires no model API key. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8ab886d..7861172 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,6 +2,7 @@ name: CI on: push: + branches: [main] pull_request: permissions: diff --git a/.gitignore b/.gitignore index 8597c6e..ac686cb 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,4 @@ quantcockpit-report.md .env.* !.env.example .gstack/ +.worktrees/ diff --git a/CHANGELOG.md b/CHANGELOG.md index ce494a1..3a0046c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ QuantCockpit 的重要变更记录在这里。版本遵循 [Semantic Versioning](https://semver.org/)。 +## [0.2.0] - 2026-07-20 + +### Added + +- 可先只读预览、再导入现有 CSV、JSON 或 JSONL 仓位快照,并用版本化映射配置适配不同列名和嵌套结构,不需要修改策略代码。 +- 可按严格组合身份查看最新仓位覆盖、gross / net、Top-1 / Top-5、HHI、资产类别、行业和国家敞口,并从页面或报告追溯到事件与映射哈希。 +- 合成演示现在同时包含策略事件和仓位快照,接入命令、API 契约、前端类型与失败状态均有自动测试。 + +### Boundaries + +- 尚不提供券商直连、成交重建、因子 Beta、VaR、压力测试、告警派发或 AI 摘要运行时。 +- 不完整的数值基础返回候选基础与覆盖率,明确空仓单独标记;两者都不会被伪装成数值零。 + ## [0.1.0] - 2026-07-20 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 799a157..097d1e7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -16,7 +16,7 @@ bun install 1. 先写能准确失败的最小测试,再写实现。 2. 事件、API 或错误语义变化时,同步更新 README 和架构文档。 -3. 示例必须完全合成,使用虚构策略名、固定时间和 `paper` 环境。 +3. 示例必须完全合成,使用虚构策略名、工具名和标的、固定时间以及 `paper` 环境。 4. 不新增交易执行、券商连接、收益承诺或自动调仓建议。 5. 保持 API 对外错误为安全摘要,不回显坏行、绝对路径或密钥。 @@ -35,7 +35,19 @@ bun audit ## 契约变更 -`schema_version` 是公共边界。新增必填字段、改变收益口径或改变幂等键都属于破坏性变更,必须提供迁移说明和旧数据测试。未知字段当前会被拒绝;不要用静默忽略来掩盖版本不匹配。 +`schema_version` 和仓位映射的 `profile_version` 都是公共边界。新增必填字段、改变收益/敞口口径或改变幂等键都属于破坏性变更,必须提供迁移说明和旧数据测试。未知字段当前会被拒绝;不要用静默忽略来掩盖版本不匹配。 + +## 贡献仓位适配 + +优先贡献声明式映射示例,只有现有 CSV / JSON / JSONL 读取器无法表达时才新增代码适配器。每个新适配必须同时提供: + +- 完全合成且不含品牌、账户或真实标的的最小输入夹具; +- 固定版本的映射配置,以及确定性的 `mapping_profile_hash` 测试; +- preview 输出测试和正式导入测试,证明 preview 不写数据库; +- 缺字段、坏数字、重复标的、半写尾行、超限输入和错误脱敏测试; +- README 接入命令或独立文档入口,并说明数据来源的时间、方向和币种口径。 + +不要把特定券商 SDK、凭据读取或订单接口塞进通用导入器。适配器的职责是把已有导出文件变成规范 `position_snapshot`,不是控制交易账户。 ## 报告问题 diff --git a/README.md b/README.md index ffed2e8..c897c9d 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > Local-first, read-only and traceable observability for quantitative strategy logs. -QuantCockpit 是一个**本地、只读、可追溯**的量化策略日志观测台:把版本化日频 JSONL 导入 DuckDB,再展示健康度、原始事件证据、UTC 日期交集相关性和本地 Markdown 报告。 +QuantCockpit 是一个**本地、只读、可追溯**的量化策略与仓位日志观测台:它把版本化策略事件,以及现有系统导出的 CSV / JSON / JSONL 仓位快照,规范化到本地 DuckDB,再展示健康度、事件证据、集中度、敞口、UTC 日期交集相关性和 Markdown 报告。 > **Public Alpha**:契约与交互仍可能调整,请勿用于生产告警或交易决策。示例数据全部合成。 @@ -51,8 +51,9 @@ export QUANTCOCKPIT_DB_PATH=/absolute/path/to/quantcockpit.duckdb 1. 看顶部的连接状态、模式与最后刷新时间。 2. 在“数据可信度”确认导入错误为空;故意导入坏行时,此处显示安全摘要,不回显原始敏感内容。 3. 在“策略健康”查看状态,再到“证据详情”核对规则、阈值、观测值和事件引用。 -4. 在“相关性”核对 `n` 和 UTC 日期窗口;不可计算时显示原因,不用 `0` 冒充结果。 -5. 复制页面底部的本地命令生成报告: +4. 在“仓位覆盖”确认快照时间、仓位数量和能力等级,再到“集中度与敞口”核对 gross、net、Top-N、HHI、分类覆盖率与事件引用。 +5. 在“相关性”核对 `n` 和 UTC 日期窗口;不可计算时显示原因,不用 `0` 冒充结果。 +6. 复制页面底部的本地命令生成报告: ```bash uv run scripts/generate_report.py --output quantcockpit-report.md --generated-at 2026-07-20T18:00:00Z @@ -60,7 +61,7 @@ uv run scripts/generate_report.py --output quantcockpit-report.md --generated-at 报告入口只提供可复制命令;只读 API 不会伪装成能够生成或下载报告。 -## JSONL 输入契约 +## 策略事件 JSONL 契约 每行是一条独立 JSON 事件。完整合成示例位于 [`examples/data`](examples/data)。 @@ -70,10 +71,10 @@ uv run scripts/generate_report.py --output quantcockpit-report.md --generated-at 公共字段: -- `schema_version`:当前固定为 `1.0`。 +- `schema_version`:既有策略事件使用 `1.0`;规范仓位快照使用 `1.1`。 - `strategy_id`:1–128 字符,只允许字母、数字、点、下划线和连字符。 - `environment`:`live` 或 `paper`;只描述日志来源环境,不代表平台连接了真实账户。 -- `event_type`:`heartbeat`、`run_status`、`nav` 或 `return`。 +- `event_type`:`heartbeat`、`run_status`、`nav`、`return` 或 `position_snapshot`;后者由仓位导入器生成,不要求现有策略直接输出。 - `event_time`:事件发生时间;必须是 UTC 且包含时区,例如尾缀 `Z`。 - `recorded_at`:日志记录时间;同样必须是 UTC。 - `source`:生成日志的来源标识。 @@ -94,6 +95,40 @@ uv run scripts/generate_report.py --output quantcockpit-report.md --generated-at uv run scripts/import_demo.py --database ./local.duckdb --data-dir ./path/to/jsonl ``` +## 零侵入接入现有仓位文件 + +你不需要改交易策略或接券商 SDK。先导出现有系统已经保存的 CSV、JSON 或 JSONL,再用一个版本化映射配置说明“哪一列是什么”。零侵入不等于零配置:不同机构没有统一的实盘仓位日志结构,QuantCockpit 把适配成本收敛到可审查、可复用的 JSON 配置,而不是散落在交易代码里的胶水逻辑。 + +先预览合成示例。`--preview` 只读取、校验和输出最多 5 个安全样本,不创建或修改数据库: + +```bash +uv run scripts/import_positions.py --input examples/positions/demo-positions.csv --profile examples/positions/demo-positions-profile.json --preview --observed-at 2026-07-20T18:00:00Z +``` + +确认 `snapshot_count`、来源字段、样本和 `mapping_profile_hash` 后再正式导入: + +```bash +uv run scripts/import_positions.py --input examples/positions/demo-positions.csv --profile examples/positions/demo-positions-profile.json --database ./quantcockpit.duckdb --observed-at 2026-07-20T18:00:00Z +``` + +正式使用时复制 [`examples/positions/demo-positions-profile.json`](examples/positions/demo-positions-profile.json),修改输入格式和字段绑定。配置的核心字段如下: + +| 区域 | 字段 | 说明 | +| --- | --- | --- | +| 配置 | `format` | `csv`、`json` 或 `jsonl` | +| 配置 | `layout` | 行式快照 `tabular_snapshot`,或含仓位数组的 `document_snapshot` | +| 配置 | `snapshot_scope` | 按元数据分组的 `grouped_rows`,或整文件一个快照的 `whole_file` | +| 快照元数据 | `strategy_id`、`environment`、`source`、`portfolio_id`、`snapshot_time` | 必填;每项可取固定 `literal` 或来源 `path` | +| 快照元数据 | `recorded_at`、`base_currency` | 可选;没有 `recorded_at` 时使用本次观察时间 | +| 仓位 | `instrument_id` | 必填;同一快照内与可选 `venue` 共同唯一 | +| 仓位数值 | `quantity`、`weight`、`market_value_base`、`exposure_value_base` | 至少映射一种;必须是有限定点十进制,不能用浮点或科学计数法 | +| 仓位分类 | `instrument_id_type`、`side`、`asset_class`、`sector`、`country` | 可选;缺失分类明确进入“未分类” | +| 转换 | `trim`、`uppercase`、`lowercase`、`decimal`、`utc_timestamp` | 按声明顺序执行;JSON 路径使用 RFC 6901 JSON Pointer | + +本地无偏移时间必须同时配置 `timestamp_format` 和 IANA `assume_timezone`;夏令时重叠或不存在的时间会被拒绝。输入文件上限 100 MiB,单条记录上限 1 MiB,单快照上限 100,000 个仓位。CSV 表头必须唯一。JSONL 半写尾行会明确报错,不会被静默吞掉。 + +仓位身份严格是 `(portfolio_id, strategy_id, environment, source)`。同一快照重放会跳过,内容变化且记录时间更新会保留修订;分析只选择评估时点及之前的最新 current 快照。映射配置按 RFC 8785 规范化后计算 SHA-256,并随结果作为证据引用,因此改列映射不会伪装成同一份数据。 + ## API API 是本机只读观察接口,不启用宽泛 CORS: @@ -104,6 +139,8 @@ GET /api/v1/strategies GET /api/v1/strategies/{strategy_id}/{environment}/health?source={source} GET /api/v1/correlations GET /api/v1/ingestion/errors +GET /api/v1/portfolios +GET /api/v1/portfolios/{portfolio_id}/exposure?strategy_id={strategy_id}&environment={environment}&source={source} ``` 快速检查: @@ -113,6 +150,8 @@ curl -fsS http://127.0.0.1:8000/healthz curl -fsS http://127.0.0.1:8000/api/v1/strategies curl -fsS 'http://127.0.0.1:8000/api/v1/strategies/healthy-demo/paper/health?source=synthetic-demo' curl -fsS http://127.0.0.1:8000/api/v1/correlations +curl -fsS http://127.0.0.1:8000/api/v1/portfolios +curl -fsS 'http://127.0.0.1:8000/api/v1/portfolios/synthetic-book/exposure?strategy_id=healthy-demo&environment=paper&source=synthetic-demo' ``` 健康接口的 `source` 查询参数必填,长度为 1–256。除 `/healthz` 外,API 每次请求都只读打开已经存在且由当前版本初始化的 DuckDB,绝不在请求路径执行 `CREATE` 或 `ALTER`。数据库缺失或未初始化时数据接口返回 `503` 且不会创建文件;`/healthz` 仍只表示 API 进程存活。先运行导入脚本,或显式执行一次 `DuckDBStore(path).close()`,再启动数据查询。 @@ -123,16 +162,19 @@ curl -fsS http://127.0.0.1:8000/api/v1/correlations 相关性是 Pearson 相关系数,严格按 UTC 日历日交集对齐,只保留双方均有限的样本,并且只读取评估时点及之前的收益。少于 3 个样本、无共同日期、零方差、重复日期或数值错误都会返回结构化 `reason` 和证据;不可计算不等于相关性为零。每一对结果还返回按左侧日期、右侧日期稳定排序的 `event_refs`,可直接追溯本次候选输入。 +仓位分析不会混拼不同数值基础。它按 `exposure_value_base`、`market_value_base`、`weight` 的顺序选择第一个覆盖全部有效仓位的基础,对应能力等级 3、2、1;没有完整基础时返回 `unavailable`、候选基础和覆盖率,不伪造 gross、net 或集中度。明确空仓返回 `empty_portfolio`,与数据缺失严格区分。分类覆盖单独报告,缺失标签归为 `unclassified`。 + 更完整的数据流、公式与故障语义见 [`docs/architecture.md`](docs/architecture.md)。 ## 架构与目录 ```text -examples/data/*.jsonl - ↓ 校验、幂等、修订、隔离 +examples/data/*.jsonl ──────────────┐ +现有 CSV / JSON / JSONL 仓位文件 ─→ 映射配置 + 安全预览 + ↓ 校验、幂等、修订、隔离 DuckDB (events / ingestion_runs / quarantine) ↓ current 只读查询 -健康度 + UTC 日期相关性 +健康度 + 仓位敞口/集中度 + UTC 日期相关性 ├── FastAPI → React 观察台 └── 本地 Markdown 报告 ``` @@ -153,11 +195,12 @@ make verify ## 局限与边界 - 本地单用户、单写者;没有认证、多租户、分布式锁或远程数据库支持。 -- 当前只处理日频事件;时区统一为 UTC,不负责交易所日历、停牌或节假日语义。 +- 策略收益仍按日频事件处理;仓位是离散快照,不是逐笔成交重建,也不负责交易所日历、停牌或节假日语义。 - 健康阈值是通用默认值,不替代策略自身的运行手册和告警系统。 - Pearson 相关性只描述选定窗口内的线性共同变化,不代表因果、未来稳定性或组合风险。 +- v0.2 不含券商直连、因子 Beta、VaR、压力测试、告警派发或 AI 报告运行时;当前报告是确定性的本地模板。 - 前端桌面优先,1024px 可用;低于 900px 会给出明确提示,不提供移动布局。 -- Alpha 不保证契约向后兼容;升级前请保留原始 JSONL。 +- Alpha 不保证契约向后兼容;升级前请保留原始输入文件和映射配置。 ## 作为求职作品的价值 @@ -166,3 +209,5 @@ make verify ## 贡献、安全与许可 贡献前请阅读 [`CONTRIBUTING.md`](CONTRIBUTING.md),安全问题请遵循 [`SECURITY.md`](SECURITY.md)。项目采用 [Apache License 2.0](LICENSE)。 + +设计判断与实施记录:[`v0.2 接入设计`](docs/superpowers/specs/2026-07-20-v0-2-position-ingestion-design.md)、[`v0.2 实施计划`](docs/superpowers/plans/2026-07-20-v0-2-position-ingestion.md)、[`v0.1 加固设计`](docs/superpowers/specs/2026-07-20-v0-1-hardening-design.md)、[`v0.1 加固计划`](docs/superpowers/plans/2026-07-20-v0-1-hardening.md) 和 [`Public Alpha 计划`](docs/superpowers/plans/2026-07-20-public-alpha.md)。这些文件保留“为什么这样做”的上下文,实际命令和支持范围以本 README 为准。 diff --git a/SECURITY.md b/SECURITY.md index 854d20d..9f3373b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -12,7 +12,7 @@ Public Alpha 只支持当前默认分支。尚未承诺长期维护窗口或安 ## 威胁模型与部署边界 -QuantCockpit 设计为本机只读观察工具:API 默认只监听 `127.0.0.1`,没有认证,也不应直接暴露到公网。若自行更改监听地址或部署到共享网络,必须另行提供 TLS、认证、授权、速率限制和日志脱敏;这不在 Public Alpha 的支持范围内。 +QuantCockpit 设计为本机只读观察工具:API 默认只监听 `127.0.0.1`,没有认证,也不应直接暴露到公网。仓位源文件只由本地进程读取,不上传到外部服务;DuckDB 保存校验后的规范事件、来源坐标、映射哈希,以及组成快照的原始记录证据。原始记录不会通过当前 API 返回,但导入前仍应删除与监控无关的敏感列。若自行更改监听地址或部署到共享网络,必须另行提供 TLS、认证、授权、速率限制和日志脱敏;这不在 Public Alpha 的支持范围内。 请把以下内容视为敏感并保持在 Git 之外: @@ -22,3 +22,5 @@ QuantCockpit 设计为本机只读观察工具:API 默认只监听 `127.0.0.1` - 含绝对路径或原始坏行的诊断输出。 仓库示例只允许固定、完全合成的 `paper` 数据。发现疑似真实数据或密钥时,请停止传播并按漏洞流程私密报告。 + +当前版本没有运行 AI 助手,也不会把日志或报告发送给模型提供商。未来若加入 AI 摘要,必须采用显式启用、数据最小化、供应商与保留策略可见、可在不配置模型时完整使用核心监控的设计;在这些边界实现并审计前,文档中的“AI 报告”只属于后续方向。 diff --git a/docs/architecture.md b/docs/architecture.md index 2009897..9edb0be 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -11,21 +11,30 @@ QuantCockpit 把外部策略日志视为不可信输入。系统只在本机导 3. 不可计算是合法结果,必须保留原因,不能用数值 `0` 代替。 4. 对外错误摘要不包含原始坏行、绝对路径或验证输入中的敏感值。 5. 公开策略身份是 `(strategy_id, environment, source)`;健康、失败、隔离与收益均不得跨来源合并。 -6. API 数据请求只读打开已初始化数据库,冷启动不得创建或迁移文件。 +6. 公开组合身份是 `(portfolio_id, strategy_id, environment, source)`;同名组合不得跨策略或来源合并。 +7. API 数据请求只读打开已初始化数据库,冷启动不得创建或迁移文件。 +8. 仓位分析只能选择一个覆盖完整的数值基础,不得把 weight、市值和敞口值拼成一个看似完整的组合。 ## 数据流 ```mermaid flowchart LR - A["版本化 JSONL"] --> B["Pydantic 严格校验"] - B -->|"有效"| C["幂等键与修订判定"] - B -->|"无效"| Q["quarantine 安全摘要"] + A["策略事件 JSONL"] --> B["事件契约校验"] + P["现有 CSV / JSON / JSONL 仓位文件"] --> SR["Source Reader"] + MP["版本化 Mapping Profile"] --> M["确定性映射"] + SR --> M + M --> PV["只读 Preview"] + M --> B + B -->|"有效"| C["幂等键、映射哈希与修订判定"] + B -->|"无效"| Q["quarantine / 安全错误摘要"] C --> D["DuckDB events"] D --> E["current 事件视图"] E --> H["健康规则"] E --> R["UTC 日期相关性"] + E --> X["Exposure / Concentration"] H --> S["只读服务层"] R --> S + X --> S Q --> S S --> API["FastAPI"] S --> MD["本地 Markdown 报告"] @@ -42,13 +51,19 @@ strategy_id + environment + event_type + event_time + source + schema_version 相同键、相同规范化内容是重复;相同键、内容变化且 `recorded_at` 更晚是修订。数据库保留原始 JSON、规范化 JSON、来源、行号、首次/最后观察时间和修订号;分析只使用 current 修订。 +仓位输入先经过有界 Source Reader:整文件最多 100 MiB、单条最多 1 MiB、单快照最多 100,000 个仓位。CSV 只支持行式快照并拒绝重复表头;JSON / JSONL 同时支持行式快照和 `positions_path` 指向数组的文档快照。半写 JSONL 尾行是显式失败,不会被当作成功 EOF。 + +`PositionMappingProfile` 只允许固定值、CSV 列名或 RFC 6901 JSON Pointer,再按声明顺序执行有限转换。配置通过 RFC 8785 规范化后计算 SHA-256。preview 与正式导入共用读取、映射和校验路径,但 preview 不打开 DuckDB,只输出来源字段、快照数、最多 5 个规范仓位样本、警告和映射哈希。 + +仓位事件使用 `schema_version=1.1` 和 `event_type=position_snapshot`。自然幂等键在通用字段外包含 `portfolio_id`;内容哈希包含规范仓位和映射哈希。没有来源 `recorded_at` 时,本次观察时间只用于版本排序,不进入内容哈希,因此同一文件重放仍能识别为重复。来源行范围单独保存用于追溯。 + 中间非法行进入隔离区。尾行解析失败被区分为 `incomplete_tail`,便于下一次导入补全。只有完整 JSON 且三个身份字段自身都有效时,隔离记录才归属某个策略三元身份。成功重放同一文件后,本次已不存在的旧隔离证据变为 `is_active=false` 并记录 `resolved_at`;同一证据再次出现会重新激活。导入批次无论成功失败都会留痕。 ### 读取 `CockpitService` 是 API 和报告的共同只读口径。FastAPI 每个数据请求通过 `DuckDBStore.open_existing` 独立只读打开并关闭 DuckDB,不执行 DDL;缺失或未初始化返回 `503`,`/healthz` 不访问数据库。schema 初始化和迁移只发生在显式写入路径 `DuckDBStore(path)`。报告直接调用服务层,不经过网络,也不会在无策略时生成看似成功的空报告。 -前端先检查 `/healthz`,再并发读取策略、相关性和导入错误;拿到策略列表后以 `source` 查询参数并发读取各个三元身份的证据。URL、React key 与证据分组都包含来源。局部请求失败只降级相应区块。`/healthz` 不可达则进入离线顶层状态并清空页面策略数据;组件卸载或外部中止后不再更新状态。 +前端先检查 `/healthz`,再并发读取策略、组合、相关性和导入错误;拿到列表后并发读取各个严格身份的健康证据与仓位敞口。URL、React key 与分组都包含来源。局部请求失败只降级相应区块。`/healthz` 不可达则进入离线顶层状态并清空页面数据;组件卸载或外部中止后不再更新状态。 ## 健康公式与规则 @@ -79,6 +94,29 @@ r = Σ((xᵢ - x̄)(yᵢ - ȳ)) / √(Σ(xᵢ - x̄)² · Σ(yᵢ - ȳ)²) - `duplicate_date`:同一侧存在重复 UTC 日期,输入不满足日频约束。 - `numerical_error`:计算中间值非有限或分母无效。 +## 仓位能力与敞口公式 + +每个评估时点只取严格四元组合身份中,`snapshot_time <= evaluated_at` 的最新 current 快照。先排除所有已提供数值均为零的仓位,再按以下优先级寻找覆盖全部有效仓位的单一基础: + +```text +exposure_value_base → Level 3 +market_value_base → Level 2 +weight → Level 1 +``` + +若没有任何基础达到 100% 覆盖,结果为 `unavailable`,只返回最佳候选基础、覆盖率、缺失字段和 `incomplete_basis`。不会跨基础填洞。明确没有有效仓位时返回 `empty_portfolio`,gross 和 net 为真实的零;Top-N 和 HHI 因分母不存在而保持不可计算。 + +对选定基础值 (v_i): + +```text +gross = Σ|vᵢ| +net = Σvᵢ +Top-N = Σ(N 个最大 |vᵢ|) / gross +HHI = Σ(|vᵢ| / gross)² +``` + +计算使用 Decimal 和 38 位局部精度,比例最多保留 18 位小数。资产类别、行业和国家按同一基础聚合;分类缺失进入 `unclassified`,同时单独返回分类字段覆盖率。响应带 `event:` 与 `mapping:sha256:` 证据引用。 + ## 故障语义 | 场景 | API / 页面语义 | 明确不做 | @@ -87,9 +125,12 @@ r = Σ((xᵢ - x̄)(yᵢ - ȳ)) / √(Σ(xᵢ - x̄)² · Σ(yᵢ - ȳ)²) | API 在线、无策略 | `empty` / 尚未导入 | 不把空数据说成健康 | | 数据库缺失或未初始化 | 数据接口 `503`;`/healthz` 仍为 200 | 不在 API 请求中创建或迁移数据库 | | 策略列表成功、某些详情失败 | `partial` / 对应区块失败 | 不丢弃已成功的策略列表 | +| 组合列表成功、某个敞口详情失败 | `partial` / 对应组合失败 | 不丢弃列表和其他详情 | | 非法 JSONL 中间行 | quarantine 安全摘要 | 不吞错,不向 API 回显原始行 | | 半写尾行 | `incomplete_tail` | 不假装导入成功 | | 相关性不可计算 | `correlation=null` + `reason` | 不返回 0 | +| 仓位数值基础覆盖不全 | `unavailable` + 候选基础 + 覆盖率 | 不跨基础拼接,不返回假指标 | +| 明确空仓 | `empty_portfolio` | 不与数据缺失混为一谈 | | 无策略生成报告 | 非零退出码 | 不写空报告 | ## 可复现演示时钟 @@ -106,4 +147,4 @@ r = Σ((xᵢ - x̄)(yᵢ - ȳ)) / √(Σ(xᵢ - x̄)² · Σ(yᵢ - ȳ)²) ## 当前局限 -本实现假设本地单用户、单写者和日频事件,不解决多写者事务、交易所日历、分钟级流处理、远程认证、告警派发、组合优化或订单执行。健康和相关性是观察信号,不是完整风险模型。 +本实现假设本地单用户、单写者;策略收益是日频事件,仓位是离散快照。它不解决多写者事务、交易所日历、逐笔成交重建、分钟级流处理、远程认证、告警派发、因子 Beta、VaR、压力测试、组合优化、AI 摘要或订单执行。健康、集中度和相关性是观察信号,不是完整风险模型。 diff --git a/docs/assets/quantcockpit-demo.png b/docs/assets/quantcockpit-demo.png index b016345..f646d7c 100644 Binary files a/docs/assets/quantcockpit-demo.png and b/docs/assets/quantcockpit-demo.png differ diff --git a/docs/superpowers/plans/2026-07-20-v0-2-position-ingestion.md b/docs/superpowers/plans/2026-07-20-v0-2-position-ingestion.md new file mode 100644 index 0000000..8ca8c84 --- /dev/null +++ b/docs/superpowers/plans/2026-07-20-v0-2-position-ingestion.md @@ -0,0 +1,737 @@ +# QuantCockpit v0.2 仓位接入实施计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 让用户把已有 CSV、JSON、JSONL 仓位文件确定性地预览、导入并转化为可追溯的仓位覆盖与集中度结果。 + +**Architecture:** 保留 FastAPI + Python 领域内核,新增声明式映射层和 `position_snapshot` 事件,不把来源差异带入分析层。DuckDB 继续保存版本化事件;`CockpitService` 是 API、前端和 Markdown 报告共享的唯一分析入口。 + +**Tech Stack:** Python 3.13、Pydantic 2、DuckDB、FastAPI、RFC 8785、pytest、uv、React、TypeScript、Vite、Vitest、Bun。 + +## Global Constraints + +- 不修改或注入用户策略,不接收券商凭据,不从远程拉取账户数据。 +- `schema_version = "1.0"` 的现有事件必须保持兼容;`position_snapshot` 只允许 `1.1`。 +- 单文件上限 100 MiB、单条物理记录上限 1 MiB、单快照上限 100,000 个 position;超限不得截断导入。 +- 数字输入最多 30 位有效数字和 18 位小数;派生计算精度 38 位并使用 `ROUND_HALF_EVEN`。 +- 时间默认要求 RFC 3339 明确偏移;本地时间必须同时配置格式和 IANA 时区,DST 歧义或不存在时间拒绝。 +- 映射不能执行代码、模板、Shell 或网络请求;映射证据使用 RFC 8785 规范化结果的 SHA-256。 +- API 不返回绝对路径、原始记录或真实账号;空值、不可计算和真实零必须有不同语义。 +- 所有 Python 命令使用 uv,所有 Node.js 命令使用 Bun。 + +--- + +## 文件结构 + +- `src/quantcockpit/models.py`:事件 1.0/1.1 契约和仓位规范模型。 +- `src/quantcockpit/ingestion/position_profile.py`:映射配置、RFC 8785 指纹和有限转换。 +- `src/quantcockpit/ingestion/position_sources.py`:CSV、JSON、JSONL 的受限流式读取和来源坐标。 +- `src/quantcockpit/ingestion/positions.py`:预览、分组、规范化、整组错误和导入编排。 +- `src/quantcockpit/analysis/exposure.py`:Decimal 仓位覆盖、gross/net、Top-N、HHI 和分类计算。 +- `src/quantcockpit/store_types.py`、`src/quantcockpit/store.py`:仓位 current 查询和行范围证据。 +- `src/quantcockpit/service.py`:portfolio 身份、快照时效和敞口查询。 +- `src/quantcockpit/api_models.py`、`src/quantcockpit/api.py`:只读 portfolio API。 +- `scripts/import_positions.py`:预览优先的仓位导入 CLI。 +- `src/quantcockpit/report.py`:共享服务结果的 Markdown 仓位章节。 +- `examples/positions/`:完全合成的仓位文件和映射配置。 +- `frontend/src/types.ts`、`frontend/src/Dashboard.tsx`、`frontend/src/styles.css`:仓位覆盖与敞口界面。 +- `README.md`、`docs/architecture.md`、`CONTRIBUTING.md`:接入教程、边界与映射贡献规范。 + +### Task 1: 事件 1.1 与仓位契约 + +**Files:** +- Modify: `src/quantcockpit/models.py` +- Modify: `tests/test_models.py` + +**Interfaces:** +- Produces: `Position`, `PositionSnapshotPayload`, `EventRecord`; `EventRecord.idempotency_key` 对仓位额外包含 `portfolio_id`。 +- Consumes: 现有 `EventRecord` 1.0 行为和 Pydantic 严格校验风格。 + +- [ ] **Step 1: 写 1.0 兼容、1.1 仓位、空仓和 portfolio 隔离失败测试** + +```python +def position_event(**payload_overrides: object) -> dict[str, object]: + payload = { + "event_type": "position_snapshot", + "portfolio_id": "paper-book-a", + "base_currency": "USD", + "mapping_profile_hash": f"sha256:{'a' * 64}", + "positions": [{"instrument_id": "AAPL", "instrument_id_type": "ticker", "weight": "0.25"}], + } + payload.update(payload_overrides) + return event_data(schema_version="1.1", event_type="position_snapshot", payload=payload) + + +def test_position_snapshot_requires_schema_1_1_and_keeps_1_0_compatible() -> None: + assert EventRecord.model_validate(event_data()).schema_version == "1.0" + assert EventRecord.model_validate(position_event()).payload.event_type == "position_snapshot" + with pytest.raises(ValidationError): + EventRecord.model_validate(position_event() | {"schema_version": "1.0"}) + + +def test_position_snapshot_accepts_authoritative_empty_portfolio() -> None: + event = EventRecord.model_validate(position_event(positions=[])) + assert event.payload.positions == () + + +def test_portfolio_id_participates_only_in_position_idempotency_key() -> None: + left = EventRecord.model_validate(position_event(portfolio_id="book-a")) + right = EventRecord.model_validate(position_event(portfolio_id="book-b")) + assert left.idempotency_key != right.idempotency_key +``` + +- [ ] **Step 2: 运行测试并确认因 1.1 和仓位模型不存在而失败** + +Run: `uv run pytest tests/test_models.py -q` + +Expected: FAIL,错误指向 `schema_version`、`position_snapshot` 或未定义仓位载荷。 + +- [ ] **Step 3: 实现严格仓位模型与自然键** + +```python +SchemaVersion = Literal["1.0", "1.1"] +EventType = Literal["heartbeat", "run_status", "nav", "return", "position_snapshot"] + +class Position(EventPayloadBase): + instrument_id: Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=256)] + instrument_id_type: Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=32)] = "custom" + venue: Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=32)] | None = None + side: Literal["long", "short"] | None = None + quantity: Decimal | None = None + weight: Decimal | None = None + market_value_base: Decimal | None = None + exposure_value_base: Decimal | None = None + asset_class: str | None = None + sector: str | None = None + country: str | None = None + +class PositionSnapshotPayload(EventPayloadBase): + event_type: Literal["position_snapshot"] + portfolio_id: StrategyId + base_currency: Annotated[str, StringConstraints(pattern=r"^[A-Z0-9]{3,12}$")] | None = None + mapping_profile_hash: Annotated[str, StringConstraints(pattern=r"^sha256:[0-9a-f]{64}$")] + positions: tuple[Position, ...] +``` + +在 model validator 中完成四项规则:每个 position 至少一个度量、拒绝 float/NaN/Infinity/超精度、非零度量方向一致、同一快照 `(instrument_id_type, instrument_id, venue or "")` 唯一;存在基础币种价值时要求 `base_currency`。`event_type = position_snapshot` 且 schema 不是 1.1 时拒绝。序列化继续使用 Decimal 字符串。 + +- [ ] **Step 4: 运行模型测试和类型检查** + +Run: `uv run pytest tests/test_models.py -q && uv run ty check src/quantcockpit/models.py` + +Expected: PASS,且现有 1.0 测试不变。 + +- [ ] **Step 5: 提交契约** + +```bash +git add src/quantcockpit/models.py tests/test_models.py +git commit -m "feat: add position snapshot event contract" +``` + +### Task 2: 声明式映射配置与确定性转换 + +**Files:** +- Create: `src/quantcockpit/ingestion/position_profile.py` +- Create: `tests/test_position_profile.py` +- Modify: `pyproject.toml` +- Modify: `uv.lock` + +**Interfaces:** +- Produces: `FieldBinding`, `PositionMappingProfile`, `profile_hash(profile) -> str`, `resolve_binding(binding, record, *, format) -> object`。 +- Consumes: `rfc8785.dumps()`、`Decimal`、`zoneinfo.ZoneInfo`。 + +- [ ] **Step 1: 用 uv 添加 RFC 8785 依赖** + +Run: `uv add rfc8785` + +Expected: `pyproject.toml` 和 `uv.lock` 只新增 RFC 8785 直接/传递依赖。 + +- [ ] **Step 2: 写映射、指纹、JSON Pointer、Decimal 和时间失败测试** + +```python +def test_profile_hash_ignores_json_key_order() -> None: + left = PositionMappingProfile.model_validate(PROFILE) + right = PositionMappingProfile.model_validate(dict(reversed(list(PROFILE.items())))) + assert profile_hash(left) == profile_hash(right) + assert profile_hash(left).startswith("sha256:") + + +def test_local_timestamp_requires_format_and_timezone_together() -> None: + with pytest.raises(ValidationError): + FieldBinding(path="as_of", transforms=("utc_timestamp",), timestamp_format="%Y-%m-%d %H:%M:%S") + + +def test_json_pointer_uses_rfc_6901_escaping() -> None: + binding = FieldBinding(path="/account~1meta/book~0name", transforms=("trim",)) + assert resolve_binding(binding, {"account/meta": {"book~name": " alpha "}}, format="json") == "alpha" +``` + +参数化覆盖每个允许转换、缺失路径、禁止未知转换、只允许 literal/path 二选一、RFC 3339 非 UTC 偏移归一化、DST 歧义、本地不存在时间、30 位有效数字和 18 位小数边界。 + +- [ ] **Step 3: 运行测试并确认映射模块不存在** + +Run: `uv run pytest tests/test_position_profile.py -q` + +Expected: FAIL with `ModuleNotFoundError: quantcockpit.ingestion.position_profile`。 + +- [ ] **Step 4: 实现映射模型和纯函数转换器** + +```python +Transform = Literal["trim", "uppercase", "lowercase", "decimal", "utc_timestamp"] +InputFormat = Literal["csv", "json", "jsonl"] +Layout = Literal["document_snapshot", "tabular_snapshot"] +SnapshotScope = Literal["grouped_rows", "whole_file"] + +class FieldBinding(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True) + literal: object | None = None + path: str | None = None + transforms: tuple[Transform, ...] = () + timestamp_format: str | None = None + assume_timezone: str | None = None + +class PositionMappingProfile(BaseModel): + model_config = ConfigDict(extra="forbid", frozen=True) + profile_version: Literal["1.0"] + name: str + format: InputFormat + layout: Layout + snapshot_scope: SnapshotScope + fields: dict[str, FieldBinding] + position_fields: dict[str, FieldBinding] + positions_path: str | None = None +``` + +`profile_hash` 对 `profile.model_dump(mode="json", exclude_none=True)` 调用 `rfc8785.dumps` 后 SHA-256。`resolve_binding` 对 CSV 使用精确列名,对 JSON 使用 JSON Pointer;转换链逐项执行,错误统一抛出包含安全错误码但不含原始值的 `MappingValueError`。 + +- [ ] **Step 5: 运行映射测试和全量模型回归** + +Run: `uv run pytest tests/test_position_profile.py tests/test_models.py -q && uv run ty check src/quantcockpit/ingestion/position_profile.py` + +Expected: PASS。 + +- [ ] **Step 6: 提交映射层** + +```bash +git add pyproject.toml uv.lock src/quantcockpit/ingestion/position_profile.py tests/test_position_profile.py +git commit -m "feat: add deterministic position mapping profiles" +``` + +### Task 3: 受限来源读取与预览 + +**Files:** +- Create: `src/quantcockpit/ingestion/position_sources.py` +- Create: `src/quantcockpit/ingestion/positions.py` +- Create: `tests/test_position_sources.py` +- Create: `tests/test_position_preview.py` + +**Interfaces:** +- Produces: `SourceRecord`, `SourceLimitError`, `read_source(path, profile) -> Iterator[SourceRecord]`, `preview_positions(path, profile, observed_at) -> PositionPreview`。 +- Consumes: Task 1 `EventRecord`、Task 2 `PositionMappingProfile`/`resolve_binding`/`profile_hash`。 + +- [ ] **Step 1: 写 CSV/JSON/JSONL 读取边界失败测试** + +```python +def test_csv_reader_preserves_multiline_record_line_range(tmp_path: Path) -> None: + path = tmp_path / "positions.csv" + path.write_text('symbol,note\nAAPL,"line one\nline two"\n', encoding="utf-8") + rows = list(read_source(path, csv_profile())) + assert [(row.start_line, row.end_line, row.value["symbol"]) for row in rows] == [(2, 3, "AAPL")] + + +def test_source_larger_than_100_mib_is_rejected_before_parse(tmp_path: Path) -> None: + path = tmp_path / "large.json" + path.write_bytes(b" " * (100 * 1024 * 1024 + 1)) + with pytest.raises(SourceLimitError, match="ingestion_limit_exceeded"): + list(read_source(path, json_profile())) +``` + +另测:1 MiB 单条记录、JSONL 半写尾行、JSON 顶层类型错误、UTF-8 错误、100,001 仓位、文件扩展名与 profile format 不一致。 + +- [ ] **Step 2: 写预览只读、脱敏、分组和空仓失败测试** + +```python +def test_preview_normalizes_short_side_without_writing_database(tmp_path: Path) -> None: + result = preview_positions( + write_csv(tmp_path, "symbol,side,weight\nAAPL,short,0.25\n"), + profile_with_literals(), + observed_at=UTC_NOW, + ) + assert result.snapshots[0].event.payload.positions[0].weight == Decimal("-0.25") + assert result.sample_positions[0]["instrument_id"] == "AAPL" + assert len(result.mapping_profile_hash) == 71 + + +def test_empty_file_is_not_an_empty_portfolio_without_whole_file_scope(tmp_path: Path) -> None: + with pytest.raises(PositionImportError, match="empty_input"): + preview_positions(write_csv(tmp_path, "symbol,weight\n"), grouped_profile(), observed_at=UTC_NOW) +``` + +另测:最多五条样本、组内身份冲突整组失败、重复标的、recorded_at 默认 import_observed_at、event_time 等于 snapshot_time、显式空数组、分类字段不猜测。 + +- [ ] **Step 3: 运行测试并确认来源与预览模块不存在** + +Run: `uv run pytest tests/test_position_sources.py tests/test_position_preview.py -q` + +Expected: FAIL with missing modules。 + +- [ ] **Step 4: 实现流式来源记录和预览领域对象** + +```python +@dataclass(frozen=True) +class SourceRecord: + value: Mapping[str, object] + start_line: int + end_line: int + raw_json: str + +@dataclass(frozen=True) +class NormalizedSnapshot: + event: EventRecord + raw_json: str + start_line: int + end_line: int + +@dataclass(frozen=True) +class PositionPreview: + format: InputFormat + record_count: int + snapshot_count: int + source_fields: tuple[str, ...] + sample_positions: tuple[dict[str, object], ...] + snapshots: tuple[NormalizedSnapshot, ...] + mapping_profile_hash: str + warnings: tuple[dict[str, object], ...] +``` + +CSV 使用 `csv.DictReader` 并以 `reader.line_num` 记录多行字段范围;JSONL 逐行读取;JSON 在读取前检查文件大小。预览先完整规范化再返回,不接收 store。样本只包含规范字段,`portfolio_id` 和原始行不进入 sample。 + +- [ ] **Step 5: 运行来源、预览、模型和映射测试** + +Run: `uv run pytest tests/test_position_sources.py tests/test_position_preview.py tests/test_position_profile.py tests/test_models.py -q` + +Expected: PASS。 + +- [ ] **Step 6: 提交来源与预览** + +```bash +git add src/quantcockpit/ingestion/position_sources.py src/quantcockpit/ingestion/positions.py tests/test_position_sources.py tests/test_position_preview.py +git commit -m "feat: preview mapped position files" +``` + +### Task 4: 仓位导入 CLI、证据范围与版本化存储 + +**Files:** +- Modify: `src/quantcockpit/store.py` +- Modify: `src/quantcockpit/store_types.py` +- Modify: `src/quantcockpit/ingestion/positions.py` +- Create: `scripts/import_positions.py` +- Create: `tests/test_position_ingestion.py` +- Modify: `tests/test_scripts.py` + +**Interfaces:** +- Produces: `import_positions(store, path, profile, observed_at) -> PositionImportResult`、CLI `--input --profile --database [--preview]`。 +- Consumes: Task 3 `preview_positions` 和现有 `DuckDBStore.record_event` 修订机制。 + +- [ ] **Step 1: 写预览不写库、导入幂等、修订、行范围和安全错误测试** + +```python +def test_position_import_reuses_event_revision_semantics(tmp_path: Path) -> None: + store = DuckDBStore(tmp_path / "cockpit.duckdb") + first = import_positions(store, CSV_PATH, PROFILE, observed_at=FIRST_SEEN) + duplicate = import_positions(store, CSV_PATH, PROFILE, observed_at=SECOND_SEEN) + assert (first.imported, duplicate.duplicates) == (1, 1) + + +def test_preview_cli_does_not_create_database(tmp_path: Path) -> None: + database = tmp_path / "must-not-exist.duckdb" + completed = run_import_positions("--preview", "--database", str(database)) + assert completed.returncode == 0 + assert not database.exists() + assert str(tmp_path) not in completed.stdout +``` + +另测:同自然键晚 recorded_at 形成 revision、早 recorded_at 为 stale、`start_line/end_line` 持久化、组错误不写半快照、资源超限记录失败批次、CLI 错误输出不含原始账号和绝对路径。 + +- [ ] **Step 2: 运行测试并确认导入接口和 CLI 不存在** + +Run: `uv run pytest tests/test_position_ingestion.py tests/test_scripts.py -q` + +Expected: FAIL,错误指向 `import_positions` 或脚本缺失。 + +- [ ] **Step 3: 扩展存储证据并实现导入** + +```python +@dataclass(frozen=True) +class PositionImportResult: + imported: int = 0 + duplicates: int = 0 + revisions: int = 0 + stale: int = 0 + rejected_snapshots: int = 0 + +def import_positions( + store: DuckDBStore, + path: str | Path, + profile: PositionMappingProfile, + *, + observed_at: datetime | None = None, +) -> PositionImportResult: + preview = preview_positions(path, profile, observed_at=observed_at or datetime.now(timezone.utc)) + return _persist_snapshots(store, Path(path), preview) +``` + +`events` 新增 `end_line_number INTEGER`,已有行迁移为 `line_number`。`record_event(..., end_line_number: int | None = None)` 在单事务内保留 current 切换;原始表格组保存为受限 JSON 数组。CLI 的 preview 输出固定 JSON 摘要,正式导入后输出计数。 + +- [ ] **Step 4: 运行导入、旧 JSONL 和安全回归测试** + +Run: `uv run pytest tests/test_position_ingestion.py tests/test_ingestion.py tests/test_security_regressions.py tests/test_scripts.py -q` + +Expected: PASS,旧 JSONL 导入语义不变。 + +- [ ] **Step 5: 提交导入闭环** + +```bash +git add src/quantcockpit/store.py src/quantcockpit/store_types.py src/quantcockpit/ingestion/positions.py scripts/import_positions.py tests/test_position_ingestion.py tests/test_scripts.py +git commit -m "feat: import versioned position snapshots" +``` + +### Task 5: 敞口分析内核 + +**Files:** +- Create: `src/quantcockpit/analysis/exposure.py` +- Create: `tests/test_exposure.py` + +**Interfaces:** +- Produces: `analyze_exposure(snapshot: PositionSnapshotPayload) -> ExposureAnalysis`。 +- Consumes: Task 1 `PositionSnapshotPayload`。 + +- [ ] **Step 1: 写能力、基础优先级、真实零和分类失败测试** + +```python +def test_exposure_prefers_complete_risk_basis_and_uses_decimal() -> None: + result = analyze_exposure(snapshot( + position("A", weight="0.6", market_value_base="60", exposure_value_base="120"), + position("B", weight="-0.4", market_value_base="-40", exposure_value_base="-80"), + )) + assert result.basis == "exposure_value_base" + assert result.gross == Decimal("200") + assert result.net == Decimal("40") + assert result.top_1_share == Decimal("0.6") + assert result.hhi == Decimal("0.52") + + +def test_incomplete_basis_returns_reason_instead_of_false_zero() -> None: + result = analyze_exposure(snapshot(position("A", weight="0.5"), position("B", quantity="2"))) + assert result.gross is None + assert result.reason == "incomplete_basis" + assert result.candidate_basis == "weight" + assert result.coverage_ratio == Decimal("0.5") +``` + +另测:quantity-only Level 0、基础优先级、Top-5 少于五项、zero gross、显式空仓、零 position 计数、long/short、unclassified、分类覆盖率和计算精度。 + +- [ ] **Step 2: 运行测试并确认分析模块不存在** + +Run: `uv run pytest tests/test_exposure.py -q` + +Expected: FAIL with missing module。 + +- [ ] **Step 3: 实现不可变分析结果和单一公式** + +```python +Basis = Literal["weight", "market_value_base", "exposure_value_base"] + +@dataclass(frozen=True) +class ExposureAnalysis: + state: Literal["ready", "unavailable", "empty_portfolio"] + level: int + basis: Basis | None + candidate_basis: Basis | None + position_count: int + zero_position_count: int + long_count: int + short_count: int + gross: Decimal | None + net: Decimal | None + top_1_share: Decimal | None + top_5_share: Decimal | None + hhi: Decimal | None + coverage_ratio: Decimal | None + coverage_by_basis: dict[Basis, Decimal | None] + categories: dict[str, tuple[CategoryExposure, ...]] + category_coverage: dict[str, Decimal | None] + reason: str | None +``` + +用同一 `_measure(position, basis)` 驱动 gross、net、排名和分类;只在完整基础上计算数值。所有除法和求和置于 `localcontext(Context(prec=38, rounding=ROUND_HALF_EVEN))`,序列化前统一量化到最多 18 位小数。 + +- [ ] **Step 4: 运行分析测试和类型检查** + +Run: `uv run pytest tests/test_exposure.py -q && uv run ty check src/quantcockpit/analysis/exposure.py` + +Expected: PASS。 + +- [ ] **Step 5: 提交分析内核** + +```bash +git add src/quantcockpit/analysis/exposure.py tests/test_exposure.py +git commit -m "feat: calculate traceable position exposure" +``` + +### Task 6: portfolio 存储查询、服务层与 API + +**Files:** +- Modify: `src/quantcockpit/store_types.py` +- Modify: `src/quantcockpit/store.py` +- Modify: `src/quantcockpit/service.py` +- Modify: `src/quantcockpit/api_models.py` +- Modify: `src/quantcockpit/api.py` +- Modify: `tests/test_api.py` +- Modify: `tests/test_openapi_export.py` +- Regenerate: `frontend/openapi.json` +- Regenerate: `frontend/src/generated/api.ts` + +**Interfaces:** +- Produces: `CockpitService.portfolios()`、`CockpitService.portfolio_exposure(...)`、`GET /api/v1/portfolios`、`GET /api/v1/portfolios/{portfolio_id}/exposure`。 +- Consumes: Task 5 `analyze_exposure` 和 store current position rows。 + +- [ ] **Step 1: 写严格四元身份、最新 current、未来快照和响应结构失败测试** + +```python +def test_portfolio_api_isolates_identity_and_returns_mapping_evidence(tmp_path: Path) -> None: + client = seeded_position_client(tmp_path) + response = client.get("/api/v1/portfolios/book-a/exposure?strategy_id=alpha&environment=paper&source=broker-export") + assert response.status_code == 200 + body = response.json() + assert body["portfolio_id"] == "book-a" + assert body["basis"] == "market_value_base" + assert body["evidence_refs"][1].startswith("mapping:sha256:") + + +def test_portfolio_detail_does_not_cross_source_or_environment(tmp_path: Path) -> None: + client = seeded_position_client(tmp_path) + assert client.get("/api/v1/portfolios/book-a/exposure?strategy_id=alpha&environment=live&source=broker-export").status_code == 404 +``` + +另测:空列表 state、快照年龄使用注入 clock、未来快照不参与、revision 只读 current、不可计算字段为 null、显式空仓 gross/net 为字符串 `"0"`、缺 query 返回 422、OpenAPI 使用命名模型。 + +- [ ] **Step 2: 运行 API 测试并确认端点不存在** + +Run: `uv run pytest tests/test_api.py tests/test_openapi_export.py -q` + +Expected: FAIL,portfolio 路由为 404。 + +- [ ] **Step 3: 实现 current 查询、服务 DTO 与响应模型** + +```python +@dataclass(frozen=True) +class PortfolioIdentity: + portfolio_id: str + strategy_id: str + environment: Environment + source: str + +@dataclass(frozen=True) +class PortfolioExposure: + identity: PortfolioIdentity + snapshot_time: datetime + age_seconds: int + mapping_profile_hash: str + analysis: ExposureAnalysis + evidence_refs: tuple[str, str] +``` + +`DuckDBStore.current_position_snapshots(evaluated_at)` 只查询 current、`event_type = position_snapshot`、`event_time <= evaluated_at`,并以四元身份和 event_time 选最新。服务解析 `normalized_json` 回 `EventRecord`,调用一次 `analyze_exposure`。API Decimal 全部序列化为字符串,避免浏览器二进制浮点改写。 + +- [ ] **Step 4: 导出 OpenAPI 并验证无漂移** + +Run: `cd frontend && bun run generate:api && cd .. && uv run pytest tests/test_api.py tests/test_openapi_export.py -q` + +Expected: PASS,生成文件包含 `PortfoliosResponse` 和 `PortfolioExposureResponse`。 + +- [ ] **Step 5: 提交服务与 API** + +```bash +git add src/quantcockpit/store_types.py src/quantcockpit/store.py src/quantcockpit/service.py src/quantcockpit/api_models.py src/quantcockpit/api.py tests/test_api.py tests/test_openapi_export.py frontend/openapi.json frontend/src/generated/api.ts +git commit -m "feat: expose portfolio coverage and exposure api" +``` + +### Task 7: Markdown 报告与合成演示 + +**Files:** +- Modify: `src/quantcockpit/report.py` +- Modify: `scripts/import_demo.py` +- Create: `examples/positions/demo-positions.csv` +- Create: `examples/positions/demo-position-profile.json` +- Modify: `tests/test_report.py` +- Modify: `tests/test_scripts.py` + +**Interfaces:** +- Produces: 报告“仓位覆盖与集中度”章节;`import_demo.py` 同时导入旧事件和合成仓位。 +- Consumes: Task 6 `CockpitService.portfolios` / `portfolio_exposure`。 + +- [ ] **Step 1: 写报告证据、不可计算原因和演示导入失败测试** + +```python +def test_report_contains_position_basis_metrics_and_evidence(tmp_path: Path) -> None: + markdown = render_markdown(position_service(tmp_path), generated_at=NOW) + assert "## 仓位覆盖与集中度" in markdown + assert "basis = market_value_base" in markdown + assert "mapping:sha256:" in markdown + assert "event:" in markdown + + +def test_demo_import_includes_synthetic_position_snapshot(tmp_path: Path) -> None: + result = run_demo_import(tmp_path / "demo.duckdb") + assert result.returncode == 0 + assert "demo-positions.csv" in result.stdout +``` + +- [ ] **Step 2: 运行报告和脚本测试并确认仓位章节缺失** + +Run: `uv run pytest tests/test_report.py tests/test_scripts.py -q` + +Expected: FAIL,报告不存在仓位章节或 demo 未导入仓位。 + +- [ ] **Step 3: 复用服务层渲染报告并加入合成文件** + +报告按 portfolio 四元身份排序;ready 显示 basis、coverage、gross、net、Top-1、Top-5、HHI 和分类;unavailable 显示 candidate basis/coverage/reason;empty portfolio 显示真实 0。所有字符串继续经过 `_markdown_inline`。演示 CSV 使用虚构账户别名和价格,不包含真实市场或个人数据。 + +- [ ] **Step 4: 运行报告、脚本和安全回归** + +Run: `uv run pytest tests/test_report.py tests/test_scripts.py tests/test_security_regressions.py -q` + +Expected: PASS。 + +- [ ] **Step 5: 提交报告与演示** + +```bash +git add src/quantcockpit/report.py scripts/import_demo.py examples/positions tests/test_report.py tests/test_scripts.py +git commit -m "feat: report synthetic portfolio exposure" +``` + +### Task 8: React 仓位观察台 + +**Files:** +- Modify: `frontend/src/types.ts` +- Modify: `frontend/src/Dashboard.tsx` +- Modify: `frontend/src/styles.css` +- Modify: `frontend/tests/api.test.ts` +- Modify: `frontend/tests/dashboard.test.tsx` + +**Interfaces:** +- Produces: “仓位覆盖”和“集中度与敞口”面板,保持 loading/empty/partial/offline 语义。 +- Consumes: Task 6 生成的 OpenAPI TypeScript 类型和两个只读端点。 + +- [ ] **Step 1: 写 portfolio 请求、ready、unavailable、empty 和 partial 失败测试** + +```typescript +it("renders portfolio coverage and exposure without inventing missing metrics", async () => { + mockApi({ portfolios: ONE_PORTFOLIO, exposure: UNAVAILABLE_EXPOSURE }); + render(); + expect(await screen.findByRole("heading", { name: "仓位覆盖" })).toBeInTheDocument(); + expect(screen.getByText("3/12 个仓位缺少基础币种市场价值")).toBeInTheDocument(); + expect(screen.queryByText(/^0\.00$/)).not.toBeInTheDocument(); +}); + + +it("keeps strategy panels when one portfolio detail request fails", async () => { + mockApi({ portfolios: TWO_PORTFOLIOS, failedPortfolio: "book-b" }); + render(); + expect(await screen.findByText("部分数据不可用")).toBeInTheDocument(); + expect(screen.getByText("book-a")).toBeInTheDocument(); + expect(screen.getByText("book-b · 敞口加载失败")).toBeInTheDocument(); +}); +``` + +- [ ] **Step 2: 运行前端测试并确认仓位 UI 不存在** + +Run: `cd frontend && bun run test` + +Expected: FAIL,找不到仓位标题或请求。 + +- [ ] **Step 3: 实现并行加载和两个无新依赖面板** + +在 `DashboardData` 增加 portfolios、exposures 和 exposureFailures。策略、相关性、错误、portfolio 列表继续 `Promise.allSettled`;portfolio 详情按严格 URL 编码并行加载。数值只格式化 API Decimal 字符串,不在浏览器重算 HHI。分类使用 CSS 条形比例,`unclassified` 明示为“未分类”。 + +- [ ] **Step 4: 运行前端测试、类型检查和构建** + +Run: `cd frontend && bun run test && bun run typecheck && bun run build` + +Expected: PASS,无 TypeScript 诊断和 Vite 构建警告。 + +- [ ] **Step 5: 提交前端闭环** + +```bash +git add frontend/src/types.ts frontend/src/Dashboard.tsx frontend/src/styles.css frontend/tests/api.test.ts frontend/tests/dashboard.test.tsx +git commit -m "feat: show portfolio exposure in cockpit" +``` + +### Task 9: 开源接入文档、全量验证与版本说明 + +**Files:** +- Modify: `README.md` +- Modify: `docs/architecture.md` +- Modify: `CONTRIBUTING.md` +- Modify: `SECURITY.md` +- Modify: `CHANGELOG.md` +- Modify: `.env.example` + +**Interfaces:** +- Produces: 10 分钟仓位接入路径、映射字段词典、隐私边界、适配器贡献规则和 v0.2 版本说明。 +- Consumes: Tasks 1–8 的实际 CLI、API、示例和界面。 + +- [ ] **Step 1: 写文档命令回归测试** + +在 `tests/test_scripts.py` 增加 README 中两条命令的真实执行: + +```python +def test_readme_position_preview_and_import_commands(tmp_path: Path) -> None: + preview = run_documented_position_command(tmp_path, preview=True) + imported = run_documented_position_command(tmp_path, preview=False) + assert preview.returncode == 0 + assert imported.returncode == 0 + assert "snapshot_count" in preview.stdout + assert "imported=1" in imported.stdout +``` + +- [ ] **Step 2: 运行测试并确认文档辅助夹具不存在** + +Run: `uv run pytest tests/test_scripts.py -q` + +Expected: FAIL,README 尚未包含可提取的仓位命令或示例路径。 + +- [ ] **Step 3: 更新文档和版本说明** + +README 依次给出 `uv sync`、preview、import、API、前端和报告命令;明确“零侵入不等于零配置”。架构文档画出 Source Reader → Mapping Profile → Event → Exposure 流;CONTRIBUTING 规定新适配器必须提供合成夹具、映射、preview 快照和安全测试;SECURITY 说明原始文件只进入本地 DuckDB、AI 助手尚未运行;CHANGELOG 在 Unreleased 下列出 v0.2 功能和明确不支持项。 + +- [ ] **Step 4: 运行完整后端与前端验证** + +Run: `make verify` + +Expected: OpenAPI 无漂移;所有 pytest/Vitest 通过;Python/TypeScript 类型检查通过;Vite 构建成功。 + +- [ ] **Step 5: 运行合成端到端冒烟测试** + +Run: `uv run scripts/import_demo.py --database /tmp/quantcockpit-v0-2-demo.duckdb && QUANTCOCKPIT_DB_PATH=/tmp/quantcockpit-v0-2-demo.duckdb uv run scripts/generate_report.py --output /tmp/quantcockpit-v0-2-report.md --generated-at 2026-07-20T18:00:00Z` + +Expected: 导入输出包含仓位快照;报告包含“仓位覆盖与集中度”、`event:` 和 `mapping:sha256:`。 + +- [ ] **Step 6: 提交文档并记录最终验证** + +```bash +git add README.md docs/architecture.md CONTRIBUTING.md SECURITY.md CHANGELOG.md .env.example tests/test_scripts.py +git commit -m "docs: document zero-intrusion position onboarding" +``` + +## 规范覆盖自审 + +- 输入布局、映射、时间、符号、数字、币种、唯一键、空仓和资源上限:Tasks 1–4。 +- 幂等、修订、原始证据和预览不写库:Task 4。 +- 能力分级、覆盖率、gross/net、Top-N、HHI 和分类:Task 5。 +- 严格 portfolio 身份、最新 current、API 和 OpenAPI:Task 6。 +- 报告和完全合成演示:Task 7。 +- 前端 ready/empty/partial/offline 和不可计算语义:Task 8。 +- 10 分钟接入、隐私、适配器贡献和全量验证:Task 9。 +- 券商直连、成交重建、AI 运行时、因子 Beta、VaR 和告警派发均保持在 v0.2 范围之外。 diff --git a/docs/superpowers/specs/2026-07-20-v0-2-position-ingestion-design.md b/docs/superpowers/specs/2026-07-20-v0-2-position-ingestion-design.md new file mode 100644 index 0000000..d315a23 --- /dev/null +++ b/docs/superpowers/specs/2026-07-20-v0-2-position-ingestion-design.md @@ -0,0 +1,477 @@ +# QuantCockpit v0.2 零侵入仓位接入与敞口分析设计 + +## 目标 + +让用户在不修改实盘或模拟盘策略代码的前提下,把已有 CSV、JSON、JSONL 日志或账户报表映射成可追溯的仓位快照,并在数据能力允许时计算 gross、net、Top-N 和 HHI 集中度。 + +v0.2 的产品重点不是增加更多风险公式,而是建立一个可扩展、低门槛、不会伪造精度的接入层。任何分析结果都必须能追溯到原始文件、映射配置和规范化事件。 + +这里的“零侵入”只表示不修改策略代码、不嵌入 SDK、不提供券商凭据;它不等于零配置。一个全新来源首次接入仍需创建映射配置,但配置可以预览、复用、版本化并由未来的 AI 助手生成候选稿。 + +## 行业现状与设计判断 + +现有标准分布在不同层级,没有一个标准同时覆盖通用日志、策略身份、订单成交、账户仓位和风险分析: + +- OpenTelemetry 定义通用日志记录、时间、资源、正文和属性,并提供文件采集能力,但不定义金融仓位语义。 +- FIX/FIXML 包含订单、成交和 PositionReport 等消息,但主要面向交易对手、券商、清算和 OMS 之间的消息交换,机构内部落地仍存在差异。 +- FINOS Common Domain Model 覆盖金融产品和交易生命周期,适合机构互操作,但作为 QuantCockpit 的最小接入契约过重。 +- 券商和平台通常提供自己的 API、CSV、XML 或报告服务,例如 IBKR Flex;这些出口允许零侵入接入,但需要来源适配器。 + +因此,QuantCockpit 不采用 FIX 或 CDM 作为内部存储模型,也不要求用户实现 SDK。系统采用“稳定的最小内部模型 + 外部声明式适配器”。标准只作为字段语义、标识体系和未来适配器的参考。 + +## 方案取舍 + +评估过三条接入路径: + +1. 要求用户在策略中发送统一事件或安装 SDK。它最适合实时、强语义数据,但直接违反“不改现有策略”的核心约束。 +2. QuantCockpit 直接连接每家券商和交易平台。它能减少部分用户配置,却引入凭据托管、限流、接口变更和大量来源维护,不适合作为 v0.2 的开源切口。 +3. 对用户已有文件做声明式映射。它需要一次字段配置,也无法制造源文件里没有的信息,但能覆盖最多现有系统,并把接入差异隔离在分析内核之外。 + +v0.2 选择第三条;后续券商适配器只是生成同一映射或同一规范事件的便捷入口,不形成第二套分析逻辑。 + +参考资料: + +- [OpenTelemetry Logs Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) +- [OpenTelemetry File Log Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) +- [FIX Trading Community machine-readable repository](https://github.com/FIXTradingCommunity/fix-repository) +- [FINOS Common Domain Model](https://cdm.finos.org/docs/cdm-overview) +- [IBKR Flex Web Service](https://www.interactivebrokers.com/campus/ibkr-api-page/flex-web-service/) + +## 范围 + +### v0.2 包含 + +- CSV、JSON 和 JSONL 的离线只读导入。 +- 两类输入布局:一份文档包含完整仓位数组;多行表格共同组成一个或多个快照。 +- 版本化 JSON 映射配置,支持字段路径、固定值和有限的确定性转换。 +- 导入前预览,包括源字段、规范化结果、缺失能力、拒绝原因和不会上传数据的声明。 +- `position_snapshot` 规范事件及其幂等、修订和隔离语义。 +- 原始记录、映射配置摘要、规范化事件和派生证据的追踪链。 +- 最新仓位快照、数据覆盖率、gross、net、Top-1、Top-5、HHI 和分类敞口。 +- API、前端观察台、Markdown 报告和合成演示数据的完整闭环。 +- 面向未来 AI 映射助手的稳定输入输出契约。 + +### v0.2 不包含 + +- 从成交记录重建权威仓位。 +- 连接券商、保存券商凭据或主动轮询远程账户。 +- 完整 FIX、FIXML、CDM、FpML 或 ISO 20022 实现。 +- 自动价格、汇率、公司行动、期货换月或期权到期处理。 +- VaR、Greeks、压力测试、因子 Beta 或组合优化。 +- 在导入和分析运行路径中调用大模型。 +- 修改、注入或重新部署用户的交易策略。 + +## 总体架构 + +```text +已有 CSV / JSON / JSONL / 报表 + │ + ▼ + Source Reader(只读) + │ 原始记录 + 来源坐标 + ▼ + Mapping Profile(版本化 JSON) + │ 有限转换、无任意代码 + ▼ + Preview + Capability + │ 用户显式执行导入 + ▼ + position_snapshot 事件契约 + │ 幂等、修订、隔离、current + ▼ + Exposure Analysis + ├── FastAPI + ├── React 观察台 + └── Markdown 报告 +``` + +接入层和分析层严格分离。接入层只回答“这份现有数据能否确定性映射成什么”;分析层只回答“在现有字段覆盖下,哪些指标可以计算”。 + +## 输入布局 + +### 文档快照 + +单个 JSON 或单条 JSONL 记录包含快照元数据和完整 `positions` 数组。每条记录天然具有原子边界,适合已经保存账户快照的系统。 + +### 表格快照 + +CSV 或扁平 JSON 行每行表示一个持仓。映射配置必须声明快照分组字段: + +- `strategy_id` +- `environment` +- `source` +- `portfolio_id` +- `snapshot_time` + +同一组的所有行组成一个原子仓位快照。缺少 `snapshot_time` 时,只允许通过显式配置把“整个文件”声明为一个快照;系统不得自行猜测时间。 + +JSONL 尾部不完整、CSV 行列错位、组内基础币种冲突或组内重复标的均进入隔离或预览错误,不产生部分仓位快照。 + +## 映射配置 + +映射配置使用 JSON,不引入可执行模板语言。配置本身经过严格 Pydantic 校验,并计算稳定摘要写入导入证据。 + +```json +{ + "profile_version": "1.0", + "name": "example-position-csv", + "format": "csv", + "layout": "tabular_snapshot", + "snapshot_scope": "grouped_rows", + "fields": { + "strategy_id": {"literal": "trend-following"}, + "environment": {"literal": "paper"}, + "source": {"literal": "daily-position-export"}, + "portfolio_id": {"path": "account_alias", "transforms": ["trim"]}, + "snapshot_time": {"path": "as_of", "transforms": ["trim", "utc_timestamp"]}, + "base_currency": {"path": "base_currency", "transforms": ["trim", "uppercase"]} + }, + "position_fields": { + "instrument_id": {"path": "symbol", "transforms": ["trim"]}, + "instrument_id_type": {"literal": "ticker"}, + "quantity": {"path": "quantity", "transforms": ["decimal"]}, + "market_value_base": {"path": "market_value_base", "transforms": ["decimal"]}, + "asset_class": {"path": "asset_class", "transforms": ["trim", "lowercase"]}, + "sector": {"path": "sector", "transforms": ["trim"]} + } +} +``` + +字段绑定只允许以下二选一: + +- `literal`:可信的固定配置值。 +- `path`:CSV 的精确列名,或 JSON/JSONL 的 RFC 6901 JSON Pointer。两类语法不混用,避免包含点号或斜杠的字段名产生歧义。 + +v0.2 只支持 `trim`、`uppercase`、`lowercase`、`decimal` 和 `utc_timestamp`。转换顺序固定执行;不支持 Python、JavaScript、正则替换、算术表达式或网络请求。复杂转换由后续来源适配器承担。 + +`utc_timestamp` 默认只接受带明确时区偏移的 RFC 3339 时间。若源文件使用本地时间,字段绑定必须同时提供 `timestamp_format` 和 IANA `assume_timezone`,例如 `Asia/Shanghai`;只提供其中一项即视为配置错误。夏令时导致的不存在时间或歧义时间必须拒绝,不能静默猜测。`snapshot_time` 必须来自映射字段或可信固定值,禁止使用文件修改时间推断。 + +映射配置的身份是 RFC 8785 JSON Canonicalization Scheme 结果的 SHA-256。证据引用固定为 `mapping:sha256:`;语义相同但字节排版不同的配置应得到相同指纹。 + +## 规范事件 + +事件契约升级到 `schema_version = "1.1"`,继续接受现有 `1.0` 事件。`position_snapshot` 只允许出现在 `1.1`。 + +```json +{ + "schema_version": "1.1", + "strategy_id": "trend-following", + "environment": "paper", + "event_type": "position_snapshot", + "event_time": "2026-07-20T16:00:00Z", + "recorded_at": "2026-07-20T16:02:00Z", + "source": "daily-position-export", + "payload": { + "event_type": "position_snapshot", + "portfolio_id": "paper-book-a", + "base_currency": "USD", + "mapping_profile_hash": "sha256:...", + "positions": [ + { + "instrument_id": "AAPL", + "instrument_id_type": "ticker", + "venue": "XNAS", + "quantity": "100", + "market_value_base": "22000", + "asset_class": "equity", + "sector": "Technology" + } + ] + } +} +``` + +### 快照身份 + +仓位快照的自然身份是: + +```text +strategy_id + environment + source + portfolio_id + event_type + event_time + schema_version +``` + +这避免同一策略、来源和时点下的不同账户或 book 相互覆盖。其他事件继续使用 v0.1 的自然身份,不发生历史键变化。 + +### Portfolio ID + +`portfolio_id` 是用户提供的稳定别名,可以代表账户、book、子组合或策略 sleeve。它必须满足与 `strategy_id` 相同的安全字符约束,不要求是真实券商账号。文档和示例要求使用别名或不可逆脱敏值,公开 API 不应暴露真实账户号码。 + +`recorded_at` 可由映射显式提供;没有提供时,导入器使用开始读取该文件时记录的 UTC 时间,并在证据元数据中标记 `recorded_at_source = import_observed_at`。它不能从文件修改时间或 `snapshot_time` 猜测。 + +规范事件的 `event_time` 固定等于映射后的 `snapshot_time`。两者不是两个可独立配置的时间,避免幂等身份和实际快照时点分叉。 + +## Position 字段 + +每个 position 必须提供 `instrument_id`,并至少提供以下度量字段之一: + +- `quantity` +- `weight` +- `market_value_base` +- `exposure_value_base` + +`positions = []` 是合法且有业务含义的权威空仓快照。它与“文件没有读到记录”不同:只有文档快照显式给出空数组,或表格映射以固定身份和 `snapshot_scope = whole_file` 明确声明空文件时才成立。 + +字段语义: + +- `quantity`:带符号数量;空头为负。 +- `weight`:用户系统给出的带符号组合权重,不强制总和为 1,但分析会报告总和。 +- `market_value_base`:按快照基础币种表示的带符号市场价值。 +- `exposure_value_base`:用户上游已经计算的带符号风险敞口;适合期货、期权或其他市场价值不能代表风险规模的工具。 +- `side`:可选的 `long` 或 `short`,用于兼容“数量恒为正、方向另列”的来源。 +- `instrument_id_type`:自由但受长度限制的标识类型,例如 `ticker`、`isin`、`figi`、`contract` 或 `custom`。 +- `venue`:可选交易场所,优先使用 MIC;同名 ticker 跨市场时必须提供。 +- `asset_class`、`sector`、`country`:可选分类字段;缺少时只影响相应分类分析。 + +规范化后所有度量字段都必须带符号。提供 `side = short` 时,正值统一转为负,负值保持不变;提供 `side = long` 时,负值属于冲突。未提供 `side` 时保留源符号。完成这一步后,同一 position 的所有非零度量必须方向一致,否则整条 position 拒绝为 `conflicting_position_side`。零值不参与 long/short 计数,但可保留在快照中。 + +所有度量规范化为不使用指数表示法的十进制定点字符串,计算使用十进制定点数,不能经二进制浮点数中转。v0.2 输入硬上限为 30 位有效数字、18 位小数;拒绝 `NaN`、正负无穷和越界值。派生计算使用 38 位十进制精度和 `ROUND_HALF_EVEN`,对外最多保留 18 位小数并去除无意义末尾零。 + +存在 `market_value_base` 或 `exposure_value_base` 时,快照必须提供 3–12 位大写字母或数字组成的 `base_currency`。法币推荐 ISO 4217,数字资产允许 `BTC`、`USDT` 等行业代码。quantity-only 或 weight-only 快照可以省略。v0.2 不做 FX 换算,也不把不同币种的原始市值相加。 + +同一快照内 `(instrument_id_type, instrument_id, venue)` 不得重复,缺失 `venue` 视为空字符串参与比较。适配器必须在规范化前合并或拒绝重复行;v0.2 默认拒绝,避免系统静默决定净额规则。 + +## 能力分级与分析口径 + +系统不把缺字段当成整个来源不可用,而是生成能力报告: + +“活跃 position”定义为任一已提供度量不为零的 position。能力完整性、覆盖率和敞口指标只以活跃 position 为分母;`position_count` 仍报告全部记录数,并单独返回 `zero_position_count`。对每种候选基础分别计算: + +```text +coverage_by_basis[basis] = 拥有该字段的活跃 position 数 / 活跃 position 总数 +``` + +没有活跃 position 时返回 `state = empty_portfolio`、`gross = 0`、`net = 0`,`basis`、各覆盖率、Top-N 和 HHI 为 `null`。这表示来源明确报告空仓,而不是缺数据。 + +### Level 0:身份可见 + +每个 position 都有标的和至少一个允许的度量,但整个快照没有共同且完整的分析基础。展示仓位数量、方向、各候选基础覆盖率和缺失项,不计算价值集中度。 + +### Level 1:权重分析 + +所有非零仓位都有 `weight`。计算: + +```text +gross = Σ |weightᵢ| +net = Σ weightᵢ +absolute_shareᵢ = |weightᵢ| / gross +HHI = Σ absolute_shareᵢ² +``` + +### Level 2:基础币种价值分析 + +所有非零仓位都有 `market_value_base`。使用同一公式,把 `weightᵢ` 替换为 `market_value_baseᵢ`。结果标记 `basis = market_value_base`。 + +### Level 3:风险敞口分析 + +所有非零仓位都有 `exposure_value_base`。使用相同公式,结果标记 `basis = exposure_value_base`。若同时存在多个完整基础,优先级是: + +```text +exposure_value_base > market_value_base > weight +``` + +系统不得混合不同基础补齐同一个快照。例如一半仓位用 `market_value_base`、另一半用 `weight` 时,集中度不可计算,并返回缺失字段和覆盖率。 + +### 输出指标 + +- `position_count` +- `zero_position_count` +- `long_count` / `short_count` +- `gross` +- `net` +- `top_1_share` +- `top_5_share` +- `hhi` +- `basis` +- `coverage_ratio` +- `coverage_by_basis` +- 按 `asset_class`、`sector`、`country` 的 gross 和 net 分组 +- 参与计算的 `event:` 和 `mapping:` 证据引用 + +有完整基础时,`coverage_ratio = 1`。无完整基础时,`basis` 和不可计算的数值字段返回 `null`,`coverage_ratio` 返回现有覆盖率最高的候选基础覆盖率;并返回该 `candidate_basis`、`coverage_by_basis` 和结构化 `reason`,不能用零代替。覆盖率相同时仍按 `exposure_value_base > market_value_base > weight` 选择候选基础。 + +Top-N 按所选基础的绝对值降序计算。当完整基础存在但 `gross = 0` 时,`gross` 和 `net` 返回真实的 0;`top_1_share`、`top_5_share` 和 `hhi` 返回 `null`,原因是 `zero_gross_exposure`,避免除以零。少于五个非零仓位时,Top-5 汇总全部非零仓位。 + +分类敞口使用同一个所选基础。缺少 `asset_class`、`sector` 或 `country` 的仓位进入该维度的 `unclassified` 桶,并同时返回分类字段覆盖率;不能从标的名称猜测分类。 + +## 预览与导入流程 + +```bash +uv run scripts/import_positions.py \ + --input ./positions.csv \ + --profile ./position-profile.json \ + --preview +``` + +预览不写数据库,输出: + +- 检测到的格式、行数和快照组数。 +- 源字段名,不默认打印完整敏感行。 +- 最多 5 条脱敏规范化 position。 +- 每个快照的能力等级和覆盖率。 +- 错误代码、行号和安全摘要。 +- 映射配置 SHA-256。 + +移除 `--preview` 后才执行导入。导入继续使用现有幂等、修订、隔离和批次留痕机制。 + +## AI 映射助手边界 + +v0.2 建立 AI 映射助手可以消费和产生的文件契约,但不绑定模型供应商,也不在导入路径调用大模型。 + +助手输入应是: + +- 文件格式和列名。 +- 用户选择的身份固定值。 +- 最多 20 行经过本地脱敏的样本。 +- 映射 JSON Schema。 + +助手输出只能是候选映射 JSON、字段置信度和缺失能力说明。候选配置仍必须经过严格校验和 `--preview`。大模型输出不能直接写数据库、执行表达式或决定风险数值。 + +后续可以提供可插拔命令调用 OpenAI、本地模型或用户自己的模型;确定性导入器和映射格式不因此改变。 + +## 存储与追踪 + +继续复用通用 `events` 表,不为每种来源创建独立仓位表。规范化 `position_snapshot` 保存在 `normalized_json`,原始输入保存在现有原始证据字段中。 + +表格快照可能包含多行原始数据,存储层需要把快照的原始行集合规范化为一个受大小限制的 JSON 数组,并保留: + +- 源文件安全引用。 +- 起止行号。 +- 映射配置摘要。 +- 导入批次 ID。 + +API 只返回文件名、行号范围和摘要,不返回绝对路径、原始账户号或原始行。 + +v0.2 的确定性资源上限是:单个源文件 100 MiB、单条物理记录 1 MiB、单个规范化快照 100,000 个 position。超过上限必须在写库前拒绝并返回结构化错误;不能截断后继续导入。CSV/JSONL 必须流式读取,JSON 文档在解析前先检查文件上限。 + +## API + +新增两个只读端点: + +```text +GET /api/v1/portfolios +GET /api/v1/portfolios/{portfolio_id}/exposure?strategy_id={id}&environment={env}&source={source} +``` + +`/portfolios` 返回严格身份、最新快照时间、快照年龄、position 数量、能力等级和缺失字段。v0.2 不擅自定义“健康/不健康”阈值。详情端点返回最新 current 快照的指标、分类敞口、缺失原因和证据引用。 + +查询身份必须包含 `portfolio_id + strategy_id + environment + source`,不得跨账户、来源或实盘/模拟盘合并。 + +OpenAPI 继续是前端类型的唯一来源;生成文件漂移由 CI 阻止。 + +## 前端 + +观察台增加两个区块: + +1. `仓位覆盖`:每个 portfolio 的最新快照、时效、仓位数、能力等级和缺失字段。 +2. `集中度与敞口`:gross、net、Top-1、Top-5、HHI 及分类分布。 + +无法计算时明确显示原因,例如“3/12 个仓位缺少基础币种市场价值”,而不是展示 0。DEMO 水印和离线、空、部分失败语义保持不变。 + +v0.2 不增加图表库。分类敞口使用现有视觉系统中的条形比例、数值和证据文本,控制依赖和作品集噪声。 + +## 报告 + +Markdown 报告增加: + +- 最新 portfolio 快照摘要。 +- 指标基础和覆盖率。 +- gross、net、Top-N、HHI。 +- 分类敞口。 +- 不可计算原因。 +- 事件和映射证据引用。 + +API、前端和报告必须调用同一个 `CockpitService` 分析结果,不能分别实现公式。 + +## 错误与故障语义 + +| 场景 | 语义 | +| --- | --- | +| 不支持的文件格式 | 预览失败,`unsupported_input_format` | +| 映射字段不存在 | 对应行或快照拒绝,`mapping_path_missing` | +| 数字或时间转换失败 | 对应行拒绝,保留安全摘要 | +| 本地时间缺时区或遇到 DST 歧义 | 对应行拒绝,`ambiguous_source_time` | +| 快照组身份冲突 | 整组拒绝,不写部分快照 | +| 重复标的 | 整组拒绝,`duplicate_instrument` | +| side 与度量符号冲突 | 对应 position 拒绝,整组不写入 | +| 多币种但无基础币种价值 | 保留快照,价值分析不可计算 | +| 部分仓位缺少分析基础 | 保留快照,返回覆盖率和 `incomplete_basis` | +| 同键同内容重放 | duplicate,不新增事件 | +| 同键更新且 recorded_at 更晚 | revision,分析读取 current | +| 文件读取中断 | 导入批次失败,不产生半个快照 | +| 超过文件、记录或仓位上限 | 写库前拒绝,`ingestion_limit_exceeded` | + +## 安全与隐私 + +- 所有输入都视为不可信数据。 +- 映射配置不能执行任意代码、Shell、模板或网络请求。 +- 预览默认只显示字段名和脱敏后的少量规范化结果。 +- `portfolio_id` 使用别名;API 不需要真实账户号。 +- 大模型助手只接收用户明确提供的本地脱敏样本。 +- 原始行、绝对路径和验证错误输入不得通过 API 或截图泄漏。 +- 单文件和单快照设置大小、行数和 position 数量上限,避免内存耗尽。 + +## 测试策略 + +### 契约测试 + +- 1.0 旧事件继续通过。 +- 1.0 `position_snapshot` 被拒绝。 +- 1.1 仓位快照接受至少一种度量字段,拒绝未知字段、重复标的和无度量 position。 +- portfolio 身份参与幂等键;不同 portfolio 不碰撞。 + +### 映射测试 + +- CSV、JSON 文档和 JSONL 文档成功映射。 +- 固定值、JSON Pointer 和每种转换有正反例。 +- 缺列、坏数字、坏时间、组内身份冲突、重复标的和半写尾行失败。 +- JSON Pointer 转义、时区偏移、本地时区、DST 歧义和映射 RFC 8785 指纹均有固定夹具。 +- side 正值空头归一化、符号冲突、数字精度上限和币种代码均有边界测试。 +- 预览不创建或修改数据库,不打印原始账户号和绝对路径。 + +### 分析测试 + +- quantity-only 返回 Level 0 和不可计算原因。 +- 显式空仓与空文件误读严格区分,空仓返回真实的 gross/net 0。 +- weight、market value 和 exposure 三种完整基础分别计算正确。 +- 基础优先级稳定,不混合基础。 +- long/short、gross/net、Top-N、HHI、分类敞口和零 gross 边界正确。 +- 部分覆盖返回 `null + incomplete_basis`,不返回伪零。 + +### API、前端和报告测试 + +- 严格四元身份不会跨 portfolio、source 或 environment 合并。 +- 空、ready、partial、offline 和不可计算状态均可见。 +- 生成的 OpenAPI 和前端类型保持同步。 +- 报告与 API 使用同一评估时点和公式。 + +### 端到端测试 + +- 从合成 CSV 和映射配置预览、导入、API 查询、页面展示到报告生成完整跑通。 +- 浏览器在 900、1024、1280 和 1440px 下无横向溢出或控制台错误。 + +## 实施分期 + +### v0.2.0 + +- 1.1 `position_snapshot` 契约。 +- JSON 映射配置和 CSV/JSON/JSONL 读取。 +- 预览、导入、能力分级和集中度分析。 +- API、前端、报告、文档和合成演示。 + +### v0.2.1 + +- 基于公开合成夹具实现首批来源适配器,候选为 IBKR Flex、MT5 报告和 CCXT/交易所持仓导出。 +- 根据真实公开样本校正字段同义词,不在 v0.2.0 中凭印象硬编码。 + +### v0.2.2 + +- 可插拔 AI 映射助手。 +- 映射配置模板库和社区贡献规范。 +- OpenTelemetry Collector 对接示例,但不要求用户采用 OpenTelemetry。 + +## 成功标准 + +- 用户只准备一个现有数据文件和一个映射 JSON,就能在 10 分钟内看到仓位覆盖和集中度结果。 +- 不修改策略源代码,不需要券商凭据,不上传输入数据。 +- 所有数值结论都能回溯到 event 和 mapping 摘要。 +- 缺字段不会产生虚假数值;用户可以明确看到还缺什么。 +- 新来源优先通过新增映射配置接入,不修改分析内核。 diff --git a/examples/positions/demo-positions-profile.json b/examples/positions/demo-positions-profile.json new file mode 100644 index 0000000..bfd7aab --- /dev/null +++ b/examples/positions/demo-positions-profile.json @@ -0,0 +1,23 @@ +{ + "profile_version": "1.0", + "name": "synthetic-demo-positions", + "format": "csv", + "layout": "tabular_snapshot", + "snapshot_scope": "grouped_rows", + "fields": { + "strategy_id": {"literal": "healthy-demo"}, + "environment": {"literal": "paper"}, + "source": {"literal": "synthetic-demo"}, + "portfolio_id": {"path": "account_alias", "transforms": ["trim"]}, + "snapshot_time": {"path": "as_of", "transforms": ["utc_timestamp"]}, + "base_currency": {"literal": "USD"} + }, + "position_fields": { + "instrument_id": {"path": "symbol", "transforms": ["trim"]}, + "instrument_id_type": {"literal": "ticker"}, + "quantity": {"path": "quantity", "transforms": ["decimal"]}, + "market_value_base": {"path": "market_value_base", "transforms": ["decimal"]}, + "asset_class": {"path": "asset_class", "transforms": ["trim", "lowercase"]}, + "country": {"path": "country", "transforms": ["trim", "uppercase"]} + } +} diff --git a/examples/positions/demo-positions.csv b/examples/positions/demo-positions.csv new file mode 100644 index 0000000..bee6b04 --- /dev/null +++ b/examples/positions/demo-positions.csv @@ -0,0 +1,4 @@ +account_alias,as_of,symbol,quantity,market_value_base,asset_class,sector,country +synthetic-book,2026-07-20T17:30:00Z,SYNTH_A,100,60000,equity,Technology,US +synthetic-book,2026-07-20T17:30:00Z,SYNTH_B,-50,-30000,equity,Financials,US +synthetic-book,2026-07-20T17:30:00Z,SYNTH_C,25,10000,bond,,GB diff --git a/frontend/openapi.json b/frontend/openapi.json index da6e789..37abe5e 100644 --- a/frontend/openapi.json +++ b/frontend/openapi.json @@ -1,6 +1,42 @@ { "components": { "schemas": { + "CategoryExposureResponse": { + "additionalProperties": false, + "properties": { + "gross": { + "title": "Gross", + "type": "string" + }, + "label": { + "title": "Label", + "type": "string" + }, + "net": { + "title": "Net", + "type": "string" + }, + "share": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Share" + } + }, + "required": [ + "label", + "gross", + "net", + "share" + ], + "title": "CategoryExposureResponse", + "type": "object" + }, "CorrelationEvidenceResponse": { "additionalProperties": false, "properties": { @@ -475,6 +511,411 @@ "title": "IngestionIssueResponse", "type": "object" }, + "PortfolioExposureResponse": { + "additionalProperties": false, + "properties": { + "age_seconds": { + "title": "Age Seconds", + "type": "integer" + }, + "analysis_state": { + "enum": [ + "ready", + "unavailable", + "empty_portfolio" + ], + "title": "Analysis State", + "type": "string" + }, + "base_currency": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Base Currency" + }, + "basis": { + "anyOf": [ + { + "enum": [ + "weight", + "market_value_base", + "exposure_value_base" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Basis" + }, + "candidate_basis": { + "anyOf": [ + { + "enum": [ + "weight", + "market_value_base", + "exposure_value_base" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Candidate Basis" + }, + "capability_level": { + "title": "Capability Level", + "type": "integer" + }, + "categories": { + "additionalProperties": { + "items": { + "$ref": "#/components/schemas/CategoryExposureResponse" + }, + "type": "array" + }, + "title": "Categories", + "type": "object" + }, + "category_coverage": { + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "title": "Category Coverage", + "type": "object" + }, + "coverage_by_basis": { + "additionalProperties": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "propertyNames": { + "enum": [ + "weight", + "market_value_base", + "exposure_value_base" + ] + }, + "title": "Coverage By Basis", + "type": "object" + }, + "coverage_ratio": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Coverage Ratio" + }, + "environment": { + "enum": [ + "live", + "paper" + ], + "title": "Environment", + "type": "string" + }, + "evidence_refs": { + "items": { + "type": "string" + }, + "title": "Evidence Refs", + "type": "array" + }, + "gross": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Gross" + }, + "hhi": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Hhi" + }, + "long_count": { + "title": "Long Count", + "type": "integer" + }, + "mapping_profile_hash": { + "title": "Mapping Profile Hash", + "type": "string" + }, + "net": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Net" + }, + "portfolio_id": { + "title": "Portfolio Id", + "type": "string" + }, + "position_count": { + "title": "Position Count", + "type": "integer" + }, + "reason": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Reason" + }, + "short_count": { + "title": "Short Count", + "type": "integer" + }, + "snapshot_time": { + "title": "Snapshot Time", + "type": "string" + }, + "source": { + "title": "Source", + "type": "string" + }, + "strategy_id": { + "title": "Strategy Id", + "type": "string" + }, + "top_1_share": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Top 1 Share" + }, + "top_5_share": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Top 5 Share" + }, + "zero_position_count": { + "title": "Zero Position Count", + "type": "integer" + } + }, + "required": [ + "portfolio_id", + "strategy_id", + "environment", + "source", + "snapshot_time", + "age_seconds", + "base_currency", + "mapping_profile_hash", + "analysis_state", + "capability_level", + "basis", + "candidate_basis", + "position_count", + "zero_position_count", + "long_count", + "short_count", + "gross", + "net", + "top_1_share", + "top_5_share", + "hhi", + "coverage_ratio", + "coverage_by_basis", + "categories", + "category_coverage", + "reason", + "evidence_refs" + ], + "title": "PortfolioExposureResponse", + "type": "object" + }, + "PortfolioSummaryResponse": { + "additionalProperties": false, + "properties": { + "age_seconds": { + "title": "Age Seconds", + "type": "integer" + }, + "analysis_state": { + "enum": [ + "ready", + "unavailable", + "empty_portfolio" + ], + "title": "Analysis State", + "type": "string" + }, + "basis": { + "anyOf": [ + { + "enum": [ + "weight", + "market_value_base", + "exposure_value_base" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Basis" + }, + "candidate_basis": { + "anyOf": [ + { + "enum": [ + "weight", + "market_value_base", + "exposure_value_base" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Candidate Basis" + }, + "capability_level": { + "title": "Capability Level", + "type": "integer" + }, + "environment": { + "enum": [ + "live", + "paper" + ], + "title": "Environment", + "type": "string" + }, + "missing_fields": { + "items": { + "type": "string" + }, + "title": "Missing Fields", + "type": "array" + }, + "portfolio_id": { + "title": "Portfolio Id", + "type": "string" + }, + "position_count": { + "title": "Position Count", + "type": "integer" + }, + "snapshot_time": { + "title": "Snapshot Time", + "type": "string" + }, + "source": { + "title": "Source", + "type": "string" + }, + "strategy_id": { + "title": "Strategy Id", + "type": "string" + }, + "zero_position_count": { + "title": "Zero Position Count", + "type": "integer" + } + }, + "required": [ + "portfolio_id", + "strategy_id", + "environment", + "source", + "snapshot_time", + "age_seconds", + "position_count", + "zero_position_count", + "capability_level", + "analysis_state", + "basis", + "candidate_basis", + "missing_fields" + ], + "title": "PortfolioSummaryResponse", + "type": "object" + }, + "PortfoliosResponse": { + "additionalProperties": false, + "properties": { + "portfolios": { + "items": { + "$ref": "#/components/schemas/PortfolioSummaryResponse" + }, + "title": "Portfolios", + "type": "array" + }, + "state": { + "enum": [ + "ready", + "empty" + ], + "title": "State", + "type": "string" + } + }, + "required": [ + "state", + "portfolios" + ], + "title": "PortfoliosResponse", + "type": "object" + }, "StrategiesResponse": { "additionalProperties": false, "properties": { @@ -669,7 +1110,7 @@ }, "info": { "title": "QuantCockpit", - "version": "0.1.0" + "version": "0.2.0" }, "openapi": "3.1.0", "paths": { @@ -709,6 +1150,98 @@ "summary": "Ingestion Errors" } }, + "/api/v1/portfolios": { + "get": { + "operationId": "portfolios_api_v1_portfolios_get", + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/PortfoliosResponse" + } + } + }, + "description": "Successful Response" + } + }, + "summary": "Portfolios" + } + }, + "/api/v1/portfolios/{portfolio_id}/exposure": { + "get": { + "operationId": "portfolio_exposure_api_v1_portfolios__portfolio_id__exposure_get", + "parameters": [ + { + "in": "path", + "name": "portfolio_id", + "required": true, + "schema": { + "title": "Portfolio Id", + "type": "string" + } + }, + { + "in": "query", + "name": "strategy_id", + "required": true, + "schema": { + "maxLength": 128, + "minLength": 1, + "title": "Strategy Id", + "type": "string" + } + }, + { + "in": "query", + "name": "environment", + "required": true, + "schema": { + "enum": [ + "live", + "paper" + ], + "title": "Environment", + "type": "string" + } + }, + { + "in": "query", + "name": "source", + "required": true, + "schema": { + "maxLength": 256, + "minLength": 1, + "title": "Source", + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/PortfolioExposureResponse" + } + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Portfolio Exposure" + } + }, "/api/v1/strategies": { "get": { "operationId": "list_strategies_api_v1_strategies_get", diff --git a/frontend/package.json b/frontend/package.json index 2e9b73e..2463639 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "quantcockpit-frontend", "private": true, - "version": "0.1.0", + "version": "0.2.0", "type": "module", "scripts": { "dev": "vite --host 127.0.0.1", diff --git a/frontend/src/Dashboard.tsx b/frontend/src/Dashboard.tsx index e967d55..2e50c8b 100644 --- a/frontend/src/Dashboard.tsx +++ b/frontend/src/Dashboard.tsx @@ -5,6 +5,9 @@ import type { CorrelationsResponse, IngestionErrorsResponse, Mode, + PortfolioExposureResponse, + PortfolioSummary, + PortfoliosResponse, StrategiesResponse, StrategyHealthResponse, StrategySummary, @@ -25,7 +28,11 @@ interface DashboardData { strategies: StrategySummary[]; health: Map; healthFailures: Set; + portfolios: PortfolioSummary[]; + exposures: Map; + exposureFailures: Set; strategyFailed: boolean; + portfolioFailed: boolean; correlations?: CorrelationsResponse; errors?: IngestionErrorsResponse; correlationFailed: boolean; @@ -36,7 +43,11 @@ const EMPTY_DATA: DashboardData = { strategies: [], health: new Map(), healthFailures: new Set(), + portfolios: [], + exposures: new Map(), + exposureFailures: new Set(), strategyFailed: false, + portfolioFailed: false, correlationFailed: false, errorsFailed: false, }; @@ -47,6 +58,17 @@ function strategyKey( return `${strategy.strategy_id}/${strategy.environment}/${strategy.source}`; } +function portfolioKey( + portfolio: Pick, +): string { + return JSON.stringify([ + portfolio.portfolio_id, + portfolio.strategy_id, + portfolio.environment, + portfolio.source, + ]); +} + function displayValue(value: unknown): string { if (value === null || value === undefined) return "无"; if (typeof value === "object") return JSON.stringify(value); @@ -62,11 +84,38 @@ function displayTime(value: string | null): string { }).format(new Date(value)) + " UTC"; } +function displayPercent(value: string | null): string { + if (value === null) return "无"; + const percent = Number(value) * 100; + return `${Number.isInteger(percent) ? percent.toFixed(0) : percent.toFixed(1)}%`; +} + +function displayMetric(value: string | null, percent = false): string { + if (value === null) return "不可计算"; + return percent ? displayPercent(value) : value; +} + +function displayAnalysisState(state: PortfolioSummary["analysis_state"]): string { + return { + ready: "可分析", + unavailable: "数据不足", + empty_portfolio: "明确空仓", + }[state]; +} + +function displayCategory(name: string): string { + return { + asset_class: "资产类别", + sector: "行业", + country: "国家 / 地区", + }[name] ?? name; +} + export function Dashboard({ mode, requestTimeoutMs = 6000 }: DashboardProps) { const [loadState, setLoadState] = useState("loading"); const [data, setData] = useState(EMPTY_DATA); const [lastRefresh, setLastRefresh] = useState(null); - const [copied, setCopied] = useState(false); + const [copyState, setCopyState] = useState<"idle" | "success" | "error">("idle"); const activeRequest = useRef(null); const load = useCallback(async () => { @@ -74,7 +123,7 @@ export function Dashboard({ mode, requestTimeoutMs = 6000 }: DashboardProps) { const controller = new AbortController(); activeRequest.current = controller; setLoadState("loading"); - setCopied(false); + setCopyState("idle"); try { await getJson<{ status: "ok" }>("/healthz", { @@ -90,22 +139,34 @@ export function Dashboard({ mode, requestTimeoutMs = 6000 }: DashboardProps) { } const requestOptions = { timeoutMs: requestTimeoutMs, signal: controller.signal }; - const [strategyResult, correlationResult, errorsResult] = await Promise.allSettled([ + const [strategyResult, correlationResult, errorsResult, portfolioResult] = await Promise.allSettled([ getJson("/api/v1/strategies", requestOptions), getJson("/api/v1/correlations", requestOptions), getJson("/api/v1/ingestion/errors", requestOptions), + getJson("/api/v1/portfolios", requestOptions), ]); if (controller.signal.aborted) return; const strategies = strategyResult.status === "fulfilled" ? strategyResult.value.strategies : []; - const healthResults = await Promise.allSettled( - strategies.map((strategy) => - getJson( - `/api/v1/strategies/${encodeURIComponent(strategy.strategy_id)}/${strategy.environment}/health?source=${encodeURIComponent(strategy.source)}`, - requestOptions, + const portfolios = portfolioResult.status === "fulfilled" ? portfolioResult.value.portfolios : []; + const [healthResults, exposureResults] = await Promise.all([ + Promise.allSettled( + strategies.map((strategy) => + getJson( + `/api/v1/strategies/${encodeURIComponent(strategy.strategy_id)}/${strategy.environment}/health?source=${encodeURIComponent(strategy.source)}`, + requestOptions, + ), ), ), - ); + Promise.allSettled( + portfolios.map((portfolio) => + getJson( + `/api/v1/portfolios/${encodeURIComponent(portfolio.portfolio_id)}/exposure?strategy_id=${encodeURIComponent(portfolio.strategy_id)}&environment=${portfolio.environment}&source=${encodeURIComponent(portfolio.source)}`, + requestOptions, + ), + ), + ), + ]); if (controller.signal.aborted) return; const health = new Map(); @@ -115,24 +176,39 @@ export function Dashboard({ mode, requestTimeoutMs = 6000 }: DashboardProps) { if (result.status === "fulfilled") health.set(key, result.value); else healthFailures.add(key); }); + const exposures = new Map(); + const exposureFailures = new Set(); + exposureResults.forEach((result, index) => { + const key = portfolioKey(portfolios[index]); + if (result.status === "fulfilled") exposures.set(key, result.value); + else exposureFailures.add(key); + }); const partial = strategyResult.status === "rejected" || correlationResult.status === "rejected" || errorsResult.status === "rejected" || - healthFailures.size > 0; + portfolioResult.status === "rejected" || + healthFailures.size > 0 || + exposureFailures.size > 0; setData({ strategies, health, healthFailures, + portfolios, + exposures, + exposureFailures, strategyFailed: strategyResult.status === "rejected", + portfolioFailed: portfolioResult.status === "rejected", correlations: correlationResult.status === "fulfilled" ? correlationResult.value : undefined, errors: errorsResult.status === "fulfilled" ? errorsResult.value : undefined, correlationFailed: correlationResult.status === "rejected", errorsFailed: errorsResult.status === "rejected", }); setLastRefresh(new Date().toISOString()); - setLoadState(partial ? "partial" : strategies.length === 0 ? "empty" : "ready"); + setLoadState( + partial ? "partial" : strategies.length === 0 && portfolios.length === 0 ? "empty" : "ready", + ); }, [requestTimeoutMs]); useEffect(() => { @@ -141,12 +217,18 @@ export function Dashboard({ mode, requestTimeoutMs = 6000 }: DashboardProps) { }, [load]); const showDemoWatermark = - mode === "demo" || data.strategies.some((strategy) => strategy.source === "synthetic-demo"); + mode === "demo" || + data.strategies.some((strategy) => strategy.source === "synthetic-demo") || + data.portfolios.some((portfolio) => portfolio.source === "synthetic-demo"); const reportCommand = showDemoWatermark ? DEMO_REPORT_COMMAND : LOCAL_REPORT_COMMAND; const copyReportCommand = async () => { - await navigator.clipboard.writeText(reportCommand); - setCopied(true); + try { + await navigator.clipboard.writeText(reportCommand); + setCopyState("success"); + } catch { + setCopyState("error"); + } }; const statusLabel = { @@ -169,7 +251,7 @@ export function Dashboard({ mode, requestTimeoutMs = 6000 }: DashboardProps) {

LOCAL OBSERVABILITY / PUBLIC ALPHA

QuantCockpit

-

策略日志的健康度、证据与相关性观察台

+

策略与仓位日志的健康度、证据、敞口与相关性观察台

+ {copyState === "error" &&

复制失败,请手动复制

} )} @@ -337,6 +523,10 @@ function PanelHeading({ index, title }: { index: string; title: string }) { return
{index}

{title}

; } +function Metric({ label, value }: { label: string; value: string }) { + return
{label}{value}
; +} + function Notice({ children, tone }: { children: React.ReactNode; tone: "good" | "danger" }) { return

{children}

; } diff --git a/frontend/src/generated/api.ts b/frontend/src/generated/api.ts index f1f1c6d..a0fb703 100644 --- a/frontend/src/generated/api.ts +++ b/frontend/src/generated/api.ts @@ -38,6 +38,40 @@ export interface paths { patch?: never; trace?: never; }; + "/api/v1/portfolios": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** Portfolios */ + get: operations["portfolios_api_v1_portfolios_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; + "/api/v1/portfolios/{portfolio_id}/exposure": { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + /** Portfolio Exposure */ + get: operations["portfolio_exposure_api_v1_portfolios__portfolio_id__exposure_get"]; + put?: never; + post?: never; + delete?: never; + options?: never; + head?: never; + patch?: never; + trace?: never; + }; "/api/v1/strategies": { parameters: { query?: never; @@ -93,6 +127,17 @@ export interface paths { export type webhooks = Record; export interface components { schemas: { + /** CategoryExposureResponse */ + CategoryExposureResponse: { + /** Gross */ + gross: string; + /** Label */ + label: string; + /** Net */ + net: string; + /** Share */ + share: string | null; + }; /** CorrelationEvidenceResponse */ CorrelationEvidenceResponse: { /** Observed Value */ @@ -194,6 +239,120 @@ export interface components { /** Source Ref */ source_ref: string; }; + /** PortfolioExposureResponse */ + PortfolioExposureResponse: { + /** Age Seconds */ + age_seconds: number; + /** + * Analysis State + * @enum {string} + */ + analysis_state: "ready" | "unavailable" | "empty_portfolio"; + /** Base Currency */ + base_currency: string | null; + /** Basis */ + basis: ("weight" | "market_value_base" | "exposure_value_base") | null; + /** Candidate Basis */ + candidate_basis: ("weight" | "market_value_base" | "exposure_value_base") | null; + /** Capability Level */ + capability_level: number; + /** Categories */ + categories: { + [key: string]: components["schemas"]["CategoryExposureResponse"][]; + }; + /** Category Coverage */ + category_coverage: { + [key: string]: string | null; + }; + /** Coverage By Basis */ + coverage_by_basis: { + [key: string]: string | null; + }; + /** Coverage Ratio */ + coverage_ratio: string | null; + /** + * Environment + * @enum {string} + */ + environment: "live" | "paper"; + /** Evidence Refs */ + evidence_refs: string[]; + /** Gross */ + gross: string | null; + /** Hhi */ + hhi: string | null; + /** Long Count */ + long_count: number; + /** Mapping Profile Hash */ + mapping_profile_hash: string; + /** Net */ + net: string | null; + /** Portfolio Id */ + portfolio_id: string; + /** Position Count */ + position_count: number; + /** Reason */ + reason: string | null; + /** Short Count */ + short_count: number; + /** Snapshot Time */ + snapshot_time: string; + /** Source */ + source: string; + /** Strategy Id */ + strategy_id: string; + /** Top 1 Share */ + top_1_share: string | null; + /** Top 5 Share */ + top_5_share: string | null; + /** Zero Position Count */ + zero_position_count: number; + }; + /** PortfolioSummaryResponse */ + PortfolioSummaryResponse: { + /** Age Seconds */ + age_seconds: number; + /** + * Analysis State + * @enum {string} + */ + analysis_state: "ready" | "unavailable" | "empty_portfolio"; + /** Basis */ + basis: ("weight" | "market_value_base" | "exposure_value_base") | null; + /** Candidate Basis */ + candidate_basis: ("weight" | "market_value_base" | "exposure_value_base") | null; + /** Capability Level */ + capability_level: number; + /** + * Environment + * @enum {string} + */ + environment: "live" | "paper"; + /** Missing Fields */ + missing_fields: string[]; + /** Portfolio Id */ + portfolio_id: string; + /** Position Count */ + position_count: number; + /** Snapshot Time */ + snapshot_time: string; + /** Source */ + source: string; + /** Strategy Id */ + strategy_id: string; + /** Zero Position Count */ + zero_position_count: number; + }; + /** PortfoliosResponse */ + PortfoliosResponse: { + /** Portfolios */ + portfolios: components["schemas"]["PortfolioSummaryResponse"][]; + /** + * State + * @enum {string} + */ + state: "ready" | "empty"; + }; /** StrategiesResponse */ StrategiesResponse: { /** @@ -318,6 +477,61 @@ export interface operations { }; }; }; + portfolios_api_v1_portfolios_get: { + parameters: { + query?: never; + header?: never; + path?: never; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["PortfoliosResponse"]; + }; + }; + }; + }; + portfolio_exposure_api_v1_portfolios__portfolio_id__exposure_get: { + parameters: { + query: { + strategy_id: string; + environment: "live" | "paper"; + source: string; + }; + header?: never; + path: { + portfolio_id: string; + }; + cookie?: never; + }; + requestBody?: never; + responses: { + /** @description Successful Response */ + 200: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["PortfolioExposureResponse"]; + }; + }; + /** @description Validation Error */ + 422: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["HTTPValidationError"]; + }; + }; + }; + }; list_strategies_api_v1_strategies_get: { parameters: { query?: never; diff --git a/frontend/src/styles.css b/frontend/src/styles.css index 85e595c..0bf9141 100644 --- a/frontend/src/styles.css +++ b/frontend/src/styles.css @@ -48,7 +48,7 @@ dd { margin: 3px 0 0; overflow-wrap: anywhere; } .notice-danger { border-color: var(--danger); background: #fff0ea; } .muted { margin: 0; color: var(--muted); } -.issues, .evidence-list, .correlation-list { display: grid; gap: 10px; } +.issues, .evidence-list, .correlation-list, .exposure-list { display: grid; gap: 10px; } .issues article, .evidence-list article, .correlation-list article { border: 1px solid var(--line); padding: 14px; background: white; } .issues article { display: grid; grid-template-columns: 1fr 1fr; } .issues p { grid-column: 1 / -1; margin: 8px 0 0; color: var(--muted); } @@ -68,6 +68,41 @@ dd { margin: 3px 0 0; overflow-wrap: anywhere; } .strategy-card dl { margin: 0; display: grid; gap: 8px; } .strategy-card dl div { display: grid; grid-template-columns: 80px 1fr; } +.portfolio-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); gap: 12px; } +.portfolio-card { border: 1px solid var(--ink); background: white; padding: 16px; box-shadow: 3px 3px 0 #c7c7c2; } +.portfolio-topline { display: flex; justify-content: space-between; gap: 12px; font: 700 12px/1 ui-monospace, monospace; } +.portfolio-topline strong { color: var(--blue); } +.portfolio-card h3 { margin: 24px 0 6px; font-size: 24px; overflow-wrap: anywhere; } +.portfolio-card > p { margin: 0 0 20px; color: var(--muted); overflow-wrap: anywhere; } +.portfolio-meta { display: grid; gap: 9px; margin: 0; border-top: 1px solid #ddd; padding-top: 14px; } +.portfolio-meta div { display: grid; grid-template-columns: 88px 1fr; } + +.exposure-group { display: grid; gap: 16px; border-top: 3px solid var(--ink); padding-top: 12px; } +.exposure-group > header { display: flex; justify-content: space-between; align-items: baseline; gap: 16px; } +.exposure-group > header > div { display: grid; gap: 4px; } +.exposure-group h3, .exposure-group h4 { margin: 0; } +.exposure-group h3 { font-size: 18px; } +.exposure-group > header span { color: var(--muted); font-size: 12px; overflow-wrap: anywhere; } +.exposure-group > header > strong { font: 800 12px/1 ui-monospace, monospace; color: var(--blue); } +.capability-note { margin: 0; border: 1px solid #d6b84d; background: #fff5c9; padding: 14px; font-weight: 700; } +.metric-grid { display: grid; grid-template-columns: repeat(5, minmax(100px, 1fr)); gap: 8px; } +.metric-card { display: grid; gap: 12px; border: 1px solid var(--line); background: white; padding: 12px; } +.metric-card > span { color: var(--muted); font: 700 11px/1 ui-monospace, monospace; text-transform: uppercase; } +.metric-value { font: 750 24px/1 ui-monospace, monospace; overflow-wrap: anywhere; } +.category-list { display: grid; gap: 14px; } +.category-list > section { display: grid; gap: 9px; border: 1px solid var(--line); background: white; padding: 13px; } +.category-list > section > header { display: flex; justify-content: space-between; gap: 12px; align-items: baseline; } +.category-list h4 { font-size: 15px; } +.category-list header span { color: var(--muted); font-size: 12px; } +.category-row { display: grid; grid-template-columns: 120px minmax(100px, 1fr) 64px 190px; align-items: center; gap: 10px; } +.category-row > span { overflow-wrap: anywhere; } +.category-row > strong { text-align: right; font: 700 13px/1 ui-monospace, monospace; } +.category-row > small { color: var(--muted); text-align: right; } +.category-bar { height: 8px; border: 1px solid var(--ink); background: #eee; } +.category-fill { display: block; height: 100%; background: var(--signal); } +.exposure-refs { display: flex; flex-wrap: wrap; gap: 6px; border-top: 1px solid #ddd; padding-top: 11px; } +.exposure-refs code { color: var(--blue); overflow-wrap: anywhere; } + .evidence-list article > div { display: flex; justify-content: space-between; gap: 20px; align-items: start; } .evidence-group { display: grid; gap: 10px; border-top: 3px solid var(--ink); padding-top: 12px; } .evidence-group > header { display: flex; justify-content: space-between; gap: 18px; align-items: baseline; } @@ -90,6 +125,7 @@ dd { margin: 3px 0 0; overflow-wrap: anywhere; } .command-row { display: flex; align-items: stretch; border: 1px solid var(--ink); background: white; } .command-row code { flex: 1; padding: 14px; overflow-wrap: anywhere; font-size: 12px; } .command-row button { border: 0; border-left: 1px solid var(--ink); background: var(--signal); min-width: 84px; font-weight: 800; } +.copy-feedback { grid-column: 2; margin: -6px 0 0; color: #a0290d; font-weight: 750; } .offline-panel { margin-top: 1px; background: var(--danger); color: #1d0904; padding: 48px; border: 2px solid var(--ink); } .offline-panel h2 { margin: 0 0 12px; font-size: 42px; } .offline-panel p { margin: 0; max-width: 680px; font-size: 16px; } @@ -100,6 +136,9 @@ dd { margin: 3px 0 0; overflow-wrap: anywhere; } .cockpit-shell { width: calc(100% - 32px); } .panel { grid-template-columns: 180px 1fr; padding: 22px; gap: 20px; } .connection-strip { grid-template-columns: 220px 1fr; } + .metric-grid { grid-template-columns: repeat(3, minmax(100px, 1fr)); } + .category-row { grid-template-columns: 100px minmax(80px, 1fr) 58px; } + .category-row > small { grid-column: 1 / -1; text-align: left; } } @media (max-width: 899px) { diff --git a/frontend/src/types.ts b/frontend/src/types.ts index 4d04c4f..4792e4d 100644 --- a/frontend/src/types.ts +++ b/frontend/src/types.ts @@ -12,3 +12,7 @@ export type CorrelationPair = Schemas["CorrelationPairResponse"]; export type CorrelationsResponse = Schemas["CorrelationsResponse"]; export type IngestionIssue = Schemas["IngestionIssueResponse"]; export type IngestionErrorsResponse = Schemas["IngestionErrorsResponse"]; +export type PortfolioSummary = Schemas["PortfolioSummaryResponse"]; +export type PortfoliosResponse = Schemas["PortfoliosResponse"]; +export type PortfolioExposureResponse = Schemas["PortfolioExposureResponse"]; +export type CategoryExposure = Schemas["CategoryExposureResponse"]; diff --git a/frontend/tests/dashboard.regression-1.test.tsx b/frontend/tests/dashboard.regression-1.test.tsx new file mode 100644 index 0000000..07503e5 --- /dev/null +++ b/frontend/tests/dashboard.regression-1.test.tsx @@ -0,0 +1,36 @@ +// Regression: ISSUE-001 — 剪贴板权限被拒绝时没有用户反馈 +// Found by /qa on 2026-07-20 +// Report: .gstack/qa-reports/qa-report-127-0-0-1-2026-07-20.md + +import { render, screen } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; + +import { Dashboard } from "../src/Dashboard"; + +function response(body: object): Response { + return { ok: true, status: 200, json: async () => body } as Response; +} + +test("剪贴板权限被拒绝时提示用户手动复制", async () => { + const user = userEvent.setup(); + vi.spyOn(globalThis, "fetch").mockImplementation(async (input) => { + const path = String(input); + if (path === "/healthz") return response({ status: "ok" }); + if (path === "/api/v1/strategies") return response({ state: "empty", strategies: [] }); + if (path === "/api/v1/correlations") return response({ state: "empty", pairs: [] }); + if (path === "/api/v1/ingestion/errors") { + return response({ state: "empty", quarantines: [], failed_runs: [] }); + } + if (path === "/api/v1/portfolios") return response({ state: "empty", portfolios: [] }); + throw new Error(`unexpected route ${path}`); + }); + vi.spyOn(navigator.clipboard, "writeText").mockRejectedValue( + new DOMException("Write permission denied", "NotAllowedError"), + ); + + render(); + await screen.findByText("尚未导入策略日志"); + await user.click(screen.getByRole("button", { name: "复制报告命令" })); + + expect(screen.getByText("复制失败,请手动复制")).toBeVisible(); +}); diff --git a/frontend/tests/dashboard.test.tsx b/frontend/tests/dashboard.test.tsx index 053ff61..aaba3a0 100644 --- a/frontend/tests/dashboard.test.tsx +++ b/frontend/tests/dashboard.test.tsx @@ -63,6 +63,56 @@ const correlation = { evidence: [], }; +const portfolio = { + portfolio_id: "book-a", + strategy_id: "atlas-demo", + environment: "paper", + source: "unit-jsonl", + snapshot_time: "2026-07-20T17:30:00Z", + age_seconds: 1800, + position_count: 2, + zero_position_count: 0, + capability_level: 2, + analysis_state: "ready", + basis: "market_value_base", + candidate_basis: null, + missing_fields: ["exposure_value_base"], +}; + +const exposure = { + portfolio_id: "book-a", + strategy_id: "atlas-demo", + environment: "paper", + source: "unit-jsonl", + snapshot_time: "2026-07-20T17:30:00Z", + age_seconds: 1800, + base_currency: "USD", + mapping_profile_hash: `sha256:${"a".repeat(64)}`, + analysis_state: "ready", + capability_level: 2, + basis: "market_value_base", + candidate_basis: null, + position_count: 2, + zero_position_count: 0, + long_count: 1, + short_count: 1, + gross: "100", + net: "20", + top_1_share: "0.6", + top_5_share: "1", + hhi: "0.52", + coverage_ratio: "1", + coverage_by_basis: { exposure_value_base: "0", market_value_base: "1", weight: "0" }, + categories: { + asset_class: [{ label: "equity", gross: "100", net: "20", share: "1" }], + sector: [{ label: "unclassified", gross: "100", net: "20", share: "1" }], + country: [], + }, + category_coverage: { asset_class: "1", sector: "0", country: "0" }, + reason: null, + evidence_refs: ["event:position-1", `mapping:sha256:${"a".repeat(64)}`], +}; + function response(body: object, ok = true): Response { return { ok, status: ok ? 200 : 500, json: async () => body } as Response; } @@ -70,7 +120,11 @@ function response(body: object, ok = true): Response { function mockApi(routes: Record): void { vi.spyOn(globalThis, "fetch").mockImplementation(async (input) => { const path = String(input); - const result = routes[path]; + const result = routes[path] ?? ( + path === "/api/v1/portfolios" + ? response({ state: "empty", portfolios: [] }) + : undefined + ); if (result instanceof Error) throw result; if (!result) throw new Error(`unexpected route ${path}`); return result; @@ -84,6 +138,8 @@ test("ready:按可信度优先顺序呈现真实 API 数据和本地报告命 "/api/v1/strategies/atlas-demo/paper/health?source=unit-jsonl": response(health), "/api/v1/correlations": response({ state: "ready", pairs: [correlation] }), "/api/v1/ingestion/errors": response({ state: "empty", quarantines: [], failed_runs: [] }), + "/api/v1/portfolios": response({ state: "ready", portfolios: [portfolio] }), + "/api/v1/portfolios/book-a/exposure?strategy_id=atlas-demo&environment=paper&source=unit-jsonl": response(exposure), }); render(); @@ -94,6 +150,9 @@ test("ready:按可信度优先顺序呈现真实 API 数据和本地报告命 expect(screen.getByText("-0.250")).toBeVisible(); expect(screen.getByText("event:left-1")).toBeVisible(); expect(screen.getByText("event:right-1")).toBeVisible(); + expect(screen.getByText("book-a")).toBeVisible(); + expect(screen.getByText("100", { selector: ".metric-value" })).toBeVisible(); + expect(screen.getByText("event:position-1")).toBeVisible(); expect(screen.getByText(/uv run scripts\/generate_report\.py/)).toBeVisible(); expect(screen.queryByText("SYNTHETIC DEMO DATA")).not.toBeInTheDocument(); @@ -103,6 +162,8 @@ test("ready:按可信度优先顺序呈现真实 API 数据和本地报告命 "数据可信度", "策略健康", "证据详情", + "仓位覆盖", + "集中度与敞口", "相关性", "本地报告", ]); @@ -242,7 +303,7 @@ test("刷新按钮可重新请求,报告命令可复制", async () => { render(); await screen.findByText("尚未导入策略日志"); await user.click(screen.getByRole("button", { name: "刷新数据" })); - await waitFor(() => expect(fetch).toHaveBeenCalledTimes(8)); + await waitFor(() => expect(fetch).toHaveBeenCalledTimes(10)); await user.click(screen.getByRole("button", { name: "复制报告命令" })); expect(clipboardWrite).toHaveBeenCalledWith( "uv run scripts/generate_report.py --output quantcockpit-report.md", @@ -301,3 +362,65 @@ test("组件卸载使 healthz 中止后 catch 不再更新状态", async () => { expect(stateUpdateAudit.afterUnmount).toBe(0); }); + +test("仓位不可计算时显示候选基础和覆盖率,不伪造零指标", async () => { + const unavailablePortfolio = { + ...portfolio, + analysis_state: "unavailable", + capability_level: 0, + basis: null, + candidate_basis: "weight", + missing_fields: ["weight"], + }; + const unavailableExposure = { + ...exposure, + analysis_state: "unavailable", + capability_level: 0, + basis: null, + candidate_basis: "weight", + gross: null, + net: null, + top_1_share: null, + top_5_share: null, + hhi: null, + coverage_ratio: "0.5", + coverage_by_basis: { exposure_value_base: "0", market_value_base: "0", weight: "0.5" }, + categories: { asset_class: [], sector: [], country: [] }, + reason: "incomplete_basis", + }; + mockApi({ + "/healthz": response({ status: "ok" }), + "/api/v1/strategies": response({ state: "ready", strategies: [strategy] }), + "/api/v1/strategies/atlas-demo/paper/health?source=unit-jsonl": response(health), + "/api/v1/correlations": response({ state: "empty", pairs: [] }), + "/api/v1/ingestion/errors": response({ state: "empty", quarantines: [], failed_runs: [] }), + "/api/v1/portfolios": response({ state: "ready", portfolios: [unavailablePortfolio] }), + "/api/v1/portfolios/book-a/exposure?strategy_id=atlas-demo&environment=paper&source=unit-jsonl": response(unavailableExposure), + }); + + render(); + + expect(await screen.findByText("候选基础 weight · 覆盖率 50% · incomplete_basis")).toBeVisible(); + expect(screen.queryByText("0.00", { selector: ".metric-value" })).not.toBeInTheDocument(); +}); + +test("一个 portfolio 详情失败时保留列表和其他面板并标记 partial", async () => { + const bookB = { ...portfolio, portfolio_id: "book-b" }; + mockApi({ + "/healthz": response({ status: "ok" }), + "/api/v1/strategies": response({ state: "ready", strategies: [strategy] }), + "/api/v1/strategies/atlas-demo/paper/health?source=unit-jsonl": response(health), + "/api/v1/correlations": response({ state: "empty", pairs: [] }), + "/api/v1/ingestion/errors": response({ state: "empty", quarantines: [], failed_runs: [] }), + "/api/v1/portfolios": response({ state: "ready", portfolios: [portfolio, bookB] }), + "/api/v1/portfolios/book-a/exposure?strategy_id=atlas-demo&environment=paper&source=unit-jsonl": response(exposure), + "/api/v1/portfolios/book-b/exposure?strategy_id=atlas-demo&environment=paper&source=unit-jsonl": response({}, false), + }); + + render(); + + expect(await screen.findByText("部分数据不可用")).toBeVisible(); + expect(screen.getByText("book-a")).toBeVisible(); + expect(screen.getByText("book-b · 敞口加载失败")).toBeVisible(); + expect(screen.getByText("atlas-demo")).toBeVisible(); +}); diff --git a/pyproject.toml b/pyproject.toml index c27ba00..08c5d6b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "quantcockpit" -version = "0.1.0" +version = "0.2.0" requires-python = ">=3.13" dependencies = [ "duckdb>=1.5.4", @@ -8,6 +8,7 @@ dependencies = [ "pydantic>=2.13.4", # DuckDB 的 TIMESTAMPTZ 查询结果映射为 Python datetime 时动态导入。 "pytz>=2026.2", + "rfc8785>=0.1.4", "uvicorn>=0.51.0", ] diff --git a/scripts/import_demo.py b/scripts/import_demo.py index fbcf2f9..20c9548 100644 --- a/scripts/import_demo.py +++ b/scripts/import_demo.py @@ -9,6 +9,8 @@ from pathlib import Path from quantcockpit.ingestion.jsonl import import_jsonl +from quantcockpit.ingestion.position_profile import PositionMappingProfile +from quantcockpit.ingestion.positions import import_positions from quantcockpit.store import DuckDBStore @@ -20,6 +22,11 @@ def main(argv: list[str] | None = None) -> int: help="DuckDB 路径;默认读取 QUANTCOCKPIT_DB_PATH 或 quantcockpit.duckdb", ) parser.add_argument("--data-dir", default="examples/data", help="JSONL 数据目录") + parser.add_argument( + "--positions-dir", + default="examples/positions", + help="合成仓位文件和 *-profile.json 映射目录", + ) args = parser.parse_args(argv) data_dir = Path(args.data_dir) @@ -28,6 +35,12 @@ def main(argv: list[str] | None = None) -> int: print(f"导入失败:未在 {data_dir} 找到 .jsonl 文件", file=sys.stderr) return 1 + positions_dir = Path(args.positions_dir) + position_profiles = sorted(positions_dir.glob("*-profile.json")) + if not position_profiles: + print(f"导入失败:未在 {positions_dir} 找到 *-profile.json", file=sys.stderr) + return 1 + store = DuckDBStore(args.database) try: for path in files: @@ -36,6 +49,17 @@ def main(argv: list[str] | None = None) -> int: f"已导入 {path}: imported={result.imported} duplicates={result.duplicates} " f"revisions={result.revisions} quarantined={result.quarantined}" ) + for profile_path in position_profiles: + profile = PositionMappingProfile.model_validate_json( + profile_path.read_text(encoding="utf-8") + ) + input_name = profile_path.name.removesuffix("-profile.json") + f".{profile.format}" + input_path = positions_dir / input_name + result = import_positions(store, input_path, profile) + print( + f"已导入 {input_path}: imported={result.imported} " + f"duplicates={result.duplicates} revisions={result.revisions}" + ) except Exception as error: print(f"导入失败:{error}", file=sys.stderr) return 1 diff --git a/scripts/import_positions.py b/scripts/import_positions.py new file mode 100644 index 0000000..f786852 --- /dev/null +++ b/scripts/import_positions.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""预览或导入一份声明式映射的本地仓位文件。""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import json +import os +from pathlib import Path +import sys +from typing import Sequence + +from pydantic import ValidationError + +from quantcockpit.ingestion.position_profile import PositionMappingProfile +from quantcockpit.ingestion.positions import PositionImportError, import_positions, preview_positions +from quantcockpit.store import DuckDBStore + + +def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--input", type=Path, required=True, help="本地 CSV、JSON 或 JSONL 文件") + parser.add_argument("--profile", type=Path, required=True, help="映射配置 JSON") + parser.add_argument( + "--database", + default=os.environ.get("QUANTCOCKPIT_DB_PATH", "quantcockpit.duckdb"), + help="正式导入目标 DuckDB;preview 不会创建它", + ) + parser.add_argument("--preview", action="store_true", help="只校验并输出安全预览") + parser.add_argument("--observed-at", help="可选的带时区 RFC 3339 观察时间") + return parser.parse_args(argv) + + +def _observed_at(value: str | None) -> datetime: + if value is None: + return datetime.now(timezone.utc) + try: + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError as error: + raise ValueError("observed_at must be a timezone-aware RFC 3339 timestamp") from error + if parsed.tzinfo is None or parsed.utcoffset() is None: + raise ValueError("observed_at must include a timezone") + return parsed.astimezone(timezone.utc) + + +def _load_profile(path: Path) -> PositionMappingProfile: + try: + decoded = json.loads(path.read_text(encoding="utf-8")) + return PositionMappingProfile.model_validate(decoded) + except (OSError, UnicodeError, json.JSONDecodeError, ValidationError) as error: + raise ValueError("mapping profile is unreadable or invalid") from error + + +def _preview_payload(preview: object) -> dict[str, object]: + from quantcockpit.ingestion.positions import PositionPreview + + if not isinstance(preview, PositionPreview): + raise TypeError("preview must be a PositionPreview") + return { + "format": preview.format, + "record_count": preview.record_count, + "snapshot_count": preview.snapshot_count, + "source_fields": list(preview.source_fields), + "sample_positions": list(preview.sample_positions), + "mapping_profile_hash": preview.mapping_profile_hash, + "warnings": list(preview.warnings), + } + + +def main(argv: Sequence[str] | None = None) -> int: + args = parse_args(argv) + try: + observed_at = _observed_at(args.observed_at) + profile = _load_profile(args.profile) + if args.preview: + preview = preview_positions(args.input, profile, observed_at=observed_at) + print(json.dumps(_preview_payload(preview), ensure_ascii=False, sort_keys=True)) + return 0 + store = DuckDBStore(args.database) + try: + result = import_positions(store, args.input, profile, observed_at=observed_at) + finally: + store.close() + except PositionImportError as error: + print(f"导入失败 [{error.code}]:{error}", file=sys.stderr) + return 1 + except ValueError as error: + print(f"导入失败:{error}", file=sys.stderr) + return 1 + print( + f"imported={result.imported} duplicates={result.duplicates} " + f"revisions={result.revisions} stale={result.stale}" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/quantcockpit/analysis/exposure.py b/src/quantcockpit/analysis/exposure.py new file mode 100644 index 0000000..f0b197d --- /dev/null +++ b/src/quantcockpit/analysis/exposure.py @@ -0,0 +1,266 @@ +"""基于规范仓位快照的确定性 Decimal 敞口分析。""" + +from __future__ import annotations + +from dataclasses import dataclass +from decimal import Context, Decimal, ROUND_HALF_EVEN, localcontext +from typing import Literal + +from quantcockpit.models import Position, PositionSnapshotPayload + + +Basis = Literal["weight", "market_value_base", "exposure_value_base"] +AnalysisState = Literal["ready", "unavailable", "empty_portfolio"] +_BASIS_PRIORITY: tuple[Basis, ...] = ( + "exposure_value_base", + "market_value_base", + "weight", +) +_LEVEL_BY_BASIS: dict[Basis, int] = { + "weight": 1, + "market_value_base": 2, + "exposure_value_base": 3, +} +_CATEGORY_NAMES = ("asset_class", "sector", "country") +_ANALYSIS_CONTEXT = Context(prec=38, rounding=ROUND_HALF_EVEN) +_RATIO_QUANTUM = Decimal("0.000000000000000001") + + +@dataclass(frozen=True) +class CategoryExposure: + label: str + gross: Decimal + net: Decimal + share: Decimal | None + + +@dataclass(frozen=True) +class ExposureAnalysis: + state: AnalysisState + level: int + basis: Basis | None + candidate_basis: Basis | None + position_count: int + zero_position_count: int + long_count: int + short_count: int + gross: Decimal | None + net: Decimal | None + top_1_share: Decimal | None + top_5_share: Decimal | None + hhi: Decimal | None + coverage_ratio: Decimal | None + coverage_by_basis: dict[Basis, Decimal | None] + categories: dict[str, tuple[CategoryExposure, ...]] + category_coverage: dict[str, Decimal | None] + reason: str | None + + +def analyze_exposure(snapshot: PositionSnapshotPayload) -> ExposureAnalysis: + """选择一个完整基础计算结果;绝不跨基础拼接仓位。""" + + with localcontext(_ANALYSIS_CONTEXT): + active = tuple(position for position in snapshot.positions if _is_active(position)) + zero_count = len(snapshot.positions) - len(active) + long_count = sum(_direction(position) > 0 for position in active) + short_count = sum(_direction(position) < 0 for position in active) + if not active: + return ExposureAnalysis( + state="empty_portfolio", + level=0, + basis=None, + candidate_basis=None, + position_count=len(snapshot.positions), + zero_position_count=zero_count, + long_count=0, + short_count=0, + gross=Decimal(0), + net=Decimal(0), + top_1_share=None, + top_5_share=None, + hhi=None, + coverage_ratio=None, + coverage_by_basis={basis: None for basis in _BASIS_PRIORITY}, + categories={name: () for name in _CATEGORY_NAMES}, + category_coverage={name: None for name in _CATEGORY_NAMES}, + reason="empty_portfolio", + ) + + active_count = Decimal(len(active)) + coverage_by_basis: dict[Basis, Decimal | None] = { + basis: _ratio( + Decimal(sum(_basis_value(position, basis) is not None for position in active)), + active_count, + ) + for basis in _BASIS_PRIORITY + } + category_coverage: dict[str, Decimal | None] = { + name: _ratio( + Decimal(sum(getattr(position, name) is not None for position in active)), + active_count, + ) + for name in _CATEGORY_NAMES + } + basis = next( + ( + candidate + for candidate in _BASIS_PRIORITY + if coverage_by_basis[candidate] == Decimal(1) + ), + None, + ) + if basis is None: + candidate_basis = _best_candidate(coverage_by_basis) + return ExposureAnalysis( + state="unavailable", + level=0, + basis=None, + candidate_basis=candidate_basis, + position_count=len(snapshot.positions), + zero_position_count=zero_count, + long_count=long_count, + short_count=short_count, + gross=None, + net=None, + top_1_share=None, + top_5_share=None, + hhi=None, + coverage_ratio=( + coverage_by_basis[candidate_basis] + if candidate_basis is not None + else Decimal(0) + ), + coverage_by_basis=coverage_by_basis, + categories={name: () for name in _CATEGORY_NAMES}, + category_coverage=category_coverage, + reason="incomplete_basis", + ) + + values = tuple(_required_basis_value(position, basis) for position in active) + absolute_values = tuple(abs(value) for value in values) + gross = sum(absolute_values, Decimal(0)) + net = sum(values, Decimal(0)) + if gross == 0: + top_1_share = None + top_5_share = None + hhi = None + reason = "zero_gross_exposure" + else: + ordered = sorted(absolute_values, reverse=True) + raw_shares = tuple(value / gross for value in absolute_values) + top_1_share = _rounded_ratio(ordered[0] / gross) + top_5_share = _rounded_ratio(sum(ordered[:5], Decimal(0)) / gross) + hhi = _rounded_ratio(sum((share * share for share in raw_shares), Decimal(0))) + reason = None + categories = { + name: _category_exposure(active, basis, name, gross) + for name in _CATEGORY_NAMES + } + return ExposureAnalysis( + state="ready", + level=_LEVEL_BY_BASIS[basis], + basis=basis, + candidate_basis=None, + position_count=len(snapshot.positions), + zero_position_count=zero_count, + long_count=long_count, + short_count=short_count, + gross=gross, + net=net, + top_1_share=top_1_share, + top_5_share=top_5_share, + hhi=hhi, + coverage_ratio=Decimal(1), + coverage_by_basis=coverage_by_basis, + categories=categories, + category_coverage=category_coverage, + reason=reason, + ) + + +def _is_active(position: Position) -> bool: + return any( + value is not None and value != 0 + for value in ( + position.quantity, + position.weight, + position.market_value_base, + position.exposure_value_base, + ) + ) + + +def _direction(position: Position) -> int: + for value in ( + position.quantity, + position.weight, + position.market_value_base, + position.exposure_value_base, + ): + if value is not None and value != 0: + return 1 if value > 0 else -1 + return 0 + + +def _basis_value(position: Position, basis: Basis) -> Decimal | None: + if basis == "weight": + return position.weight + if basis == "market_value_base": + return position.market_value_base + return position.exposure_value_base + + +def _required_basis_value(position: Position, basis: Basis) -> Decimal: + value = _basis_value(position, basis) + if value is None: + raise AssertionError("selected exposure basis is incomplete") + return value + + +def _best_candidate(coverage: dict[Basis, Decimal | None]) -> Basis | None: + best: Basis | None = None + best_coverage = Decimal(0) + for basis in _BASIS_PRIORITY: + value = coverage[basis] + if value is not None and value > best_coverage: + best = basis + best_coverage = value + return best + + +def _category_exposure( + positions: tuple[Position, ...], + basis: Basis, + category_name: str, + total_gross: Decimal, +) -> tuple[CategoryExposure, ...]: + buckets: dict[str, list[Decimal]] = {} + for position in positions: + label_value = getattr(position, category_name) + label = label_value if isinstance(label_value, str) else "unclassified" + buckets.setdefault(label, []).append(_required_basis_value(position, basis)) + result = [ + CategoryExposure( + label=label, + gross=sum((abs(value) for value in values), Decimal(0)), + net=sum(values, Decimal(0)), + share=( + _rounded_ratio(sum((abs(value) for value in values), Decimal(0)) / total_gross) + if total_gross != 0 + else None + ), + ) + for label, values in buckets.items() + ] + return tuple(sorted(result, key=lambda item: (-item.gross, item.label))) + + +def _ratio(numerator: Decimal, denominator: Decimal) -> Decimal: + return _rounded_ratio(numerator / denominator) + + +def _rounded_ratio(value: Decimal) -> Decimal: + rounded = value.quantize(_RATIO_QUANTUM, rounding=ROUND_HALF_EVEN) + if rounded == rounded.to_integral_value(): + return rounded.to_integral_value() + return rounded.normalize() diff --git a/src/quantcockpit/api.py b/src/quantcockpit/api.py index fdbd539..d2a93bb 100644 --- a/src/quantcockpit/api.py +++ b/src/quantcockpit/api.py @@ -4,6 +4,7 @@ import os from datetime import datetime, timezone +from importlib.metadata import version from pathlib import Path from typing import Annotated, Callable, Literal @@ -17,11 +18,21 @@ HealthzResponse, IngestionErrorsResponse, IngestionIssueResponse, + PortfolioExposureResponse, + PortfoliosResponse, + PortfolioSummaryResponse, StrategiesResponse, StrategyHealthResponse, StrategySummaryResponse, ) -from quantcockpit.service import CockpitService, correlation_payload, evidence_payload, utc_text +from quantcockpit.service import ( + CockpitService, + correlation_payload, + evidence_payload, + portfolio_exposure_payload, + portfolio_summary_payload, + utc_text, +) from quantcockpit.store import DatabaseUnavailableError, DuckDBStore @@ -51,7 +62,12 @@ def create_app( configured_path = Path(database_path or os.environ.get("QUANTCOCKPIT_DB_PATH", "quantcockpit.duckdb")) service_clock = clock if clock is not None else _demo_clock_from_environment() - app = FastAPI(title="QuantCockpit", docs_url=None, redoc_url=None) + app = FastAPI( + title="QuantCockpit", + version=version("quantcockpit"), + docs_url=None, + redoc_url=None, + ) def get_service(): try: @@ -129,6 +145,39 @@ def ingestion_errors(service: CockpitService = Depends(get_service)) -> Ingestio failed_runs=[IngestionIssueResponse.model_validate(item) for item in errors.failed_runs], ) + @app.get("/api/v1/portfolios") + def portfolios(service: CockpitService = Depends(get_service)) -> PortfoliosResponse: + items = service.portfolios() + return PortfoliosResponse( + state="ready" if items else "empty", + portfolios=[ + PortfolioSummaryResponse.model_validate(portfolio_summary_payload(item)) + for item in items + ], + ) + + @app.get("/api/v1/portfolios/{portfolio_id}/exposure") + def portfolio_exposure( + portfolio_id: str, + strategy_id: Annotated[str, Query(min_length=1, max_length=128)], + environment: Literal["live", "paper"], + source: Annotated[str, Query(min_length=1, max_length=256)], + service: CockpitService = Depends(get_service), + ) -> PortfolioExposureResponse: + strategy_id = strategy_id.strip() + source = source.strip() + if not strategy_id or not source: + raise HTTPException(status_code=422, detail="identity query values must not be blank") + result = service.portfolio_exposure( + portfolio_id, + strategy_id, + environment, + source, + ) + if result is None: + raise HTTPException(status_code=404, detail="portfolio identity not found") + return PortfolioExposureResponse.model_validate(portfolio_exposure_payload(result)) + return app diff --git a/src/quantcockpit/api_models.py b/src/quantcockpit/api_models.py index 3ae9bfe..7f2ed8f 100644 --- a/src/quantcockpit/api_models.py +++ b/src/quantcockpit/api_models.py @@ -12,6 +12,8 @@ Environment = Literal["live", "paper"] HealthStatus = Literal["healthy", "warning", "critical", "unknown"] State = Literal["ready", "empty"] +AnalysisState = Literal["ready", "unavailable", "empty_portfolio"] +Basis = Literal["weight", "market_value_base", "exposure_value_base"] class ApiResponseModel(BaseModel): @@ -95,3 +97,60 @@ class IngestionErrorsResponse(ApiResponseModel): state: State quarantines: list[IngestionIssueResponse] failed_runs: list[IngestionIssueResponse] + + +class PortfolioIdentityResponse(ApiResponseModel): + portfolio_id: str + strategy_id: str + environment: Environment + source: str + + +class PortfolioSummaryResponse(PortfolioIdentityResponse): + snapshot_time: str + age_seconds: int + position_count: int + zero_position_count: int + capability_level: int + analysis_state: AnalysisState + basis: Basis | None + candidate_basis: Basis | None + missing_fields: list[str] + + +class PortfoliosResponse(ApiResponseModel): + state: State + portfolios: list[PortfolioSummaryResponse] + + +class CategoryExposureResponse(ApiResponseModel): + label: str + gross: str + net: str + share: str | None + + +class PortfolioExposureResponse(PortfolioIdentityResponse): + snapshot_time: str + age_seconds: int + base_currency: str | None + mapping_profile_hash: str + analysis_state: AnalysisState + capability_level: int + basis: Basis | None + candidate_basis: Basis | None + position_count: int + zero_position_count: int + long_count: int + short_count: int + gross: str | None + net: str | None + top_1_share: str | None + top_5_share: str | None + hhi: str | None + coverage_ratio: str | None + coverage_by_basis: dict[Basis, str | None] + categories: dict[str, list[CategoryExposureResponse]] + category_coverage: dict[str, str | None] + reason: str | None + evidence_refs: list[str] diff --git a/src/quantcockpit/ingestion/position_profile.py b/src/quantcockpit/ingestion/position_profile.py new file mode 100644 index 0000000..b38ec52 --- /dev/null +++ b/src/quantcockpit/ingestion/position_profile.py @@ -0,0 +1,265 @@ +"""仓位来源的声明式映射配置与无副作用转换。""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from datetime import datetime, timezone +from decimal import Decimal +from hashlib import sha256 +import re +from typing import Literal, cast +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from pydantic import BaseModel, ConfigDict, StringConstraints, TypeAdapter, model_validator +from typing_extensions import Annotated + +import rfc8785 + +from quantcockpit.models import PositionDecimal + + +Transform = Literal["trim", "uppercase", "lowercase", "decimal", "utc_timestamp"] +InputFormat = Literal["csv", "json", "jsonl"] +Layout = Literal["document_snapshot", "tabular_snapshot"] +SnapshotScope = Literal["grouped_rows", "whole_file"] +LiteralValue = str | int | bool +MetadataField = Literal[ + "strategy_id", + "environment", + "source", + "portfolio_id", + "snapshot_time", + "recorded_at", + "base_currency", +] +PositionField = Literal[ + "instrument_id", + "instrument_id_type", + "venue", + "side", + "quantity", + "weight", + "market_value_base", + "exposure_value_base", + "asset_class", + "sector", + "country", +] + + +_RFC3339_PATTERN = re.compile( + r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,6})?(?:Z|[+-]\d{2}:\d{2})$" +) +_INVALID_POINTER_ESCAPE = re.compile(r"~(?:[^01]|$)") +_POSITION_DECIMAL_ADAPTER = TypeAdapter(PositionDecimal) + + +class MappingValueError(ValueError): + """不包含原始值的安全映射错误。""" + + def __init__(self, code: str, message: str) -> None: + self.code = code + super().__init__(f"{code}: {message}") + + +class FieldBinding(BaseModel): + """一个固定值或来源路径,以及有限的确定性转换链。""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + literal: LiteralValue | None = None + path: Annotated[str, StringConstraints(min_length=1, max_length=512)] | None = None + transforms: tuple[Transform, ...] = () + timestamp_format: Annotated[str, StringConstraints(min_length=1, max_length=128)] | None = None + assume_timezone: Annotated[str, StringConstraints(min_length=1, max_length=128)] | None = None + + @model_validator(mode="after") + def require_safe_binding(self) -> FieldBinding: + if (self.literal is None) == (self.path is None): + raise ValueError("field binding requires exactly one of literal or path") + if (self.timestamp_format is None) != (self.assume_timezone is None): + raise ValueError("timestamp_format and assume_timezone must be configured together") + if self.timestamp_format is not None and "utc_timestamp" not in self.transforms: + raise ValueError("local timestamp configuration requires utc_timestamp transform") + if self.assume_timezone is not None: + try: + ZoneInfo(self.assume_timezone) + except ZoneInfoNotFoundError as error: + raise ValueError("assume_timezone must be a known IANA timezone") from error + return self + + +class PositionMappingProfile(BaseModel): + """一类仓位文件的版本化映射契约。""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + profile_version: Literal["1.0"] + name: Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=1, max_length=128), + ] + format: InputFormat + layout: Layout + snapshot_scope: SnapshotScope + fields: dict[MetadataField, FieldBinding] + position_fields: dict[PositionField, FieldBinding] + positions_path: Annotated[str, StringConstraints(min_length=1, max_length=512)] | None = None + + @model_validator(mode="after") + def require_complete_mapping(self) -> PositionMappingProfile: + required_metadata = { + "strategy_id", + "environment", + "source", + "portfolio_id", + "snapshot_time", + } + missing_metadata = required_metadata.difference(self.fields) + if missing_metadata: + raise ValueError(f"mapping fields missing required names: {sorted(missing_metadata)}") + if "instrument_id" not in self.position_fields: + raise ValueError("position_fields must map instrument_id") + if not {"quantity", "weight", "market_value_base", "exposure_value_base"}.intersection( + self.position_fields + ): + raise ValueError("position_fields must map at least one measure") + if self.layout == "document_snapshot" and self.positions_path is None: + raise ValueError("document_snapshot requires positions_path") + if self.layout == "tabular_snapshot" and self.positions_path is not None: + raise ValueError("tabular_snapshot does not accept positions_path") + if self.format == "csv" and self.layout != "tabular_snapshot": + raise ValueError("csv supports only tabular_snapshot layout") + if self.format != "csv": + paths = [ + binding.path + for binding in (*self.fields.values(), *self.position_fields.values()) + if binding.path is not None + ] + if self.positions_path is not None: + paths.append(self.positions_path) + if any(not path.startswith("/") for path in paths): + raise ValueError("JSON and JSONL paths must use RFC 6901 JSON Pointer syntax") + return self + + +def profile_hash(profile: PositionMappingProfile) -> str: + """返回 RFC 8785 规范化映射的稳定 SHA-256 证据引用。""" + + canonical = rfc8785.dumps(profile.model_dump(mode="json", exclude_none=True)) + return f"sha256:{sha256(canonical).hexdigest()}" + + +def resolve_binding( + binding: FieldBinding, + record: Mapping[str, object], + *, + format: InputFormat, +) -> object: + """读取一个绑定并按声明顺序转换,失败时不泄漏原始值。""" + + if binding.path is None: + value: object = binding.literal + elif format == "csv": + if binding.path not in record: + raise MappingValueError("mapping_path_missing", "configured CSV column is absent") + value = record[binding.path] + else: + value = _resolve_json_pointer(record, binding.path) + + for transform in binding.transforms: + value = _apply_transform(value, transform, binding) + return value + + +def _resolve_json_pointer(record: object, pointer: str) -> object: + if not pointer.startswith("/") or _INVALID_POINTER_ESCAPE.search(pointer): + raise MappingValueError("mapping_path_invalid", "configured JSON Pointer is invalid") + current = record + for raw_token in pointer[1:].split("/"): + token = raw_token.replace("~1", "/").replace("~0", "~") + if isinstance(current, Mapping): + current_mapping = cast(Mapping[str, object], current) + if token not in current_mapping: + raise MappingValueError("mapping_path_missing", "configured JSON member is absent") + current = current_mapping[token] + elif isinstance(current, Sequence) and not isinstance(current, (str, bytes, bytearray)): + if not token.isdigit(): + raise MappingValueError("mapping_path_missing", "configured JSON array index is invalid") + index = int(token) + if index >= len(current): + raise MappingValueError("mapping_path_missing", "configured JSON array index is absent") + current = current[index] + else: + raise MappingValueError("mapping_path_missing", "configured JSON path cannot be traversed") + return current + + +def _apply_transform(value: object, transform: Transform, binding: FieldBinding) -> object: + if transform == "trim": + return _require_text(value, transform).strip() + if transform == "uppercase": + return _require_text(value, transform).upper() + if transform == "lowercase": + return _require_text(value, transform).lower() + if transform == "decimal": + try: + return _POSITION_DECIMAL_ADAPTER.validate_python(value) + except ValueError as error: + raise MappingValueError( + "mapping_decimal_invalid", + "source value is not an allowed fixed-point decimal", + ) from error + return _utc_timestamp(value, binding) + + +def _require_text(value: object, transform: str) -> str: + if not isinstance(value, str): + raise MappingValueError( + "mapping_transform_invalid", + f"{transform} requires a text source value", + ) + return value + + +def _utc_timestamp(value: object, binding: FieldBinding) -> datetime: + text = _require_text(value, "utc_timestamp") + if binding.timestamp_format is None: + if not _RFC3339_PATTERN.fullmatch(text): + raise MappingValueError( + "mapping_timestamp_invalid", + "timestamp must be RFC 3339 with an explicit offset", + ) + try: + parsed = datetime.fromisoformat(text.replace("Z", "+00:00")) + except ValueError as error: + raise MappingValueError( + "mapping_timestamp_invalid", + "timestamp is not a valid calendar time", + ) from error + return parsed.astimezone(timezone.utc) + + try: + naive = datetime.strptime(text, binding.timestamp_format) + except ValueError as error: + raise MappingValueError( + "mapping_timestamp_invalid", + "timestamp does not match configured local format", + ) from error + if naive.tzinfo is not None or binding.assume_timezone is None: + raise MappingValueError( + "mapping_timestamp_invalid", + "configured local timestamp must not contain an offset", + ) + zone = ZoneInfo(binding.assume_timezone) + candidates: set[datetime] = set() + for fold in (0, 1): + candidate = naive.replace(tzinfo=zone, fold=fold).astimezone(timezone.utc) + if candidate.astimezone(zone).replace(tzinfo=None) == naive: + candidates.add(candidate) + if len(candidates) != 1: + raise MappingValueError( + "ambiguous_source_time", + "local timestamp is ambiguous or does not exist in configured timezone", + ) + return candidates.pop() diff --git a/src/quantcockpit/ingestion/position_sources.py b/src/quantcockpit/ingestion/position_sources.py new file mode 100644 index 0000000..d018872 --- /dev/null +++ b/src/quantcockpit/ingestion/position_sources.py @@ -0,0 +1,262 @@ +"""受资源限制的 CSV、JSON 与 JSONL 仓位来源读取器。""" + +from __future__ import annotations + +from collections.abc import Iterator, Mapping +import csv +from dataclasses import dataclass +import json +from pathlib import Path +import re +import stat +from typing import TextIO, cast + +from quantcockpit.ingestion.position_profile import PositionMappingProfile + + +MAX_SOURCE_BYTES = 100 * 1024 * 1024 +MAX_RECORD_BYTES = 1024 * 1024 +_EXTENSION_BY_FORMAT = {"csv": ".csv", "json": ".json", "jsonl": ".jsonl"} + + +class SourceReadError(ValueError): + """不回显路径或原始内容的来源读取错误。""" + + def __init__(self, code: str, message: str, *, line_number: int | None = None) -> None: + self.code = code + self.line_number = line_number + location = "" if line_number is None else f" at line {line_number}" + super().__init__(f"{code}{location}: {message}") + + +@dataclass(frozen=True) +class SourceRecord: + value: Mapping[str, object] + start_line: int + end_line: int + raw_json: str + + +class _TrackedLines: + def __init__(self, source: TextIO) -> None: + self._source = source + self.total_bytes = 0 + + def __iter__(self) -> _TrackedLines: + return self + + def __next__(self) -> str: + line = next(self._source) + self.total_bytes += len(line.encode("utf-8")) + if self.total_bytes > MAX_SOURCE_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "source file exceeds the 100 MiB limit", + ) + return line + + +def read_source( + path: str | Path, + profile: PositionMappingProfile, +) -> Iterator[SourceRecord]: + """按 profile 读取来源记录;不保留文件句柄,也不吞掉格式错误。""" + + source_path = Path(path) + _validate_source_file(source_path, profile) + if profile.format == "csv": + yield from _read_csv(source_path) + elif profile.format == "jsonl": + yield from _read_jsonl(source_path) + else: + yield from _read_json(source_path, document=profile.layout == "document_snapshot") + + +def _validate_source_file(source_path: Path, profile: PositionMappingProfile) -> None: + if source_path.suffix.lower() != _EXTENSION_BY_FORMAT[profile.format]: + raise SourceReadError( + "unsupported_input_format", + "file extension does not match mapping profile format", + ) + try: + source_stat = source_path.stat() + except OSError as error: + raise SourceReadError("file_read_error", "source file cannot be inspected") from error + if not stat.S_ISREG(source_stat.st_mode): + raise SourceReadError("file_read_error", "source path must be a regular file") + if source_stat.st_size > MAX_SOURCE_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "source file exceeds the 100 MiB limit", + ) + + +def _read_csv(source_path: Path) -> Iterator[SourceRecord]: + try: + with source_path.open("r", encoding="utf-8", newline="") as source: + tracked = _TrackedLines(source) + reader = csv.DictReader(tracked) + fieldnames = reader.fieldnames + if fieldnames is None: + return + if any(not name or not name.strip() for name in fieldnames) or len(set(fieldnames)) != len( + fieldnames + ): + raise SourceReadError( + "invalid_tabular_header", + "CSV header names must be nonblank and unique", + line_number=1, + ) + if tracked.total_bytes > MAX_RECORD_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "CSV header exceeds the 1 MiB record limit", + line_number=1, + ) + previous_line = reader.line_num + previous_bytes = tracked.total_bytes + for row in reader: + start_line = previous_line + 1 + end_line = reader.line_num + record_bytes = tracked.total_bytes - previous_bytes + if record_bytes > MAX_RECORD_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "CSV record exceeds the 1 MiB limit", + line_number=start_line, + ) + if None in row or any(value is None for value in row.values()): + raise SourceReadError( + "invalid_tabular_record", + "CSV row does not match the declared header", + line_number=start_line, + ) + normalized = cast(dict[str, object], row) + yield SourceRecord( + value=normalized, + start_line=start_line, + end_line=end_line, + raw_json=_safe_json(normalized), + ) + previous_line = end_line + previous_bytes = tracked.total_bytes + except SourceReadError: + raise + except (OSError, UnicodeError) as error: + raise SourceReadError("file_read_error", "source CSV cannot be read as UTF-8") from error + except csv.Error as error: + raise SourceReadError("invalid_tabular_record", "source CSV is malformed") from error + + +def _read_jsonl(source_path: Path) -> Iterator[SourceRecord]: + try: + file_size = source_path.stat().st_size + total_bytes = 0 + with source_path.open("rb") as source: + for line_number, raw_line in enumerate(source, start=1): + total_bytes += len(raw_line) + if total_bytes > MAX_SOURCE_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "source file exceeds the 100 MiB limit", + ) + if len(raw_line) > MAX_RECORD_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "JSONL record exceeds the 1 MiB limit", + line_number=line_number, + ) + if not raw_line.strip(): + continue + try: + text = raw_line.decode("utf-8") + decoded = json.loads(text) + except UnicodeError as error: + raise SourceReadError( + "file_read_error", + "source JSONL cannot be read as UTF-8", + line_number=line_number, + ) from error + except json.JSONDecodeError as error: + is_tail = source.tell() == file_size and _incomplete_json_error(error, text) + raise SourceReadError( + "incomplete_tail" if is_tail else "invalid_json", + "JSONL record is incomplete" if is_tail else "JSONL record is malformed", + line_number=line_number, + ) from error + record = _require_object(decoded, line_number=line_number) + yield SourceRecord( + value=record, + start_line=line_number, + end_line=line_number, + raw_json=_safe_json(record), + ) + except SourceReadError: + raise + except OSError as error: + raise SourceReadError("file_read_error", "source JSONL cannot be read") from error + + +def _read_json(source_path: Path, *, document: bool) -> Iterator[SourceRecord]: + try: + with source_path.open("rb") as source: + raw = source.read(MAX_SOURCE_BYTES + 1) + if len(raw) > MAX_SOURCE_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "source file exceeds the 100 MiB limit", + ) + text = raw.decode("utf-8") + except SourceReadError: + raise + except (OSError, UnicodeError) as error: + raise SourceReadError("file_read_error", "source JSON cannot be read as UTF-8") from error + try: + decoded = json.loads(text) + except json.JSONDecodeError as error: + raise SourceReadError("invalid_json", "source JSON is malformed", line_number=error.lineno) from error + line_count = max(text.count("\n") + 1, 1) + if document: + record = _require_object(decoded, line_number=1) + if len(_safe_json(record).encode("utf-8")) > MAX_SOURCE_BYTES: + raise SourceReadError("ingestion_limit_exceeded", "JSON document exceeds source limit") + yield SourceRecord(value=record, start_line=1, end_line=line_count, raw_json=_safe_json(record)) + return + if not isinstance(decoded, list): + raise SourceReadError("invalid_json_layout", "tabular JSON must contain a top-level array") + for index, item in enumerate(decoded, start=1): + record = _require_object(item, line_number=index) + raw_json = _safe_json(record) + if len(raw_json.encode("utf-8")) > MAX_RECORD_BYTES: + raise SourceReadError( + "ingestion_limit_exceeded", + "JSON array record exceeds the 1 MiB limit", + line_number=index, + ) + yield SourceRecord(value=record, start_line=index, end_line=index, raw_json=raw_json) + + +def _require_object(value: object, *, line_number: int) -> Mapping[str, object]: + if not isinstance(value, dict) or any(not isinstance(key, str) for key in value): + raise SourceReadError( + "invalid_json_layout", + "source record must be a JSON object", + line_number=line_number, + ) + return cast(dict[str, object], value) + + +def _safe_json(value: object) -> str: + return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + + +def _incomplete_json_error(error: json.JSONDecodeError, raw_line: str) -> bool: + stripped = raw_line.strip() + return ( + error.pos >= len(stripped) + or error.msg.startswith("Unterminated string") + or ( + error.msg == "Invalid \\uXXXX escape" + and re.search(r"\\u[0-9a-fA-F]{0,3}$", stripped) is not None + ) + ) diff --git a/src/quantcockpit/ingestion/positions.py b/src/quantcockpit/ingestion/positions.py new file mode 100644 index 0000000..8a07657 --- /dev/null +++ b/src/quantcockpit/ingestion/positions.py @@ -0,0 +1,449 @@ +"""仓位文件的只读预览与规范事件构造。""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import TYPE_CHECKING, Literal, cast + +from pydantic import TypeAdapter, ValidationError + +from quantcockpit.ingestion.position_profile import ( + FieldBinding, + MappingValueError, + PositionMappingProfile, + profile_hash, + resolve_binding, +) +from quantcockpit.ingestion.position_sources import SourceReadError, SourceRecord, read_source +from quantcockpit.models import EventRecord, PositionDecimal + +if TYPE_CHECKING: + from quantcockpit.store import DuckDBStore + + +MAX_POSITIONS_PER_SNAPSHOT = 100_000 +_MEASURE_FIELDS = ("quantity", "weight", "market_value_base", "exposure_value_base") +_POSITION_DECIMAL_ADAPTER = TypeAdapter(PositionDecimal) + + +class PositionImportError(ValueError): + """可供 CLI 安全展示的仓位预览错误。""" + + def __init__(self, code: str, message: str, *, line_number: int | None = None) -> None: + self.code = code + self.line_number = line_number + location = "" if line_number is None else f" at line {line_number}" + super().__init__(f"{code}{location}: {message}") + + +@dataclass(frozen=True) +class NormalizedSnapshot: + event: EventRecord + raw_json: str + start_line: int + end_line: int + recorded_at_source: Literal["mapped", "import_observed_at"] + + +@dataclass(frozen=True) +class PositionPreview: + format: str + record_count: int + snapshot_count: int + source_fields: tuple[str, ...] + sample_positions: tuple[dict[str, object], ...] + snapshots: tuple[NormalizedSnapshot, ...] + mapping_profile_hash: str + warnings: tuple[dict[str, object], ...] + + +@dataclass(frozen=True) +class PositionImportResult: + imported: int = 0 + duplicates: int = 0 + revisions: int = 0 + stale: int = 0 + rejected_snapshots: int = 0 + + +@dataclass +class _SnapshotGroup: + metadata: dict[str, object] + positions: list[dict[str, object]] + records: list[SourceRecord] + + +def preview_positions( + path: str | Path, + profile: PositionMappingProfile, + *, + observed_at: datetime, +) -> PositionPreview: + """完整校验仓位文件并返回只读预览;本函数没有持久化入口。""" + + if observed_at.tzinfo is None or observed_at.utcoffset() is None: + raise ValueError("observed_at must be timezone-aware") + observed_at = observed_at.astimezone(timezone.utc) + try: + records = tuple(read_source(path, profile)) + if profile.layout == "document_snapshot": + snapshots = _preview_documents(records, profile, observed_at) + else: + snapshots = _preview_tabular(records, profile, observed_at) + except PositionImportError: + raise + except SourceReadError as error: + raise PositionImportError(error.code, "source file could not be read safely", line_number=error.line_number) from error + except MappingValueError as error: + raise PositionImportError(error.code, "configured source value could not be mapped") from error + + source_fields = tuple(sorted({key for record in records for key in record.value})) + samples: list[dict[str, object]] = [] + for snapshot in snapshots: + payload = snapshot.event.payload + if payload.event_type != "position_snapshot": + raise AssertionError("position preview created a non-position event") + for position in payload.positions: + if len(samples) == 5: + break + samples.append(position.model_dump(mode="json", exclude_none=True)) + if len(samples) == 5: + break + return PositionPreview( + format=profile.format, + record_count=len(records), + snapshot_count=len(snapshots), + source_fields=source_fields, + sample_positions=tuple(samples), + snapshots=snapshots, + mapping_profile_hash=profile_hash(profile), + warnings=(), + ) + + +def import_positions( + store: DuckDBStore, + path: str | Path, + profile: PositionMappingProfile, + *, + observed_at: datetime | None = None, +) -> PositionImportResult: + """先完整预览,再把每个合法快照写入现有版本化事件存储。""" + + observed = observed_at or datetime.now(timezone.utc) + if observed.tzinfo is None or observed.utcoffset() is None: + raise ValueError("observed_at must be timezone-aware") + observed = observed.astimezone(timezone.utc) + source_path = Path(path) + run_id = store.begin_run(source_path, observed) + try: + preview = preview_positions(source_path, profile, observed_at=observed) + except PositionImportError as error: + store.fail_run(run_id, observed, error_code=error.code, error_message=str(error)) + raise + except Exception as error: + store.fail_run( + run_id, + observed, + error_code="processing_error", + error_message="position preview failed", + ) + raise + + counts = {"imported": 0, "duplicates": 0, "revisions": 0, "stale": 0} + try: + for snapshot in preview.snapshots: + disposition = store.record_event( + snapshot.event, + raw_json=snapshot.raw_json, + source_file=source_path, + line_number=snapshot.start_line, + end_line_number=snapshot.end_line, + observed_at=observed, + ignore_recorded_at_for_content=( + snapshot.recorded_at_source == "import_observed_at" + ), + ) + count_key = {"revision": "revisions", "duplicate": "duplicates"}.get( + disposition, + disposition, + ) + counts[count_key] += 1 + except Exception: + store.fail_run( + run_id, + observed, + error_code="processing_error", + error_message="position persistence failed", + ) + raise + store.finish_run( + run_id, + observed, + imported=counts["imported"], + duplicates=counts["duplicates"], + revisions=counts["revisions"], + stale=counts["stale"], + quarantined=0, + incomplete_tail=0, + blank=0, + ) + return PositionImportResult(**counts) + + +def _preview_documents( + records: tuple[SourceRecord, ...], + profile: PositionMappingProfile, + observed_at: datetime, +) -> tuple[NormalizedSnapshot, ...]: + if not records: + raise PositionImportError("empty_input", "document source contains no snapshot") + snapshots: list[NormalizedSnapshot] = [] + for record in records: + if profile.positions_path is None: + raise AssertionError("validated document profile is missing positions_path") + metadata = _resolve_metadata(profile, record.value, observed_at, line_number=record.start_line) + try: + raw_positions = resolve_binding( + FieldBinding(path=profile.positions_path), + record.value, + format=profile.format, + ) + except MappingValueError as error: + raise PositionImportError(error.code, "positions array path could not be mapped") from error + if not isinstance(raw_positions, Sequence) or isinstance(raw_positions, (str, bytes, bytearray)): + raise PositionImportError( + "invalid_json_layout", + "document positions path must resolve to an array", + line_number=record.start_line, + ) + if len(raw_positions) > MAX_POSITIONS_PER_SNAPSHOT: + raise PositionImportError( + "ingestion_limit_exceeded", + "snapshot exceeds the position limit", + line_number=record.start_line, + ) + positions = [ + _resolve_position(profile, _require_mapping(item, record.start_line), record.start_line) + for item in raw_positions + ] + snapshots.append( + _build_snapshot( + metadata, + positions, + raw_json=record.raw_json, + start_line=record.start_line, + end_line=record.end_line, + mapping_hash=profile_hash(profile), + ) + ) + return tuple(snapshots) + + +def _preview_tabular( + records: tuple[SourceRecord, ...], + profile: PositionMappingProfile, + observed_at: datetime, +) -> tuple[NormalizedSnapshot, ...]: + if not records: + if profile.snapshot_scope != "whole_file" or any( + profile.fields[name].literal is None + for name in ("strategy_id", "environment", "source", "portfolio_id", "snapshot_time") + ): + raise PositionImportError("empty_input", "tabular source contains no position rows") + metadata = _resolve_metadata(profile, {}, observed_at, line_number=1) + return ( + _build_snapshot( + metadata, + [], + raw_json="[]", + start_line=1, + end_line=1, + mapping_hash=profile_hash(profile), + ), + ) + + groups: dict[tuple[object, ...], _SnapshotGroup] = {} + whole_file_key: tuple[object, ...] | None = None + for record in records: + metadata = _resolve_metadata(profile, record.value, observed_at, line_number=record.start_line) + identity = tuple( + metadata[name] + for name in ("strategy_id", "environment", "source", "portfolio_id", "snapshot_time") + ) + if profile.snapshot_scope == "whole_file": + if whole_file_key is None: + whole_file_key = identity + elif identity != whole_file_key: + raise PositionImportError( + "snapshot_identity_conflict", + "whole-file snapshot contains conflicting identity fields", + line_number=record.start_line, + ) + group = groups.get(identity) + if group is None: + group = _SnapshotGroup(metadata=metadata, positions=[], records=[]) + groups[identity] = group + elif any( + group.metadata.get(name) != metadata.get(name) + for name in ("base_currency", "recorded_at") + ): + raise PositionImportError( + "snapshot_identity_conflict", + "snapshot group contains conflicting metadata", + line_number=record.start_line, + ) + group.positions.append(_resolve_position(profile, record.value, record.start_line)) + group.records.append(record) + if len(group.positions) > MAX_POSITIONS_PER_SNAPSHOT: + raise PositionImportError( + "ingestion_limit_exceeded", + "snapshot exceeds the position limit", + line_number=record.start_line, + ) + + snapshots = [ + _build_snapshot( + group.metadata, + group.positions, + raw_json=f"[{','.join(record.raw_json for record in group.records)}]", + start_line=group.records[0].start_line, + end_line=group.records[-1].end_line, + mapping_hash=profile_hash(profile), + ) + for group in groups.values() + ] + return tuple(snapshots) + + +def _resolve_metadata( + profile: PositionMappingProfile, + record: Mapping[str, object], + observed_at: datetime, + *, + line_number: int, +) -> dict[str, object]: + try: + metadata: dict[str, object] = { + str(name): resolve_binding(binding, record, format=profile.format) + for name, binding in profile.fields.items() + } + except MappingValueError as error: + raise PositionImportError(error.code, "snapshot metadata could not be mapped", line_number=line_number) from error + snapshot_time = metadata.get("snapshot_time") + if not isinstance(snapshot_time, datetime): + raise PositionImportError( + "mapping_timestamp_invalid", + "snapshot_time must use utc_timestamp transform", + line_number=line_number, + ) + recorded_at = metadata.get("recorded_at") + if recorded_at is None: + metadata["recorded_at"] = observed_at + metadata["recorded_at_source"] = "import_observed_at" + elif isinstance(recorded_at, datetime): + metadata["recorded_at_source"] = "mapped" + else: + raise PositionImportError( + "mapping_timestamp_invalid", + "recorded_at must use utc_timestamp transform", + line_number=line_number, + ) + return metadata + + +def _resolve_position( + profile: PositionMappingProfile, + record: Mapping[str, object], + line_number: int, +) -> dict[str, object]: + try: + position: dict[str, object] = { + str(name): resolve_binding(binding, record, format=profile.format) + for name, binding in profile.position_fields.items() + } + except MappingValueError as error: + raise PositionImportError(error.code, "position field could not be mapped", line_number=line_number) from error + side = position.get("side") + if side is not None and side not in {"long", "short"}: + raise PositionImportError( + "mapping_value_invalid", + "position side must normalize to long or short", + line_number=line_number, + ) + for name in _MEASURE_FIELDS: + if name not in position: + continue + try: + measure = _POSITION_DECIMAL_ADAPTER.validate_python(position[name]) + except ValidationError as error: + raise PositionImportError( + "mapping_decimal_invalid", + "position measure is not a valid fixed-point decimal", + line_number=line_number, + ) from error + position[name] = -measure if side == "short" and measure > 0 else measure + return position + + +def _build_snapshot( + metadata: Mapping[str, object], + positions: list[dict[str, object]], + *, + raw_json: str, + start_line: int, + end_line: int, + mapping_hash: str, +) -> NormalizedSnapshot: + payload: dict[str, object] = { + "event_type": "position_snapshot", + "portfolio_id": metadata["portfolio_id"], + "mapping_profile_hash": mapping_hash, + "positions": positions, + } + if "base_currency" in metadata: + payload["base_currency"] = metadata["base_currency"] + event_data = { + "schema_version": "1.1", + "strategy_id": metadata["strategy_id"], + "environment": metadata["environment"], + "event_type": "position_snapshot", + "event_time": metadata["snapshot_time"], + "recorded_at": metadata["recorded_at"], + "source": metadata["source"], + "payload": payload, + } + try: + event = EventRecord.model_validate(event_data) + except ValidationError as error: + safe_errors = error.errors(include_input=False, include_context=False, include_url=False) + duplicate = any("duplicate instrument" in item["msg"] for item in safe_errors) + raise PositionImportError( + "duplicate_instrument" if duplicate else "contract_invalid", + "normalized snapshot does not satisfy the position contract", + line_number=start_line, + ) from error + recorded_at_source = cast( + Literal["mapped", "import_observed_at"], + metadata["recorded_at_source"], + ) + return NormalizedSnapshot( + event=event, + raw_json=raw_json, + start_line=start_line, + end_line=end_line, + recorded_at_source=recorded_at_source, + ) + + +def _require_mapping(value: object, line_number: int) -> Mapping[str, object]: + if not isinstance(value, dict) or any(not isinstance(key, str) for key in value): + raise PositionImportError( + "invalid_json_layout", + "document positions must contain JSON objects", + line_number=line_number, + ) + return cast(dict[str, object], value) diff --git a/src/quantcockpit/models.py b/src/quantcockpit/models.py index 5788c70..05e7ac4 100644 --- a/src/quantcockpit/models.py +++ b/src/quantcockpit/models.py @@ -8,12 +8,20 @@ import re from typing import Annotated, Literal -from pydantic import BaseModel, ConfigDict, Field, StringConstraints, field_validator, model_validator +from pydantic import ( + BaseModel, + BeforeValidator, + ConfigDict, + Field, + StringConstraints, + field_validator, + model_validator, +) -SchemaVersion = Literal["1.0"] +SchemaVersion = Literal["1.0", "1.1"] Environment = Literal["live", "paper"] -EventType = Literal["heartbeat", "run_status", "nav", "return"] +EventType = Literal["heartbeat", "run_status", "nav", "return", "position_snapshot"] StrategyId = Annotated[ str, StringConstraints( @@ -24,6 +32,51 @@ ), ] Source = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, max_length=256)] +InstrumentText = Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=1, max_length=256), +] +ClassificationText = Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=1, max_length=128), +] + + +_FIXED_DECIMAL_PATTERN = re.compile(r"^[+-]?(?:0|[1-9]\d*)(?:\.\d+)?$") + + +def _position_decimal(value: object) -> Decimal: + """把仓位度量限制为可审计的定点 Decimal,拒绝 float 和指数输入。""" + + if isinstance(value, bool) or isinstance(value, float): + raise ValueError("position decimal must not use binary float") + if isinstance(value, str): + if not _FIXED_DECIMAL_PATTERN.fullmatch(value): + raise ValueError("position decimal must use finite fixed-point notation") + parsed = Decimal(value) + elif isinstance(value, int): + parsed = Decimal(value) + elif isinstance(value, Decimal): + parsed = value + else: + raise ValueError("position decimal must be a string, integer, or Decimal") + + if not parsed.is_finite(): + raise ValueError("position decimal must be finite") + decimal_tuple = parsed.as_tuple() + exponent = decimal_tuple.exponent + if not isinstance(exponent, int): + raise ValueError("position decimal must be finite") + if isinstance(value, Decimal) and exponent > 0: + raise ValueError("position decimal must use finite fixed-point notation") + fractional_places = max(-exponent, 0) + integer_places = max(parsed.adjusted() + 1, 1) if parsed else 1 + if len(decimal_tuple.digits) > 30 or fractional_places > 18 or integer_places > 30: + raise ValueError("position decimal exceeds 30 digits or 18 decimal places") + return parsed + + +PositionDecimal = Annotated[Decimal, BeforeValidator(_position_decimal)] class EventPayloadBase(BaseModel): @@ -60,8 +113,88 @@ class ReturnPayload(EventPayloadBase): ] +class Position(EventPayloadBase): + """一个已经完成方向归一化的规范仓位。""" + + instrument_id: InstrumentText + instrument_id_type: Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=1, max_length=32), + ] = "custom" + venue: Annotated[ + str, + StringConstraints(strip_whitespace=True, min_length=1, max_length=32), + ] | None = None + side: Literal["long", "short"] | None = None + quantity: PositionDecimal | None = None + weight: PositionDecimal | None = None + market_value_base: PositionDecimal | None = None + exposure_value_base: PositionDecimal | None = None + asset_class: ClassificationText | None = None + sector: ClassificationText | None = None + country: ClassificationText | None = None + + @model_validator(mode="after") + def require_consistent_measures(self) -> Position: + measures = tuple( + value + for value in ( + self.quantity, + self.weight, + self.market_value_base, + self.exposure_value_base, + ) + if value is not None + ) + if not measures: + raise ValueError("position requires at least one measure") + nonzero_signs = {value.compare(Decimal(0)) for value in measures if value != 0} + if len(nonzero_signs) > 1: + raise ValueError("position nonzero measures must have the same direction") + if self.side == "long" and any(value < 0 for value in measures): + raise ValueError("long position measures must be non-negative") + if self.side == "short" and any(value > 0 for value in measures): + raise ValueError("short position measures must be non-positive") + return self + + +class PositionSnapshotPayload(EventPayloadBase): + """同一 portfolio 和时点的一组原子仓位。""" + + event_type: Literal["position_snapshot"] + portfolio_id: StrategyId + base_currency: Annotated[ + str, + StringConstraints(strip_whitespace=True, pattern=r"^[A-Z0-9]{3,12}$"), + ] | None = None + mapping_profile_hash: Annotated[ + str, + StringConstraints(pattern=r"^sha256:[0-9a-f]{64}$"), + ] + positions: Annotated[tuple[Position, ...], Field(max_length=100_000)] + + @model_validator(mode="after") + def require_snapshot_consistency(self) -> PositionSnapshotPayload: + identities: set[tuple[str, str, str]] = set() + for position in self.positions: + identity = ( + position.instrument_id_type, + position.instrument_id, + position.venue or "", + ) + if identity in identities: + raise ValueError("position snapshot contains duplicate instrument identity") + identities.add(identity) + if self.base_currency is None and any( + position.market_value_base is not None or position.exposure_value_base is not None + for position in self.positions + ): + raise ValueError("base_currency is required for base value measures") + return self + + EventPayload = Annotated[ - HeartbeatPayload | RunStatusPayload | NavPayload | ReturnPayload, + HeartbeatPayload | RunStatusPayload | NavPayload | ReturnPayload | PositionSnapshotPayload, Field(discriminator="event_type"), ] @@ -102,22 +235,34 @@ def require_utc_aware_timestamp(cls, value: datetime) -> datetime: def payload_type_matches_event_type(self) -> EventRecord: if self.payload.event_type != self.event_type: raise ValueError("payload event_type must match event_type") + if self.event_type == "position_snapshot" and self.schema_version != "1.1": + raise ValueError("position_snapshot requires schema 1.1") return self @property def idempotency_key(self) -> str: """返回事件自然身份的稳定 SHA-256 摘要,不包含可修订的载荷。""" - identity = "\x1f".join( - ( + if isinstance(self.payload, PositionSnapshotPayload): + identity_parts = [ + self.strategy_id, + self.environment, + self.source, + self.payload.portfolio_id, + self.event_type, + self.event_time.isoformat().replace("+00:00", "Z"), + self.schema_version, + ] + else: + identity_parts = [ self.strategy_id, self.environment, self.event_type, self.event_time.isoformat().replace("+00:00", "Z"), self.source, self.schema_version, - ) - ) + ] + identity = "\x1f".join(identity_parts) return sha256(identity.encode("utf-8")).hexdigest() def is_late_revision_of(self, previous: EventRecord) -> bool: diff --git a/src/quantcockpit/report.py b/src/quantcockpit/report.py index e77e9cc..ccdec79 100644 --- a/src/quantcockpit/report.py +++ b/src/quantcockpit/report.py @@ -5,7 +5,13 @@ from datetime import datetime, timezone import re -from quantcockpit.service import CockpitService, StrategyHealth, utc_text +from quantcockpit.service import ( + CockpitService, + PortfolioExposure, + StrategyHealth, + decimal_text, + utc_text, +) class NoStrategyDataError(ValueError): @@ -25,7 +31,7 @@ def render_markdown(service: CockpitService, *, generated_at: datetime) -> str: lines = [ "# QuantCockpit 本地健康报告", "", - "- 数据来源:DuckDB 中的 current 版本化 JSONL 事件", + "- 数据来源:DuckDB 中的 current 版本化事件", f"- 生成时间:{utc_text(generated_at)}", "", "## 策略健康", @@ -55,6 +61,14 @@ def render_markdown(service: CockpitService, *, generated_at: datetime) -> str: ) lines.append("") + lines.extend(["## 仓位覆盖与集中度", ""]) + portfolios = service.portfolios(evaluated_at=generated_at) + if not portfolios: + lines.extend(["- 无 current 仓位快照。", ""]) + else: + for item in portfolios: + _append_portfolio(lines, item.exposure) + errors = service.ingestion_errors() lines.extend(["## 导入错误摘要", ""]) if errors.quarantines: @@ -110,6 +124,48 @@ def _append_strategy(lines: list[str], item: StrategyHealth) -> None: lines.append("") +def _append_portfolio(lines: list[str], item: PortfolioExposure) -> None: + analysis = item.analysis + lines.extend( + [ + f"### {_markdown_inline(item.identity.portfolio_id)} / " + f"{_markdown_inline(item.identity.strategy_id)} / " + f"{_markdown_inline(item.identity.environment)}", + "", + f"- source = {_markdown_inline(item.identity.source)}", + f"- snapshot\\_time = {utc_text(item.snapshot_time)};age\\_seconds = {item.age_seconds}", + f"- state = {_markdown_inline(analysis.state)};level = {analysis.level};" + f"basis = {_markdown_inline(analysis.basis)};candidate\\_basis = " + f"{_markdown_inline(analysis.candidate_basis)}", + f"- position\\_count = {analysis.position_count};zero\\_position\\_count = " + f"{analysis.zero_position_count};long = {analysis.long_count};short = {analysis.short_count}", + f"- coverage = {_markdown_inline(decimal_text(analysis.coverage_ratio))};" + f"reason = {_markdown_inline(analysis.reason)}", + f"- gross = {_markdown_inline(decimal_text(analysis.gross))};" + f"net = {_markdown_inline(decimal_text(analysis.net))};" + f"Top-1 = {_markdown_inline(decimal_text(analysis.top_1_share))};" + f"Top-5 = {_markdown_inline(decimal_text(analysis.top_5_share))};" + f"HHI = {_markdown_inline(decimal_text(analysis.hhi))}", + ] + ) + for dimension, categories in analysis.categories.items(): + if not categories: + continue + rendered = ";".join( + f"{_markdown_inline(category.label)}: gross={_markdown_inline(decimal_text(category.gross))}, " + f"net={_markdown_inline(decimal_text(category.net))}, " + f"share={_markdown_inline(decimal_text(category.share))}" + for category in categories + ) + lines.append(f"- {_markdown_inline(dimension)}:{rendered}") + lines.extend( + [ + f"- evidence = {_markdown_inline(', '.join(item.evidence_refs))}", + "", + ] + ) + + def _markdown_inline(value: object) -> str: """把数据库中的非可信文本限制为单行普通 inline 文本。""" diff --git a/src/quantcockpit/service.py b/src/quantcockpit/service.py index e66d430..d7abd2c 100644 --- a/src/quantcockpit/service.py +++ b/src/quantcockpit/service.py @@ -4,6 +4,7 @@ from dataclasses import dataclass from datetime import datetime, timezone +from decimal import Decimal from itertools import combinations from typing import Callable, Mapping @@ -14,7 +15,8 @@ calculate_daily_correlation, ) from quantcockpit.analysis.health import HealthAssessment, HealthEvidence, assess_strategy_health -from quantcockpit.models import Environment +from quantcockpit.analysis.exposure import ExposureAnalysis, analyze_exposure +from quantcockpit.models import Environment, EventRecord, PositionSnapshotPayload from quantcockpit.store import DuckDBStore @@ -54,6 +56,31 @@ class IngestionErrors: failed_runs: tuple[dict[str, object], ...] +@dataclass(frozen=True) +class PortfolioIdentity: + portfolio_id: str + strategy_id: str + environment: Environment + source: str + + +@dataclass(frozen=True) +class PortfolioExposure: + identity: PortfolioIdentity + snapshot_time: datetime + age_seconds: int + base_currency: str | None + mapping_profile_hash: str + analysis: ExposureAnalysis + evidence_refs: tuple[str, str] + + +@dataclass(frozen=True) +class PortfolioSummary: + exposure: PortfolioExposure + missing_fields: tuple[str, ...] + + class CockpitService: """将持久化事件转换为可审计的策略观测,不产生模拟数据。""" @@ -145,6 +172,74 @@ def ingestion_errors(self) -> IngestionErrors: failed_runs=tuple(_limited_message(row) for row in raw["failed_runs"]), ) + def portfolios( + self, + *, + evaluated_at: datetime | None = None, + ) -> tuple[PortfolioSummary, ...]: + return tuple( + PortfolioSummary(exposure=item, missing_fields=_missing_fields(item.analysis)) + for item in self._portfolio_exposures(evaluated_at=evaluated_at) + ) + + def portfolio_exposure( + self, + portfolio_id: str, + strategy_id: str, + environment: Environment, + source: str, + *, + evaluated_at: datetime | None = None, + ) -> PortfolioExposure | None: + identity = (portfolio_id, strategy_id, environment, source) + return next( + ( + item + for item in self._portfolio_exposures(evaluated_at=evaluated_at) + if ( + item.identity.portfolio_id, + item.identity.strategy_id, + item.identity.environment, + item.identity.source, + ) + == identity + ), + None, + ) + + def _portfolio_exposures( + self, + *, + evaluated_at: datetime | None, + ) -> tuple[PortfolioExposure, ...]: + point_in_time = evaluated_at or self.evaluated_at() + results: list[PortfolioExposure] = [] + for row in self._store.current_position_snapshots(point_in_time): + event = EventRecord.model_validate_json(row["normalized_json"]) + payload = event.payload + if not isinstance(payload, PositionSnapshotPayload): + raise ValueError("position snapshot query returned a non-position payload") + results.append( + PortfolioExposure( + identity=PortfolioIdentity( + portfolio_id=payload.portfolio_id, + strategy_id=event.strategy_id, + environment=event.environment, + source=event.source, + ), + snapshot_time=event.event_time, + age_seconds=max(int((point_in_time - event.event_time).total_seconds()), 0), + base_currency=payload.base_currency, + mapping_profile_hash=payload.mapping_profile_hash, + analysis=analyze_exposure(payload), + evidence_refs=( + f"event:{row['event_id']}", + f"mapping:{payload.mapping_profile_hash}", + ), + ) + ) + return tuple(results) + def _limited_message(row: Mapping[str, object]) -> dict[str, object]: result = dict(row) @@ -162,6 +257,95 @@ def _supported_environment(value: str) -> Environment: raise ValueError(f"unsupported strategy environment in database: {value!r}") +def _missing_fields(analysis: ExposureAnalysis) -> tuple[str, ...]: + if analysis.state == "empty_portfolio" or analysis.level == 3: + return () + if analysis.level == 2: + return ("exposure_value_base",) + if analysis.level == 1: + return ("exposure_value_base", "market_value_base") + if analysis.candidate_basis is not None: + return (analysis.candidate_basis,) + return ("exposure_value_base", "market_value_base", "weight") + + +def decimal_text(value: Decimal | None) -> str | None: + if value is None: + return None + text = format(value, "f") + if "." in text: + text = text.rstrip("0").rstrip(".") + return "0" if text in {"", "-0"} else text + + +def portfolio_summary_payload(item: PortfolioSummary) -> dict[str, object]: + exposure = item.exposure + analysis = exposure.analysis + return { + "portfolio_id": exposure.identity.portfolio_id, + "strategy_id": exposure.identity.strategy_id, + "environment": exposure.identity.environment, + "source": exposure.identity.source, + "snapshot_time": utc_text(exposure.snapshot_time), + "age_seconds": exposure.age_seconds, + "position_count": analysis.position_count, + "zero_position_count": analysis.zero_position_count, + "capability_level": analysis.level, + "analysis_state": analysis.state, + "basis": analysis.basis, + "candidate_basis": analysis.candidate_basis, + "missing_fields": list(item.missing_fields), + } + + +def portfolio_exposure_payload(item: PortfolioExposure) -> dict[str, object]: + analysis = item.analysis + return { + "portfolio_id": item.identity.portfolio_id, + "strategy_id": item.identity.strategy_id, + "environment": item.identity.environment, + "source": item.identity.source, + "snapshot_time": utc_text(item.snapshot_time), + "age_seconds": item.age_seconds, + "base_currency": item.base_currency, + "mapping_profile_hash": item.mapping_profile_hash, + "analysis_state": analysis.state, + "capability_level": analysis.level, + "basis": analysis.basis, + "candidate_basis": analysis.candidate_basis, + "position_count": analysis.position_count, + "zero_position_count": analysis.zero_position_count, + "long_count": analysis.long_count, + "short_count": analysis.short_count, + "gross": decimal_text(analysis.gross), + "net": decimal_text(analysis.net), + "top_1_share": decimal_text(analysis.top_1_share), + "top_5_share": decimal_text(analysis.top_5_share), + "hhi": decimal_text(analysis.hhi), + "coverage_ratio": decimal_text(analysis.coverage_ratio), + "coverage_by_basis": { + basis: decimal_text(value) for basis, value in analysis.coverage_by_basis.items() + }, + "categories": { + name: [ + { + "label": category.label, + "gross": decimal_text(category.gross), + "net": decimal_text(category.net), + "share": decimal_text(category.share), + } + for category in categories + ] + for name, categories in analysis.categories.items() + }, + "category_coverage": { + name: decimal_text(value) for name, value in analysis.category_coverage.items() + }, + "reason": analysis.reason, + "evidence_refs": list(item.evidence_refs), + } + + def evidence_payload(item: HealthEvidence | CorrelationEvidence) -> dict[str, object]: return { "rule_id": item.rule_id, diff --git a/src/quantcockpit/store.py b/src/quantcockpit/store.py index 4a2551b..7abc8bb 100644 --- a/src/quantcockpit/store.py +++ b/src/quantcockpit/store.py @@ -13,6 +13,7 @@ from quantcockpit.models import EventRecord from quantcockpit.store_types import ( CurrentReturnPointRow, + CurrentPositionSnapshotRow, HealthInputs, SafeIngestionErrors, SafeIngestionIssue, @@ -93,6 +94,7 @@ def _create_schema(self) -> None: raw_json VARCHAR NOT NULL, source_file VARCHAR NOT NULL, line_number INTEGER NOT NULL, + end_line_number INTEGER NOT NULL, event_time TIMESTAMPTZ NOT NULL, recorded_at TIMESTAMPTZ NOT NULL, first_observed_at TIMESTAMPTZ NOT NULL, @@ -145,6 +147,10 @@ def _create_schema(self) -> None: ) self.connection.execute("UPDATE quarantine SET is_active = TRUE WHERE is_active IS NULL") self.connection.execute("ALTER TABLE quarantine ADD COLUMN IF NOT EXISTS resolved_at TIMESTAMPTZ") + self.connection.execute("ALTER TABLE events ADD COLUMN IF NOT EXISTS end_line_number INTEGER") + self.connection.execute( + "UPDATE events SET end_line_number = line_number WHERE end_line_number IS NULL" + ) def begin_run(self, source_file: Path, observed_at: datetime) -> str: run_id = str(uuid4()) @@ -198,12 +204,19 @@ def record_event( raw_json: str, source_file: Path, line_number: int, + end_line_number: int | None = None, observed_at: datetime, + ignore_recorded_at_for_content: bool = False, ) -> str: """保存事件并返回 imported、duplicate、revision 或 stale。""" - normalized_json = json.dumps(event.model_dump(mode="json"), sort_keys=True, separators=(",", ":")) - content_hash = sha256(normalized_json.encode("utf-8")).hexdigest() + normalized_data = event.model_dump(mode="json") + normalized_json = json.dumps(normalized_data, sort_keys=True, separators=(",", ":")) + content_data = dict(normalized_data) + if ignore_recorded_at_for_content: + content_data.pop("recorded_at", None) + content_json = json.dumps(content_data, sort_keys=True, separators=(",", ":")) + content_hash = sha256(content_json.encode("utf-8")).hexdigest() matching = self.connection.execute( """ SELECT event_id FROM events @@ -243,13 +256,14 @@ def record_event( """ INSERT INTO events( event_id, idempotency_key, revision, normalized_json, raw_json, - source_file, line_number, event_time, recorded_at, first_observed_at, + source_file, line_number, end_line_number, event_time, recorded_at, first_observed_at, last_observed_at, content_hash, is_current - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, TRUE) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, TRUE) """, [ str(uuid4()), event.idempotency_key, revision, normalized_json, raw_json, - str(source_file.resolve()), line_number, event.event_time, event.recorded_at, observed_at, + str(source_file.resolve()), line_number, end_line_number or line_number, + event.event_time, event.recorded_at, observed_at, observed_at, content_hash, ], ) @@ -527,6 +541,53 @@ def current_return_points(self, evaluated_at: datetime) -> list[CurrentReturnPoi for event_id, strategy_id, environment, source, event_time, simple_return in rows ] + def current_position_snapshots( + self, + evaluated_at: datetime, + ) -> list[CurrentPositionSnapshotRow]: + """返回每个严格 portfolio 四元身份在计算时点前的最新 current 快照。""" + + rows = self.connection.execute( + """ + SELECT event_id, event_time, recorded_at, normalized_json + FROM ( + SELECT + event_id, + event_time, + recorded_at, + normalized_json, + ROW_NUMBER() OVER ( + PARTITION BY + json_extract_string(normalized_json, '$.payload.portfolio_id'), + json_extract_string(normalized_json, '$.strategy_id'), + json_extract_string(normalized_json, '$.environment'), + json_extract_string(normalized_json, '$.source') + ORDER BY event_time DESC, recorded_at DESC, event_id DESC + ) AS row_number + FROM events + WHERE is_current = TRUE + AND json_extract_string(normalized_json, '$.event_type') = 'position_snapshot' + AND event_time <= ? + ) latest + WHERE row_number = 1 + ORDER BY + json_extract_string(normalized_json, '$.payload.portfolio_id'), + json_extract_string(normalized_json, '$.strategy_id'), + json_extract_string(normalized_json, '$.environment'), + json_extract_string(normalized_json, '$.source') + """, + [evaluated_at], + ).fetchall() + return [ + { + "event_id": event_id, + "event_time": event_time, + "recorded_at": recorded_at, + "normalized_json": normalized_json, + } + for event_id, event_time, recorded_at, normalized_json in rows + ] + def safe_ingestion_errors(self) -> SafeIngestionErrors: """返回可公开的导入失败摘要,绝不读取原始行内容。""" diff --git a/src/quantcockpit/store_types.py b/src/quantcockpit/store_types.py index 1ef19d1..3990cd3 100644 --- a/src/quantcockpit/store_types.py +++ b/src/quantcockpit/store_types.py @@ -40,6 +40,13 @@ class CurrentReturnPointRow(StrategyIdentityRow): simple_return: str +class CurrentPositionSnapshotRow(TypedDict): + event_id: str + event_time: datetime + recorded_at: datetime + normalized_json: str + + class SafeIngestionIssue(TypedDict): error_code: str error_message: str diff --git a/tests/test_api.py b/tests/test_api.py index 6caa0e9..524eaba 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -47,6 +47,52 @@ def event_line( ) +def position_event_line( + *, + portfolio_id: str = "book-a", + strategy_id: str = "alpha", + environment: str = "paper", + source: str = "broker-export", + event_time: str = "2026-07-20T12:00:00Z", + recorded_at: str = "2026-07-20T12:01:00Z", + positions: list[dict[str, object]] | None = None, +) -> str: + return json.dumps( + { + "schema_version": "1.1", + "strategy_id": strategy_id, + "environment": environment, + "event_type": "position_snapshot", + "event_time": event_time, + "recorded_at": recorded_at, + "source": source, + "payload": { + "event_type": "position_snapshot", + "portfolio_id": portfolio_id, + "base_currency": "USD", + "mapping_profile_hash": f"sha256:{'a' * 64}", + "positions": positions + if positions is not None + else [ + { + "instrument_id": "AAPL", + "instrument_id_type": "ticker", + "market_value_base": "60", + "asset_class": "equity", + }, + { + "instrument_id": "MSFT", + "instrument_id_type": "ticker", + "market_value_base": "-40", + "asset_class": "equity", + }, + ], + }, + }, + sort_keys=True, + ) + + def seed_database(database_path: Path, tmp_path: Path, *lines: str) -> None: source_path = tmp_path / "events.jsonl" source_path.write_text("\n".join(lines) + "\n", encoding="utf-8") @@ -93,6 +139,8 @@ def test_openapi_uses_named_response_models(tmp_path: Path) -> None: "/api/v1/strategies": "StrategiesResponse", "/api/v1/correlations": "CorrelationsResponse", "/api/v1/ingestion/errors": "IngestionErrorsResponse", + "/api/v1/portfolios": "PortfoliosResponse", + "/api/v1/portfolios/{portfolio_id}/exposure": "PortfolioExposureResponse", } for path, model_name in expected.items(): response_schema = schema["paths"][path]["get"]["responses"]["200"]["content"]["application/json"]["schema"] @@ -356,3 +404,139 @@ def test_ingestion_errors_expose_traceable_metadata_without_raw_sensitive_lines( assert body["failed_runs"] == [] assert "do-not-disclose" not in response.text assert "raw_line" not in response.text + + +def test_empty_database_explicitly_reports_empty_portfolio_state(tmp_path: Path) -> None: + database_path = tmp_path / "empty-portfolios.duckdb" + DuckDBStore(database_path).close() + + response = client(database_path).get("/api/v1/portfolios") + + assert response.status_code == 200 + assert response.json() == {"state": "empty", "portfolios": []} + + +def test_portfolio_api_returns_latest_strict_identity_with_mapping_evidence(tmp_path: Path) -> None: + database_path = tmp_path / "portfolios.duckdb" + seed_database(database_path, tmp_path, position_event_line()) + app_client = client(database_path) + + portfolios = app_client.get("/api/v1/portfolios") + detail = app_client.get( + "/api/v1/portfolios/book-a/exposure" + "?strategy_id=alpha&environment=paper&source=broker-export" + ) + + assert portfolios.status_code == 200 + summary = portfolios.json()["portfolios"][0] + assert summary == { + "portfolio_id": "book-a", + "strategy_id": "alpha", + "environment": "paper", + "source": "broker-export", + "snapshot_time": "2026-07-20T12:00:00Z", + "age_seconds": 21600, + "position_count": 2, + "zero_position_count": 0, + "capability_level": 2, + "analysis_state": "ready", + "basis": "market_value_base", + "candidate_basis": None, + "missing_fields": ["exposure_value_base"], + } + assert detail.status_code == 200 + body = detail.json() + assert body["portfolio_id"] == "book-a" + assert body["base_currency"] == "USD" + assert body["basis"] == "market_value_base" + assert body["gross"] == "100" + assert body["net"] == "20" + assert body["top_1_share"] == "0.6" + assert body["hhi"] == "0.52" + assert body["coverage_by_basis"] == { + "exposure_value_base": "0", + "market_value_base": "1", + "weight": "0", + } + assert body["evidence_refs"][0].startswith("event:") + assert body["evidence_refs"][1] == f"mapping:sha256:{'a' * 64}" + + +def test_portfolio_detail_never_crosses_source_environment_or_portfolio(tmp_path: Path) -> None: + database_path = tmp_path / "isolation.duckdb" + seed_database( + database_path, + tmp_path, + position_event_line(portfolio_id="book-a", environment="paper", source="source-a"), + position_event_line(portfolio_id="book-a", environment="live", source="source-a"), + position_event_line(portfolio_id="book-b", environment="paper", source="source-b"), + ) + app_client = client(database_path) + + assert app_client.get( + "/api/v1/portfolios/book-a/exposure?strategy_id=alpha&environment=paper&source=source-a" + ).status_code == 200 + assert app_client.get( + "/api/v1/portfolios/book-a/exposure?strategy_id=alpha&environment=paper&source=source-b" + ).status_code == 404 + assert app_client.get( + "/api/v1/portfolios/book-b/exposure?strategy_id=alpha&environment=paper&source=source-a" + ).status_code == 404 + + +def test_portfolio_query_ignores_future_snapshot_and_reads_current_revision(tmp_path: Path) -> None: + database_path = tmp_path / "as-of-position.duckdb" + seed_database( + database_path, + tmp_path, + position_event_line( + positions=[{"instrument_id": "AAPL", "market_value_base": "50"}], + ), + position_event_line( + recorded_at="2026-07-20T12:02:00Z", + positions=[{"instrument_id": "AAPL", "market_value_base": "75"}], + ), + position_event_line( + event_time="2026-07-21T12:00:00Z", + recorded_at="2026-07-21T12:01:00Z", + positions=[{"instrument_id": "AAPL", "market_value_base": "999"}], + ), + ) + + body = client(database_path).get( + "/api/v1/portfolios/book-a/exposure" + "?strategy_id=alpha&environment=paper&source=broker-export" + ).json() + + assert body["gross"] == "75" + assert body["snapshot_time"] == "2026-07-20T12:00:00Z" + + +def test_portfolio_api_preserves_explicit_empty_portfolio_zero_semantics(tmp_path: Path) -> None: + database_path = tmp_path / "flat.duckdb" + seed_database(database_path, tmp_path, position_event_line(positions=[])) + + body = client(database_path).get( + "/api/v1/portfolios/book-a/exposure" + "?strategy_id=alpha&environment=paper&source=broker-export" + ).json() + + assert body["analysis_state"] == "empty_portfolio" + assert body["gross"] == "0" + assert body["net"] == "0" + assert body["basis"] is None + assert body["coverage_ratio"] is None + + +def test_portfolio_detail_validates_query_and_unknown_identity(tmp_path: Path) -> None: + database_path = tmp_path / "validation.duckdb" + seed_database(database_path, tmp_path, position_event_line()) + app_client = client(database_path) + + assert app_client.get("/api/v1/portfolios/book-a/exposure").status_code == 422 + assert app_client.get( + "/api/v1/portfolios/book-a/exposure?strategy_id=alpha&environment=backtest&source=broker-export" + ).status_code == 422 + assert app_client.get( + "/api/v1/portfolios/missing/exposure?strategy_id=alpha&environment=paper&source=broker-export" + ).status_code == 404 diff --git a/tests/test_exposure.py b/tests/test_exposure.py new file mode 100644 index 0000000..bc3321d --- /dev/null +++ b/tests/test_exposure.py @@ -0,0 +1,148 @@ +from decimal import Decimal + +from quantcockpit.analysis.exposure import analyze_exposure +from quantcockpit.models import Position, PositionSnapshotPayload + + +def position(instrument_id: str, **values: object) -> Position: + return Position.model_validate( + {"instrument_id": instrument_id, "instrument_id_type": "ticker", **values} + ) + + +def snapshot(*positions: Position) -> PositionSnapshotPayload: + needs_currency = any( + item.market_value_base is not None or item.exposure_value_base is not None + for item in positions + ) + return PositionSnapshotPayload( + event_type="position_snapshot", + portfolio_id="book-a", + base_currency="USD" if needs_currency else None, + mapping_profile_hash=f"sha256:{'a' * 64}", + positions=positions, + ) + + +def test_exposure_prefers_complete_risk_basis_and_uses_decimal() -> None: + result = analyze_exposure( + snapshot( + position( + "A", + weight="0.6", + market_value_base="60", + exposure_value_base="120", + asset_class="equity", + ), + position( + "B", + weight="-0.4", + market_value_base="-40", + exposure_value_base="-80", + asset_class="equity", + ), + ) + ) + + assert result.state == "ready" + assert result.level == 3 + assert result.basis == "exposure_value_base" + assert result.gross == Decimal("200") + assert result.net == Decimal("40") + assert result.top_1_share == Decimal("0.6") + assert result.top_5_share == Decimal("1") + assert result.hhi == Decimal("0.52") + assert result.coverage_ratio == Decimal("1") + + +def test_incomplete_basis_returns_reason_instead_of_false_zero() -> None: + result = analyze_exposure( + snapshot(position("A", weight="0.5"), position("B", quantity="2")) + ) + + assert result.state == "unavailable" + assert result.gross is None + assert result.net is None + assert result.reason == "incomplete_basis" + assert result.candidate_basis == "weight" + assert result.coverage_ratio == Decimal("0.5") + assert result.coverage_by_basis["weight"] == Decimal("0.5") + + +def test_quantity_only_is_level_zero_without_a_candidate_value_basis() -> None: + result = analyze_exposure(snapshot(position("A", quantity="2"), position("B", quantity="-1"))) + + assert result.level == 0 + assert result.basis is None + assert result.candidate_basis is None + assert result.coverage_ratio == Decimal("0") + assert (result.long_count, result.short_count) == (1, 1) + + +def test_explicit_empty_portfolio_returns_real_zero_without_claiming_a_basis() -> None: + result = analyze_exposure(snapshot()) + + assert result.state == "empty_portfolio" + assert result.position_count == 0 + assert result.zero_position_count == 0 + assert result.gross == Decimal("0") + assert result.net == Decimal("0") + assert result.basis is None + assert result.top_1_share is None + assert result.hhi is None + assert all(value is None for value in result.coverage_by_basis.values()) + + +def test_zero_selected_exposure_has_zero_gross_reason_instead_of_dividing_by_zero() -> None: + result = analyze_exposure( + snapshot( + position("A", quantity="2", exposure_value_base="0"), + position("B", quantity="-1", exposure_value_base="0"), + ) + ) + + assert result.state == "ready" + assert result.basis == "exposure_value_base" + assert result.gross == Decimal("0") + assert result.net == Decimal("0") + assert result.top_1_share is None + assert result.top_5_share is None + assert result.hhi is None + assert result.reason == "zero_gross_exposure" + + +def test_top_five_and_categories_use_same_basis_with_unclassified_bucket() -> None: + positions = tuple( + position( + f"S{index}", + weight=str(value), + sector="Technology" if index < 2 else None, + country="US", + ) + for index, value in enumerate((6, 5, 4, 3, 2, -1)) + ) + result = analyze_exposure(snapshot(*positions)) + + assert result.gross == Decimal("21") + assert result.net == Decimal("19") + assert result.top_5_share == Decimal("0.952380952380952381") + sectors = {item.label: item for item in result.categories["sector"]} + assert sectors["Technology"].gross == Decimal("11") + assert sectors["unclassified"].gross == Decimal("10") + assert sectors["unclassified"].net == Decimal("8") + assert result.category_coverage["sector"] == Decimal("0.333333333333333333") + assert result.category_coverage["country"] == Decimal("1") + + +def test_zero_positions_are_counted_but_excluded_from_coverage_denominator() -> None: + result = analyze_exposure( + snapshot( + position("A", weight="1"), + position("B", quantity="0"), + ) + ) + + assert result.position_count == 2 + assert result.zero_position_count == 1 + assert result.coverage_by_basis["weight"] == Decimal("1") + assert result.gross == Decimal("1") diff --git a/tests/test_models.py b/tests/test_models.py index 9337927..1006782 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -1,5 +1,6 @@ from datetime import datetime, timezone from decimal import Decimal +from hashlib import sha256 import pytest from pydantic import ValidationError @@ -26,6 +27,28 @@ def event_data(**overrides: object) -> dict[str, object]: return data +def position_event(**payload_overrides: object) -> dict[str, object]: + payload: dict[str, object] = { + "event_type": "position_snapshot", + "portfolio_id": "paper-book-a", + "base_currency": "USD", + "mapping_profile_hash": f"sha256:{'a' * 64}", + "positions": [ + { + "instrument_id": "AAPL", + "instrument_id_type": "ticker", + "weight": "0.25", + } + ], + } + payload.update(payload_overrides) + return event_data( + schema_version="1.1", + event_type="position_snapshot", + payload=payload, + ) + + def test_return_event_preserves_strategy_identity_and_decimal_return() -> None: event = EventRecord.model_validate(event_data()) @@ -114,3 +137,129 @@ def test_event_rejects_unsupported_event_type() -> None: EventRecord.model_validate( event_data(event_type="order", payload={"event_type": "order"}) ) + + +def test_position_snapshot_requires_schema_1_1_and_keeps_1_0_compatible() -> None: + assert EventRecord.model_validate(event_data()).schema_version == "1.0" + assert EventRecord.model_validate(position_event()).payload.event_type == "position_snapshot" + + with pytest.raises(ValidationError, match="schema 1.1"): + EventRecord.model_validate(position_event() | {"schema_version": "1.0"}) + + +def test_position_snapshot_accepts_authoritative_empty_portfolio() -> None: + event = EventRecord.model_validate(position_event(positions=[])) + + assert event.payload.positions == () + + +def test_portfolio_id_participates_in_position_idempotency_key() -> None: + left = EventRecord.model_validate(position_event(portfolio_id="book-a")) + right = EventRecord.model_validate(position_event(portfolio_id="book-b")) + + assert left.idempotency_key != right.idempotency_key + + +def test_position_idempotency_key_uses_the_documented_identity_order() -> None: + event = EventRecord.model_validate(position_event(portfolio_id="book-a")) + material = "\x1f".join( + ( + "trend-following-v1", + "paper", + "daily-jsonl", + "book-a", + "position_snapshot", + "2026-07-20T09:30:00Z", + "1.1", + ) + ) + + assert event.idempotency_key == sha256(material.encode("utf-8")).hexdigest() + + +def test_position_requires_a_measure_and_rejects_conflicting_directions() -> None: + without_measure = { + "instrument_id": "AAPL", + "instrument_id_type": "ticker", + } + conflicting = without_measure | {"quantity": "1", "weight": "-0.25"} + + with pytest.raises(ValidationError, match="at least one measure"): + EventRecord.model_validate(position_event(positions=[without_measure])) + with pytest.raises(ValidationError, match="same direction"): + EventRecord.model_validate(position_event(positions=[conflicting])) + + +def test_position_side_must_match_normalized_measure_signs() -> None: + with pytest.raises(ValidationError, match="short position measures must be non-positive"): + EventRecord.model_validate( + position_event( + positions=[ + { + "instrument_id": "AAPL", + "instrument_id_type": "ticker", + "side": "short", + "quantity": "1", + } + ] + ) + ) + + +def test_position_snapshot_requires_base_currency_for_base_values() -> None: + with pytest.raises(ValidationError, match="base_currency"): + EventRecord.model_validate( + position_event( + base_currency=None, + positions=[ + { + "instrument_id": "AAPL", + "instrument_id_type": "ticker", + "market_value_base": "22000", + } + ], + ) + ) + + +def test_position_identity_includes_venue_and_rejects_exact_duplicates() -> None: + same_symbol_different_venues = [ + {"instrument_id": "ABC", "instrument_id_type": "ticker", "venue": "XNAS", "weight": "0.4"}, + {"instrument_id": "ABC", "instrument_id_type": "ticker", "venue": "XNYS", "weight": "0.6"}, + ] + assert len( + EventRecord.model_validate(position_event(positions=same_symbol_different_venues)).payload.positions + ) == 2 + + with pytest.raises(ValidationError, match="duplicate instrument"): + EventRecord.model_validate( + position_event(positions=[same_symbol_different_venues[0], same_symbol_different_venues[0]]) + ) + + +@pytest.mark.parametrize( + "invalid_measure", + [ + 0.25, + "NaN", + "Infinity", + "1e-3", + "1234567890123456789012345678901", + "0.1234567890123456789", + ], +) +def test_position_measure_rejects_binary_float_nonfinite_exponent_and_precision( + invalid_measure: object, +) -> None: + with pytest.raises(ValidationError, match="position decimal"): + EventRecord.model_validate( + position_event( + positions=[ + { + "instrument_id": "AAPL", + "instrument_id_type": "ticker", + "weight": invalid_measure, + } + ] + ) + ) diff --git a/tests/test_position_ingestion.py b/tests/test_position_ingestion.py new file mode 100644 index 0000000..0efc464 --- /dev/null +++ b/tests/test_position_ingestion.py @@ -0,0 +1,130 @@ +from datetime import datetime, timezone +from pathlib import Path + +import pytest + +from quantcockpit.ingestion.position_profile import PositionMappingProfile +from quantcockpit.ingestion.positions import PositionImportError, import_positions +from quantcockpit.store import DuckDBStore + + +FIRST_SEEN = datetime(2026, 7, 20, 10, 0, tzinfo=timezone.utc) +SECOND_SEEN = datetime(2026, 7, 20, 11, 0, tzinfo=timezone.utc) + + +def profile() -> PositionMappingProfile: + return PositionMappingProfile.model_validate( + { + "profile_version": "1.0", + "name": "position-import-test", + "format": "csv", + "layout": "tabular_snapshot", + "snapshot_scope": "grouped_rows", + "fields": { + "strategy_id": {"literal": "alpha"}, + "environment": {"literal": "paper"}, + "source": {"literal": "broker-export"}, + "portfolio_id": {"path": "account"}, + "snapshot_time": {"path": "as_of", "transforms": ["utc_timestamp"]}, + "base_currency": {"literal": "USD"}, + }, + "position_fields": { + "instrument_id": {"path": "symbol"}, + "instrument_id_type": {"literal": "ticker"}, + "weight": {"path": "weight", "transforms": ["decimal"]}, + }, + } + ) + + +def write_positions(tmp_path: Path, *, weight: str = "1") -> Path: + path = tmp_path / "positions.csv" + path.write_text( + "account,as_of,symbol,weight\n" + f"book-a,2026-07-20T09:30:00Z,AAPL,{weight}\n", + encoding="utf-8", + ) + return path + + +def test_position_import_replay_is_duplicate_even_when_import_observed_at_changes( + tmp_path: Path, +) -> None: + store = DuckDBStore(tmp_path / "cockpit.duckdb") + path = write_positions(tmp_path) + + first = import_positions(store, path, profile(), observed_at=FIRST_SEEN) + duplicate = import_positions(store, path, profile(), observed_at=SECOND_SEEN) + + assert (first.imported, duplicate.duplicates) == (1, 1) + assert len(store.event_rows()) == 1 + + +def test_changed_position_content_creates_revision_and_preserves_line_range(tmp_path: Path) -> None: + store = DuckDBStore(tmp_path / "cockpit.duckdb") + path = write_positions(tmp_path, weight="0.5") + import_positions(store, path, profile(), observed_at=FIRST_SEEN) + path = write_positions(tmp_path, weight="0.6") + + revised = import_positions(store, path, profile(), observed_at=SECOND_SEEN) + + assert revised.revisions == 1 + rows = store.connection.execute( + "SELECT revision, is_current, line_number, end_line_number FROM events ORDER BY revision" + ).fetchall() + assert rows == [(1, False, 2, 2), (2, True, 2, 2)] + + +def test_invalid_snapshot_fails_run_without_writing_partial_event(tmp_path: Path) -> None: + store = DuckDBStore(tmp_path / "cockpit.duckdb") + path = tmp_path / "positions.csv" + path.write_text( + "account,as_of,symbol,weight\n" + "book-a,2026-07-20T09:30:00Z,AAPL,0.5\n" + "book-a,2026-07-20T09:30:00Z,AAPL,0.5\n", + encoding="utf-8", + ) + + with pytest.raises(PositionImportError) as captured: + import_positions(store, path, profile(), observed_at=FIRST_SEEN) + + assert captured.value.code == "duplicate_instrument" + assert store.event_rows() == [] + assert store.ingestion_runs()[0]["status"] == "failed" + assert store.ingestion_runs()[0]["error_code"] == "duplicate_instrument" + + +def test_position_import_stores_grouped_raw_evidence_as_bounded_json_array(tmp_path: Path) -> None: + store = DuckDBStore(tmp_path / "cockpit.duckdb") + path = tmp_path / "positions.csv" + path.write_text( + "account,as_of,symbol,weight\n" + "book-a,2026-07-20T09:30:00Z,AAPL,0.5\n" + "book-a,2026-07-20T09:30:00Z,MSFT,0.5\n", + encoding="utf-8", + ) + + result = import_positions(store, path, profile(), observed_at=FIRST_SEEN) + + assert result.imported == 1 + row = store.connection.execute( + "SELECT raw_json, line_number, end_line_number FROM events" + ).fetchone() + assert row is not None + raw_json, start_line, end_line = row + assert raw_json.startswith("[") and raw_json.endswith("]") + assert (start_line, end_line) == (2, 3) + + +def test_position_import_rejects_naive_observation_time_before_starting_run(tmp_path: Path) -> None: + store = DuckDBStore(tmp_path / "cockpit.duckdb") + + with pytest.raises(ValueError, match="timezone-aware"): + import_positions( + store, + write_positions(tmp_path), + profile(), + observed_at=datetime(2026, 7, 20, 10, 0), + ) + + assert store.ingestion_runs() == [] diff --git a/tests/test_position_preview.py b/tests/test_position_preview.py new file mode 100644 index 0000000..f36eaec --- /dev/null +++ b/tests/test_position_preview.py @@ -0,0 +1,178 @@ +from datetime import datetime, timezone +from decimal import Decimal +from pathlib import Path + +import pytest + +from quantcockpit.ingestion.position_profile import PositionMappingProfile +from quantcockpit.ingestion.positions import NormalizedSnapshot, PositionImportError, preview_positions +from quantcockpit.models import PositionSnapshotPayload + + +UTC_NOW = datetime(2026, 7, 20, 10, 0, tzinfo=timezone.utc) + + +def csv_profile(*, scope: str = "grouped_rows") -> PositionMappingProfile: + return PositionMappingProfile.model_validate( + { + "profile_version": "1.0", + "name": "test-position-csv", + "format": "csv", + "layout": "tabular_snapshot", + "snapshot_scope": scope, + "fields": { + "strategy_id": {"literal": "alpha"}, + "environment": {"literal": "paper"}, + "source": {"literal": "broker-export"}, + "portfolio_id": {"path": "account", "transforms": ["trim"]}, + "snapshot_time": {"path": "as_of", "transforms": ["utc_timestamp"]}, + "base_currency": {"literal": "USD"}, + }, + "position_fields": { + "instrument_id": {"path": "symbol", "transforms": ["trim"]}, + "instrument_id_type": {"literal": "ticker"}, + "side": {"path": "side", "transforms": ["trim", "lowercase"]}, + "weight": {"path": "weight", "transforms": ["decimal"]}, + }, + } + ) + + +def write_csv(tmp_path: Path, body: str) -> Path: + path = tmp_path / "positions.csv" + path.write_text(body, encoding="utf-8") + return path + + +def snapshot_payload(snapshot: NormalizedSnapshot) -> PositionSnapshotPayload: + payload = snapshot.event.payload + assert isinstance(payload, PositionSnapshotPayload) + return payload + + +def test_preview_normalizes_short_side_without_database_access(tmp_path: Path) -> None: + result = preview_positions( + write_csv( + tmp_path, + "account,as_of,symbol,side,weight\n" + " private-book ,2026-07-20T09:30:00Z,AAPL,SHORT,0.25\n", + ), + csv_profile(), + observed_at=UTC_NOW, + ) + + snapshot = result.snapshots[0] + assert snapshot_payload(snapshot).positions[0].weight == Decimal("-0.25") + assert snapshot.event.event_time == datetime(2026, 7, 20, 9, 30, tzinfo=timezone.utc) + assert snapshot.event.recorded_at == UTC_NOW + assert snapshot.recorded_at_source == "import_observed_at" + assert result.sample_positions[0]["instrument_id"] == "AAPL" + assert "portfolio_id" not in result.sample_positions[0] + assert len(result.mapping_profile_hash) == 71 + + +def test_preview_groups_rows_by_strict_snapshot_identity(tmp_path: Path) -> None: + result = preview_positions( + write_csv( + tmp_path, + "account,as_of,symbol,side,weight\n" + "book-a,2026-07-20T09:30:00Z,AAPL,long,0.5\n" + "book-a,2026-07-20T09:30:00Z,MSFT,long,0.5\n" + "book-b,2026-07-20T09:30:00Z,AAPL,long,1\n", + ), + csv_profile(), + observed_at=UTC_NOW, + ) + + assert result.record_count == 3 + assert result.snapshot_count == 2 + assert [len(snapshot_payload(item).positions) for item in result.snapshots] == [2, 1] + + +def test_whole_file_scope_rejects_conflicting_identity(tmp_path: Path) -> None: + with pytest.raises(PositionImportError) as captured: + preview_positions( + write_csv( + tmp_path, + "account,as_of,symbol,side,weight\n" + "book-a,2026-07-20T09:30:00Z,AAPL,long,1\n" + "book-b,2026-07-20T09:30:00Z,MSFT,long,1\n", + ), + csv_profile(scope="whole_file"), + observed_at=UTC_NOW, + ) + assert captured.value.code == "snapshot_identity_conflict" + + +def test_empty_grouped_file_is_not_silently_treated_as_empty_portfolio(tmp_path: Path) -> None: + with pytest.raises(PositionImportError) as captured: + preview_positions( + write_csv(tmp_path, "account,as_of,symbol,side,weight\n"), + csv_profile(), + observed_at=UTC_NOW, + ) + assert captured.value.code == "empty_input" + + +def test_document_snapshot_accepts_explicit_empty_positions(tmp_path: Path) -> None: + path = tmp_path / "positions.json" + path.write_text( + '{"strategy":"alpha","environment":"paper","source":"broker-export",' + '"portfolio":"book-a","as_of":"2026-07-20T09:30:00Z","positions":[]}', + encoding="utf-8", + ) + profile = PositionMappingProfile.model_validate( + { + "profile_version": "1.0", + "name": "document-snapshot", + "format": "json", + "layout": "document_snapshot", + "snapshot_scope": "whole_file", + "positions_path": "/positions", + "fields": { + "strategy_id": {"path": "/strategy"}, + "environment": {"path": "/environment"}, + "source": {"path": "/source"}, + "portfolio_id": {"path": "/portfolio"}, + "snapshot_time": {"path": "/as_of", "transforms": ["utc_timestamp"]}, + }, + "position_fields": { + "instrument_id": {"path": "/symbol"}, + "weight": {"path": "/weight", "transforms": ["decimal"]}, + }, + } + ) + + result = preview_positions(path, profile, observed_at=UTC_NOW) + + assert result.snapshot_count == 1 + assert snapshot_payload(result.snapshots[0]).positions == () + + +def test_preview_returns_at_most_five_normalized_samples(tmp_path: Path) -> None: + rows = "".join( + f"book-a,2026-07-20T09:30:00Z,SYM{index},long,0.1\n" for index in range(6) + ) + result = preview_positions( + write_csv(tmp_path, "account,as_of,symbol,side,weight\n" + rows), + csv_profile(), + observed_at=UTC_NOW, + ) + + assert len(result.sample_positions) == 5 + + +def test_preview_rejects_duplicate_instrument_as_an_atomic_snapshot(tmp_path: Path) -> None: + with pytest.raises(PositionImportError) as captured: + preview_positions( + write_csv( + tmp_path, + "account,as_of,symbol,side,weight\n" + "book-a,2026-07-20T09:30:00Z,AAPL,long,0.5\n" + "book-a,2026-07-20T09:30:00Z,AAPL,long,0.5\n", + ), + csv_profile(), + observed_at=UTC_NOW, + ) + assert captured.value.code == "duplicate_instrument" + assert str(tmp_path) not in str(captured.value) diff --git a/tests/test_position_profile.py b/tests/test_position_profile.py new file mode 100644 index 0000000..8d4b046 --- /dev/null +++ b/tests/test_position_profile.py @@ -0,0 +1,165 @@ +from datetime import datetime, timezone +from decimal import Decimal + +import pytest +from pydantic import ValidationError + +from quantcockpit.ingestion.position_profile import ( + FieldBinding, + MappingValueError, + PositionMappingProfile, + Transform, + profile_hash, + resolve_binding, +) + + +PROFILE: dict[str, object] = { + "profile_version": "1.0", + "name": "example-position-csv", + "format": "csv", + "layout": "tabular_snapshot", + "snapshot_scope": "grouped_rows", + "fields": { + "strategy_id": {"literal": "trend-following"}, + "environment": {"literal": "paper"}, + "source": {"literal": "daily-export"}, + "portfolio_id": {"path": "account_alias", "transforms": ["trim"]}, + "snapshot_time": {"path": "as_of", "transforms": ["utc_timestamp"]}, + }, + "position_fields": { + "instrument_id": {"path": "symbol", "transforms": ["trim"]}, + "instrument_id_type": {"literal": "ticker"}, + "weight": {"path": "weight", "transforms": ["decimal"]}, + }, +} + + +def test_profile_hash_ignores_json_key_order() -> None: + left = PositionMappingProfile.model_validate(PROFILE) + right = PositionMappingProfile.model_validate(dict(reversed(list(PROFILE.items())))) + + assert profile_hash(left) == profile_hash(right) + assert profile_hash(left).startswith("sha256:") + assert len(profile_hash(left)) == 71 + + +def test_field_binding_requires_exactly_one_literal_or_path() -> None: + with pytest.raises(ValidationError, match="exactly one"): + FieldBinding() + with pytest.raises(ValidationError, match="exactly one"): + FieldBinding(literal="alpha", path="strategy") + + +def test_local_timestamp_requires_format_and_timezone_together() -> None: + with pytest.raises(ValidationError, match="together"): + FieldBinding( + path="as_of", + transforms=("utc_timestamp",), + timestamp_format="%Y-%m-%d %H:%M:%S", + ) + + +def test_json_pointer_uses_rfc_6901_escaping() -> None: + binding = FieldBinding(path="/account~1meta/book~0name", transforms=("trim",)) + + assert ( + resolve_binding( + binding, + {"account/meta": {"book~name": " alpha "}}, + format="json", + ) + == "alpha" + ) + + +def test_csv_path_is_an_exact_header_and_missing_path_is_safe() -> None: + binding = FieldBinding(path="Account Alias", transforms=("trim",)) + assert resolve_binding(binding, {"Account Alias": " book-a "}, format="csv") == "book-a" + + with pytest.raises(MappingValueError) as captured: + resolve_binding(binding, {"account alias": "book-a"}, format="csv") + assert captured.value.code == "mapping_path_missing" + assert "book-a" not in str(captured.value) + + +@pytest.mark.parametrize( + ("transforms", "raw", "expected"), + [ + (("trim", "uppercase"), " usdt ", "USDT"), + (("trim", "lowercase"), " EQUITY ", "equity"), + (("decimal",), "-12.2500", Decimal("-12.2500")), + (("utc_timestamp",), "2026-07-20T17:30:00+08:00", datetime(2026, 7, 20, 9, 30, tzinfo=timezone.utc)), + ], +) +def test_allowed_transforms_execute_in_order( + transforms: tuple[Transform, ...], raw: str, expected: object +) -> None: + binding = FieldBinding(path="value", transforms=transforms) + + assert resolve_binding(binding, {"value": raw}, format="csv") == expected + + +def test_local_timestamp_uses_explicit_iana_timezone() -> None: + binding = FieldBinding( + path="as_of", + transforms=("utc_timestamp",), + timestamp_format="%Y-%m-%d %H:%M:%S", + assume_timezone="Asia/Shanghai", + ) + + assert resolve_binding(binding, {"as_of": "2026-07-20 17:30:00"}, format="csv") == datetime( + 2026, 7, 20, 9, 30, tzinfo=timezone.utc + ) + + +@pytest.mark.parametrize( + ("raw", "zone"), + [ + ("2026-11-01 01:30:00", "America/New_York"), + ("2026-03-08 02:30:00", "America/New_York"), + ], +) +def test_local_timestamp_rejects_dst_ambiguous_and_nonexistent_times(raw: str, zone: str) -> None: + binding = FieldBinding( + path="as_of", + transforms=("utc_timestamp",), + timestamp_format="%Y-%m-%d %H:%M:%S", + assume_timezone=zone, + ) + + with pytest.raises(MappingValueError) as captured: + resolve_binding(binding, {"as_of": raw}, format="csv") + assert captured.value.code == "ambiguous_source_time" + assert raw not in str(captured.value) + + +@pytest.mark.parametrize( + "raw", + [ + 0.25, + "NaN", + "1e-3", + "1234567890123456789012345678901", + "0.1234567890123456789", + ], +) +def test_decimal_transform_reuses_position_decimal_limits(raw: object) -> None: + binding = FieldBinding(path="weight", transforms=("decimal",)) + + with pytest.raises(MappingValueError) as captured: + resolve_binding(binding, {"weight": raw}, format="csv") + assert captured.value.code == "mapping_decimal_invalid" + assert str(raw) not in str(captured.value) + + +def test_profile_rejects_unknown_fields_and_invalid_json_path_syntax() -> None: + invalid = dict(PROFILE) + invalid["network"] = "https://example.invalid" + with pytest.raises(ValidationError, match="Extra inputs"): + PositionMappingProfile.model_validate(invalid) + + json_profile = dict(PROFILE) + json_profile["format"] = "json" + with pytest.raises(ValidationError, match="JSON Pointer"): + PositionMappingProfile.model_validate(json_profile) diff --git a/tests/test_position_sources.py b/tests/test_position_sources.py new file mode 100644 index 0000000..0878495 --- /dev/null +++ b/tests/test_position_sources.py @@ -0,0 +1,146 @@ +from pathlib import Path + +import pytest + +from quantcockpit.ingestion import position_sources +from quantcockpit.ingestion.position_profile import PositionMappingProfile +from quantcockpit.ingestion.position_sources import SourceReadError, read_source + + +def profile(format: str, *, layout: str = "tabular_snapshot") -> PositionMappingProfile: + json_path = format != "csv" + prefix = "/" if json_path else "" + data: dict[str, object] = { + "profile_version": "1.0", + "name": f"test-{format}", + "format": format, + "layout": layout, + "snapshot_scope": "grouped_rows", + "fields": { + "strategy_id": {"literal": "alpha"}, + "environment": {"literal": "paper"}, + "source": {"literal": "test-export"}, + "portfolio_id": {"literal": "book-a"}, + "snapshot_time": {"literal": "2026-07-20T09:30:00Z", "transforms": ["utc_timestamp"]}, + }, + "position_fields": { + "instrument_id": {"path": f"{prefix}symbol"}, + "weight": {"path": f"{prefix}weight", "transforms": ["decimal"]}, + }, + } + if layout == "document_snapshot": + data["positions_path"] = "/positions" + data["snapshot_scope"] = "whole_file" + return PositionMappingProfile.model_validate(data) + + +def test_csv_reader_preserves_multiline_record_line_range(tmp_path: Path) -> None: + path = tmp_path / "positions.csv" + path.write_text('symbol,weight,note\nAAPL,0.5,"line one\nline two"\n', encoding="utf-8") + + rows = list(read_source(path, profile("csv"))) + + assert [(row.start_line, row.end_line, row.value["symbol"]) for row in rows] == [ + (2, 3, "AAPL") + ] + + +def test_source_larger_than_100_mib_is_rejected_before_parse(tmp_path: Path) -> None: + path = tmp_path / "large.json" + with path.open("wb") as file: + file.truncate(100 * 1024 * 1024 + 1) + + with pytest.raises(SourceReadError) as captured: + list(read_source(path, profile("json"))) + assert captured.value.code == "ingestion_limit_exceeded" + + +@pytest.mark.parametrize( + ("format", "contents"), + [ + ("csv", "symbol,weight\nAAPL,0.5\nMSFT,0.5\n"), + ("jsonl", '{"symbol":"AAPL","weight":"0.5"}\n{"symbol":"MSFT","weight":"0.5"}\n'), + ("json", '[{"symbol":"AAPL","weight":"0.5"},{"symbol":"MSFT","weight":"0.5"}]'), + ], +) +def test_reader_enforces_total_byte_limit_during_read( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + format: str, + contents: str, +) -> None: + path = tmp_path / f"positions.{format}" + path.write_text(contents, encoding="utf-8") + monkeypatch.setattr(position_sources, "MAX_SOURCE_BYTES", 24) + monkeypatch.setattr(position_sources, "_validate_source_file", lambda *_: None) + + with pytest.raises(SourceReadError) as captured: + list(read_source(path, profile(format))) + + assert captured.value.code == "ingestion_limit_exceeded" + + +def test_jsonl_record_larger_than_one_mib_is_rejected(tmp_path: Path) -> None: + path = tmp_path / "positions.jsonl" + path.write_text('{"symbol":"' + ("A" * (1024 * 1024)) + '","weight":"1"}\n', encoding="utf-8") + + with pytest.raises(SourceReadError) as captured: + list(read_source(path, profile("jsonl"))) + assert captured.value.code == "ingestion_limit_exceeded" + assert captured.value.line_number == 1 + + +def test_jsonl_incomplete_tail_has_distinct_error(tmp_path: Path) -> None: + path = tmp_path / "positions.jsonl" + path.write_text('{"symbol":"AAPL","weight":"1"}\n{"symbol":', encoding="utf-8") + + with pytest.raises(SourceReadError) as captured: + list(read_source(path, profile("jsonl"))) + assert captured.value.code == "incomplete_tail" + assert captured.value.line_number == 2 + + +def test_json_document_and_tabular_array_have_explicit_record_boundaries(tmp_path: Path) -> None: + document = tmp_path / "snapshot.json" + document.write_text('{"positions":[{"symbol":"AAPL","weight":"1"}]}', encoding="utf-8") + tabular = tmp_path / "rows.json" + tabular.write_text('[{"symbol":"AAPL","weight":"1"},{"symbol":"MSFT","weight":"-1"}]', encoding="utf-8") + + document_rows = list(read_source(document, profile("json", layout="document_snapshot"))) + tabular_rows = list(read_source(tabular, profile("json"))) + + assert len(document_rows) == 1 + assert len(tabular_rows) == 2 + assert tabular_rows[1].value["symbol"] == "MSFT" + + +def test_source_extension_must_match_declared_format(tmp_path: Path) -> None: + path = tmp_path / "positions.csv" + path.write_text("symbol,weight\nAAPL,1\n", encoding="utf-8") + + with pytest.raises(SourceReadError) as captured: + list(read_source(path, profile("json"))) + assert captured.value.code == "unsupported_input_format" + + +def test_csv_duplicate_or_blank_headers_are_rejected_instead_of_overwritten(tmp_path: Path) -> None: + duplicate = tmp_path / "duplicate.csv" + duplicate.write_text("symbol,symbol,weight\nAAPL,MSFT,1\n", encoding="utf-8") + blank = tmp_path / "blank.csv" + blank.write_text("symbol,,weight\nAAPL,ignored,1\n", encoding="utf-8") + + for path in (duplicate, blank): + with pytest.raises(SourceReadError) as captured: + list(read_source(path, profile("csv"))) + assert captured.value.code == "invalid_tabular_header" + + +def test_source_errors_do_not_include_raw_values_or_absolute_paths(tmp_path: Path) -> None: + path = tmp_path / "secret-account.jsonl" + path.write_text('{"account":"REAL-ACCOUNT-123"', encoding="utf-8") + + with pytest.raises(SourceReadError) as captured: + list(read_source(path, profile("jsonl"))) + message = str(captured.value) + assert "REAL-ACCOUNT-123" not in message + assert str(tmp_path) not in message diff --git a/tests/test_report.py b/tests/test_report.py index d4138f7..f5a4f20 100644 --- a/tests/test_report.py +++ b/tests/test_report.py @@ -33,6 +33,28 @@ def event_line(strategy_id: str, day: int, value: str) -> str: ) +def position_line(*, portfolio_id: str, positions: list[dict[str, object]]) -> str: + return json.dumps( + { + "schema_version": "1.1", + "strategy_id": "alpha", + "environment": "paper", + "event_type": "position_snapshot", + "event_time": "2026-07-20T17:30:00Z", + "recorded_at": "2026-07-20T17:31:00Z", + "source": "report-fixture", + "payload": { + "event_type": "position_snapshot", + "portfolio_id": portfolio_id, + "base_currency": "USD", + "mapping_profile_hash": f"sha256:{'b' * 64}", + "positions": positions, + }, + }, + sort_keys=True, + ) + + def populated_store(tmp_path: Path) -> DuckDBStore: source = tmp_path / "report-events.jsonl" source.write_text( @@ -67,7 +89,7 @@ def test_report_is_deterministic_and_contains_traceable_health_correlation_and_e finally: store.close() - assert "数据来源:DuckDB 中的 current 版本化 JSONL 事件" in report + assert "数据来源:DuckDB 中的 current 版本化事件" in report assert "生成时间:2026-07-20T18:00:00Z" in report assert "规则 `latest_current_event_freshness`" in report assert "引用 = event:" in report @@ -184,3 +206,52 @@ def test_report_script_writes_only_the_explicit_output_target(tmp_path: Path) -> assert "报告已写入" in result.stdout assert output_path.exists() assert "# QuantCockpit 本地健康报告" in output_path.read_text(encoding="utf-8") + + +def test_report_contains_position_metrics_missing_reason_and_evidence(tmp_path: Path) -> None: + from quantcockpit.report import render_markdown + + source = tmp_path / "position-report.jsonl" + source.write_text( + "\n".join( + ( + event_line("alpha", 19, "0.01"), + position_line( + portfolio_id="ready-book", + positions=[ + {"instrument_id": "AAPL", "market_value_base": "60", "sector": "Technology"}, + {"instrument_id": "MSFT", "market_value_base": "-40"}, + ], + ), + position_line( + portfolio_id="partial-book", + positions=[ + {"instrument_id": "AAPL", "weight": "0.5"}, + {"instrument_id": "MSFT", "quantity": "2"}, + ], + ), + ) + ) + + "\n", + encoding="utf-8", + ) + store = DuckDBStore(":memory:") + try: + import_jsonl(store, source, observed_at=GENERATED_AT) + report = render_markdown(CockpitService(store), generated_at=GENERATED_AT) + finally: + store.close() + + assert "## 仓位覆盖与集中度" in report + assert r"ready\-book" in report + assert r"basis = market\_value\_base" in report + assert "gross = 100" in report + assert "net = 20" in report + assert "Top-1 = 0.6" in report + assert "HHI = 0.52" in report + assert "unclassified" in report + assert r"partial\-book" in report + assert r"incomplete\_basis" in report + assert "coverage = 0.5" in report + assert "mapping:sha256:" in report + assert "event:" in report diff --git a/tests/test_scripts.py b/tests/test_scripts.py index 16a73a5..f06e2a9 100644 --- a/tests/test_scripts.py +++ b/tests/test_scripts.py @@ -3,6 +3,8 @@ from __future__ import annotations import subprocess +import json +import shlex from pathlib import Path from datetime import datetime, timezone @@ -10,6 +12,50 @@ from quantcockpit.service import CockpitService +def write_position_cli_inputs(tmp_path: Path) -> tuple[Path, Path]: + input_path = tmp_path / "positions.csv" + input_path.write_text( + "account,as_of,symbol,weight\n" + "book-a,2026-07-20T09:30:00Z,AAPL,1\n", + encoding="utf-8", + ) + profile_path = tmp_path / "profile.json" + profile_path.write_text( + json.dumps( + { + "profile_version": "1.0", + "name": "cli-test", + "format": "csv", + "layout": "tabular_snapshot", + "snapshot_scope": "grouped_rows", + "fields": { + "strategy_id": {"literal": "alpha"}, + "environment": {"literal": "paper"}, + "source": {"literal": "broker-export"}, + "portfolio_id": {"path": "account"}, + "snapshot_time": {"path": "as_of", "transforms": ["utc_timestamp"]}, + }, + "position_fields": { + "instrument_id": {"path": "symbol"}, + "weight": {"path": "weight", "transforms": ["decimal"]}, + }, + } + ), + encoding="utf-8", + ) + return input_path, profile_path + + +def run_position_cli(*args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + ["uv", "run", "scripts/import_positions.py", *args], + cwd=Path(__file__).parents[1], + capture_output=True, + text=True, + check=False, + ) + + def test_import_demo_script_imports_example_jsonl_into_explicit_database(tmp_path: Path) -> None: database_path = tmp_path / "demo.duckdb" @@ -50,3 +96,120 @@ def test_import_demo_script_imports_example_jsonl_into_explicit_database(tmp_pat assert correlations[0].result.reason is None finally: store.close() + + +def test_position_preview_cli_does_not_create_database_or_print_absolute_paths(tmp_path: Path) -> None: + input_path, profile_path = write_position_cli_inputs(tmp_path) + database = tmp_path / "must-not-exist.duckdb" + + result = run_position_cli( + "--input", + str(input_path), + "--profile", + str(profile_path), + "--database", + str(database), + "--preview", + "--observed-at", + "2026-07-20T10:00:00Z", + ) + + assert result.returncode == 0 + assert not database.exists() + body = json.loads(result.stdout) + assert body["snapshot_count"] == 1 + assert body["sample_positions"][0]["instrument_id"] == "AAPL" + assert str(tmp_path) not in result.stdout + + +def test_position_import_cli_writes_explicit_database_and_reports_counts(tmp_path: Path) -> None: + input_path, profile_path = write_position_cli_inputs(tmp_path) + database = tmp_path / "positions.duckdb" + + result = run_position_cli( + "--input", + str(input_path), + "--profile", + str(profile_path), + "--database", + str(database), + "--observed-at", + "2026-07-20T10:00:00Z", + ) + + assert result.returncode == 0 + assert database.exists() + assert "imported=1" in result.stdout + + +def test_demo_import_includes_synthetic_position_snapshot(tmp_path: Path) -> None: + database = tmp_path / "demo-with-positions.duckdb" + + result = subprocess.run( + [ + "uv", + "run", + "scripts/import_demo.py", + "--database", + str(database), + "--data-dir", + "examples/data", + "--positions-dir", + "examples/positions", + ], + cwd=Path(__file__).parents[1], + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode == 0 + assert "demo-positions.csv" in result.stdout + store = DuckDBStore(database) + try: + assert len(store.current_position_snapshots(datetime(2026, 7, 20, 18, tzinfo=timezone.utc))) == 1 + finally: + store.close() + + +def test_readme_position_preview_and_import_commands_are_executable(tmp_path: Path) -> None: + root = Path(__file__).parents[1] + readme = (root / "README.md").read_text(encoding="utf-8") + preview_command = ( + "uv run scripts/import_positions.py " + "--input examples/positions/demo-positions.csv " + "--profile examples/positions/demo-positions-profile.json " + "--preview --observed-at 2026-07-20T18:00:00Z" + ) + import_command = ( + "uv run scripts/import_positions.py " + "--input examples/positions/demo-positions.csv " + "--profile examples/positions/demo-positions-profile.json " + "--database ./quantcockpit.duckdb " + "--observed-at 2026-07-20T18:00:00Z" + ) + assert preview_command in readme + assert import_command in readme + + preview = subprocess.run( + shlex.split(preview_command), + cwd=root, + capture_output=True, + text=True, + check=False, + ) + imported = subprocess.run( + [ + str(tmp_path / "documented-import") if part == "./quantcockpit.duckdb" else part + for part in shlex.split(import_command) + ], + cwd=root, + capture_output=True, + text=True, + check=False, + ) + + assert preview.returncode == 0 + assert imported.returncode == 0 + assert '"snapshot_count": 1' in preview.stdout + assert "imported=1" in imported.stdout diff --git a/uv.lock b/uv.lock index 52a3c93..d6d9384 100644 --- a/uv.lock +++ b/uv.lock @@ -280,13 +280,14 @@ wheels = [ [[package]] name = "quantcockpit" -version = "0.1.0" +version = "0.2.0" source = { editable = "." } dependencies = [ { name = "duckdb" }, { name = "fastapi" }, { name = "pydantic" }, { name = "pytz" }, + { name = "rfc8785" }, { name = "uvicorn" }, ] @@ -303,6 +304,7 @@ requires-dist = [ { name = "fastapi", specifier = ">=0.139.2" }, { name = "pydantic", specifier = ">=2.13.4" }, { name = "pytz", specifier = ">=2026.2" }, + { name = "rfc8785", specifier = ">=0.1.4" }, { name = "uvicorn", specifier = ">=0.51.0" }, ] @@ -313,6 +315,15 @@ dev = [ { name = "ty", specifier = ">=0.0.61" }, ] +[[package]] +name = "rfc8785" +version = "0.1.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ef/2f/fa1d2e740c490191b572d33dbca5daa180cb423c24396b856f5886371d8b/rfc8785-0.1.4.tar.gz", hash = "sha256:e545841329fe0eee4f6a3b44e7034343100c12b4ec566dc06ca9735681deb4da", size = 14321, upload-time = "2024-09-27T16:33:31.206Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4d/78/119878110660b2ad709888c8a1614fce7e2fab39080ab960656dc8605bf6/rfc8785-0.1.4-py3-none-any.whl", hash = "sha256:520d690b448ecf0703691c76e1a34a24ddcd4fc5bc41d589cb7c58ec651bcd48", size = 9240, upload-time = "2024-09-27T16:33:29.683Z" }, +] + [[package]] name = "starlette" version = "1.3.1"