diff --git a/doc/README.md b/doc/README.md index 66308595..463aae44 100644 --- a/doc/README.md +++ b/doc/README.md @@ -35,6 +35,7 @@ | 文档 | 状态 | |---|---| | [`features/provider-registry.md`](features/provider-registry.md) | 已实现——五种后端的统一注册表 | +| [`features/knowledge-use-trace.md`](features/knowledge-use-trace.md) | 知识检索、内容提供、规则检查与证据的逐步追踪 | | [`features/strict-review-deep-engine.md`](features/strict-review-deep-engine.md) | 已实现（默认开） | | [`features/review-recall.md`](features/review-recall.md) | 已实现（默认开）——v14/v15 召回攻坚 | | [`features/auto-run.md`](features/auto-run.md) | **draft，未实现**——GitHub 事件触发 | diff --git a/doc/architecture/SPEC/README.md b/doc/architecture/SPEC/README.md index 6124ba24..b6fd5517 100644 --- a/doc/architecture/SPEC/README.md +++ b/doc/architecture/SPEC/README.md @@ -68,7 +68,7 @@ | 跨切 | `review/{diff_summary,triggers,reviewer}` `memory/{debug_memory,skills}` `run_trace` `notify` `metrics` | | 后端 | `providers/{registry,base,claude_code,codex,cursor,deepseek,zcode,audit,harness_llm}` `tool_bridge` | | MCP 面 | `thin_mcp_server` `mcp_server` `mcp_policy` `run_status` `knowledge_docs` `__main__` | -| 共享服务 | `kb_service` `rfc_service`（包规范覆盖各自实现模块） | +| 共享服务 | `kb_service` `rfc_service` `knowledge_trace`（包规范覆盖各自实现模块） | 平凡的 `__init__.py` 再导出文件不单独出规范 （`engine/steps/__init__` 例外，因为它定义了 `register_builtin_steps`）。 diff --git a/doc/architecture/SPEC/adapters/base.md b/doc/architecture/SPEC/adapters/base.md index ebf8e5fe..a2643b2a 100644 --- a/doc/architecture/SPEC/adapters/base.md +++ b/doc/architecture/SPEC/adapters/base.md @@ -1,6 +1,6 @@ # adapters/base.py —— 规范 - + `LOC ~423 · 边缘（仓库知识） · refactor-status: ok` @@ -64,3 +64,11 @@ ## 2026-09-28 knowledge_lifecycle `knowledge_lifecycle` 加入 `HIGH_RISK_SECTIONS`：知识服务的开关与 `auto_merge` 切换只能经人工审阅修改。 + +## 2026-11-04 逐文档 briefing 观测 + +`render_briefing_docs(..., on_document=None)` 和 `RepoAdapter.briefing` 接受可选 +回调，报告实际经过 visible-text、frontmatter 去除、首尾空白去除和 briefing +上限后保留的文档片段、相对文档 ID、源偏移与截断状态。零长度片段不代表 +模型获得正文；后续请求级裁剪仍由 agent runtime 核实。未传回调时渲染行为不变。 +回调不另读知识、不增加 briefing 预算，也不改变 manifest 写入门。 diff --git a/doc/architecture/SPEC/app/workflow_execution.md b/doc/architecture/SPEC/app/workflow_execution.md index b56ee041..0e901a8e 100644 --- a/doc/architecture/SPEC/app/workflow_execution.md +++ b/doc/architecture/SPEC/app/workflow_execution.md @@ -1,6 +1,6 @@ # app/workflow_execution.py — workflow execution boundary - + `WorkflowExecution` runs an already selected playbook with the caller's current LLM dependency. It rechecks the repository context before starting, @@ -36,3 +36,12 @@ not belong in this module. `Copilot._execute` remains the compatibility entry point and records `last_run_dir`/`last_blocked_reason` from the returned result. Existing run state keys, traces, exit codes, and checkpoint behavior stay stable. + +Knowledge-use tracing is finalized in the same execution `finally` boundary, +after restoring signal handlers. An existing session receives `complete` only +when the executor returned `done` without shutdown; other exits receive +`interrupted`. Finalization writes `KNOWLEDGE_TRACE.json` and +`KNOWLEDGE_TRACE.md` through the agent-runtime owner. No session is created +merely to finalize a workflow, and run finalizers, interruption handling and +lock ownership retain their existing behavior. Required-mode trace failures +propagate rather than allowing an audit-complete result. diff --git a/doc/architecture/SPEC/cli.md b/doc/architecture/SPEC/cli.md index 1ea4e489..215934aa 100644 --- a/doc/architecture/SPEC/cli.md +++ b/doc/architecture/SPEC/cli.md @@ -1,6 +1,6 @@ # cli/ —— 规范 - + `LOC ~1420（6 个文件） · 接口 + 编排门面 · refactor-status: ok` @@ -141,3 +141,12 @@ RFC 子命令原样转发与共享入口：`test_rfc_integration.py`；服务契 保留原始 run，不自动重放已经完成的任务。托管安装的执行还在 worker 内持有 跨进程 checkout/PR 锁，以覆盖不同 MCP 进程和 release；运行时固定与控制面契约见 [upgrade.md](upgrade.md)。 + +## 2026-11-04 知识使用追踪 + +`infermatrix-copilot knowledge-trace …` 在通用 argparse、Copilot 和模型构造前 +分派到 `knowledge_trace.cli.main`。`show` 支持原始事件分页；`export` 读取全部 +事件页后输出 JSON 或 Markdown；`query --repo … --rule …` 返回有界事件摘要。 +`--root` 指定追踪运行目录；读取已有记录不受采集模式 `off` 限制。该入口不读取 +产品知识来补造历史记录，也不调用模型或发布结果。验证见 +`test/test_knowledge_trace_cli.py`。 diff --git a/doc/architecture/SPEC/engine/agent_runtime.md b/doc/architecture/SPEC/engine/agent_runtime.md index 2ba51213..faaef02b 100644 --- a/doc/architecture/SPEC/engine/agent_runtime.md +++ b/doc/architecture/SPEC/engine/agent_runtime.md @@ -1,6 +1,6 @@ # engine/agent_runtime/ —— 规范 - + `LOC ~1690（7 个文件） · 引擎（受治理的 agent 运行时） · refactor-status: ok` @@ -119,3 +119,26 @@ cheap-seat 与按 pass 路由、MoA harness 成员、考古工具）。有两条 ## 自进化接入（2026-10-04） 影子 agent 使用严格只读 scope，知识检索与 repo map 受读根约束，拒绝知识写入工具。API 与 harness 调用均记录 trace/1；工作流的采集和配置指纹由执行器绑定，普通生产运行保持既有工具能力。 + +## 2026-11-04 知识使用追踪 + +`knowledge_trace.py` 是 Strict 到共享 `knowledge_trace/` 协议的唯一桥接： +每次执行尝试建立一份会话，并行步骤和 MoA 成员保留完整的限定 step ID。 +`dispatch` 增加可选 `knowledge_assessments`，把规则、具体检查、证据和 finding +关联；缺失声明保持未评估。每次模型调用的证据 ID 独立，重复 step 不覆盖旧证据。 + +文档工具与 briefing 先记录检索，再排队候选内容；runner 在最终请求组装和 +裁剪后确认实际存在的片段，才记录 `model_request` 提供边界。自适应服务返回的 +`documents` 只作引用；`adaptive.packet_fragments` 仅解析实际 `model_content` +中完整、哈希一致的 provider 围栏，不再读取源文档。保留 repo、source pins 和 +上下文会话元数据；continuation 或变换片段的源区间未知，预算截断明确保留。 +已授权的跨仓库自适应内容保持原执行行为，由单仓库追踪记录 +`context_cross_repository_unobserved` 缺口，不扩大追踪权限。累计预算、反馈回执、 +containment 与用户协议门继续由原知识上下文服务拥有。 + +API 模式可以观测模型请求；harness 模式记录提供给外部 CLI 的 prompt 边界， +外部 CLI 的原生读取和内部请求不能据此宣称已观测。skill/debug memory 保留 +检索身份并说明其缺少 curated-document 来源。`required` 采集失败阻止后续受控 +工作，`best_effort` 保持执行并标为 degraded；终结时写 JSON/Markdown 报告。 +验证见 `test/test_knowledge_trace_strict.py`、 +`test/test_knowledge_trace_adaptive_strict.py`。 diff --git a/doc/architecture/SPEC/knowledge_trace.md b/doc/architecture/SPEC/knowledge_trace.md new file mode 100644 index 00000000..7b047f77 --- /dev/null +++ b/doc/architecture/SPEC/knowledge_trace.md @@ -0,0 +1,332 @@ +# knowledge_trace/ — Knowledge-use trace, issue #442 + + + +Issue: https://github.com/InferMatrixLab/InferMatrixCopilot/issues/442 + +## Problem and outcome + +Before this change, the Direct router returned owner pages, selection reasons, quick maps, +related prose, and a knowledge snapshot. Strict records selected pages and injected +skills. TraceStore records calls, decisions, and outcomes. These pieces do not +provide a unified account of **step → knowledge → check → evidence → decision**. + +Provide a durable, queryable record for Direct and Strict and a deterministic +step-by-step report. It must answer which entries were selected, what content was +actually provided, why a rule was applicable, which evidence was checked, and what +decision followed. This records concise, reviewable justifications and observable +evidence, not private model chain-of-thought. A finding need not cite knowledge; +an assessed rule need not produce a finding. + +## Existing owners + +- `direct_routing.py`: repository-neutral bounded owner/model selection. +- `sdk/v1/direct.py`: review context, source head, resource revision, document + excerpts, hashes, and context-bound reads. +- `thin_mcp_server.py`: Direct/Strict review, document reads, and completion tools. +- `engine/agent_runtime/`: Strict briefings, skills, knowledge tools, and dispatch. +- `trace_store.py`: redacted JSONL, content-addressed blobs, derived SQLite index. +- `run_trace.py`: per-run factual timeline; it can link to the knowledge trace. + +The final serialized SDK response includes `DocumentRef.excerpt`, not just quick +maps. Capture all content actually returned. Capture Strict content after prompt +assembly and clipping, rather than equating retrieval with model input. + +## Semantics + +| State | Meaning | Observer | +|---|---|---| +| retrieved | Routing/search/briefing selected an entry | Copilot | +| provided | Exact content crossed a named output/input boundary | Boundary owner | +| assessed/applied | Agent explicitly connected a rule to a check or decision | Agent declaration, references validated | + +Provided boundaries are `provider_response`, `host_context`, and `model_request`. +Returning content does not prove the host supplied it to a model. Supplying it +does not prove model attention or application. A quick map mentioning a rule ID +does not prove delivery of that rule's body. Unreported application stays unknown. + +Assessment has independent dimensions: + +- `applicability`: `applicable`, `not_applicable`, `unknown`. +- `verification`: `supported`, `contradicted`, `unresolved`. + +An applicable rule with unavailable hardware is unresolved. A rule contradicted +by current source keeps the conflicting evidence. No assessment means +`not_assessed`, not `not_applicable`. Rules can be provided and assessed repeatedly +across steps; this is not a single linear state machine. + +## Architecture and ownership + +One `knowledge_trace/` package owns event semantics, identifiers, validation, +recording, query, and reporting. Direct and Strict adapters submit observations; +they do not implement separate schemas. Reuse TraceStore and its redaction and +blobs. JSONL and blobs are the source; SQLite and Markdown are derived views. +There is no additional service, message queue, model call, or retrieval pass. + +Package ownership and public contract: + +- `references.py`: knowledge identities, scoped document paths, content hashes, + explicit rule-heading sections and conservative source ranges. +- `recorder.py`: `KnowledgeTrace.start_session`, `retrieve`, `provide`, + `record_events`, `finish_session`, `report` and `query`; protocol validation, + append ordering and writer serialization. +- `report.py`: deterministic event projection and `markdown(report)`; findings + retain assessment/evidence links without inferring causation. +- `adaptive.py`: `packet_fragments(packet)` extracts exact delivered fragments + from already metered provider packets; no source reads or permission changes. +- `cli.py`: model-free show/export/query commands, including atomic file export. +- `__init__.py`: public `KnowledgeTrace`, `KnowledgeTraceError`, + `extract_rule_sections` and `markdown` exports. + +Dependencies are stdlib plus the shared TraceStore owner. The package does not +import SDK, transports, planner or model runtime; those owners call downward +into this protocol. Existing router behavior and knowledge read budgets remain +intact. + +## Data model + +### Session + +`trace_id`, `run_id`/`review_id`/`review_context_id`, canonical repository, mode, +source head or workspace fingerprint (or explicit unknown), knowledge snapshot, +resource revision, instrumentation capabilities, and lifecycle status. + +`review_context_id` binds inputs and versions; `trace_id` identifies an execution +attempt. Within a DirectClient, repeating an identical plan request reuses its +trace, including after completion. Use a new `review_id` for a new attempt. +Strict creates a new trace for each execution/resume attempt and retains the run +ID for correlation. A trace ID is not an authorization token. + +### KnowledgeRef + +`knowledge_ref_id`, namespace (`general` or repository), document-relative path, +optional rule ID/section anchor, snapshot, document hash, and optional section +hash. Identity includes namespace, path, and version, not only `DIFF-1b`. Pages +without IDs remain page/section references; never infer a rule ID from prose. + +### Exposure + +`exposure_id`, knowledge reference, step, boundary, representation (search hit, +quick map, excerpt, full section), payload blob, included ranges, explicit offset +unit (`utf8_bytes` or `unicode_codepoints`), truncation, and delivery status. + +Offsets refer to served visible text, not original disk lines after retired-rule +filtering. Unknown or disjoint ranges are empty with `range_status: unknown`, not +fabricated zero offsets. UTF-8 byte slices preserve their original source length +even when decoding introduces replacement characters. Keep document identity and +served-content hashes distinct from the redacted archive blob hash. Only explicit +rule headings attest rule-body delivery. + +### EvidenceRef + +Evidence ID, origin (`captured` or `host_reported`), kind (source/test/tool), source +head or workspace identity, relative path/range or command, output reference, +execution environment where available, and verification availability. References +from another source head are stale, not silently applicable. + +### Assessment + +Assessment ID, step, knowledge reference, exposure IDs, applicability, +verification, short rationale, concrete check, evidence IDs, decision/finding +IDs, reporter, and reference-validation result. Missing evidence is explicit. +Mechanical reference validity does not prove semantic correctness. + +```json +{ + "assessment_id": "a-17", + "step_id": "review.defaults", + "knowledge_ref_id": "k-42", + "exposure_ids": ["x-12"], + "applicability": "applicable", + "verification": "supported", + "rationale": "The parameter permits zero; inspect default-value handling.", + "check": "Verify guidance=0 reaches its consumer.", + "evidence_ids": ["source-8", "test-3"], + "decision_ids": ["finding-2"], + "reported_by": "host" +} +``` + +## Event protocol and persistence + +Add `knowledge_event` to the existing `trace/1` kind set. Its `result` contains +`schema: knowledge-use/1`, event type, trace/event IDs, producer ID, producer +sequence, parent IDs, and the event-specific payload. Preserve existing query +columns (`run_id`, `step`, `repo`, `rule_ids`); no initial index migration. + +Event types: + +- `session_started` / `session_finished`. +- `retrieval_completed`: mechanism, query, candidates, reasons, ranks, no hits. +- `content_provided`: actual content, representation, range, named boundary. +- `evidence_registered`. +- `assessment_recorded`. +- `gap_recorded`: failed reads, version mismatch, unavailable observation. + +Cross-process recording must serialize session updates. Event IDs are idempotent; +same ID and same content retries return the previous result, conflicting content +is rejected. Producer sequences order local events; parent links express causal +order across producers. Wall-clock timestamps are not proof of causality. +Assessment/gap corrections append a same-type `supersedes` link rather than +rewriting old events. Observations and evidence remain immutable and cannot be +superseded; corrected evidence gets a new identity and can support a replacement +assessment. This prevents host corrections from hiding provider observations or +leaving assessments with dangling evidence references. + +Store redacted payloads before records. Indexes and reports can be rebuilt. Old +readers/exporters must tolerate the added kind; do not count knowledge events as +model calls, outcomes, or training examples by accident. + +## Direct integration + +1. Start a session bound to repository, review context, and source head. +2. Record route/search candidates, selection reasons, fallbacks, unmatched scope. +3. Record all knowledge in the final response: quick maps, related prose, + mandatory-guide excerpts, maps, and SDK document excerpts. +4. Return trace metadata, knowledge references, and exposure references. +5. Host optionally acknowledges exact content inserted into its context. +6. Host submits evidence and assessments in batches per step or at completion. +7. Completion returns separate review and knowledge-trace status. + +Without a host acknowledgement, report "provider returned; host delivery unknown". +Shell/native file reads can be declared with hashes and tool references, but +remain host-reported when Copilot cannot observe them. + +Traced MCP reads/searches accept an optional trace/context. Reads use the pinned +knowledge view; an unavailable old context fails explicitly instead of falling +through to the current tree. Untraced legacy calls retain their behavior. + +SDK additions are optional/defaulted, with a capability flag and protocol version. +`KnowledgeTraceClient` exposes start, record, finish, and report. MCP tools are +`start_knowledge_trace`, `record_knowledge_usage`, `finish_knowledge_trace`, and +`get_knowledge_trace`. Review can auto-start a session. Existing clients need not +report assessments. + +## Adaptive context integration + +The existing `KnowledgeContextService` owns frozen views, cumulative budgets, +repository authorization, consent, feedback receipts and containment. Tracing +observes its final packet without repeating retrieval or changing those gates. +`documents` are metadata references and cannot attest delivery. Only complete +provider-rendered `` JSON blocks in `model_content` yield +exposures; verify `model_content_sha256` when supplied and treat malformed or +unframed content as a capture failure. + +Preserve fragment `repo_id`, `source_pins` and context session ID in retrieval +metadata. Record only the budgeted fragment body, with its partial/truncated +state. Continuations and transformed/search fragments retain unknown source +ranges rather than inventing offsets. Empty/no-new-context packets provide no +new content; referenced sibling documents do not become exposures. + +The context service may authorize cross-repository dependencies. This trace +protocol remains scoped to `general/` and one selected repository. Record an +explicit cross-repository observation gap and let the already authorized +context operation retain its original behavior, including in required mode. +Do not silently copy that other repository's content into this trace or use a +trace ID as a new grant. + +Direct records the provider-return boundary. Strict queues each exact packet +fragment, then confirms it survives the final prompt/tool-response assembly +before recording a model-request boundary. JSON escaping of a tool result must +not turn a delivered fragment into an unexplained missing observation. + +## Strict integration + +Capture briefing/skill/related selection, then final model inputs after clipping. +Instrument `doc_search`, `doc_read`, and `doc_related`; associate actual tool +results with the model request that receives them. Add optional +`knowledge_assessments` to the output contract, with stable check/finding refs. + +API mode can observe model requests. Harness mode records the supplied prompt +boundary and Copilot tools; external CLI-native file reads remain unknown unless +the host reports them. Do not parse arbitrary terminal output to invent usage. +Concurrent steps share a run session and retain qualified step IDs. + +## Reports, query, and CLI + +Deterministically project events into session identity, instrumentation coverage, +ordered steps, retrievals, exposures, assessments, evidence, findings, and gaps. +Support bounded/paginated event reads and repository/rule/run queries. + +```bash +infermatrix-copilot knowledge-trace show +infermatrix-copilot knowledge-trace export --format json --out trace.json +infermatrix-copilot knowledge-trace export --format markdown --out KNOWLEDGE_TRACE.md +infermatrix-copilot knowledge-trace query --repo vllm-omni --rule DIFF-1b +``` + +| Step | Entry | Selection reason | Provided content | Assessment/evidence | +|---|---|---|---|---| +| Parameter defaults | DIFF-1b | guidance signal | Full section in model request | Applicable; source and zero-value test | +| Parameter defaults | Supporting guide | Source-path match | Truncated excerpt | Not assessed | +| Output contract | None | Search had no hit | — | Explicit knowledge gap | + +Counts describe observation, not correctness or causation. "12 rules provided, +4 explicitly assessed" is valid. "Knowledge caused four findings" is not proven. +Historical records are reconstructed only where evidence permits, with unknown +fields retained; no fabricated backfill. + +## Validation, scope, and failure behavior + +Core validates page/rule/version/exposure references, repository scope, source +head consistency, event identity, and parent/evidence links. Readable knowledge +is shared `general/` plus exactly one selected repository. Reports do not disclose +another repository's runtime evidence. + +`KNOWLEDGE_TRACE_MODE=off|best_effort|required`: + +- `off`: no trace I/O; legacy execution behavior. +- `best_effort` (default): storage failure is visible as degraded telemetry but + does not stop the task. +- `required`: failed critical recording stops subsequent controlled work and + refuses an audit-complete result; it does not pretend to undo previous actions. + +Review completion and trace completion are independent. Missing observation is +partial/unknown, never implicit proof that knowledge was unused. + +`KNOWLEDGE_TRACE_ROOT` defaults to the Copilot runtime state directory. Keep all +artifacts outside the product knowledge tree, use relative public paths, reuse +secret redaction, and never automatically upload traces or publish comments. +Retention follows trace management; cleaned payloads are explicitly unavailable. +Content-addressed blobs deduplicate content; repeated observations remain events. +Bound event sizes/counts and report explicit limits rather than silently dropping. + +## Delivery plan + +1. Core protocol, references, persistence, deterministic reports, and CLI. +2. Strict briefing, document tools, actual request capture, and artifacts. +3. Direct SDK/MCP sessions, pinned reads, host receipts, backwards compatibility. +4. Assessments, evidence/finding links, completion diagnostics, and integration tests. + +## Validation plan + +Test retrieval without delivery/application; quick-map ID versus rule body; SDK +excerpt delivery without a doc read; prompt clipping; partial/multi-page reads; +snapshot activation mid-session; same rule ID in different repos; retries, +concurrency, and restart; stale-head evidence; successful checks without findings; +harness visibility gaps; required/best-effort/off modes; interrupted runs; index +rebuild; redaction; traversal and cross-repo rejection; and CLI JSON/Markdown export. +Run adjacent SDK, MCP, Strict, TraceStore, reporting, and CLI compatibility tests. + +## Tradeoffs + +Parsing only existing logs is cheaper but cannot reconstruct absent rule-to-check +links. A separate telemetry service provides scale at unnecessary operational +cost. Extending existing TraceStore with explicit producer observations preserves +the current ownership and permits incremental adoption. Remaining limits are +model attention, semantic evidence sufficiency, and unobserved native host reads; +the report must state these rather than infer them. + +## Implemented validation ownership + +`test/test_knowledge_trace.py` covers the shared protocol, references, immutable +observations, producer ordering, idempotency, index recovery, scopes, redaction +and event/report semantics. `test/test_knowledge_trace_cli.py` covers pagination +and complete JSON/Markdown exports. `test/test_knowledge_trace_direct.py` covers +SDK/MCP provider boundaries, pinned reads and host declarations. +`test/test_knowledge_trace_strict.py` covers final prompt clipping, assessed +checks, evidence identity and mode failure behavior. Adaptive capture and +authorization/budget compatibility are covered by +`test/test_knowledge_trace_adaptive.py` and +`test/test_knowledge_trace_adaptive_strict.py`. Adjacent SDK, MCP, agreement, +feedback and containment tests remain part of integration validation. diff --git a/doc/architecture/SPEC/sdk.md b/doc/architecture/SPEC/sdk.md index 96e90883..b4175d96 100644 --- a/doc/architecture/SPEC/sdk.md +++ b/doc/architecture/SPEC/sdk.md @@ -1,6 +1,6 @@ # sdk/ —— 规范 - + `Python SDK v1 · 跨仓库唯一 typed 边界 · refactor-status: ok` @@ -217,3 +217,27 @@ Direct 1.1 / Strict 1.3 accept typed `CarriedFinding` inputs (unique IDs, source Opt-in protocol 1 exports `knowledge_maintenance_status`, `knowledge_usage_record`, `knowledge_availability_check`, `knowledge_publication_guard` and `knowledge_policy_install`. DirectClient and StrictRuntimeConfig accept explicit maintenance configuration. Usage accepts an exact privately issued provider context; arbitrary caller hashes cannot mint provenance. Retrieved and actually injected units are separate from conservative Strict snapshot scope. Current generation is checked at Direct validation and Strict polling; held Strict results retain the original result with top-level state `held`. Hosts hold the publication guard around final admissibility and the external write; the installer drains that same lock before signed policy publication and ACK. ACK includes actual current knowledge snapshot/tree and corrected-content observations, separately from protocol readiness. No service private key crosses this SDK boundary. Direct plan receipts bind a provider-private durable delivery session. Adaptive follow-up read/search/related deliveries and continuations after expansion join the same session before returning; legacy `read_document(review_context_id=...)` pages join their original review's session as well. `validate` and the generic final `knowledge_availability_check` resolve all delivered dependencies, including when the host reuses its initial receipt after another SDK instance resumes an adaptive session. Caller-supplied session IDs cannot create that binding. Enforced document reads require the existing provider-issued `review_context_id` and include it in `DocumentPage.to_dict()` so identical pages in concurrent reviews retain separate provenance. The optional field is omitted when empty; disabled standalone reads retain compatibility. SDK retrieval does not claim model injection; the host records actual injection separately. + +## 2026-11-04 知识使用追踪 + +`sdk.v1.KnowledgeTraceClient` 为惰性公共导出，提供 `start_session`、 +`record_events`（`record_usage` 别名）、`finish_session` 和 `get_report`。 +宿主只能提交 host-context 回执、host-reported 证据、assessment 与缺口；回执 +引用既有 exposure，不能改写 provider 内容、哈希或规则身份。批量提交有界且 +可幂等重试，所有引用仍经共享协议校验。导入公共 SDK 不初始化追踪存储。 + +`Capabilities` 声明知识追踪支持与协议版本。`DirectReviewPlan`、`DocumentRef` +和 `DocumentPage` 增加默认可空的追踪/引用元数据，旧客户端可忽略。相同计划 +请求复用其 trace；新的 `review_id` 开始新的执行尝试。后续携带 trace/context +的读取绑定原冻结 view，失效时明确拒绝，不能回退当前快照。trace ID 不授予 +读取权限。存储故障持续反映在 capture/finish 状态，不能在恢复后隐去缺口。 + +legacy plan 记录最终返回的 quick map、related prose 和 SDK excerpt；自适应 +plan/read/search/related 仅从经过累计预算的 `model_content` 获取实际提供片段， +不为追踪重复读取 `documents` 引用或额外返回 excerpt。检索记录保留 repo、 +source pins 和上下文会话；continuation 的源区间保持未知。已由上下文服务授权 +的跨仓库内容继续提供，单仓库追踪记录 `context_cross_repository_unobserved` +而不扩大追踪范围。原反馈、用户协议和 containment 签发/校验顺序保持不变。 +服务返回仅证明 `provider_response`，宿主另报 `host_context` 才证明其交付声明。 +验证见 `test/test_knowledge_trace_direct.py`、 +`test/test_knowledge_trace_adaptive.py`；协议详见 [knowledge_trace.md](knowledge_trace.md)。 diff --git a/doc/architecture/SPEC/thin_mcp_server.md b/doc/architecture/SPEC/thin_mcp_server.md index 80196a9a..89235ff1 100644 --- a/doc/architecture/SPEC/thin_mcp_server.md +++ b/doc/architecture/SPEC/thin_mcp_server.md @@ -1,6 +1,6 @@ # thin_mcp_server.py —— 规范 - + `LOC ~490 · 默认 MCP：Direct 门面 + Strict 入口 · refactor-status: ok` @@ -172,3 +172,21 @@ Skill 与物理引用路径；不兼容协议拒绝执行。写知识的工作 authoring checkout，不能修改安装制品。`get_update_status()` 仅读取当前、活动、 待安装版本和升级状态，不下载、不启动 updater。贡献入口同时支持 wheel 内的 `_runtime/doc/knowledge/`。升级与进程固定契约见 [upgrade.md](upgrade.md)。 + +## 2026-11-04 知识使用追踪 + +Direct review 返回可选 `knowledge_trace` 元数据；`doc_read`、`doc_search` +及自适应上下文工具携带 trace/context 时使用原固定知识视图并记录实际交付， +不可用的上下文明示失败。自适应工具仅观测 budget 后 `model_content` 的正文， +不重复解析 `documents` 作为全文，也不改变反馈、用户协议、containment 或 +累计预算服务的授权。已授权跨仓库交付只产生追踪范围缺口。 + +四个增量工具是 `start_knowledge_trace`、`record_knowledge_usage`、 +`finish_knowledge_trace` 和 `get_knowledge_trace`，统一委托公开 +`KnowledgeTraceClient`。宿主回执属于 `host_context` 声明，不能假冒 +`provider_response` 或捕获的工具证据；证据、assessment 和幂等重试由共享 +协议校验。读取/报告可在进程重启后恢复，带 repository 的调用校验会话归属。 +`best_effort` 故障独立报告 degraded，`required` 阻止后续受控工作；Direct +完成校验与追踪完成状态分别返回。相关宿主步骤见随 release 提供的 imreview +工作流。验证见 `test/test_thin_mcp_server.py` 与 +`test/test_knowledge_trace_direct.py`。 diff --git a/doc/architecture/SPEC/trace_store.md b/doc/architecture/SPEC/trace_store.md index 363385d2..2e885ec3 100644 --- a/doc/architecture/SPEC/trace_store.md +++ b/doc/architecture/SPEC/trace_store.md @@ -1,6 +1,6 @@ # trace_store.py —— 规范 - + `LOC ~500 · trace/1：Copilot 与 RB 共用的模型调用/工具调用/判定/结果记录 · refactor-status: stable` @@ -8,7 +8,7 @@ - `TraceStore(root)`：追加式 `records/<日期>.jsonl`（事实来源）+ 内容寻址 blob （`blobs//.gz`，gzip，读取时复核哈希）+ SQLite 索引 `index.db` （按 kind、run、仓库、变更集、PR、规则 ID、角色、模型、结果查询；`rebuild_index` 可从 JSONL 重建）。 -- 记录（`schema: trace/1`）：`kind` ∈ `model_call`/`tool_call`/`decision`/`outcome`/`replay`；`context`（run_id、 +- 记录（`schema: trace/1`）：`kind` ∈ `model_call`/`tool_call`/`decision`/`outcome`/`replay`/`knowledge_event`；`context`（run_id、 playbook、step、repo、changeset_id、pr、rule_ids、workflow、unit_id、item、fingerprint、of…）、`model`（role、provider、model、effort、served_model）、 `usage`、`seconds`、`inputs`/`outputs`（只存 blob 引用，从不内联正文）、`result`、`error`、`env` （copilot 版本与 SHA）。`outcome` 是事后金标（merged、closed_unmerged、head_changed、rule_retired）。 @@ -50,3 +50,13 @@ 重建到 schema 0 + 旧代码位置式插入、只读比较、两处 choke point 的采集、声明与指纹、执行器上下文、CLI 门）； `test_trace_store.py`（schema 与去重 blob、索引与重建、脱敏、网关记录成功与失败调用、回放、导出的判定/结果 关联与校准集泄漏过滤、SDK 导出、知识服务的 decision/outcome 记录、派发前输入留档、原生事件及中断保留）。 + +## 2026-11-04 knowledge_event + +`trace/1` 允许 `kind=knowledge_event`，`result` 包含 +`schema=knowledge-use/1` 的共享知识使用协议。session、retrieval、content、 +evidence、assessment、gap 和 finish 事件沿用脱敏 blob、追加 JSONL 与派生索引； +run/step/repository/rule 等现有查询列保持可用，不触发索引迁移。会话级排序、 +引用与幂等校验属于 `knowledge_trace/`，TraceStore 只拥有通用持久化。 +新 kind 不得被当作模型调用、事后 outcome 或训练样本。协议与验证见 +[knowledge_trace.md](knowledge_trace.md) 及 `test/test_knowledge_trace.py`。 diff --git a/doc/features/knowledge-use-trace.md b/doc/features/knowledge-use-trace.md new file mode 100644 index 00000000..7beba934 --- /dev/null +++ b/doc/features/knowledge-use-trace.md @@ -0,0 +1,102 @@ +# 知识使用追踪 + +追踪把执行步骤、命中的知识条目、实际提供的内容、规则检查与证据关联起来。 +完整设计见 [knowledge_trace](../architecture/SPEC/knowledge_trace.md)。 + +## 开关和存储 + +通过进程环境配置；不需要模型凭据，也不会增加模型调用： + +```bash +export KNOWLEDGE_TRACE_MODE=best_effort +# 可选：将 Direct 和 Strict 的记录放入同一个运行状态目录 +export KNOWLEDGE_TRACE_ROOT=/path/to/runtime/knowledge-traces +``` + +模式为 `off`、`best_effort`（默认）或 `required`。关闭时不采集记录； +best_effort 的存储故障单独标为 degraded；required 在关键记录失败时阻止继续 +宣称追踪完整。它不撤回此前已发生的动作，也不改变审查规则本身。 + +Direct 默认使用用户运行状态目录中的 `knowledge-traces/`。Strict 未设置 +`KNOWLEDGE_TRACE_ROOT` 时使用 `/knowledge_trace/`，运行结束生成 +`KNOWLEDGE_TRACE.json` 和 `KNOWLEDGE_TRACE.md`；JSON 中包含 trace ID 和存储位置。 +内容通过 TraceStore 脱敏后保存为内容寻址 blob，记录不进入产品知识树，也不会 +自动发布到 GitHub。 + +## 查看与导出 + +```bash +infermatrix-copilot knowledge-trace show --root +infermatrix-copilot knowledge-trace show --root --json +infermatrix-copilot knowledge-trace export --root \ + --format markdown --out KNOWLEDGE_TRACE.md +infermatrix-copilot knowledge-trace export --root \ + --format json --out knowledge-trace.json +infermatrix-copilot knowledge-trace query --root \ + --repo vllm-omni --rule DIFF-1b --limit 100 +``` + +`show --json` 的原始事件列表支持 `--cursor` 和 `--limit`；报告中的步骤投影 +覆盖该会话。`export` 收集所有事件页。`query` 返回有界的匹配事件摘要。 +即使采集模式为 off，也可以读取已有记录。 + +## Direct 宿主接入 + +Direct review 的返回值增加 `knowledge_trace`，SDK plan 也带追踪元数据。 +随版本提供的 `imreview` 工作流会携带固定的 head、trace ID 和仓库信息，按步骤 +批量记录实际使用，并在结束时关闭追踪；旧宿主需采用更新后的工作流才能记录这些回执。 +服务返回知识时记录 `provider_response`；宿主确认提供到 agent 上下文时才记录 +`host_context`。SDK `DocumentRef.excerpt` 是实际提供内容的一部分，即使宿主 +没有再调用文档读取工具，也会被记录。 + +自适应知识会话只记录累计预算后 `model_content` 中实际提供的正文， +不会为追踪重新读取 `documents` 引用或增加片段。检索元数据保留来源仓库、 +source pins 和上下文会话；continuation 或无法确认坐标的片段标为区间未知， +预算截断明确保留。上下文服务已授权的跨仓库内容继续正常提供；单仓库追踪 +为其记录观测缺口。原用户协议、反馈回执和 containment 校验仍然生效。 + +MCP 提供四个工具： + +- `start_knowledge_trace`：显式启动一次追踪。 +- `record_knowledge_usage`：批量提交宿主回执、证据和规则检查。 +- `finish_knowledge_trace`：结束追踪。 +- `get_knowledge_trace`：读取报告。 + +文档读取或搜索携带 `trace_id` 时使用会话固定的知识版本。过期或不可用的上下文 +明确报错，不会切换到新快照。调用方必须使用同一个仓库；trace ID 不赋予额外访问权限。 + +嵌入式调用使用 `sdk.v1.KnowledgeTraceClient`。以下示例只演示宿主提交契约： + +```python +from infermatrix_copilot.sdk.v1 import KnowledgeTraceClient + +client = KnowledgeTraceClient(root="/path/to/runtime/knowledge-traces") +report = client.get_report(trace_id, repository="vllm-omni") +exposure = report["exposures"][0] +client.record_usage(trace_id, [{ + "event_type": "content_provided", + "step_id": "review.defaults", + "exposure": {"source_exposure_id": exposure["exposure_id"]}, +}], repository="vllm-omni", idempotency_key="defaults-delivery-1") +``` + +宿主回执只能引用既有内容，不能覆盖来源内容、哈希或规则身份。 +assessment 引用版本固定的 `knowledge_ref_id`，或无歧义的 `document_id` 与 +`rule_id`；支持/反驳结论必须关联证据。没有运行证据时记录 unresolved。 + +## 如何解释报告 + +- retrieved：条目被选择；不代表正文已经提供。 +- provided：具体片段在指定边界出现；不证明模型注意到它。 +- assessed：agent 显式记录适用性、检查与证据；引用有效不等于结论已独立证明。 +- 未评估：没有应用声明，不能推断为“不适用”或“未使用”。 + +quick map 中出现规则 ID 只证明提供了引用，不证明完整规则正文已提供。 +截断、无命中、失败读取、缺失证据和旧 head 的证据会保留在报告中。 + +Strict API 可以观测提交给模型的请求；harness 记录提供给外部 CLI 的 prompt， +外部 CLI 自带的文件读取和内部模型请求仍可能不可观测。skills/debug memory +来自独立存储时，保留检索元数据并标注与知识文档引用的区别。 + +报告支持从 finding 追溯规则和证据，但不据此宣称知识导致了该 finding。 +计数描述记录覆盖范围，不能当作规则正确率。 diff --git a/plugins/infermatrix-copilot/skills/imreview/SKILL.md b/plugins/infermatrix-copilot/skills/imreview/SKILL.md index e0241191..f265b9b9 100644 --- a/plugins/infermatrix-copilot/skills/imreview/SKILL.md +++ b/plugins/infermatrix-copilot/skills/imreview/SKILL.md @@ -9,8 +9,8 @@ Immediately update the host conversation with the pinned head SHA, current CI status, mergeability, and any early findings before reading knowledge, searching source, or running tests. Within 60 seconds, do this. Then call InferMatrixCopilot `review` once -with `mode="direct"` plus the -collected `title`, `body`, and `changed_files`. Use the embedded `quick_map` in +with `mode="direct"`, the selected `repo`, plus the +collected `title`, `body`, `changed_files`, and pinned `expected_head_sha`. Use the embedded `quick_map` in each returned `knowledge_routes` item and the bounded prose in `related_knowledge`. That prose includes source pins and explicit missing facets; verify claims at the PR head, and treat inferred tradeoffs as background. Do not open the full route file @@ -65,6 +65,37 @@ classify `subtraction_signal`. Use `none` without a minimality proof when the diff does not add or expand a helper, class, fallback, compatibility branch, or public behavior. Use `triggered` for those changes and only then provide subtraction evidence. +When the response includes `knowledge_trace.trace_id`, retain it for this review. +Pass that `trace_id`, the selected `repo`, and a stable `step_id` to subsequent +`doc_read`/`doc_search` calls; pass `trace_id` and `repo` to +`validate_direct_review` so completion also closes the trace. When the knowledge +usage tools are available, batch `record_knowledge_usage` once per meaningful +step: acknowledge only exposures actually supplied to your context using +`content_provided` with `source_exposure_id` and boundary `host_context`, then +record only rules you explicitly checked using `assessment_recorded` with the +provided document/rule identity, applicability, verification, concise rationale, +check, and evidence IDs. Register the referenced source/test evidence in the same +batch, bound to the pinned head and actual validation environment. Reuse the +existing evidence packet; do not add source reads or re-evaluate every returned +rule just for telemetry. Missing evidence stays unresolved, quick-map references +do not prove the full rule was read, and unavailable capture stays unknown. +The batch uses this shape; replace placeholders with this review's actual +references and evidence, omit `rule_id` when the page has no explicit rule ID, +and choose a verification status supported by the evidence: + +```json +[ + {"event_type":"content_provided","step_id":"review.source","exposure":{"source_exposure_id":"","boundary":"host_context"}}, + {"event_type":"evidence_registered","step_id":"review.source","evidence":{"evidence_id":"source-check-1","kind":"source","path":"","source_revision":"","summary":""}}, + {"event_type":"assessment_recorded","step_id":"review.source","assessment":{"document_id":"","rule_id":"","exposure_ids":[""],"applicability":"applicable","verification":"unresolved","rationale":"","check":"","evidence_ids":["source-check-1"]}} +] +``` + +Submit these batches with `trace_id`, the trace's selected `repo`, and an +idempotency key unique to that step's batch **before completion validation** +closes the session. A terminal partial/blocked attempt calls `finish_knowledge_trace` +with `status="interrupted"` after validation; leave it running if continuing the +same attempt. Ordinary successful validation closes it automatically. Format each actionable finding like a normal GitHub inline review comment: identify the exact path and line/hunk, then explain the concrete triggering input or call path, the observed behavior, why it matters, and the smallest diff --git a/src/infermatrix_copilot/adapters/base.py b/src/infermatrix_copilot/adapters/base.py index 98e00b7c..54f6d72f 100644 --- a/src/infermatrix_copilot/adapters/base.py +++ b/src/infermatrix_copilot/adapters/base.py @@ -13,6 +13,7 @@ import time from dataclasses import dataclass from pathlib import Path +from typing import Callable import yaml @@ -70,13 +71,16 @@ def _without_frontmatter(text: str) -> str: def render_briefing_docs(root: Path | str, docs: list[str], *, header: str = "", cap: int = _BRIEFING_CAP, - warnings: list[str] | None = None) -> str: + warnings: list[str] | None = None, + on_document: Callable[..., None] | None = None) -> str: """Concatenate the `docs` (paths relative to `root`) into one capped briefing slice, prefixed by `header`. Missing docs are skipped (never fatal); page frontmatter is stripped. Returns "" when none exist. Shared by the general and repo-specific briefing layers so both render identically.""" root = Path(root).resolve() parts: list[str] = [] + sources: list[str] = [] + offsets: list[int] = [] for rel in docs: p = (root / rel).resolve() try: @@ -90,10 +94,20 @@ def render_briefing_docs(root: Path | str, docs: list[str], *, header: str = "", warnings.append(f"briefing document missing or not Markdown: {rel}") continue text = visible_text(p.read_text(encoding="utf-8", errors="replace")) - parts.append(_without_frontmatter(text).strip()) + visible_body = _without_frontmatter(text) + parts.append(visible_body.strip()) + offsets.append(len(text) - len(visible_body) + len(visible_body) - len(visible_body.lstrip())) + sources.append(p.relative_to(root).as_posix()) if not parts: return "" body = ((header + "\n\n") if header else "") + "\n\n---\n\n".join(parts) + if on_document is not None: + position = len(header) + 2 if header else 0 + for rel, text, offset in zip(sources, parts, offsets): + included = text[:max(0, cap - position)] + on_document(document_id=rel, content=included, + truncated=len(included) < len(text), offset=offset) + position += len(text) + len("\n\n---\n\n") if len(body) > cap: body = (body[:cap] + f"\n\n...[briefing capped at {cap} chars — use doc_read for the " @@ -227,7 +241,8 @@ def capabilities(self) -> set[str]: def briefing(self, knowledge_root: Path | None = None, warnings: list[str] | None = None, - mode: str = "eco") -> str: + mode: str = "eco", + on_document: Callable[..., None] | None = None) -> str: """The repo's always-on prompt slice — the REPO-SPECIFIC part only. Reads this adapter's `knowledge.briefing_docs` (its `repos//` hard-gate rules + navigation index) from the SHARED `knowledge_root` (the vendored @@ -273,12 +288,12 @@ def scoped(items: list[str]) -> list[str]: knowledge_root, docs, header=(f"Repo-specific knowledge (source: {kn.get('source', '')}; " "open the linked deeper docs with doc_read / doc_search):"), - warnings=warnings)) + warnings=warnings, on_document=on_document)) if extra_docs: parts.append(render_briefing_docs( knowledge_root, extra_docs, header="Extended review knowledge router:", - warnings=warnings)) + warnings=warnings, on_document=on_document)) body = "\n\n".join(part for part in parts if part) if body: return body diff --git a/src/infermatrix_copilot/app/workflow_execution.py b/src/infermatrix_copilot/app/workflow_execution.py index b8b8c033..ab07aed1 100644 --- a/src/infermatrix_copilot/app/workflow_execution.py +++ b/src/infermatrix_copilot/app/workflow_execution.py @@ -135,6 +135,7 @@ def run( # cancelled through the same finalizers, and the run is left # resumable and marked `interrupted` — never `running`. restore = install_run_signal_handlers(run_dir, run_dir.name) + outcome = None try: outcome = asyncio.run( run_guarded(executor.run(playbook, state), run_dir)) @@ -145,6 +146,11 @@ def run( return self._interrupted(run_dir, trace, req) finally: restore() + from ..engine.agent_runtime.knowledge_trace import finish_knowledge_trace + + finish_knowledge_trace( + state, "complete" if outcome is not None and outcome.status == "done" + and shutdown_requested() is None else "interrupted") req = shutdown_requested() if req is not None: # the cooperative path: a step noticed the request and diff --git a/src/infermatrix_copilot/cli/entry.py b/src/infermatrix_copilot/cli/entry.py index cbc0f2e4..8196da09 100644 --- a/src/infermatrix_copilot/cli/entry.py +++ b/src/infermatrix_copilot/cli/entry.py @@ -103,6 +103,10 @@ def main(argv: list[str] | None = None) -> int: configured (and not --no-chat), else the plain command REPL. `argv` defaults to sys.argv. Returns the process exit code.""" raw = list(sys.argv[1:] if argv is None else argv) + if raw and raw[0] == "knowledge-trace": + from ..knowledge_trace.cli import main as knowledge_trace_main + + return knowledge_trace_main(raw[1:]) if raw and raw[0] == "upgrade": from ..upgrade.cli import main as upgrade_main diff --git a/src/infermatrix_copilot/engine/agent_runtime/dispatch.py b/src/infermatrix_copilot/engine/agent_runtime/dispatch.py index 874ffed0..310501bc 100644 --- a/src/infermatrix_copilot/engine/agent_runtime/dispatch.py +++ b/src/infermatrix_copilot/engine/agent_runtime/dispatch.py @@ -58,7 +58,7 @@ def render(self) -> str: treats it as data, not instructions. Returns the single prompt string.""" parts = [ "## OUTPUT CONTRACT\nYour FINAL message must be a single JSON object " - "with exactly these fields:\n" + "with these fields (fields marked OPTIONAL may be omitted):\n" + json.dumps(self.output_contract, ensure_ascii=False, indent=1), "## PERMISSIONS\n" + json.dumps(self.permissions, ensure_ascii=False, indent=1), diff --git a/src/infermatrix_copilot/engine/agent_runtime/knowledge.py b/src/infermatrix_copilot/engine/agent_runtime/knowledge.py index ff74ef90..69bb4e08 100644 --- a/src/infermatrix_copilot/engine/agent_runtime/knowledge.py +++ b/src/infermatrix_copilot/engine/agent_runtime/knowledge.py @@ -152,13 +152,17 @@ def _retrieve_memories(ctx: StepContext, query: str) -> list[str]: return hits[:3] -def _knowledge_tools(store: "_ScopedKnowledge", ctx: StepContext) -> dict[str, ToolDef]: +def _knowledge_tools(store: "_ScopedKnowledge", ctx: StepContext, + knowledge_trace=None) -> dict[str, ToolDef]: """Read-only knowledge tools + governed skill-candidate proposals.""" def skill_search(query: str, **_: Any) -> str: """Tool: return up to 5 matching skills rendered as name + description + body head, or a `(no matching skills)` sentinel.""" hits = store.find(query=query, k=5) + if knowledge_trace is not None: + knowledge_trace.external_context( + [{"name": s.name} for s in hits], "skill_search", query=query) if not hits: return "(no matching skills)" return "\n\n".join(f"# {s.name}\n{s.description}\n{s.body[:1500]}" @@ -168,6 +172,9 @@ def memory_search(query: str, **_: Any) -> str: """Tool: return the debug-memory hits for `query` as newline-joined lines, or a `(no matching debug memories)` sentinel.""" hits = _retrieve_memories(ctx, query) + if knowledge_trace is not None: + knowledge_trace.external_context( + [{"summary": h} for h in hits], "memory_search", query=query) return "\n".join(hits) or "(no matching debug memories)" def skill_update_candidate(name: str, description: str, body: str, @@ -250,7 +257,7 @@ def repo_map(query: str, **_: Any) -> str: "required": ["query"]}, repo_map)} -def _repo_docs_tool(ctx: StepContext, adapter) -> dict[str, ToolDef]: +def _repo_docs_tool(ctx: StepContext, adapter, knowledge_trace=None) -> dict[str, ToolDef]: """Shared read/search tools plus bounded implementation context. General docs are readable; related prose is selected only from the active @@ -291,30 +298,53 @@ def _repo_docs_tool(ctx: StepContext, adapter) -> dict[str, ToolDef]: session_id = adaptive.open_session(selector, source_pin=head, review_id=ctx.run_dir.name, budget=budget)["session_id"] + def adaptive_result(packet: dict, mechanism: str, query: str = "") -> str: + # The budgeted model packet owns the delivered bytes. Its document + # references contain metadata only; reading the pages again here would + # falsely attribute unbudgeted content to the model. + if knowledge_trace is not None: + knowledge_trace.adaptive_packet(packet, mechanism, query=query) + return json.dumps(packet, ensure_ascii=False) + def doc_search(query: str, repository: str | None = None, **_: Any) -> str: """Tool: grep the knowledge base's markdown for `query`; return knowledge-relative `path:line:text` matches (capped), or a sentinel.""" try: if adaptive is not None: - return json.dumps(adaptive.search(session_id, query, repository=repository), ensure_ascii=False) + return adaptive_result(adaptive.search(session_id, query, repository=repository), + "doc_search", query) hits = docs.search(query) except KnowledgeDocsError as exc: + if knowledge_trace is not None: + knowledge_trace.gap("doc_search_refused", detail=str(exc)) return f"refused: {exc}" - return "\n".join( + result = "\n".join( f"{h['path']}:{h['line']}:{h['text']}" for h in hits ) or "(no matching docs)" + if knowledge_trace is not None: + knowledge_trace.tool_result("doc_search", {"hits": hits, "query": query}, result) + return result def doc_read(path: str, offset: int = 0, repository: str | None = None, **_: Any) -> str: """Tool: read a knowledge-base doc by its knowledge-relative path (windowed 24k chars; page with offset). Refuses paths escaping the base.""" try: if adaptive is not None: - return json.dumps(adaptive.read(session_id, path, offset=offset, repository=repository), ensure_ascii=False) + return adaptive_result(adaptive.read(session_id, path, offset=offset, repository=repository), + "doc_read") page = docs.read(path, offset=offset) except FileNotFoundError: + if knowledge_trace is not None: + knowledge_trace.gap("doc_read_missing", document_id=path) return f"(no such doc: {path})" except KnowledgeDocsError as exc: + if knowledge_trace is not None: + knowledge_trace.gap("doc_read_refused", detail=str(exc)) return f"refused: {exc}" + if knowledge_trace is not None: + knowledge_trace.tool_result("doc_read", { + "path": page["path"], "offset": offset, + "truncated": page["next_offset"] is not None}, page["content"]) window = page["content"] if page["next_offset"] is not None: window += (f"\n...[truncated — call doc_read again with " @@ -330,9 +360,15 @@ def doc_related(changed_files: list[str], query: str = "", repository: str | Non model_content_sha256=packet["model_content_sha256"], model_content=packet["model_content"], documents=packet["documents"], consumed_tokens=packet["consumed_tokens"], token_accounting=packet["token_accounting"]) - return json.dumps(packet, ensure_ascii=False) - return json.dumps(docs.related(changed_files, query=query), ensure_ascii=False) + return adaptive_result(packet, "doc_related", query) + related = docs.related(changed_files, query=query) + result = json.dumps(related, ensure_ascii=False) + if knowledge_trace is not None: + knowledge_trace.tool_result("doc_related", {**related, "query": query}, result) + return result except KnowledgeDocsError as exc: + if knowledge_trace is not None: + knowledge_trace.gap("doc_related_refused", detail=str(exc)) return json.dumps({"status": "refused", "reason": str(exc), "documents": []}) s = {"type": "string"} @@ -364,7 +400,8 @@ def context_briefing(**_: Any) -> str: if adapter is not None: paths += list((adapter.manifest.get("knowledge") or {}).get("briefing_docs") or []) rows = [{"path": path} for path in dict.fromkeys(paths) if (view.root / path).is_file()] - return json.dumps(adaptive.inject(session_id, rows, purpose="general-and-owner-briefing"), ensure_ascii=False) + return adaptive_result(adaptive.inject(session_id, rows, purpose="general-and-owner-briefing"), + "briefing") def context_expand(target_tokens: int, reason: str, **_: Any) -> str: return json.dumps(adaptive.expand(session_id, target_tokens=target_tokens, reason=reason), ensure_ascii=False) diff --git a/src/infermatrix_copilot/engine/agent_runtime/knowledge_trace.py b/src/infermatrix_copilot/engine/agent_runtime/knowledge_trace.py new file mode 100644 index 00000000..71b639f2 --- /dev/null +++ b/src/infermatrix_copilot/engine/agent_runtime/knowledge_trace.py @@ -0,0 +1,425 @@ +"""Strict knowledge-use capture at the rendered prompt and API boundaries. + +The model reports assessments; only the runtime can attest which content and +tool results it supplied. Harness sessions expose the prompt handoff, while +their native file reads and internal model requests remain unobserved. +""" + +from __future__ import annotations + +import hashlib +import json +import os +from pathlib import Path +import subprocess +from typing import Any +import uuid + +from ..step import StepContext + +_STATE_KEY = "_knowledge_use_trace" + + +class StrictKnowledgeTrace: + def __init__(self, ctx: StepContext, mode: str): + from ...knowledge_trace import KnowledgeTrace + + self.ctx, self.mode = ctx, mode + self.root = Path(os.environ.get("KNOWLEDGE_TRACE_ROOT") + or ctx.run_dir / "knowledge_trace") + self.service = KnowledgeTrace(root=self.root, mode=mode) + self.error = "" + spec = ctx.state.get("task_spec") or {} + self.source_revision = str(ctx.state.get("pr_head_sha") + or spec.get("expected_head_sha") or "") + if not self.source_revision and ctx.state.get("repo_path"): + try: + result = subprocess.run( + ["git", "rev-parse", "HEAD"], cwd=ctx.state["repo_path"], + capture_output=True, text=True, timeout=5, check=False) + if result.returncode == 0: + self.source_revision = result.stdout.strip() + except (OSError, subprocess.TimeoutExpired): + pass # An unknown revision is explicit session metadata. + try: + pin = json.loads((ctx.run_dir / "knowledge.json").read_text(encoding="utf-8")) + except (OSError, ValueError): + pin = {} + from ...knowledge_view import KnowledgeView, _load_view + from ...kb_service.repo_spec import resolve_snapshot_repo + + root = Path(ctx.settings.knowledge_dir).resolve() + current = KnowledgeView.current() + if current.root.resolve() == root: + view = current + elif (root.parent / "MANIFEST.json").is_file(): + view = _load_view(str(root.parent)) + else: + view = KnowledgeView(root, f"unverified:{root}") + self.snapshot = str(pin.get("snapshot") or view.public_snapshot) + from .knowledge import _resolve_adapter + + adapter = _resolve_adapter(ctx) + subdir = str(((adapter.manifest.get("knowledge") or {}).get("repo_subdir") + if adapter else "") or "") + repository = (subdir.split("/")[1] if subdir.startswith("repos/") + else str(spec.get("repo") or "unknown").replace("_", "-")) + binding = resolve_snapshot_repo(view, str(spec.get("repo") or "")) + if binding is not None: + repository = binding.repo_id + self.repository = repository + identity = hashlib.sha256(str(ctx.run_dir.resolve()).encode()).hexdigest()[:24] + self.trace_id = "strict-" + identity + "-" + uuid.uuid4().hex[:8] + started = self.service.start_session( + repository=repository, mode="strict", + run_id=ctx.run_dir.name, trace_id=self.trace_id, + source_revision=self.source_revision, knowledge_snapshot=self.snapshot, + resource_revision=str(pin.get("tree_sha256") or view.tree_sha256 or ""), + instrumentation={"prompt_boundary": "observed", + "copilot_document_tools": "observed", + "native_file_reads": "unknown", + "applications": "host_reported"}) + ctx.state["knowledge_trace_id"] = self.trace_id + ctx.state["knowledge_trace_root"] = str(self.root) + ctx.state["knowledge_trace_status"] = "running" + if started.get("status") == "degraded": + self.failed("start_session: " + str(started.get("error", "storage unavailable"))) + ctx.trace.record("knowledge_trace", trace_id=self.trace_id, + trace_root=str(self.root), status=ctx.state["knowledge_trace_status"]) + + def call(self, method: str, *args, **kwargs): + try: + result = getattr(self.service, method)(self.trace_id, *args, **kwargs) + if isinstance(result, dict) and result.get("status") == "degraded" and result.get("error"): + self.failed(method + ": " + str(result["error"])) + return None + return result + except Exception as exc: + self.failed(f"{method}: {type(exc).__name__}: {exc}") + return None + + def failed(self, error: str): + self.error = error + self.ctx.state["knowledge_trace_status"] = "degraded" + self.ctx.trace.record("knowledge_trace_error", trace_id=self.trace_id, error=error) + self.check() + + def check(self): + if self.error and self.mode == "required": + raise RuntimeError("required knowledge trace failed: " + self.error) + + def finish(self, status: str): + self.call("finish_session", status="degraded" if self.error else status) + report = self.call("report") + if report is None: + return + from ...knowledge_trace import markdown + + try: + artifact = {**report, "trace_root": str(self.root), "trace_id": self.trace_id} + self.ctx.run_dir.mkdir(parents=True, exist_ok=True) + (self.ctx.run_dir / "KNOWLEDGE_TRACE.json").write_text( + json.dumps(artifact, ensure_ascii=False, indent=2), encoding="utf-8") + (self.ctx.run_dir / "KNOWLEDGE_TRACE.md").write_text( + markdown(report) + f"\nTrace store: `{self.root}`.\n", encoding="utf-8") + except OSError as exc: + self.failed(f"artifact_export: {type(exc).__name__}: {exc}") + self.ctx.state["knowledge_trace_status"] = "degraded" if self.error else status + + +def get_knowledge_trace(ctx: StepContext) -> StrictKnowledgeTrace | None: + """Lazy per-run initialization; ensemble steps share the same trace object.""" + if _STATE_KEY in ctx.state: + return ctx.state[_STATE_KEY] + mode = str(getattr(ctx.settings, "knowledge_trace_mode", "") + or os.environ.get("KNOWLEDGE_TRACE_MODE", "best_effort")) + if mode == "off": + ctx.state[_STATE_KEY] = None + return None + try: + trace = StrictKnowledgeTrace(ctx, mode) + except Exception as exc: + ctx.state["knowledge_trace_status"] = "degraded" + ctx.trace.record("knowledge_trace_error", error=f"{type(exc).__name__}: {exc}") + if mode == "required": + raise RuntimeError(f"required knowledge trace failed: {exc}") from exc + trace = None + ctx.state[_STATE_KEY] = trace + return trace + + +def finish_knowledge_trace(state: dict, status: str) -> None: + trace = state.get(_STATE_KEY) + if trace is not None: + trace.finish(status) + + +class StepKnowledgeTrace: + """One step's pending content, observed evidence, and optional assessments.""" + + def __init__(self, trace: StrictKnowledgeTrace, step_id: str): + self.trace, self.step_id = trace, step_id + # A playbook may invoke the same runtime step for multiple foreach + # items or retries. Local evidence names (input.*, tool.*) belong to + # one invocation, even when the caller reuses the step name or a tool + # provider reuses its call IDs. + self.invocation_id = "ki-" + uuid.uuid4().hex + self.pending: list[dict] = [] + self.seen_tools: set[str] = set() + self.evidence: dict[str, dict] = {} + self.new_references: list[dict] = [] + + def retrieve(self, entries: list[dict], mechanism: str, *, query: str = "", + status: str = "ready"): + return self.trace.call("retrieve", step_id=self.step_id, entries=entries, mechanism=mechanism, + query=query, status=status) + + def queue(self, document_id: str, content: str, *, representation="excerpt", + offset=None, truncated=False, match_text: str | list[str] | None = None, + line: int | None = None, fingerprint=True, + offset_unit="unicode_codepoints", source_length: int | None = None): + if content: + root = Path(self.trace.ctx.settings.knowledge_dir).resolve() + path = (root / document_id).resolve() + sha = "" + if fingerprint and path.is_relative_to(root) and path.is_file(): + from ...knowledge_service.lifecycle import visible_text + + try: + raw = path.read_bytes() + sha = hashlib.sha256(raw).hexdigest() + visible = visible_text(raw.decode("utf-8", errors="replace")) + if line is not None: + offset = sum(len(part) for part in visible.splitlines(keepends=True)[:line - 1]) + elif offset is None: + found = visible.find(content) + offset = found if found >= 0 and found == visible.rfind(content) else None + except OSError as exc: + self.trace.failed(f"knowledge document fingerprint unavailable: {type(exc).__name__}") + self.pending.append({"document_id": document_id, "content": content, + "representation": representation, "offset": offset, + "document_sha256": sha, + "truncated": truncated, + "offset_unit": offset_unit, "source_length": source_length, + "match_text": content if match_text is None else match_text}) + + def adaptive_packet(self, packet: dict, mechanism: str, *, query: str = ""): + """Queue only provider-rendered fragments surviving the context budget. + + The context service already verified, scoped, and metered this payload. + Reading its document references again would lose truncation and could + introduce content which never reached the model. + """ + from ...knowledge_trace.adaptive import packet_fragments + from ...knowledge_trace.references import KnowledgeTraceError, document_scope + + try: + fragments = packet_fragments(packet) + except (ValueError, TypeError, KeyError) as exc: + self.gap("adaptive_packet_invalid", detail=str(exc)) + return + accepted = [] + for fragment in fragments: + document_id = fragment["document_id"] + try: + document_scope(document_id, self.trace.repository) + except KnowledgeTraceError: + self.gap("context_cross_repository_unobserved", document_id=document_id, + repo_id=fragment.get("repo_id", ""), + detail="Context service authorized delivery; this trace supports one repository scope.") + continue + accepted.append(fragment) + self.retrieve([{"document_id": f["document_id"], + "repo_id": f.get("repo_id", ""), "source_pins": f.get("source_pins", []), + "context_session_id": packet.get("session_id", "")} + for f in accepted], mechanism, query=query, status=packet.get("status", "ready")) + for fragment in accepted: + wire = fragment["wire_content"] + # Briefings inject the fence directly; tool responses JSON-encode + # the containing model_content field once more. + self.queue(fragment["document_id"], fragment["content"], + representation="adaptive_excerpt", offset=fragment.get("offset"), + truncated=bool(fragment.get("truncated")), fingerprint=False, + offset_unit=fragment.get("offset_unit", "unicode_codepoints"), + source_length=fragment.get("source_length"), + match_text=[wire, json.dumps(wire, ensure_ascii=False)[1:-1]]) + + def briefing_document(self, *, document_id: str, content: str, truncated: bool, offset=0): + self.retrieve([{"document_id": document_id}], "briefing") + self.queue(document_id, content, representation="briefing", truncated=truncated, offset=offset) + + def gap(self, reason: str, **details): + self.trace.call("record_events", [{"event_type": "gap_recorded", + "step_id": self.step_id, "gap": {"reason": reason, **details}}]) + + def external_context(self, entries: list[dict], mechanism: str, query=""): + self.retrieve(entries, mechanism, query=query, + status="ready" if entries else "no_matches") + if entries: + self.gap("external_context_not_a_knowledge_document", mechanism=mechanism, + detail="Skill/debug-memory identity is recorded; curated document provenance is unavailable.") + + def tool_result(self, name: str, arguments: dict, result: str): + """Called by document tools; retrieval is distinct from later delivery.""" + if name == "doc_read": + path = str(arguments["path"]) + self.retrieve([{"document_id": path}], "doc_read") + self.queue(path, result, representation=("full_document" if not arguments.get("offset") + and not arguments.get("truncated") else "excerpt"), offset=arguments.get("offset", 0), + truncated=bool(arguments.get("truncated"))) + elif name == "doc_search": + hits = arguments["hits"] + self.retrieve([{"document_id": h["path"], "line": h["line"]} for h in hits], + name, query=arguments.get("query", ""), + status="ready" if hits else "no_matches") + for hit in hits: + self.queue(hit["path"], hit["text"], representation="search_hit", + match_text=result, line=hit["line"]) + elif name == "doc_related": + documents = arguments.get("documents", []) + self.retrieve([{"document_id": d["path"], "reason": d.get("match_reason", "")} + for d in documents], name, query=arguments.get("query", ""), + status=arguments.get("status", "ready")) + for doc in documents: + content = str(doc.get("content") or "") + # Related context is encoded as JSON inside the prompt/tool + # result; preserve its exact decoded excerpt and match wire text. + encoded = json.dumps(content, ensure_ascii=False)[1:-1] + self.queue(doc["path"], content, representation="related_excerpt", + truncated=bool(doc.get("more_available")), match_text=encoded) + + def observe(self, messages: list[dict], boundary="model_request"): + self.trace.check() + revision = {"source_revision": "", "workspace_state": "unknown", "commit_verified": False} + repo = self.trace.ctx.state.get("repo_path") + if repo: + try: + result = subprocess.run( + ["git", "status", "--porcelain=v2", "--branch", "--untracked-files=normal"], + cwd=repo, capture_output=True, text=True, timeout=5, check=False) + if result.returncode == 0: + lines = result.stdout.splitlines() + head = next((line.removeprefix("# branch.oid ") for line in lines + if line.startswith("# branch.oid ")), "") + dirty = any(line and not line.startswith("#") for line in lines) + revision.update(observed_head=head, workspace_state="dirty" if dirty else "clean", + source_revision=head if not dirty and len(head) == 40 else "") + except (OSError, subprocess.TimeoutExpired): + pass + for evidence in self.evidence.values(): + if "workspace_state" not in evidence: + evidence.update(revision) + texts: list[str] = [] + for message in messages: + content = message.get("content", "") + if isinstance(content, str): + texts.append(content) + continue + for block in content if isinstance(content, list) else []: + if block.get("type") == "tool_result": + result = str(block.get("content", "")) + texts.append(result) + tool_id = str(block.get("tool_use_id", "")) + if tool_id and tool_id not in self.seen_tools: + self.seen_tools.add(tool_id) + self.evidence["tool." + tool_id] = { + "kind": "tool", "tool_call_id": tool_id, "content": result, **revision} + elif block.get("type") == "text": + texts.append(str(block.get("text", ""))) + remaining = [] + for item in self.pending: + needles = item["match_text"] if isinstance(item["match_text"], list) else [item["match_text"]] + # The review briefing escapes '<' inside related context. + if any(needle in text or needle.replace("<", "\\u003c") in text + for text in texts for needle in needles): + fields = {k: v for k, v in item.items() if k != "match_text"} + recorded = self.trace.call("provide", step_id=self.step_id, boundary=boundary, + snapshot=self.trace.snapshot, **fields) + if recorded: + result = recorded.get("result", {}) + for ref in [{"knowledge_ref_id": result.get("knowledge_ref_id")}, + *result.get("rule_refs", [])]: + self.new_references.append({"document_id": item["document_id"], + "knowledge_ref_id": ref.get("knowledge_ref_id"), + "rule_id": ref.get("rule_id", ""), + "exposure_ids": [result.get("exposure_id")]}) + else: + remaining.append(item) + self.pending = remaining + + def assessments(self, output: dict): + assessments = output.get("knowledge_assessments") + if not assessments: + return + if not isinstance(assessments, list): + self.trace.call("record_events", [{"event_type": "gap_recorded", + "step_id": self.step_id, "gap": {"reason": "knowledge_assessments must be a list"}}]) + return + events, registered = [], set() + decisions = {} + for index, finding in enumerate(output.get("findings") or []): + identity = str(finding.get("finding_id") or finding.get("id") or f"finding:{index}") \ + if isinstance(finding, dict) else f"finding:{index}" + decisions[identity] = {"decision_id": identity, "finding": finding} + for assessment in assessments: + if not isinstance(assessment, dict): + self.gap("invalid_knowledge_assessment", detail="assessment must be an object") + continue + if not isinstance(assessment.get("evidence_ids", []), list) or any( + not isinstance(eid, str) for eid in assessment.get("evidence_ids", [])): + self.gap("invalid_knowledge_assessment", detail="evidence_ids must be a list of strings") + continue + updated = dict(assessment) + updated["invocation_id"] = self.invocation_id + decision_ids = assessment.get("decision_ids") or [] + if not isinstance(decision_ids, list) or any(not isinstance(did, str) for did in decision_ids): + self.gap("invalid_decision_ids", detail="decision_ids must be a list of strings") + decision_ids = [] + missing = [did for did in decision_ids if did not in decisions] + if missing: + self.gap("unknown_decision_reference", decision_ids=missing) + updated["decision_ids"] = [did for did in decision_ids if did in decisions] + updated["decision_refs"] = [decisions[did] for did in updated["decision_ids"]] + evidence_key = lambda eid: "kevidence-" + hashlib.sha256( + (self.invocation_id + ":" + str(eid)).encode()).hexdigest()[:32] + updated["evidence_ids"] = [evidence_key(eid) + for eid in assessment.get("evidence_ids", [])] + for local_id in assessment.get("evidence_ids", []): + evidence_id = evidence_key(local_id) + if local_id in self.evidence and evidence_id not in registered: + registered.add(evidence_id) + events.append({"event_type": "evidence_registered", "step_id": self.step_id, + "evidence": {**self.evidence[local_id], "evidence_id": evidence_id, + "invocation_id": self.invocation_id, + "evidence_origin": "captured"}}) + events.append({"event_type": "assessment_recorded", "step_id": self.step_id, + "assessment": {**updated, "reported_by": "host"}}) + if events: + for start in range(0, len(events), 100): + self.trace.call("record_events", events[start:start + 100]) + + def wrap_llm(self, llm): + step_trace = self + + class ObservedLLM: + def __getattr__(self, name: str) -> Any: + return getattr(llm, name) + + def create(self, **kwargs): + step_trace.observe(kwargs.get("messages", [])) + if step_trace.new_references and kwargs.get("messages"): + references = step_trace.new_references[:32] + if len(step_trace.new_references) > len(references): + step_trace.gap("reference_map_truncated", supplied=len(references), + available=len(step_trace.new_references)) + note = "Knowledge trace references (optional assessments may cite these exact IDs):\n" + json.dumps( + references, ensure_ascii=False) + step_trace.new_references.clear() + message = kwargs["messages"][-1] + if isinstance(message.get("content"), list): + message["content"].append({"type": "text", "text": note}) + else: + message["content"] = str(message.get("content", "")) + "\n\n" + note + return llm.create(**kwargs) + + return ObservedLLM() diff --git a/src/infermatrix_copilot/engine/agent_runtime/runner.py b/src/infermatrix_copilot/engine/agent_runtime/runner.py index 2193f143..782c62fa 100644 --- a/src/infermatrix_copilot/engine/agent_runtime/runner.py +++ b/src/infermatrix_copilot/engine/agent_runtime/runner.py @@ -29,6 +29,7 @@ _retrieve_skills, ) from .utils import _build_evidence, _coerce_output, _permissions_view, _to_step_result +from .knowledge_trace import StepKnowledgeTrace, get_knowledge_trace async def run_agent_step( @@ -87,6 +88,25 @@ async def run_agent_step( if hasattr(ctx.llm, "for_target") else ctx.llm) contract = {**BASE_OUTPUT_SCHEMA, **(output_extension or {})} capped, refs = _build_evidence(ctx, evidence) + try: + run_knowledge_trace = get_knowledge_trace(ctx) + except RuntimeError as exc: + return StepResult(False, FailureKind.BLOCKED, str(exc)), {} + knowledge_trace = (StepKnowledgeTrace(run_knowledge_trace, step_name) + if run_knowledge_trace is not None else None) + if knowledge_trace is not None: + contract["knowledge_assessments"] = ( + "OPTIONAL list (may omit): {knowledge_ref_id and exposure_ids: exact IDs from the trace reference map, " + "or document_id: knowledge-relative page path, " + "rule_id: exact rule heading ID if applicable, applicability: " + "applicable|not_applicable|unknown, verification: supported|contradicted|unresolved, " + "rationale: brief checkable reason, check: concrete check when applicable, " + "evidence_ids: list of input. or tool. actually observed, " + "decision_ids: optional finding: or finding object ID}. Assess only rules you examined; " + "supported/contradicted requires evidence. Do not report internal deliberation.") + knowledge_trace.evidence.update({"input." + name: { + "kind": "source", "input_name": name, "content": content} + for name, content in capped.items()}) query = " ".join(str(x) for x in [ spec.get("kind", ""), step_name, @@ -94,11 +114,16 @@ async def run_agent_step( *list(evidence.keys())[:5]]) skills, store = _retrieve_skills(ctx, query) memories = _retrieve_memories(ctx, query) - knowledge = _knowledge_tools(store, ctx) + if knowledge_trace is not None: + knowledge_trace.external_context([{"name": s["name"]} for s in skills], + "skill_injection", query) + knowledge_trace.external_context([{"summary": m} for m in memories], + "memory_injection", query) + knowledge = _knowledge_tools(store, ctx, knowledge_trace) adapter = _resolve_adapter(ctx) all_extra = {**knowledge, **_repo_map_tool(ctx, adapter), - **_repo_docs_tool(ctx, adapter), **(extra_tools or {})} + **_repo_docs_tool(ctx, adapter, knowledge_trace), **(extra_tools or {})} # Briefing = GENERAL knowledge (shared, cross-repo) + this repo's # REPO-SPECIFIC slice — both from the shared knowledge base. Deeper docs are # pulled on demand via the doc tools (all_extra above). @@ -122,10 +147,12 @@ async def run_agent_step( kroot, ctx.settings.knowledge_general_docs, header="General knowledge (cross-repo; open deeper " "docs with doc_read / doc_search):", - warnings=briefing_warnings)) + warnings=briefing_warnings, + on_document=knowledge_trace.briefing_document if knowledge_trace else None)) if adapter is not None: parts.append(adapter.briefing( - kroot, warnings=briefing_warnings, mode=tier)) + kroot, warnings=briefing_warnings, mode=tier, + on_document=knowledge_trace.briefing_document if knowledge_trace else None)) briefing = "\n\n".join(p for p in parts if p) for warning in briefing_warnings: ctx.trace.record("knowledge_warning", step=step_name, @@ -202,8 +229,15 @@ async def run_agent_step( prompt = dispatch_ctx.render() if guidance: prompt += f"\n\n## STEP GUIDANCE\n{guidance}" - if harness_member is not None or (target.kind == "harness" - and llm_override is None): + harness_backend = harness_member is not None or (target.kind == "harness" and llm_override is None) + if knowledge_trace is not None: + if harness_backend: + knowledge_trace.gap("harness_model_requests_unobserved", + detail="Prompt handoff observed; native file reads and internal model requests are unknown.") + knowledge_trace.observe([{"content": prompt}], boundary="host_context") + else: + step_llm = knowledge_trace.wrap_llm(step_llm) + if harness_backend: # Harness backend (doc/features/provider-registry.md): the vendor CLI owns # the tool loop for this whole step — same prompt bundle and output # contract, builtin tools served back through the MCP bridge under @@ -228,6 +262,8 @@ async def run_agent_step( output = _coerce_output(outcome.text, ctx, contract) if output is not None: output.setdefault("_tools_used", outcome.tools_used[:40]) + if knowledge_trace is not None: + knowledge_trace.assessments(output) ctx.trace.record( "agent_output", step=step_name, ok=output is not None, tool_calls=outcome.tool_calls, tools_used=outcome.tools_used[:30], diff --git a/src/infermatrix_copilot/knowledge_trace/__init__.py b/src/infermatrix_copilot/knowledge_trace/__init__.py new file mode 100644 index 00000000..64889b4e --- /dev/null +++ b/src/infermatrix_copilot/knowledge_trace/__init__.py @@ -0,0 +1,7 @@ +"""Auditable knowledge retrieval, delivery and explicit application records.""" + +from .recorder import KnowledgeTrace +from .references import KnowledgeTraceError, extract_rule_sections +from .report import markdown + +__all__ = ["KnowledgeTrace", "KnowledgeTraceError", "extract_rule_sections", "markdown"] diff --git a/src/infermatrix_copilot/knowledge_trace/adaptive.py b/src/infermatrix_copilot/knowledge_trace/adaptive.py new file mode 100644 index 00000000..ca4c3317 --- /dev/null +++ b/src/infermatrix_copilot/knowledge_trace/adaptive.py @@ -0,0 +1,51 @@ +"""Observe exact adaptive provider packets without reading their sources again.""" + +from __future__ import annotations + +import hashlib +import json +import re + +from .references import KnowledgeTraceError + +_FENCE = re.compile(r"\n\n(.*?)\n\n", re.DOTALL) + + +def packet_fragments(packet: dict) -> list[dict]: + """Extract delivered text and its conservative source coordinates. + + ``documents`` contains references, not the budgeted delivery. Only complete + provider-rendered fences in ``model_content`` can produce exposures. A + continuation or a transformed/search fragment has unknown source offsets. + ``wire_content`` is retained for matching the final Strict request boundary. + """ + content = packet.get("model_content", "") + if not isinstance(content, str): + raise KnowledgeTraceError("adaptive model_content must be a string") + expected = packet.get("model_content_sha256") + if expected and expected != hashlib.sha256(content.encode("utf-8")).hexdigest(): + raise KnowledgeTraceError("adaptive model_content hash mismatch") + fragments = [] + matches = list(_FENCE.finditer(content)) + # A malformed packet must not silently claim an empty, complete capture. + if _FENCE.sub("", content).strip(): + raise KnowledgeTraceError("adaptive model_content has unframed content") + for match in matches: + try: + fragment = json.loads(match[1]) + except (ValueError, TypeError) as exc: + raise KnowledgeTraceError("adaptive source fragment is not valid JSON") from exc + if (not isinstance(fragment, dict) or not isinstance(fragment.get("path"), str) + or not fragment["path"] or not isinstance(fragment.get("content"), str)): + raise KnowledgeTraceError("adaptive source fragment must name its path and content") + offset = fragment.get("offset") + if (isinstance(offset, bool) or not isinstance(offset, int) or offset < 0 + or fragment.get("continuation")): + offset = None + fragments.append({**fragment, "document_id": fragment["path"], + "offset": offset, "offset_unit": "unicode_codepoints", + "source_length": len(fragment["content"]), + "truncated": bool(fragment.get("partial")), + "source_pins": fragment.get("source_pins", []), + "representation": "excerpt", "wire_content": match[0]}) + return fragments diff --git a/src/infermatrix_copilot/knowledge_trace/cli.py b/src/infermatrix_copilot/knowledge_trace/cli.py new file mode 100644 index 00000000..df9b218e --- /dev/null +++ b/src/infermatrix_copilot/knowledge_trace/cli.py @@ -0,0 +1,91 @@ +"""Read and export knowledge-use records without loading a model backend.""" + +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path +import sys +import tempfile + + +def _write_export(destination: str, content: str) -> None: + if destination == "-": + print(content) + return + path = Path(destination).expanduser() + path.parent.mkdir(parents=True, exist_ok=True) + temporary = None + try: + with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", dir=path.parent, + prefix=f".{path.name}.", delete=False) as handle: + temporary = Path(handle.name) + handle.write(content) + handle.write("\n") + os.replace(temporary, path) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + + +def main(argv: list[str] | None = None) -> int: + from . import KnowledgeTrace, KnowledgeTraceError, markdown + + parser = argparse.ArgumentParser(prog="infermatrix-copilot knowledge-trace", + description="Inspect per-step knowledge and decision evidence.") + parser.add_argument("--root", type=Path, help="Trace store (Strict: /knowledge_trace)") + commands = parser.add_subparsers(dest="command", required=True) + for name in ("show", "export", "query"): + sub = commands.add_parser(name) + sub.add_argument("--root", type=Path, default=argparse.SUPPRESS, + help="Trace store; defaults to KNOWLEDGE_TRACE_ROOT") + if name == "query": + sub.add_argument("--repo", required=True) + sub.add_argument("--rule", default="") + sub.add_argument("--limit", type=int, default=100) + else: + sub.add_argument("trace_id") + if name == "show": + sub.add_argument("--json", action="store_true") + sub.add_argument("--cursor") + sub.add_argument("--limit", type=int, default=100) + else: + sub.add_argument("--format", choices=("json", "markdown"), default="markdown") + sub.add_argument("--out", required=True, help="Output file, or - for stdout") + args = parser.parse_args(argv) + try: + # Inspecting existing records works even when collection is disabled. + recorder = KnowledgeTrace(root=args.root, mode="required") + if args.command == "query": + result = recorder.query(repository=args.repo, rule_id=args.rule, limit=args.limit) + print(json.dumps(result, ensure_ascii=False, indent=2)) + else: + options = ({"cursor": args.cursor, "limit": args.limit} + if args.command == "show" else {}) + report = recorder.report(args.trace_id, **options) + if args.command == "export": + # Export all ordered events, rather than silently stopping at + # the report API's first page. Session size is bounded upstream. + events = list(report["events"]) + cursor = report.get("next_cursor") + while cursor is not None: + page = recorder.report(args.trace_id, cursor=cursor) + events.extend(page["events"]) + cursor = page.get("next_cursor") + report["events"] = events + report["next_cursor"] = None + as_json = args.json if args.command == "show" else args.format == "json" + content = json.dumps(report, ensure_ascii=False, indent=2) if as_json else markdown(report) + if args.command == "show": + print(content) + else: + _write_export(args.out, content) + return 0 + except (KnowledgeTraceError, OSError, ValueError, KeyError) as exc: + print(f"knowledge-trace: {exc}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/infermatrix_copilot/knowledge_trace/recorder.py b/src/infermatrix_copilot/knowledge_trace/recorder.py new file mode 100644 index 00000000..7f0e431b --- /dev/null +++ b/src/infermatrix_copilot/knowledge_trace/recorder.py @@ -0,0 +1,450 @@ +"""Append-only knowledge-use protocol backed by the shared trace/1 store. + +The root lock serializes duplicate detection and append across processes. No +knowledge, source or model work is performed by this module. +""" + +from __future__ import annotations + +import copy +import os +import re +import sqlite3 +import uuid +from pathlib import Path + +from ..trace_store import TraceStore, file_lock +from .references import (KnowledgeTraceError, digest, document_scope, + extract_rule_sections, knowledge_reference) +from .report import project + +SCHEMA = "knowledge-use/1" +MAX_EVENTS = 10_000 +MAX_CONTENT = 1_048_576 +EVENT_TYPES = {"session_started", "retrieval_completed", "content_provided", "evidence_registered", + "assessment_recorded", "gap_recorded", "session_finished"} +BOUNDARIES = {"provider_response", "host_context", "model_request"} +_IDENTIFIER = re.compile(r"^[A-Za-z0-9_.:-]{1,200}$") + + +def _identifier(value: str, name: str) -> str: + if not isinstance(value, str) or not _IDENTIFIER.fullmatch(value): + raise KnowledgeTraceError(f"{name} must be a bounded identifier") + return value + + +def _rule_ids(values) -> list[str]: + if values is None: + return [] + if not isinstance(values, list) or any(not isinstance(v, str) or not v or len(v) > 100 for v in values): + raise KnowledgeTraceError("rule_ids must be a list of bounded strings") + return list(dict.fromkeys(values)) + + +class KnowledgeTrace: + """Collect bounded, versioned knowledge-use sessions. + + Invalid claims always raise :class:`KnowledgeTraceError`. Storage failures + return ``status=degraded`` in best_effort mode and raise in required mode. + The optional constructor ``mode`` is the recording policy, while + ``start_session(mode=...)`` selects direct or strict execution. + """ + + def __init__(self, root: Path | str | None = None, mode: str | None = None): + self.mode = mode or os.environ.get("KNOWLEDGE_TRACE_MODE", "best_effort") + if self.mode not in {"off", "best_effort", "required"}: + raise KnowledgeTraceError("KNOWLEDGE_TRACE_MODE must be off, best_effort or required") + self.root = Path(root or os.environ.get("KNOWLEDGE_TRACE_ROOT") or + Path.home() / ".infermatrix-copilot" / "knowledge-traces").expanduser() + self.store = TraceStore(self.root) + self.producer_id = "producer-" + uuid.uuid4().hex + + @property + def enabled(self) -> bool: + return self.mode != "off" + + def _operate(self, trace_id, operation): + if not self.enabled: + return {"trace_id": "", "status": "off"} + try: + self.root.mkdir(parents=True, exist_ok=True, mode=0o700) + with file_lock(self.root / "knowledge.lock") as held: + if not held: + raise OSError("knowledge trace writer lock timed out") + # The JSONL archive is authoritative even after deleting an index. + if not self.store.index_path.exists() and any((self.root / "records").glob("*.jsonl")): + self.store.rebuild_index(offline_confirmed=True) + return operation() + except (OSError, sqlite3.Error) as exc: + if self.mode == "required": + raise KnowledgeTraceError(f"required knowledge recording failed: {type(exc).__name__}") from exc + return {"trace_id": trace_id, "status": "degraded", "error": type(exc).__name__, + "gaps": [{"reason": "trace_storage_unavailable"}]} + + def _records(self, trace_id: str) -> list[dict]: + _identifier(trace_id, "trace_id") + records = self.store.query(kind="knowledge_event", changeset_id=trace_id, limit=MAX_EVENTS + 8) + if len(records) > MAX_EVENTS + 2: + raise KnowledgeTraceError("knowledge trace exceeds its bounded event budget") + return sorted(records, key=lambda r: r["result"].get("session_seq", 0)) + + def _session(self, records: list[dict]) -> dict: + if not records or records[0]["result"].get("event_type") != "session_started": + raise KnowledgeTraceError("unknown knowledge trace session") + return records[0]["result"]["session"] + + def _append(self, trace_id: str, session: dict, records: list[dict], event: dict) -> dict: + event_type = event.get("event_type") + if event_type not in EVENT_TYPES: + raise KnowledgeTraceError("unsupported knowledge trace event_type") + event = copy.deepcopy(event) + step_id = event.get("step_id", "") + # Step names are opaque engine labels, not paths or entity IDs. MoA + # names include '#moa//' and must retain that identity. + if (not isinstance(step_id, str) or len(step_id) > 200 + or any(ord(character) < 32 or ord(character) == 127 for character in step_id)): + raise KnowledgeTraceError("step_id must be a bounded printable string") + event_id = _identifier(event.pop("event_id", "ke-" + uuid.uuid4().hex), "event_id") + # Fingerprint the declaration before automatically generated identity fields. + fingerprint = digest(event) + prior = next((r for r in records if r["result"].get("event_id") == event_id), None) + if prior: + if prior["result"].get("declaration_sha256") != fingerprint: + raise KnowledgeTraceError("event_id conflicts with an immutable prior event") + return prior + if records and records[-1]["result"].get("event_type") == "session_finished": + raise KnowledgeTraceError("knowledge trace session is already finished") + if len(records) >= MAX_EVENTS and event_type not in {"session_finished", "gap_recorded"}: + raise KnowledgeTraceError("knowledge trace event budget exhausted") + parent_ids = event.get("parent_event_ids", []) + known_ids = {r["result"]["event_id"] for r in records} + if not isinstance(parent_ids, list) or any(p not in known_ids for p in parent_ids): + raise KnowledgeTraceError("parent_event_ids must refer to existing events in this session") + supersedes = event.get("supersedes") + if supersedes and supersedes not in known_ids: + raise KnowledgeTraceError("supersedes must refer to an existing event") + if supersedes: + replaced = next(r["result"] for r in records if r["result"]["event_id"] == supersedes) + if (replaced["event_type"] != event_type + or event_type not in {"assessment_recorded", "gap_recorded"}): + raise KnowledgeTraceError("only assessments and gaps may supersede events of the same type; observations and evidence are immutable") + producer_id = event.pop("producer_id", self.producer_id) + _identifier(producer_id, "producer_id") + previous = max((r["result"].get("producer_seq", 0) for r in records + if r["result"].get("producer_id") == producer_id), default=0) + producer_seq = event.pop("producer_seq", previous + 1) + if isinstance(producer_seq, bool) or not isinstance(producer_seq, int) or producer_seq <= previous: + raise KnowledgeTraceError("producer_seq must increase monotonically") + result = {"schema": SCHEMA, "trace_id": trace_id, "event_id": event_id, + "producer_id": producer_id, "producer_seq": producer_seq, + "session_seq": max((r["result"].get("session_seq", 0) for r in records), default=0) + 1, + "declaration_sha256": fingerprint, **event} + rule_ids = list(dict.fromkeys(self._event_rules(result))) + record = self.store.append("knowledge_event", context={ + "run_id": session.get("run_id") or trace_id, "changeset_id": trace_id, + "step": event.get("step_id", ""), "repo": session["repository"], "rule_ids": rule_ids, + }, result=result) + records.append(record) + return record + + @staticmethod + def _event_rules(event): + for entry in event.get("entries", []): + yield from entry.get("rule_ids", []) + if entry.get("rule_id"): + yield entry["rule_id"] + exposure = event.get("exposure", {}) + yield from exposure.get("rule_ids", []) + if event.get("assessment", {}).get("rule_id"): + yield event["assessment"]["rule_id"] + + def start_session(self, repository: str, mode: str, run_id: str = "", review_context_id: str = "", + source_revision: str = "", knowledge_snapshot: str = "", resource_revision: str = "", + instrumentation: dict | None = None, trace_id: str = "") -> dict: + _identifier(repository, "repository") + if mode not in {"direct", "strict"}: + raise KnowledgeTraceError("session mode must be direct or strict") + if instrumentation is not None and not isinstance(instrumentation, dict): + raise KnowledgeTraceError("instrumentation must be an object") + trace_id = _identifier(trace_id or "kt-" + uuid.uuid4().hex, "trace_id") + session = {"trace_id": trace_id, "repository": repository, "mode": mode, "run_id": run_id, + "review_context_id": review_context_id, "source_revision": source_revision, + "knowledge_snapshot": knowledge_snapshot, "resource_revision": resource_revision, + "instrumentation": instrumentation or {}, "status": "running"} + + def write(): + records = self._records(trace_id) + if records: + if self._session(records) != session: + raise KnowledgeTraceError("trace_id conflicts with an immutable session") + else: + self._append(trace_id, session, records, {"event_type": "session_started", "session": session}) + return {**session, "status": "running"} + return self._operate(trace_id, write) + + def retrieve(self, trace_id: str, *, step_id: str, entries: list[dict], mechanism: str, + query: str = "", reason: str = "", status: str = "ready") -> dict: + if not isinstance(entries, list) or len(entries) > 1000: + raise KnowledgeTraceError("retrieval entries must be a bounded list") + + def write(): + records = self._records(trace_id) + session = self._session(records) + for entry in entries: + path = entry.get("document_id") or entry.get("path") + if path: + document_scope(path, session["repository"]) + _rule_ids(entry.get("rule_ids")) + return self._append(trace_id, session, records, {"event_type": "retrieval_completed", + "step_id": step_id, "entries": entries, "mechanism": mechanism, + "query": query, "reason": reason, "status": status}) + return self._operate(trace_id, write) + + def provide(self, trace_id: str, *, step_id: str, document_id: str, content: str, boundary: str, + representation: str = "excerpt", rule_ids: list[str] | None = None, offset: int | None = 0, + offset_unit: str = "unicode_codepoints", truncated: bool = False, snapshot: str = "", + document_sha256: str = "", parent_event_ids: list[str] | None = None, + source_length: int | None = None) -> dict: + if not isinstance(content, str) or len(content.encode("utf-8")) > MAX_CONTENT: + raise KnowledgeTraceError("knowledge exposure content exceeds the 1 MiB budget") + if boundary not in BOUNDARIES: + raise KnowledgeTraceError("unsupported knowledge delivery boundary") + if offset is not None and (isinstance(offset, bool) or not isinstance(offset, int) or offset < 0): + raise KnowledgeTraceError("offset must be a non-negative integer or unknown (None)") + if source_length is not None and (isinstance(source_length, bool) + or not isinstance(source_length, int) or source_length < 0): + raise KnowledgeTraceError("source_length must be a non-negative integer") + if offset_unit not in {"unicode_codepoints", "utf8_bytes"}: + raise KnowledgeTraceError("unsupported offset_unit") + ids = _rule_ids(rule_ids) + + def write(): + records = self._records(trace_id) + session = self._session(records) + effective_snapshot = snapshot or session.get("knowledge_snapshot", "") + if snapshot and session.get("knowledge_snapshot") and snapshot != session["knowledge_snapshot"]: + raise KnowledgeTraceError("knowledge snapshot differs from the immutable session") + document_scope(document_id, session["repository"]) + content_sha = digest(content) + payload_ref = self.store.put_blob(content) + # A pinned snapshot supplies a stable document version across + # paginated reads. With neither hash nor snapshot, only the visible + # payload version is known; never claim a partial hash is a file hash. + source_hash = document_sha256 + ref = knowledge_reference(document_id, session["repository"], effective_snapshot, source_hash) + if not effective_snapshot and not source_hash: + ref = knowledge_reference(document_id, session["repository"], "visible:" + content_sha, "") + effective_snapshot = ref["snapshot_id"] + sections = extract_rule_sections(content) + section_ids = [s["rule_id"] for s in sections] + ids_all = list(dict.fromkeys([*ids, *section_ids])) + rule_refs = [] + for rule_id in ids_all: + section = next((s for s in sections if s["rule_id"] == rule_id), None) + rule_ref = knowledge_reference(document_id, session["repository"], effective_snapshot, source_hash, + rule_id=rule_id, section_sha256=section["section_sha256"] if section else "") + complete = bool(section and section["has_body"] and representation in {"full_document", "full_section"} + and (not truncated or section["closed_by_heading"])) + rule_refs.append({**rule_ref, "coverage": "body_complete" if complete else + "body_excerpt" if section and section["has_body"] else "reference_only"}) + length = (source_length if source_length is not None else + len(content.encode("utf-8")) if offset_unit == "utf8_bytes" else len(content)) + exposure_id = "kx-" + uuid.uuid4().hex + exposure = {"exposure_id": exposure_id, "knowledge_ref_id": ref["knowledge_ref_id"], + "reference": ref, "document_id": document_id, "rule_ids": ids_all, "rule_refs": rule_refs, + "step_id": step_id, "boundary": boundary, "representation": representation, + "payload_ref": payload_ref, "visible_text_sha256": content_sha, + "archived_sha256": payload_ref.removeprefix("sha256:"), + "included_ranges": [] if offset is None else [[offset, offset + length]], + "range_status": "unknown" if offset is None else "known", "offset_unit": offset_unit, + "truncated": bool(truncated), "delivery_status": "observed", + "reported_by": "provider"} + return self._append(trace_id, session, records, {"event_type": "content_provided", "step_id": step_id, + "exposure_id": exposure_id, "knowledge_ref_id": ref["knowledge_ref_id"], "rule_refs": rule_refs, + "exposure": exposure, "parent_event_ids": parent_event_ids or []}) + return self._operate(trace_id, write) + + @staticmethod + def _entities(records): + refs, exposures, evidence = {}, {}, {} + for record in records: + result = record["result"] + exposure = result.get("exposure") + if exposure: + exposures[exposure["exposure_id"]] = exposure + ref = exposure["reference"] + refs[ref["knowledge_ref_id"]] = ref + for rule in exposure.get("rule_refs", []): + refs[rule["knowledge_ref_id"]] = rule + if result.get("evidence"): + item = result["evidence"] + evidence[item["evidence_id"]] = item + return refs, exposures, evidence + + def _validate_declaration(self, event: dict, records: list[dict], session: dict) -> dict: + event = copy.deepcopy(event) + event_type = event.get("event_type") + if event_type not in {"assessment_recorded", "evidence_registered", "content_provided", "gap_recorded"}: + raise KnowledgeTraceError("record_events accepts usage declarations and gaps only") + refs, exposures, evidence = self._entities(records) + if event_type == "gap_recorded": + if not isinstance(event.get("gap"), dict) or not event["gap"].get("reason"): + raise KnowledgeTraceError("gap_recorded requires a gap reason") + elif event_type == "evidence_registered": + item = event.get("evidence") + if not isinstance(item, dict): + raise KnowledgeTraceError("evidence_registered requires evidence") + item.setdefault("evidence_id", "kevidence-" + digest(item)[:32]) + _identifier(item["evidence_id"], "evidence_id") + path = item.get("path", "") + if path and (path.startswith(("/", "\\")) or ".." in path.replace("\\", "/").split("/")): + raise KnowledgeTraceError("evidence path must be repository-relative") + revision = item.get("source_revision", "") + expected = session.get("source_revision", "") + item["revision_status"] = "stale" if revision and expected and revision != expected else \ + "matched" if revision and expected else "unknown" + item.setdefault("evidence_origin", "host_reported") + if "content" in item: + content = item.pop("content") + if not isinstance(content, str) or len(content.encode("utf-8")) > MAX_CONTENT: + raise KnowledgeTraceError("evidence content exceeds the 1 MiB budget") + item["payload_ref"] = self.store.put_blob(content) + if item["evidence_id"] in evidence and evidence[item["evidence_id"]] != item: + raise KnowledgeTraceError("evidence_id conflicts with immutable evidence") + elif event_type == "content_provided": + item = event.get("exposure") + if not isinstance(item, dict) or item.get("boundary") != "host_context": + raise KnowledgeTraceError("host delivery receipts must name the host_context boundary") + source = exposures.get(item.get("source_exposure_id")) + if source is None: + raise KnowledgeTraceError("host receipt references an unknown source_exposure_id") + allowed = {"source_exposure_id", "boundary", "exposure_id", "reported_by", "delivered_at"} + if set(item) - allowed: + raise KnowledgeTraceError("host delivery receipts cannot override source content identity") + exposure_id = item.get("exposure_id") or "kx-" + digest(event)[:32] + _identifier(exposure_id, "exposure_id") + event["exposure"] = {**source, **item, "exposure_id": exposure_id, + "reported_by": "host", "delivery_status": "host_reported"} + if exposure_id in exposures and exposures[exposure_id] != event["exposure"]: + raise KnowledgeTraceError("exposure_id conflicts with immutable exposure") + event["exposure_id"] = exposure_id + event["knowledge_ref_id"] = source["knowledge_ref_id"] + event["rule_refs"] = source.get("rule_refs", []) + else: + item = event.get("assessment") + if not isinstance(item, dict): + raise KnowledgeTraceError("assessment_recorded requires assessment") + if not item.get("knowledge_ref_id") and item.get("document_id"): + document_scope(item["document_id"], session["repository"]) + candidates = [ref for ref in refs.values() if ref["document_id"] == item["document_id"] + and ref.get("rule_id", "") == item.get("rule_id", "")] + if len(candidates) != 1: + raise KnowledgeTraceError("assessment requires an unambiguous versioned knowledge reference") + item["knowledge_ref_id"] = candidates[0]["knowledge_ref_id"] + ref = refs.get(item.get("knowledge_ref_id")) + if ref is None: + raise KnowledgeTraceError("assessment references unknown knowledge_ref_id") + item.setdefault("document_id", ref["document_id"]) + item.setdefault("rule_id", ref.get("rule_id", "")) + if item["document_id"] != ref["document_id"] or item["rule_id"] != ref.get("rule_id", ""): + raise KnowledgeTraceError("assessment document/rule identity conflicts with knowledge_ref_id") + if item.get("applicability") not in {"applicable", "not_applicable", "unknown"}: + raise KnowledgeTraceError("assessment requires explicit applicability") + if item.get("verification") not in {"supported", "contradicted", "unresolved"}: + raise KnowledgeTraceError("assessment requires explicit verification") + if not isinstance(item.get("rationale"), str) or not item["rationale"].strip(): + raise KnowledgeTraceError("assessment requires a brief rationale") + if item["applicability"] == "applicable" and not str(item.get("check", "")).strip(): + raise KnowledgeTraceError("applicable assessment requires the check performed") + if item["verification"] in {"supported", "contradicted"} and not item.get("evidence_ids"): + raise KnowledgeTraceError("verified assessments require evidence references") + for exposure_id in item.get("exposure_ids", []): + exposure = exposures.get(exposure_id) + if exposure is None: + raise KnowledgeTraceError("assessment references unknown exposure_id") + allowed_refs = {exposure["knowledge_ref_id"], *(r["knowledge_ref_id"] for r in exposure.get("rule_refs", []))} + if item["knowledge_ref_id"] not in allowed_refs: + raise KnowledgeTraceError("assessment exposure does not contain the referenced knowledge") + for evidence_id in item.get("evidence_ids", []): + if evidence_id not in evidence: + raise KnowledgeTraceError("assessment references unknown evidence_id") + if not isinstance(item.get("rationale", ""), str) or len(item.get("rationale", "")) > 4000: + raise KnowledgeTraceError("assessment rationale must be at most 4000 characters") + item.setdefault("assessment_id", "ka-" + digest(item)[:32]) + _identifier(item["assessment_id"], "assessment_id") + for name in ("evidence_ids", "exposure_ids", "decision_ids"): + _rule_ids(item.get(name)) + item.setdefault("reported_by", "host") + item["reference_validation"] = "valid" + item["exposure_status"] = "reported" if item.get("exposure_ids") else "not_reported" + stale = [e for e in item.get("evidence_ids", []) if evidence[e].get("revision_status") == "stale"] + item["evidence_revision_status"] = "stale" if stale else "current_or_unknown" + previous = next((r["result"] for r in reversed(records) + if r["result"].get("assessment", {}).get("assessment_id") == item["assessment_id"]), None) + if previous and previous["assessment"] != item: + if (event.get("supersedes") != previous["event_id"] or + previous["assessment"]["knowledge_ref_id"] != item["knowledge_ref_id"]): + raise KnowledgeTraceError("assessment_id conflicts; corrections must supersede the same knowledge assessment") + return event + + def record_events(self, trace_id: str, events: list[dict], idempotency_key: str = "") -> dict: + if not isinstance(events, list) or len(events) > 100: + raise KnowledgeTraceError("events must be a batch of at most 100 declarations") + + def write(): + records = self._records(trace_id) + session = self._session(records) + saved = [] + # Validation is sequential so an assessment may refer to earlier evidence in the batch. + for index, raw in enumerate(events): + if not isinstance(raw, dict): + raise KnowledgeTraceError("each event must be an object") + event = copy.deepcopy(raw.get("payload", raw)) + if "payload" in raw: + event.setdefault("event_type", raw.get("event_type")) + event.setdefault("step_id", raw.get("step_id", "")) + if idempotency_key: + event.setdefault("event_id", "ke-" + digest([idempotency_key, index])[:32]) + event = self._validate_declaration(event, records, session) + saved.append(self._append(trace_id, session, records, event)) + return {"trace_id": trace_id, "status": "recorded", "events": saved} + return self._operate(trace_id, write) + + def finish_session(self, trace_id: str, status: str = "complete") -> dict: + if status not in {"complete", "interrupted", "degraded"}: + raise KnowledgeTraceError("invalid finished session status") + + def write(): + records = self._records(trace_id) + session = self._session(records) + if records[-1]["result"].get("event_type") == "session_finished": + if records[-1]["result"]["status"] != status: + raise KnowledgeTraceError("finished session status is immutable") + else: + self._append(trace_id, session, records, {"event_type": "session_finished", "status": status}) + return {"trace_id": trace_id, "status": status} + return self._operate(trace_id, write) + + def report(self, trace_id: str, cursor: str | int | None = None, limit: int | None = None) -> dict: + def read(): + records = self._records(trace_id) + self._session(records) + return project(records, self.store, cursor=cursor, limit=limit) + return self._operate(trace_id, read) + + def query(self, repository: str = "", rule_id: str = "", limit: int = 100) -> list[dict] | dict: + if isinstance(limit, bool) or not isinstance(limit, int) or not 1 <= limit <= 1000: + raise KnowledgeTraceError("query limit must be between 1 and 1000") + if repository: + _identifier(repository, "repository") + if not self.enabled: + return [] + + def read(): + records = self.store.query(kind="knowledge_event", repo=repository or None, + rule_id=rule_id or None, limit=limit) + return [{"trace_id": r["result"]["trace_id"], "event_id": r["result"]["event_id"], + "event_type": r["result"]["event_type"], "repository": r["context"]["repo"], + "step_id": r["result"].get("step_id", ""), "at": r["at"], + "rule_ids": r["context"].get("rule_ids", [])} for r in records] + return self._operate("", read) diff --git a/src/infermatrix_copilot/knowledge_trace/references.py b/src/infermatrix_copilot/knowledge_trace/references.py new file mode 100644 index 00000000..9faee8cb --- /dev/null +++ b/src/infermatrix_copilot/knowledge_trace/references.py @@ -0,0 +1,90 @@ +"""Versioned knowledge identities and conservative Markdown rule references.""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import PurePosixPath + + +class KnowledgeTraceError(ValueError): + """Invalid knowledge trace input or unavailable required recording.""" + + +def digest(value) -> str: + data = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(data.encode("utf-8")).hexdigest() + + +def document_scope(document_id: str, repository: str) -> str: + """Accept only a relative Markdown path inside general or this repository.""" + if not isinstance(document_id, str) or not document_id or "\\" in document_id: + raise KnowledgeTraceError("document_id must be a knowledge-relative Markdown path") + path = PurePosixPath(document_id) + if path.is_absolute() or any(p in ("", ".", "..") for p in document_id.split("/")): + raise KnowledgeTraceError("document_id escapes the knowledge root") + if path.suffix.lower() != ".md": + raise KnowledgeTraceError("knowledge references must name Markdown documents") + if document_id in {"AGENTS.md", "README.md", "repos/_index.md"}: + return "navigation" + parts = path.parts + if len(parts) >= 2 and parts[0] == "general": + return "general" + if len(parts) >= 3 and parts[:2] == ("repos", repository): + return repository + raise KnowledgeTraceError("knowledge document is outside the selected repository scope") + + +_HEADING = re.compile(r"^(#{1,6})[ \t]+(.+?)[ \t]*#*[ \t]*$", re.MULTILINE) +_RULE = re.compile(r"^`?([A-Z][A-Z0-9_]*(?:-[A-Z0-9_]+)*-\d+[a-z]*)`?(?=\s|[—–:：]|$)") + + +def extract_rule_sections(content: str) -> list[dict]: + """Only exact heading IDs count; prose and quick-map mentions never do. + + Positions are Unicode code points within the supplied visible text. Fenced + examples are ignored, and a section stops at the next peer/ancestor heading. + A section reaching the end of an excerpt is not evidence of completeness. + """ + headings = [] + fenced = False + fence_character = "" + offset = 0 + for line in content.splitlines(keepends=True): + stripped = line.lstrip() + if stripped.startswith(("```", "~~~")): + char = stripped[0] + if not fenced: + fenced, fence_character = True, char + elif char == fence_character: + fenced = False + offset += len(line) + continue + match = _HEADING.match(line.rstrip("\r\n")) if not fenced else None + if match: + headings.append((offset, len(match.group(1)), match.group(2), offset + len(line))) + offset += len(line) + sections = [] + for position, (start, level, title, body_start) in enumerate(headings): + match = _RULE.match(title) + if not match: + continue + following = next((h[0] for h in headings[position + 1:] if h[1] <= level), len(content)) + sections.append({"rule_id": match.group(1), "start": start, "end": following, + "body_start": body_start, "has_body": bool(content[body_start:following].strip()), + "closed_by_heading": following < len(content), + "section_sha256": digest(content[start:following])}) + return sections + + +def knowledge_reference(document_id: str, repository: str, snapshot: str, document_sha256: str, + *, rule_id: str = "", section_sha256: str = "") -> dict: + namespace = document_scope(document_id, repository) + identity = {"knowledge_namespace": namespace, "document_id": document_id, "rule_id": rule_id, + "snapshot_id": snapshot, "document_sha256": document_sha256, + "section_sha256": section_sha256} + # An excerpt's section hash is evidence about that exposure, not a new + # rule identity. The versioned document and exact heading own the identity. + identity_fields = {k: v for k, v in identity.items() if k != "section_sha256"} + return {"knowledge_ref_id": "kr-" + digest(identity_fields)[:32], **identity} diff --git a/src/infermatrix_copilot/knowledge_trace/report.py b/src/infermatrix_copilot/knowledge_trace/report.py new file mode 100644 index 00000000..d4badef6 --- /dev/null +++ b/src/infermatrix_copilot/knowledge_trace/report.py @@ -0,0 +1,189 @@ +"""Deterministic, bounded projections of knowledge-use declarations.""" + +from __future__ import annotations + +from .references import KnowledgeTraceError + + +def project(records, store, *, cursor=None, limit=None) -> dict: + """Project a complete bounded session; paginate the raw event list only. + + A complete session means capture ended successfully, not that the model + understood the content or that a conclusion has been independently verified. + """ + try: + offset = int(cursor or 0) + except (TypeError, ValueError) as exc: + raise KnowledgeTraceError("invalid knowledge trace cursor") from exc + if offset < 0: + raise KnowledgeTraceError("invalid knowledge trace cursor") + page_size = 1000 if limit is None else limit + if isinstance(page_size, bool) or not isinstance(page_size, int) or not 1 <= page_size <= 1000: + raise KnowledgeTraceError("report limit must be between 1 and 1000") + session = dict(records[0]["result"]["session"]) + retrievals, exposures, evidence, assessments, gaps = [], [], [], [], [] + references = {} + steps = {} + finished = False + superseded = {r["result"].get("supersedes") for r in records if r["result"].get("supersedes")} + available = {} + for record in records: + event = record["result"] + if event["event_id"] in superseded: + continue + event_type = event["event_type"] + step_id = event.get("step_id", "") + if step_id: + steps.setdefault(step_id, {"step_id": step_id, "retrieval_event_ids": [], "exposure_ids": [], + "assessment_ids": [], "evidence_ids": [], "gaps": []}) + if event_type == "retrieval_completed": + item = {k: event[k] for k in ("event_id", "step_id", "mechanism", "query", "reason", "status", "entries") if k in event} + retrievals.append(item) + if step_id: + steps[step_id]["retrieval_event_ids"].append(event["event_id"]) + elif event_type == "content_provided": + item = dict(event["exposure"]) + blob_ref = item.get("payload_ref", "") + if blob_ref not in available: + try: + store.blob(blob_ref) + available[blob_ref] = True + except (OSError, KeyError, ValueError, EOFError): + available[blob_ref] = False + item["content_available"] = available[blob_ref] + exposures.append(item) + references[item["knowledge_ref_id"]] = item["reference"] + for ref in item.get("rule_refs", []): + references[ref["knowledge_ref_id"]] = ref + if step_id: + steps[step_id]["exposure_ids"].append(item["exposure_id"]) + if not item["content_available"]: + gaps.append({"reason": "archived_content_unavailable", "exposure_id": item["exposure_id"], "step_id": step_id}) + elif event_type == "evidence_registered": + item = dict(event["evidence"]) + if item.get("payload_ref"): + try: + store.blob(item["payload_ref"]) + item["content_available"] = True + except (OSError, KeyError, ValueError, EOFError): + item["content_available"] = False + gaps.append({"reason": "evidence_content_unavailable", "evidence_id": item["evidence_id"], "step_id": step_id}) + evidence.append(item) + if step_id: + steps[step_id]["evidence_ids"].append(item["evidence_id"]) + elif event_type == "assessment_recorded": + item = dict(event["assessment"]) + item["step_id"] = step_id + assessments.append(item) + if item.get("evidence_revision_status") == "stale": + gaps.append({"reason": "assessment_evidence_revision_stale", "assessment_id": item["assessment_id"], "step_id": step_id}) + if not item.get("exposure_ids"): + gaps.append({"reason": "assessment_exposure_not_reported", "assessment_id": item["assessment_id"], "step_id": step_id}) + if item.get("applicability") == "applicable" and not item.get("evidence_ids"): + gaps.append({"reason": "assessment_evidence_not_reported", "assessment_id": item["assessment_id"], "step_id": step_id}) + if step_id: + steps[step_id]["assessment_ids"].append(item["assessment_id"]) + elif event_type == "gap_recorded": + item = {"step_id": step_id, **event["gap"]} + gaps.append(item) + if step_id: + steps[step_id]["gaps"].append(item) + elif event_type == "session_finished": + session["status"] = event["status"] + finished = True + assessed = {a["knowledge_ref_id"] for a in assessments} + provided_rule_ids = {(r["knowledge_ref_id"]) for x in exposures for r in x.get("rule_refs", [])} + unassessed = sorted(provided_rule_ids - assessed) + next_offset = offset + page_size + return {"schema": "knowledge-use-report/1", "trace_id": session["trace_id"], + "status": session["status"], "session": session, + "trace_status": "complete" if finished and session["status"] == "complete" and not gaps else "partial", + "instrumentation": session.get("instrumentation", {}), + "retrievals": retrievals, "exposures": exposures, "evidence": evidence, + "assessments": assessments, "references": list(references.values()), "gaps": gaps, + "steps": list(steps.values()), "unassessed_reference_ids": unassessed, + "counts": {"events": len(records), "retrievals": len(retrievals), "exposures": len(exposures), + "provided_rules": len(provided_rule_ids), "assessments": len(assessments), + "unassessed_rules": len(unassessed), "evidence": len(evidence), "gaps": len(gaps)}, + "events": records[offset:next_offset], + "next_cursor": str(next_offset) if next_offset < len(records) else None, + "limits": {"session_events": 10000, "page_events": page_size}, + "semantics": {"provided": "content observed at the recorded boundary; attention is not observable", + "assessment": "explicit producer declaration; reference validation does not prove its conclusion", + "unassessed": "no application declaration; applicability is unknown"}} + + +def _cell(value) -> str: + return str("—" if value is None or value == "" else value).replace("|", "\\|").replace("\r", " ").replace("\n", " ") + + +def markdown(report: dict) -> str: + """Render a portable reasoning-evidence report, without archived raw text.""" + session = report.get("session", {}) + counts = report.get("counts", {}) + lines = ["# Knowledge-use trace", "", + f"Trace: `{_cell(report.get('trace_id'))}`. Status: **{_cell(report.get('status'))}**; capture: **{_cell(report.get('trace_status'))}**.", "", + f"Repository: `{_cell(session.get('repository'))}`; mode: `{_cell(session.get('mode'))}`.", + f"Source revision: `{_cell(session.get('source_revision'))}`; knowledge snapshot: `{_cell(session.get('knowledge_snapshot'))}`.", "", + "Delivery records describe the stated boundary. Assessments are explicit declarations; references and evidence allow review, and do not establish causation or model attention.", "", + f"Retrievals: {counts.get('retrievals', 0)}; exposures: {counts.get('exposures', 0)}; provided rules: {counts.get('provided_rules', 0)}; assessments: {counts.get('assessments', 0)}; unassessed rules: {counts.get('unassessed_rules', 0)}.", "", + "## Capture boundaries", ""] + instrumentation = report.get("instrumentation", {}) + lines.extend(f"- {_cell(key)}: {_cell(value)}" for key, value in sorted(instrumentation.items())) + if not instrumentation: + lines.append("Capture capabilities were not declared.") + lines.extend(["", "## Steps", "", "| Step | Retrieved knowledge / reason | Provided content | Assessment / evidence |", + "| --- | --- | --- | --- |"]) + retrievals = {r["event_id"]: r for r in report.get("retrievals", [])} + exposures = {r["exposure_id"]: r for r in report.get("exposures", [])} + assessments = {r["assessment_id"]: r for r in report.get("assessments", [])} + for step in report.get("steps", []): + retrieved = [] + for event_id in step["retrieval_event_ids"]: + event = retrievals[event_id] + if not event["entries"]: + retrieved.append(f"no match ({event.get('mechanism', '')})") + for entry in event["entries"]: + path = entry.get("document_id") or entry.get("path") or entry.get("name") or "candidate" + reason = entry.get("reason") or event.get("reason") or event.get("mechanism", "") + retrieved.append(f"{path}: {reason}") + supplied = [] + for exposure_id in step["exposure_ids"]: + exposure = exposures[exposure_id] + rules = ", ".join(f"{r['rule_id']} ({r['coverage']})" for r in exposure.get("rule_refs", [])) + supplied.append(f"{exposure['document_id']} @ {exposure['boundary']} [{exposure['representation']}]" + + (f"; {rules}" if rules else "") + + ("; truncated" if exposure.get("truncated") else "")) + assessed = [] + for assessment_id in step["assessment_ids"]: + assessment = assessments[assessment_id] + evidence = ", ".join(assessment.get("evidence_ids", [])) or "none declared" + decisions = ", ".join(assessment.get("decision_ids", [])) + decision_text = "; ".join( + f"{ref.get('decision_id', '')}: {ref.get('finding', '')}" + for ref in assessment.get("decision_refs", []) if isinstance(ref, dict)) + assessed.append(f"{assessment.get('rule_id') or assessment.get('document_id')}: " + f"{assessment['applicability']}/{assessment['verification']}; {assessment.get('rationale', '')}; " + f"check: {assessment.get('check', 'not declared')}; " + f"evidence: {evidence}" + (f"; decisions: {decisions}" if decisions else "") + + (f"; findings: {decision_text}" if decision_text else "")) + lines.append("| " + " | ".join(_cell(x) for x in + (step["step_id"], "; ".join(retrieved), "; ".join(supplied), "; ".join(assessed) or "not assessed")) + " |") + if not report.get("steps"): + lines.append("| — | — | — | not assessed |") + lines.extend(["", "## Evidence", ""]) + for item in report.get("evidence", []): + lines.append(f"- `{_cell(item['evidence_id'])}`: {_cell(item.get('kind'))}; " + f"{_cell(item.get('path') or item.get('tool_call_id'))}; revision: {_cell(item.get('revision_status'))}; " + f"source: {_cell(item.get('source_revision'))}; origin: {_cell(item.get('evidence_origin'))}.") + for key in ("command", "result_summary", "summary", "environment_fingerprint", "payload_ref", + "workspace_state", "observed_head", "commit_verified"): + if key in item and item[key] is not None and item[key] != "": + lines.append(f" - {_cell(key)}: {_cell(item[key])}") + if not report.get("evidence"): + lines.append("No evidence declared.") + lines.extend(["", "## Gaps", ""]) + lines.extend(f"- {_cell(g.get('step_id'))}: {_cell(g.get('reason'))}" for g in report.get("gaps", [])) + if not report.get("gaps"): + lines.append("No capture gaps recorded. Unassessed knowledge remains unknown.") + return "\n".join(lines) + "\n" diff --git a/src/infermatrix_copilot/sdk/v1/__init__.py b/src/infermatrix_copilot/sdk/v1/__init__.py index 5565b70e..c715aafb 100644 --- a/src/infermatrix_copilot/sdk/v1/__init__.py +++ b/src/infermatrix_copilot/sdk/v1/__init__.py @@ -7,9 +7,11 @@ ContainmentError, knowledge_maintenance_status, knowledge_usage_record, knowledge_availability_check, knowledge_publication_guard, knowledge_policy_install, ) +from .knowledge_trace import KnowledgeTraceClient from .models import ( DIRECT_API_VERSION, KNOWLEDGE_API_VERSION, + KNOWLEDGE_TRACE_API_VERSION, QUALITY_API_VERSION, SDK_API_VERSION, STRICT_API_VERSION, @@ -66,6 +68,8 @@ "SourceRef", "DIRECT_API_VERSION", "KNOWLEDGE_API_VERSION", + "KNOWLEDGE_TRACE_API_VERSION", + "KnowledgeTraceClient", "QUALITY_API_VERSION", "SDK_API_VERSION", "STRICT_API_VERSION", diff --git a/src/infermatrix_copilot/sdk/v1/direct.py b/src/infermatrix_copilot/sdk/v1/direct.py index b202b8f5..6188a138 100644 --- a/src/infermatrix_copilot/sdk/v1/direct.py +++ b/src/infermatrix_copilot/sdk/v1/direct.py @@ -8,6 +8,7 @@ import re import threading import uuid +from copy import deepcopy from collections import OrderedDict from dataclasses import dataclass, replace from functools import lru_cache @@ -44,6 +45,25 @@ _CONTEXT_ID = re.compile(r"sha256:[0-9a-f]{64}") +def _quick_map_rule_ids(content: str, source_content: str) -> list[str]: + """Resolve explicit map references against actual heading IDs in the page. + + Maps sometimes abbreviate adjacent IDs (``SCHED-1a/1b``). Only expansions + that exist as actual source sections are accepted; prose never invents IDs. + """ + from ...knowledge_trace import extract_rule_sections + + known = {section["rule_id"] for section in extract_rule_sections(source_content)} + found = set() + for match in re.finditer(r"(? str: try: installed = importlib.metadata.version("infermatrix-copilot") @@ -129,6 +149,8 @@ class _IssuedContext: knowledge_usage: dict | None = None task_id: str = "" repository: str = "" + trace_id: str = "" + trace_metadata: Any = None class DirectClient: @@ -136,24 +158,29 @@ class DirectClient: Completion validation is intentionally bound to plans issued by this client instance. The registry is bounded so a long-running review host - cannot grow memory without limit. + cannot grow memory without limit. Repeating an identical review context + reuses its trace; use a new review_id for a new attempt. """ def __init__(self, *, max_issued_contexts: int = _DEFAULT_CONTEXT_LIMIT, knowledge_context_path: str | Path | None = None, allowed_knowledge_repositories: tuple[str, ...] = (), - knowledge_maintenance: dict | None = None) -> None: + knowledge_maintenance: dict | None = None, + trace_root: str | Path | None = None, trace_mode: str | None = None) -> None: if max_issued_contexts < 1: raise InvalidRequestError("max_issued_contexts must be >= 1") self._adapters = adapters_root() self._max_issued_contexts = int(max_issued_contexts) self._issued_contexts: OrderedDict[str, _IssuedContext] = OrderedDict() - self._context_lock = threading.Lock() + self._context_lock = threading.RLock() self._knowledge_context_path = Path(knowledge_context_path) if knowledge_context_path else \ Path.home() / ".infermatrix-copilot" / "knowledge-context.sqlite" self._knowledge_context_services: dict[str, Any] = {} self._allowed_knowledge_repositories = tuple(allowed_knowledge_repositories) self._knowledge_maintenance = knowledge_maintenance + from .knowledge_trace import KnowledgeTraceClient + + self.knowledge_trace = KnowledgeTraceClient(root=trace_root, mode=trace_mode) @staticmethod def _view(): @@ -273,6 +300,8 @@ def read_document( review_context_id: str = "", task_id: str = "", repository: str = "", + trace_id: str = "", + step_id: str = "document.read", ) -> DocumentPage: """Read one knowledge document page. @@ -289,6 +318,7 @@ def read_document( "offset must be >= 0 and max_bytes must be within 1..65536" ) view = None + issued = None context_id = str(review_context_id).strip().casefold() if configuration()["enabled"] and not context_id: raise ContainmentError("enforced Direct document reads require a provider-issued review_context_id") @@ -300,11 +330,24 @@ def read_document( "review_context_id was not issued by this DirectClient" ) view = issued.knowledge_view + if trace_id and trace_id != issued.trace_id: + raise InvalidRequestError("trace_id does not match the issued review context") + trace_id = issued.trace_id + elif trace_id: + raise InvalidRequestError("traced document reads require review_context_id") else: view = self._view() - data = self._served_bytes(self._document_path(document_id, view)) + try: + if issued: + self._check_document_scope(document_id, issued.repository) + data = self._served_bytes(self._document_path(document_id, view)) + except Exception: + if trace_id: + self.knowledge_trace.record_gap(trace_id, "document_read_failed") + raise page = data[offset:offset + max_bytes] next_offset = offset + len(page) if offset + len(page) < len(data) else None + trace_metadata = {} result = DocumentPage( document_id=document_id, sha256=_sha256(data), @@ -312,6 +355,7 @@ def read_document( content=page.decode("utf-8", errors="replace"), next_offset=next_offset, review_context_id=context_id if configuration()["enabled"] else "", + knowledge_trace=trace_metadata, ) session = issued.knowledge_usage.get("delivery_session") if context_id and issued.knowledge_usage else None if configuration()["enabled"] and not session: @@ -353,6 +397,21 @@ def read_document( repository, ([{"path": document_id, "content": result.content}] if PurePosixPath(document_id).suffix.casefold() == ".md" else []))) result = replace(result, knowledge_feedback=feedback) + trace_metadata = {} + if trace_id and self.knowledge_trace.enabled: + trace_metadata = self._provide( + trace_id, step_id=step_id, document_id=document_id, content=page.decode("utf-8", errors="replace"), + boundary="provider_response", representation="full_document" if offset == 0 and next_offset is None else "excerpt", offset=offset, + offset_unit="utf8_bytes", source_length=len(page), truncated=next_offset is not None, + snapshot=view.public_snapshot, document_sha256=_sha256(data), + ) + if self.knowledge_trace.is_degraded(trace_id): + with self._context_lock: + current = self._issued_contexts.get(context_id) + if current is not None: + self._issued_contexts[context_id] = replace(current, trace_metadata={ + **(current.trace_metadata or {}), "status": "degraded"}) + result = replace(result, knowledge_trace=trace_metadata) _issue_usage(result.to_dict(), view=view, session_id=session) return result @@ -402,7 +461,8 @@ def open_knowledge_context(self, request: DirectReviewRequest, *, budget=None) - status = service.open_session(alias, source_pin=request.expected_head_sha.casefold(), review_id=request.review_id, budget=budget) self._knowledge_context_services[status["session_id"]] = service - return status + metadata = self._trace_adaptive(status["session_id"]) + return {**status, "knowledge_trace": metadata} if metadata else status def _knowledge_context(self, session_id: str): service = self._knowledge_context_services.get(session_id) @@ -462,29 +522,122 @@ def plan_adaptive(self, request: DirectReviewRequest, *, budget=None) -> dict: packet["knowledge_feedback"] = self._feedback( service.view, request.review_id, alias, model_content_documents(content)) + packet = self._trace_context_packet(session_id, packet, step_id="review.plan", + mechanism="adaptive_review", query=request.title) usage = _issue_usage(packet, view=service.view, session_id=session_id) + metadata = packet.get("knowledge_trace", {}) self._remember_context(context_id, request.expected_head_sha.casefold(), tuple(row.to_dict() for row in request.carried_findings), service.view, usage, - task_id=request.review_id, repository=alias) + task_id=request.review_id, repository=alias, + trace_id=metadata.get("trace_id", ""), trace_metadata=metadata) return packet def read_knowledge_context(self, session_id: str, document_id: str, *, offset: int = 0, repository: str | None = None) -> dict: - return self._knowledge_context(session_id).read(session_id, document_id, offset=offset, repository=repository) + packet = self._knowledge_context(session_id).read(session_id, document_id, offset=offset, repository=repository) + return self._trace_context_packet(session_id, packet, step_id="knowledge.read", mechanism="adaptive_read") def search_knowledge_context(self, session_id: str, query: str, *, repository: str | None = None) -> dict: - return self._knowledge_context(session_id).search(session_id, query, repository=repository) + packet = self._knowledge_context(session_id).search(session_id, query, repository=repository) + return self._trace_context_packet(session_id, packet, step_id="knowledge.search", mechanism="adaptive_search", query=query) def related_knowledge_context(self, session_id: str, changed_files: list[str], *, query: str = "", repository: str | None = None) -> dict: - return self._knowledge_context(session_id).related(session_id, changed_files, query=query, repository=repository) + packet = self._knowledge_context(session_id).related(session_id, changed_files, query=query, repository=repository) + return self._trace_context_packet(session_id, packet, step_id="knowledge.related", mechanism="adaptive_related", query=query) def expand_knowledge_context(self, session_id: str, *, target_tokens: int, reason: str) -> dict: return self._knowledge_context(session_id).expand(session_id, target_tokens=target_tokens, reason=reason) + def _trace_adaptive(self, session_id: str, packet: dict | None = None, *, + step_id: str = "knowledge.context", mechanism: str = "adaptive_context", + query: str = "") -> dict: + """Observe the exact budgeted packet without re-reading or selecting knowledge.""" + if not self.knowledge_trace.enabled: + return {} + service = self._knowledge_context(session_id) + status = service.status(session_id) + identity = status["identity"] + repository = identity["repository"]["knowledge_slice"].removeprefix("repos/") + trace_id = "kt-adaptive-" + session_id + try: + session = self.knowledge_trace.start_session( + repository, mode="direct", trace_id=trace_id, + run_id=identity["review_id"], + source_revision=identity["source_pin"], + knowledge_snapshot=service.view.public_snapshot, + resource_revision=self._revision(service.view.root), + instrumentation={"provider": "observed", "host_context": "unknown", "applications": "host_reported", + "adaptive_context_session_id": session_id}, + ) + if session.get("status") == "degraded": + return session + exposures = [] + if packet is not None: + from ...knowledge_trace.adaptive import packet_fragments + from ...knowledge_trace.references import document_scope + + fragments = packet_fragments(packet) + entries = [] + for fragment in fragments: + path = fragment["document_id"] + try: + document_scope(path, repository) + except ValueError: + # The context service already authorized this delivery. + # A single-repository trace must not extend that authority + # or prevent a permitted cross-repository context read. + self.knowledge_trace.record_gap(trace_id, "context_cross_repository_unobserved") + continue + entries.append({"document_id": path, "repo_id": fragment.get("repo_id", ""), + "source_pins": fragment.get("source_pins", []), + "partial": fragment.get("partial", False), + "continuation": fragment.get("continuation", False), "session_id": session_id}) + provided = self._provide( + trace_id, step_id=step_id, document_id=path, + content=fragment["content"], boundary="provider_response", + representation=fragment.get("representation", "excerpt"), + offset=fragment.get("offset"), source_length=fragment.get("source_length"), + offset_unit=fragment.get("offset_unit", "unicode_codepoints"), + truncated=bool(fragment.get("truncated") or fragment.get("partial")), + snapshot=service.view.public_snapshot, + ) + exposures.append({"document_id": path, "source_pins": fragment.get("source_pins", []), + "repo_id": fragment.get("repo_id", ""), **provided}) + result = self.knowledge_trace.capture( + "retrieve", trace_id, step_id=step_id, entries=entries, + mechanism=mechanism, query=query, + status=packet.get("status", "ready" if entries else "no_match"), + ) + if result.get("status") == "degraded": + return result + return {"trace_id": trace_id, + "status": "degraded" if self.knowledge_trace.is_degraded(trace_id) else "ready", + "exposures": exposures, "host_context": "unconfirmed", "applications": "not_reported"} + except Exception as exc: + self.knowledge_trace.mark_degraded(trace_id) + if self.knowledge_trace.mode == "required": + raise + return {"trace_id": trace_id, "status": "degraded", "error": str(exc)} + + def _trace_context_packet(self, session_id: str, packet: dict, *, step_id: str, + mechanism: str, query: str = "") -> dict: + metadata = self._trace_adaptive(session_id, packet, step_id=step_id, mechanism=mechanism, query=query) + if not metadata: + return packet + result = {**packet, "knowledge_trace": metadata} + if result.get("documents"): + # Containment issuance binds the exact returned payload. Trace + # metadata is part of that object, while the model content and + # its cumulative context budget remain unchanged. + service = self._knowledge_context(session_id) + with configured(self._knowledge_maintenance): + _issue_usage(result, view=service.view, session_id=session_id) + return result + def _remember_context( self, context_id: str, expected_head_sha: str, carried=(), view=None, usage=None, - *, task_id: str = "", repository: str = "", + *, task_id: str = "", repository: str = "", trace_id: str = "", trace_metadata=None, ) -> None: view = view if view is not None else self._view() from ...direct_routing import _normalize_repo @@ -498,6 +651,8 @@ def _remember_context( knowledge_usage=usage, task_id=task_id, repository=repository, + trace_id=trace_id, + trace_metadata=deepcopy(trace_metadata), ) self._issued_contexts.move_to_end(context_id) while len(self._issued_contexts) > self._max_issued_contexts: @@ -611,11 +766,137 @@ def plan(self, request: DirectReviewRequest) -> DirectReviewPlan: "content": row["document"]["excerpt"]} for row in related.get("documents", []) if row.get("document")) plan = replace(plan, knowledge_feedback=self._feedback(view, request.review_id, alias, documents)) - usage = _issue_usage(plan.to_dict(), view=view, session_id=review_context_id.removeprefix("sha256:")) - self._remember_context(review_context_id, expected_head, carried, view, usage, - task_id=request.review_id, repository=alias) + # A review_context_id identifies this client's attempt, including its + # review_id. Retrying the same plan must never redirect its reads or + # completion to a different trace. Serialize publication of that binding. + with self._context_lock: + issued = self._issued_contexts.get(review_context_id) + trace_metadata = deepcopy(issued.trace_metadata) if issued is not None else self._trace_plan(plan, request, view) + trace_metadata = trace_metadata or {} + plan = replace(plan, knowledge_trace=trace_metadata) + usage = _issue_usage(plan.to_dict(), view=view, session_id=review_context_id.removeprefix("sha256:")) + self._remember_context( + review_context_id, expected_head, carried, view, repository=alias, usage=usage, task_id=request.review_id, + trace_id=str(trace_metadata.get("trace_id") or ""), trace_metadata=trace_metadata, + ) return plan + @staticmethod + def _check_document_scope(document_id: str, repository: str) -> None: + value = str(document_id).replace("\\", "/") + if value in {"AGENTS.md", "README.md", "repos/_index.md"}: + return + if value.startswith("general/") or value.startswith(f"repos/{repository}/"): + return + raise DocumentNotFoundError("document is outside the review repository knowledge scope") + + def _provide(self, trace_id: str, **kwargs) -> dict: + try: + event = self.knowledge_trace.capture("provide", trace_id, **kwargs) + if event.get("status") in {"degraded", "off"}: + return {"trace_id": trace_id, **event} + result = event.get("result", {}) + return {"trace_id": trace_id, "status": "degraded" if self.knowledge_trace.is_degraded(trace_id) else "ready", **{ + name: result[name] for name in ("exposure_id", "knowledge_ref_id") if name in result + }} + except Exception as exc: + if self.knowledge_trace.mode == "required": + raise + self.knowledge_trace.mark_degraded(trace_id) + return {"trace_id": trace_id, "status": "degraded", "error": str(exc)} + + def _trace_plan(self, plan: DirectReviewPlan, request: DirectReviewRequest, view) -> dict: + if not self.knowledge_trace.enabled: + return {} + trace_id = "" + try: + session = self.knowledge_trace.start_session( + self._request_values(request, view=view)[0], mode="direct", run_id=request.review_id, + review_context_id=plan.review_context_id, + source_revision=plan.expected_head_sha, + knowledge_snapshot=plan.knowledge_snapshot, + resource_revision=plan.resource_revision, + instrumentation={"provider": "observed", "host_context": "unknown", "applications": "host_reported"}, + ) + trace_id = str(session.get("trace_id") or "") + if not trace_id or session.get("status") == "degraded": + return session + entries = [{"document_id": route.document.document_id, "owner": route.owner, + "reason": route.reason, "rank": rank} + for rank, route in enumerate(plan.knowledge_routes, 1)] + retrieved = self.knowledge_trace.capture("retrieve", trace_id, step_id="review.plan", entries=entries, + mechanism="direct_routing", query=request.title, + status="ready" if entries else "no_match") + if retrieved.get("status") == "degraded": + return retrieved + self.knowledge_trace.capture("retrieve", trace_id, step_id="review.plan", mechanism="mandatory_review_guides", + entries=[{"document_id": doc.document_id, "reason": "mandatory_review_guide"} + for doc in plan.mandatory_review_guides]) + self.knowledge_trace.capture("retrieve", trace_id, step_id="review.plan", mechanism="navigation", + entries=[{"document_id": doc.document_id, "reason": "knowledge_navigation"} + for doc in [plan.knowledge_entry, *plan.document_maps]]) + related_entries = [{"document_id": doc["path"], "reason": doc.get("match_reason", ""), + "matched_files": doc.get("matched_files", []), "rank": rank} + for rank, doc in enumerate(plan.related_knowledge.get("documents", []), 1)] + self.knowledge_trace.capture("retrieve", trace_id, step_id="review.plan", mechanism="related_knowledge", + entries=related_entries, query=request.title, + status="ready" if related_entries else "no_match") + exposures = [] + documents = [] + def visit(value): + if isinstance(value, dict): + if {"document_id", "sha256", "excerpt"}.issubset(value): + documents.append(value) + else: + for item in value.values(): + visit(item) + elif isinstance(value, (tuple, list)): + for item in value: + visit(item) + visit(plan.to_dict()) + seen = set() + def provide(document_id, content, representation, truncated=False, sha256="", rule_ids=None): + key = (document_id, content, representation) + if key in seen: + return + seen.add(key) + source_data = self._served_bytes(self._document_path(document_id, view)) + source_content = source_data.decode("utf-8", errors="replace") + if representation in {"quick_map", "related_excerpt"}: + position = source_content.find(content) + offset = position if content and position >= 0 and source_content.find(content, position + 1) < 0 else None + offset_unit, source_length = "unicode_codepoints", None + else: + offset, offset_unit = 0, "utf8_bytes" + source_length = min(len(source_data), _EXCERPT_BYTES) + info = self._provide(trace_id, step_id="review.plan", document_id=document_id, + content=content, boundary="provider_response", + representation=representation, truncated=truncated, offset=offset, + offset_unit=offset_unit, source_length=source_length, + snapshot=view.public_snapshot, document_sha256=sha256, rule_ids=rule_ids) + exposures.append({"document_id": document_id, "representation": representation, **info}) + for document in documents: + provide(document["document_id"], document["excerpt"], "excerpt" if document.get("truncated") else "full_document", + document.get("truncated", False), document["sha256"]) + for route in plan.knowledge_routes: + if route.quick_map: + provide(route.document.document_id, route.quick_map, "quick_map", + route.quick_map_status == "truncated", route.document.sha256, + _quick_map_rule_ids(route.quick_map, self._served_bytes( + self._document_path(route.document.document_id, view)).decode("utf-8", errors="replace"))) + for document in plan.related_knowledge.get("documents", []): + if document.get("content"): + provide(document["path"], document["content"], "related_excerpt", + bool(document.get("more_available")), document.get("document", {}).get("sha256", "")) + status = "degraded" if self.knowledge_trace.is_degraded(trace_id) or any(item.get("status") == "degraded" for item in exposures) else "ready" + return {"trace_id": trace_id, "status": status, "exposures": exposures, + "host_context": "unconfirmed", "applications": "not_reported"} + except Exception as exc: + self.knowledge_trace.mark_degraded(trace_id) + if self.knowledge_trace.mode == "required": + raise + return {"trace_id": trace_id, "status": "degraded", "error": str(exc)} + @with_containment def validate( self, request: DirectCompletionRequest @@ -685,6 +966,19 @@ def validate( # provider and SDK guards. unique_missing = tuple(dict.fromkeys(missing)) complete = not unique_missing and bool(details.get("publish_ready")) + if issued and issued.trace_id: + try: + if complete: + self.knowledge_trace.finish_session(issued.trace_id, "complete", repository=issued.repository) + trace_report = self.knowledge_trace.get_report(issued.trace_id, repository=issued.repository) + details["knowledge_trace"] = {"trace_id": issued.trace_id, + "status": trace_report.get("trace_status", trace_report.get("status", "partial")), + "session_status": trace_report.get("session", {}).get("status", "unknown"), + "counts": trace_report.get("counts", {})} + except Exception as exc: + if self.knowledge_trace.mode == "required": + raise + details["knowledge_trace"] = {"trace_id": issued.trace_id, "status": "degraded", "error": str(exc)} return DirectCompletionDecision( protocol_version=DIRECT_API_VERSION, review_context_id=context_id, diff --git a/src/infermatrix_copilot/sdk/v1/knowledge_trace.py b/src/infermatrix_copilot/sdk/v1/knowledge_trace.py new file mode 100644 index 00000000..4340d3c0 --- /dev/null +++ b/src/infermatrix_copilot/sdk/v1/knowledge_trace.py @@ -0,0 +1,183 @@ +"""Embedded facade for durable, evidence-backed knowledge-use traces. + +Host submissions are declarations. They cannot impersonate provider delivery or +captured tool evidence; reference validation is performed by the trace service. +""" + +from __future__ import annotations + +from copy import deepcopy +import hashlib +import threading +from pathlib import Path +from typing import Any + +from .models import InvalidRequestError, RepositoryRef + + +class KnowledgeTraceClient: + """Record host receipts/assessments and read deterministic trace reports.""" + + def __init__(self, *, root: str | Path | None = None, mode: str | None = None): + # Keep public SDK imports independent of server/config initialization. + from ...knowledge_trace import KnowledgeTrace + + self.recorder = KnowledgeTrace(root=root, mode=mode) + self._degraded: set[str] = set() + self._persisted_gaps: set[str] = set() + self._capture_lock = threading.RLock() + + @property + def enabled(self) -> bool: + return self.recorder.enabled + + @property + def mode(self) -> str: + return self.recorder.mode + + def is_degraded(self, trace_id: str) -> bool: + with self._capture_lock: + return trace_id in self._degraded + + def _flush_capture_gap(self, trace_id: str) -> None: + """Persist a sticky marker after a transient write failure recovers.""" + with self._capture_lock: + if not trace_id or trace_id not in self._degraded or trace_id in self._persisted_gaps or not self.enabled: + return + event = {"event_type": "gap_recorded", "step_id": "knowledge.capture", + "event_id": "ke-capture-gap-" + hashlib.sha256(trace_id.encode()).hexdigest()[:32], + "gap": {"reason": "knowledge_capture_degraded", + "detail": "At least one knowledge observation was not persisted."}} + try: + result = self.recorder.record_events(trace_id, [event]) + if result.get("status") == "recorded": + self._persisted_gaps.add(trace_id) + except Exception: + # Keep the in-memory marker for the next operation. The original + # failure retains its required/best_effort behavior. + pass + + def mark_degraded(self, trace_id: str) -> None: + if trace_id and self.enabled: + with self._capture_lock: + self._degraded.add(trace_id) + self._flush_capture_gap(trace_id) + + def capture(self, operation: str, trace_id: str, **kwargs) -> dict: + """Internal provider boundary: observe failures returned as values too.""" + if operation not in {"retrieve", "provide"}: + raise InvalidRequestError("unsupported provider capture operation") + try: + result = getattr(self.recorder, operation)(trace_id, **kwargs) + except Exception as exc: + self.mark_degraded(trace_id) + if self.mode == "required": + raise + return {"trace_id": trace_id, "status": "degraded", "error": str(exc)} + if result.get("status") == "degraded": + self.mark_degraded(trace_id) + else: + self._flush_capture_gap(trace_id) + return result + + def record_gap(self, trace_id: str, reason: str) -> None: + """Record a failed scoped operation without copying rejected path data.""" + try: + result = self.recorder.record_events(trace_id, [{"event_type": "gap_recorded", + "step_id": "knowledge.access", "gap": {"reason": reason}}]) + if result.get("status") == "degraded": + self.mark_degraded(trace_id) + except Exception: + self.mark_degraded(trace_id) + + @staticmethod + def _repository(repository: RepositoryRef | str) -> str: + value = repository.alias if isinstance(repository, RepositoryRef) else repository + value = str(value).strip().replace("_", "-").casefold() + if not value or "/" in value or "\\" in value or value in {".", ".."}: + raise InvalidRequestError("repository must be a canonical knowledge repository alias") + return value + + def start_session(self, repository: RepositoryRef | str, *, mode: str = "direct", **kwargs) -> dict: + result = self.recorder.start_session(self._repository(repository), mode=mode, **kwargs) + if result.get("status") == "degraded": + self.mark_degraded(result.get("trace_id", "")) + return result + + def get_report( + self, trace_id: str, *, repository: RepositoryRef | str = "", + cursor: str | int | None = None, limit: int | None = None, + ) -> dict: + report = self.recorder.report(trace_id, cursor=cursor, limit=limit) + if not report.get("session"): + if report.get("status") == "degraded": + self.mark_degraded(trace_id) + return report + if repository: + expected = self._repository(repository) + if report["session"].get("repository") != expected: + raise InvalidRequestError("trace belongs to a different repository") + if any(gap.get("reason") == "knowledge_capture_degraded" for gap in report.get("gaps", [])): + with self._capture_lock: + self._degraded.add(trace_id) + self._persisted_gaps.add(trace_id) + self._flush_capture_gap(trace_id) + if self.is_degraded(trace_id): + report = self.recorder.report(trace_id, cursor=cursor, limit=limit) + report["trace_status"] = "partial" + if not any(g.get("reason") == "knowledge_capture_degraded" for g in report.get("gaps", [])): + report.setdefault("gaps", []).append({"reason": "knowledge_capture_degraded", "persisted": False}) + if report.get("counts"): + report["counts"]["gaps"] = len(report["gaps"]) + return report + + def record_events( + self, trace_id: str, events: list[dict[str, Any]], *, + repository: RepositoryRef | str = "", idempotency_key: str = "", + ) -> dict: + """Submit host claims; all references remain subject to validation.""" + report = self.get_report(trace_id, repository=repository) + if not report.get("session"): + return report + if not isinstance(events, list) or len(events) > 100: + raise InvalidRequestError("events must be a list of at most 100 entries") + submitted = deepcopy(events) + allowed = {"content_provided", "evidence_registered", "assessment_recorded", "gap_recorded"} + for event in submitted: + if not isinstance(event, dict) or event.get("event_type") not in allowed: + raise InvalidRequestError("unsupported host knowledge event") + if "payload" in event: + raise InvalidRequestError("host events must use the flat knowledge-use event format") + kind = event["event_type"] + payload_key = {"content_provided": "exposure", "assessment_recorded": "assessment", + "evidence_registered": "evidence", "gap_recorded": "gap"}[kind] + if not isinstance(event.get(payload_key), dict): + raise InvalidRequestError(f"{kind} requires a {payload_key} object") + if kind == "content_provided": + exposure = event.setdefault("exposure", {}) + if exposure.get("boundary", "host_context") != "host_context": + raise InvalidRequestError("host receipts must use the host_context boundary") + exposure["boundary"] = "host_context" + exposure["reported_by"] = "host" + elif kind == "assessment_recorded": + event.setdefault("assessment", {})["reported_by"] = "host" + elif kind == "evidence_registered": + event.setdefault("evidence", {})["evidence_origin"] = "host_reported" + result = self.recorder.record_events(trace_id, submitted, idempotency_key=idempotency_key) + if result.get("status") == "degraded": + self.mark_degraded(trace_id) + return result + + # A deliberately small vocabulary for hosts: receipt + assessment batches. + record_usage = record_events + + def finish_session(self, trace_id: str, status: str = "complete", *, repository: RepositoryRef | str = "") -> dict: + report = self.get_report(trace_id, repository=repository) + if not report.get("session"): + return report + if status == "complete" and self.is_degraded(trace_id): + status = "degraded" + result = self.recorder.finish_session(trace_id, status=status) + if result.get("status") == "degraded" and not result.get("session"): + self.mark_degraded(trace_id) + return result diff --git a/src/infermatrix_copilot/sdk/v1/models.py b/src/infermatrix_copilot/sdk/v1/models.py index ef6755a9..7ae5de4e 100644 --- a/src/infermatrix_copilot/sdk/v1/models.py +++ b/src/infermatrix_copilot/sdk/v1/models.py @@ -10,6 +10,7 @@ STRICT_API_VERSION = "1.4.0" # StrictRuntimeConfig.max_workers QUALITY_API_VERSION = "1.0.0" KNOWLEDGE_API_VERSION = "1.1.0" +KNOWLEDGE_TRACE_API_VERSION = "1.0.0" class _Serializable: @@ -78,6 +79,7 @@ class DocumentPage(_Serializable): next_offset: int | None review_context_id: str = "" knowledge_feedback: dict[str, Any] = field(default_factory=dict) + knowledge_trace: dict[str, Any] = field(default_factory=dict) def to_dict(self) -> dict[str, Any]: result = super().to_dict() @@ -209,6 +211,11 @@ class Capabilities(_Serializable): max_strict_workers: int knowledge_maintenance_protocol_version: int = 1 supports_knowledge_containment: bool = True + supports_knowledge_usage_trace: bool = True + knowledge_trace_api_version: str = KNOWLEDGE_TRACE_API_VERSION + knowledge_trace_boundaries: tuple[str, ...] = ( + "provider_response", "host_context", "model_request", + ) @dataclass(frozen=True) @@ -274,6 +281,7 @@ class DirectReviewPlan(_Serializable): untested_public_api: dict[str, Any] = field(default_factory=dict) related_knowledge: dict[str, Any] = field(default_factory=dict) knowledge_feedback: dict[str, Any] = field(default_factory=dict) + knowledge_trace: dict[str, Any] = field(default_factory=dict) FeedbackStatus = Literal[ "checked", "disabled", "unavailable", "not_applicable" diff --git a/src/infermatrix_copilot/thin_mcp_server.py b/src/infermatrix_copilot/thin_mcp_server.py index 8d4e93aa..73bc7199 100644 --- a/src/infermatrix_copilot/thin_mcp_server.py +++ b/src/infermatrix_copilot/thin_mcp_server.py @@ -2,7 +2,11 @@ from __future__ import annotations +import hashlib +import json import re +import threading +from collections import OrderedDict import signal import sqlite3 import sys @@ -41,6 +45,7 @@ re.IGNORECASE, ) _PR_NUMBER = re.compile(r"^(?:pr\s*#?\s*)?(\d+)$", re.IGNORECASE) +_DEFAULT_KNOWLEDGE_REPO = "vllm-omni" def __getattr__(name: str): # formerly an import-time constant; now the knowledge root of the current view if name == "_KNOWLEDGE": @@ -275,10 +280,92 @@ def build_mcp( context_client = DirectClient(knowledge_context_path=core.settings.run_root / "knowledge-context.sqlite", allowed_knowledge_repositories=core.settings.allowed_knowledge_repositories) + from .sdk.v1.models import SDKError + from .knowledge_trace import KnowledgeTraceError + + trace_client = context_client.knowledge_trace + trace_views: OrderedDict[str, tuple[str, KnowledgeView]] = OrderedDict() + trace_lock = threading.Lock() + + def trace_guard(fn): + try: + return _guard(fn) + except (SDKError, KnowledgeTraceError) as exc: + return {"error": str(exc)} + + def pin_trace(trace_id: str, selected_repo: str) -> tuple[dict, KnowledgeView]: + report = trace_client.get_report(trace_id, repository=selected_repo) + with trace_lock: + pinned = trace_views.get(trace_id) + if pinned is not None: + return report, pinned[1] + # A restarted provider may reuse the current tree only when its full + # identity matches. A different active snapshot never replaces the + # unavailable tree silently. + from .sdk.v1.direct import _resource_revision + from .sdk._resources import adapters_root + view = KnowledgeView.current() + session = report.get("session", {}) + revision = _resource_revision(str(view.root), str(adapters_root())) + if (session.get("knowledge_snapshot") != view.public_snapshot + or session.get("resource_revision") != revision): + raise ValueError("trace knowledge snapshot is unavailable; start a new trace") + remember_trace(trace_id, selected_repo, view) + return report, view + + def remember_trace(trace_id: str, selected_repo: str, view: KnowledgeView) -> None: + with trace_lock: + trace_views[trace_id] = (selected_repo, view) + trace_views.move_to_end(trace_id) + while len(trace_views) > 256: + trace_views.popitem(last=False) + + def begin_trace(selected_repo: str, view: KnowledgeView, *, source_revision="", review_context_id="", run_id="") -> dict: + from .sdk.v1.direct import _resource_revision + from .sdk._resources import adapters_root + result = trace_client.start_session( + selected_repo, mode="direct", run_id=run_id, + source_revision=source_revision, review_context_id=review_context_id, + knowledge_snapshot=view.public_snapshot, + resource_revision=_resource_revision(str(view.root), str(adapters_root())), + instrumentation={"provider": "observed", "host_context": "unknown", "applications": "host_reported"}, + ) + if result.get("trace_id"): + remember_trace(result["trace_id"], selected_repo, view) + return result + + def exposure(trace_id: str, view: KnowledgeView, *, step_id: str, path: str, content: str, + representation: str, offset=None, truncated=False) -> dict: + from .knowledge_service.lifecycle import visible_text + served = visible_text(view.path(path).read_text(encoding="utf-8", errors="replace")) + if offset is None: + position = served.find(content) + offset = position if content and position >= 0 and served.find(content, position + 1) < 0 else None + from .sdk.v1.direct import _quick_map_rule_ids + rule_ids = _quick_map_rule_ids(content, served) if representation == "quick_map" else None + event = trace_client.capture("provide", trace_id, step_id=step_id, document_id=path, content=content, + boundary="provider_response", representation=representation, + offset=offset, offset_unit="unicode_codepoints", truncated=truncated, + snapshot=view.public_snapshot, rule_ids=rule_ids, + document_sha256="sha256:" + hashlib.sha256(served.encode("utf-8")).hexdigest(), + ) + if event.get("status") == "degraded": + raise OSError(event.get("error", "knowledge trace storage is unavailable")) + result = event.get("result", {}) + return {key: result[key] for key in ("exposure_id", "knowledge_ref_id") if key in result} + + def trace_failure(trace_id: str, exc: Exception) -> dict: + trace_client.mark_degraded(trace_id) + if trace_client.mode == "required": + raise exc + return {"trace_id": trace_id, "status": "degraded", "error": str(exc)} + # Approval-gating hosts (codex asks per-call approval for tools without # hints, and cancels them outright in headless runs) read these # annotations, so they must stay truthful: `review` reserves run state - # and its Strict child reaches the network; everything else only reads. + # and its Strict child reaches the network. Explicit trace tools write + # local records; document operations only read knowledge and append audit + # records when their optional trace context is supplied. read_only = ToolAnnotations( readOnlyHint=True, idempotentHint=True, openWorldHint=False) mcp = FastMCP( @@ -370,7 +457,7 @@ def get_update_status() -> dict: openWorldHint=True)) def review( target: str, - repo: str = "vllm-omni", + repo: str = _DEFAULT_KNOWLEDGE_REPO, mode: str = "direct", post: bool = False, review_depth: str = "", @@ -383,6 +470,7 @@ def review( diff: str = "", knowledge_profile: str = "legacy", task_id: str = "", + trace_id: str = "", ) -> dict: """Begin a Direct or Strict review. @@ -396,7 +484,7 @@ def review( ``"truncated"`` (only part of one) — must be opened, and the budget grants a read for each. Strict runs the packaged PR-review workflow and accepts an optional local checkout through ``repo_path``. - ``expected_head_sha`` (Strict only) pins the review to one snapshot: pass + ``expected_head_sha`` pins the review to one source snapshot: pass the full 40-hex head you observed, and the run stops as stale rather than reviewing a different commit if the PR moved in between. ``diff`` (Direct only, optional) is the frozen unified diff; with it the @@ -420,6 +508,8 @@ def run() -> dict: raise ValueError("mode must be 'direct' or 'strict'") if selected_mode == "direct": if knowledge_profile == "adaptive": + if trace_id: + raise ValueError("adaptive traces are bound to their context session; omit trace_id") from .sdk.v1.models import ChangedPath, DirectReviewRequest, RepositoryRef request = DirectReviewRequest(task_id or idempotency_key or f"direct:{target}:{uuid.uuid4().hex}", RepositoryRef(repo), 0, expected_head_sha, title, body, @@ -427,15 +517,73 @@ def run() -> dict: return _model_context_delivery(context_client.plan_adaptive(request)) if knowledge_profile != "legacy": raise ValueError("knowledge_profile must be legacy or adaptive") + selected_repo = _normalize_repo(repo) + if expected_head_sha and not re.fullmatch(r"[0-9a-fA-F]{40}", expected_head_sha): + raise ValueError("expected_head_sha must be exactly 40 hexadecimal characters") view = KnowledgeView.current() - return _legacy_plan_delivery(direct_review_plan( - repo, - title=title, - body=body, - changed_files=changed_files, - diff=diff, - view=view, + active_trace = trace_id + if active_trace: + report, view = pin_trace(active_trace, selected_repo) + source = report.get("session", {}).get("source_revision", "") + if expected_head_sha and source != expected_head_sha.casefold(): + raise ValueError("expected_head_sha does not match the trace session") + result = _legacy_plan_delivery(direct_review_plan( + repo, title=title, body=body, changed_files=changed_files, diff=diff, view=view, ), task_id=task_id or idempotency_key, repository=repo, view=view) + if trace_client.enabled and selected_repo in _supported_repos(): + try: + if not active_trace: + context = hashlib.sha256(json.dumps({ + "target": target, "repo": selected_repo, "head": expected_head_sha, + "title": title, "body": body, "changed_files": changed_files, + "snapshot": view.public_snapshot, + }, sort_keys=True).encode("utf-8")).hexdigest() + session = begin_trace(selected_repo, view, source_revision=expected_head_sha.casefold(), + review_context_id="sha256:" + context) + active_trace = session.get("trace_id", "") + if session.get("status") == "degraded": + result["knowledge_trace"] = session + return result + if active_trace: + routes = result.get("knowledge_routes") or [] + entries = [{"document_id": view.relative(route["path"]), + "reason": route.get("reason", ""), "owner": route.get("owner", ""), "rank": rank} + for rank, route in enumerate(routes, 1)] + retrieved = trace_client.capture("retrieve", active_trace, step_id="review.plan", entries=entries, + mechanism="direct_routing", query=title, + status="ready" if entries else "no_match") + if retrieved.get("status") == "degraded": + result["knowledge_trace"] = retrieved + return result + trace_client.capture("retrieve", active_trace, step_id="review.plan", mechanism="mandatory_review_guides", + entries=[{"document_id": view.relative(path), "reason": "mandatory_review_guide"} + for path in result.get("mandatory_review_guides", [])]) + trace_client.capture("retrieve", active_trace, step_id="review.plan", mechanism="navigation", + entries=[{"document_id": view.relative(result["knowledge_entry"]), "reason": "knowledge_navigation"}]) + related_entries = [{"document_id": doc["path"], "reason": doc.get("match_reason", ""), + "matched_files": doc.get("matched_files", []), "rank": rank} + for rank, doc in enumerate(result.get("related_knowledge", {}).get("documents", []), 1)] + trace_client.capture("retrieve", active_trace, step_id="review.plan", mechanism="related_knowledge", + entries=related_entries, query=title, status="ready" if related_entries else "no_match") + provided = [] + for route in routes: + if route.get("quick_map"): + provided.append({"document_id": view.relative(route["path"]), + **exposure(active_trace, view, step_id="review.plan", path=view.relative(route["path"]), + content=route["quick_map"], representation="quick_map", + truncated=route.get("quick_map_status") == "truncated")}) + for document in result.get("related_knowledge", {}).get("documents", []): + provided.append({"document_id": document["path"], + **exposure(active_trace, view, step_id="review.plan", path=document["path"], + content=document["content"], representation="related_excerpt", + truncated=bool(document.get("more_available")))}) + result["knowledge_trace"] = {"trace_id": active_trace, "status": "degraded" if trace_client.is_degraded(active_trace) else "ready", "exposures": provided, + "host_context": "unconfirmed", "applications": "not_reported"} + else: + result["knowledge_trace"] = session + except Exception as exc: + result["knowledge_trace"] = trace_failure(active_trace, exc) + return result if post: # Refused at the surface as well as in the policy, so the @@ -473,7 +621,7 @@ def run() -> dict: }, } - return _guard(run) + return trace_guard(run) @mcp.tool(annotations=read_only) def validate_direct_review( @@ -485,6 +633,8 @@ def validate_direct_review( existing_feedback_status: str = "", finding_dispositions: list[dict[str, str]] | None = None, knowledge_usage: dict | None = None, + trace_id: str = "", + repo: str = _DEFAULT_KNOWLEDGE_REPO, ) -> dict: """Validate the Direct completion gate before the only final comment. @@ -514,6 +664,23 @@ def validate_direct_review( finding_dispositions=finding_dispositions, knowledge_usage=knowledge_usage, ) + if trace_id: + try: + report = trace_client.get_report(trace_id, repository=_normalize_repo(repo)) + source = report.get("session", {}).get("source_revision", "") + if source and source != evidence_head_sha.casefold(): + result["review_complete"] = False + result["publish_ready"] = False + result["status"] = "partial_review" + result.setdefault("missing", []).append("evidence_head_sha does not match the trace session") + if result.get("publish_ready"): + trace_client.finish_session(trace_id, "complete", repository=_normalize_repo(repo)) + report = trace_client.get_report(trace_id, repository=_normalize_repo(repo)) + result["knowledge_trace"] = {"trace_id": trace_id, "status": report.get("trace_status", report.get("status", "partial")), + "session_status": report.get("session", {}).get("status", "unknown"), + "counts": report.get("counts", {})} + except Exception as exc: + result["knowledge_trace"] = trace_failure(trace_id, exc) result.setdefault("diagnostics", {})["timing_ms"] = { "validate_direct_review": int( (time.perf_counter() - started) * 1000 @@ -552,17 +719,19 @@ def run() -> dict: return _guard(run) @mcp.tool(annotations=read_only) - def update_knowledge(repo: str = "vllm-omni") -> dict: + def update_knowledge(repo: str = _DEFAULT_KNOWLEDGE_REPO) -> dict: """Return the knowledge contribution entrypoint for the host to follow.""" return _guard(lambda: {"knowledge_entry": _contributing_entry()}) @mcp.tool(annotations=read_only) def doc_search( query: str, - repo: str = "vllm-omni", + repo: str = _DEFAULT_KNOWLEDGE_REPO, limit: int = 20, knowledge_usage: dict | None = None, task_id: str = "", + trace_id: str = "", + step_id: str = "knowledge.search", ) -> dict: """Search knowledge; enforced reads require the review plan's knowledge_usage receipt.""" def run() -> dict: @@ -578,8 +747,40 @@ def run() -> dict: f"Supported: {supported}." ) } - matches = _docs(selected_repo, view).search(query, limit=limit) + if trace_id: + _report, trace_view = pin_trace(trace_id, selected_repo) + if session and (trace_view.root != view.root or trace_view.public_snapshot != view.public_snapshot): + raise ValueError("trace snapshot does not match the containment delivery context") + view = trace_view + try: + matches = _docs(selected_repo, view).search(query, limit=limit) + except Exception: + if trace_id: + trace_client.record_gap(trace_id, "document_search_failed") + raise + delivery = _raw_document_delivery({"repo": selected_repo, "query": query, "matches": matches}, + knowledge_usage, session, view, task_id) + metadata = {} + if trace_id: + try: + trace_client.capture("retrieve", trace_id, step_id=step_id, entries=[ + {"document_id": hit["path"], "line": hit["line"], "rank": rank} + for rank, hit in enumerate(matches, 1)], mechanism="doc_search", query=query, + status="ready" if matches else "no_match") + provided = [] + from .knowledge_service.lifecycle import visible_text + for hit in matches: + lines = visible_text(view.path(hit["path"]).read_text(encoding="utf-8", errors="replace")).splitlines(keepends=True) + start = sum(len(line) for line in lines[:hit["line"] - 1]) + provided.append({"document_id": hit["path"], **exposure( + trace_id, view, step_id=step_id, path=hit["path"], content=hit["text"], + representation="search_hit", offset=start, truncated=True)}) + metadata = {"knowledge_trace": {"trace_id": trace_id, "status": "degraded" if trace_client.is_degraded(trace_id) else "ready", "exposures": provided}} + except Exception as exc: + metadata = {"knowledge_trace": trace_failure(trace_id, exc)} result = { + **delivery, + **metadata, "query": query, "repo": selected_repo, "matches": matches, @@ -589,17 +790,19 @@ def run() -> dict: "knowledge_entry." )} if not matches else {}), } - return _raw_document_delivery(result, knowledge_usage, session, view, task_id) + return result - return _guard(run) + return trace_guard(run) @mcp.tool(annotations=read_only) def doc_read( path: str, - repo: str = "vllm-omni", + repo: str = _DEFAULT_KNOWLEDGE_REPO, offset: int = 0, knowledge_usage: dict | None = None, task_id: str = "", + trace_id: str = "", + step_id: str = "knowledge.read", ) -> dict: """Read a doc_search page; enforced reads require the review plan's knowledge_usage receipt.""" def run() -> dict: @@ -607,14 +810,58 @@ def run() -> dict: session, view = _usage_session_view(knowledge_usage) view = view or KnowledgeView.current() selected_repo = _normalize_repo(repo, view) - result = { - "repo": selected_repo, - **_docs(selected_repo, view).read(path, offset=offset), - } - return _raw_document_delivery(result, knowledge_usage, session, view, task_id) - - return _guard(run) + if trace_id: + _report, trace_view = pin_trace(trace_id, selected_repo) + if session and (trace_view.root != view.root or trace_view.public_snapshot != view.public_snapshot): + raise ValueError("trace snapshot does not match the containment delivery context") + view = trace_view + try: + result = {"repo": selected_repo, **_docs(selected_repo, view).read(path, offset=offset)} + except Exception: + if trace_id: + trace_client.record_gap(trace_id, "document_read_failed") + raise + result = _raw_document_delivery(result, knowledge_usage, session, view, task_id) + if trace_id: + try: + result["knowledge_trace"] = {"trace_id": trace_id, "status": "degraded" if trace_client.is_degraded(trace_id) else "ready", **exposure( + trace_id, view, step_id=step_id, path=result["path"], content=result["content"], + representation="full_document" if offset == 0 and result.get("next_offset") is None else "excerpt", + offset=offset, truncated=result.get("next_offset") is not None)} + except Exception as exc: + result["knowledge_trace"] = trace_failure(trace_id, exc) + return result + + return trace_guard(run) + + local_write = ToolAnnotations(readOnlyHint=False, destructiveHint=False, idempotentHint=False, openWorldHint=False) + + @mcp.tool(annotations=local_write) + def start_knowledge_trace(repo: str = _DEFAULT_KNOWLEDGE_REPO, run_id: str = "", source_revision: str = "") -> dict: + """Start local knowledge-use recording pinned to the current knowledge snapshot.""" + def run(): + selected_repo = _normalize_repo(repo) + if selected_repo not in _supported_repos(): + raise ValueError("unsupported knowledge repository") + if source_revision and not re.fullmatch(r"[0-9a-fA-F]{40}", source_revision): + raise ValueError("source_revision must be a full 40-hex head SHA") + return begin_trace(selected_repo, KnowledgeView.current(), source_revision=source_revision.casefold(), run_id=run_id) + return trace_guard(run) + + @mcp.tool(annotations=local_write) + def record_knowledge_usage(trace_id: str, events: list[dict], repo: str = _DEFAULT_KNOWLEDGE_REPO, idempotency_key: str = "") -> dict: + """Record host context receipts, evidence and explicit rule assessments locally.""" + return trace_guard(lambda: trace_client.record_events(trace_id, events, repository=_normalize_repo(repo), idempotency_key=idempotency_key)) + + @mcp.tool(annotations=local_write) + def finish_knowledge_trace(trace_id: str, status: str = "complete", repo: str = _DEFAULT_KNOWLEDGE_REPO) -> dict: + """Finish a local trace; this does not publish a review or certify its reasoning.""" + return trace_guard(lambda: trace_client.finish_session(trace_id, status, repository=_normalize_repo(repo))) + @mcp.tool(annotations=read_only) + def get_knowledge_trace(trace_id: str, repo: str = _DEFAULT_KNOWLEDGE_REPO, cursor: str = "", limit: int = 100) -> dict: + """Read the deterministic per-step report, including missing receipts and evidence.""" + return trace_guard(lambda: trace_client.get_report(trace_id, repository=_normalize_repo(repo), cursor=cursor or None, limit=limit)) @mcp.tool(annotations=ToolAnnotations( title="Inspect or revoke the knowledge learning agreement", readOnlyHint=False, destructiveHint=False, idempotentHint=True, openWorldHint=False)) diff --git a/src/infermatrix_copilot/trace_store.py b/src/infermatrix_copilot/trace_store.py index bfcbf953..35234832 100644 --- a/src/infermatrix_copilot/trace_store.py +++ b/src/infermatrix_copilot/trace_store.py @@ -77,7 +77,7 @@ from typing import Any, Callable, Iterator, Mapping SCHEMA = "trace/1" -KINDS = ("model_call", "tool_call", "decision", "outcome", "replay") +KINDS = ("model_call", "tool_call", "decision", "outcome", "replay", "knowledge_event") SCHEMA_VERSION = 2 _CONTEXT: contextvars.ContextVar[dict] = contextvars.ContextVar("trace_context", default={}) _STORE: contextvars.ContextVar["TraceStore | None"] = contextvars.ContextVar("trace_store", default=None) diff --git a/test/conftest.py b/test/conftest.py index 713f36d1..bc3eeb62 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -99,6 +99,7 @@ def _no_real_run_roots(tmp_path, monkeypatch): the real ~/.infermatrix-copilot/runs. Feedback tests also never write a user's ledger or contact a configured team collector unless opted in.""" monkeypatch.setenv("KB_RUN_ROOTS", str(tmp_path / "no-runs")) + monkeypatch.setenv("KNOWLEDGE_TRACE_ROOT", str(tmp_path / "knowledge-traces")) monkeypatch.setenv("KNOWLEDGE_LEARNING_STATE_DIR", str(tmp_path / "learning-state")) monkeypatch.setenv("KNOWLEDGE_FEEDBACK_URL", "") monkeypatch.setenv("KNOWLEDGE_FEEDBACK_TOKEN", "") diff --git a/test/test_knowledge_trace.py b/test/test_knowledge_trace.py new file mode 100644 index 00000000..bd017497 --- /dev/null +++ b/test/test_knowledge_trace.py @@ -0,0 +1,335 @@ +"""Knowledge-use traces distinguish routing, delivery and declared application.""" + +from __future__ import annotations + +import concurrent.futures +import gzip +import json +from pathlib import Path + +import pytest + +from infermatrix_copilot.knowledge_trace import KnowledgeTrace, KnowledgeTraceError, extract_rule_sections, markdown + + +DOC = "repos/demo/components/rules.md" +BODY = "### DEMO-1a — Preserve zero\n\nZero is an explicit request value.\n" + + +def session(tmp_path, **kwargs): + recorder = KnowledgeTrace(tmp_path, mode="required") + trace_id = recorder.start_session("demo", "direct", source_revision="head-1", knowledge_snapshot="snap-1", **kwargs)["trace_id"] + return recorder, trace_id + + +def exposed(recorder, trace_id, **kwargs): + return recorder.provide(trace_id, step_id="defaults", document_id=DOC, content=BODY, + boundary="provider_response", representation="full_document", **kwargs)["result"] + + +def assessment(exposure, **kwargs): + return {"event_type": "assessment_recorded", "step_id": "defaults", "assessment": { + "knowledge_ref_id": exposure["rule_refs"][0]["knowledge_ref_id"], + "exposure_ids": [exposure["exposure_id"]], "applicability": "applicable", "verification": "supported", + "rationale": "Zero must survive the default-value branch.", "check": "Run explicit-zero input through the consumer.", + "evidence_ids": ["test-1"], "decision_ids": ["finding-2"], **kwargs}} + + +def test_full_chain_survives_restart_and_exports_evidence(tmp_path): + recorder, trace_id = session(tmp_path, instrumentation={"host_context": "host_reported"}) + recorder.retrieve(trace_id, step_id="defaults", entries=[{"path": DOC, "reason": "guidance zero"}], mechanism="routing") + exposure = exposed(recorder, trace_id) + recorder.record_events(trace_id, [{"event_type": "evidence_registered", "step_id": "defaults", "evidence": { + "evidence_id": "test-1", "kind": "test", "source_revision": "head-1", "command": "pytest test_zero.py", + "result_summary": "one passed", "environment_fingerprint": "python-3.11", "content": "1 passed"}}, assessment(exposure)]) + recorder.finish_session(trace_id) + other = KnowledgeTrace(tmp_path, mode="required") + report = other.report(trace_id) + assert report["schema"] == "knowledge-use-report/1" + assert report["status"] == report["trace_status"] == "complete" + assert report["counts"]["provided_rules"] == report["counts"]["assessments"] == 1 + assert report["counts"]["unassessed_rules"] == 0 + assert report["exposures"][0]["rule_refs"][0]["coverage"] == "body_complete" + text = markdown(report) + for expected in ("guidance zero", "pytest test_zero.py", "one passed", "python-3.11", "finding-2", "Run explicit-zero"): + assert expected in text + assert other.query(repository="demo", rule_id="DEMO-1a") + + +def test_retrieval_is_not_exposure_and_quick_map_is_not_rule_body(tmp_path): + recorder, trace_id = session(tmp_path) + recorder.retrieve(trace_id, step_id="route", entries=[{"path": DOC, "rule_ids": ["DEMO-1a"]}], mechanism="route") + assert recorder.report(trace_id)["counts"]["exposures"] == 0 + recorder.provide(trace_id, step_id="route", document_id=DOC, content="- DEMO-1a: preserve zero", + boundary="provider_response", representation="quick_map", rule_ids=["DEMO-1a"]) + report = recorder.report(trace_id) + assert report["exposures"][0]["rule_refs"][0]["coverage"] == "reference_only" + assert report["counts"]["unassessed_rules"] == 1 + assert not report["assessments"] + + +def test_partial_body_and_utf8_ranges_are_precise(tmp_path): + recorder, trace_id = session(tmp_path) + event = recorder.provide(trace_id, step_id="read", document_id=DOC, content=BODY, + boundary="model_request", representation="excerpt", truncated=True, offset=9, offset_unit="utf8_bytes") + exposure = event["result"]["exposure"] + assert exposure["included_ranges"] == [[9, 9 + len(BODY.encode("utf-8"))]] + assert exposure["rule_refs"][0]["coverage"] == "body_excerpt" + + +def test_unknown_and_split_utf8_source_ranges_are_not_fabricated(tmp_path): + recorder, trace_id = session(tmp_path) + unknown = exposed(recorder, trace_id, offset=None)["exposure"] + assert unknown["included_ranges"] == [] and unknown["range_status"] == "unknown" + split = recorder.provide(trace_id, step_id="read", document_id=DOC, + content="�", boundary="provider_response", offset=7, + offset_unit="utf8_bytes", source_length=1)["result"]["exposure"] + assert split["included_ranges"] == [[7, 8]] + + +def test_host_corrections_cannot_remove_observed_content_or_evidence(tmp_path): + recorder, trace_id = session(tmp_path) + exposure = exposed(recorder, trace_id) + content_event = recorder.report(trace_id)["events"][-1]["result"]["event_id"] + evidence_event = recorder.record_events(trace_id, [{"event_type": "evidence_registered", + "evidence": {"evidence_id": "test-1", "kind": "test", "source_revision": "head-1"}}])["events"][0] + for event in [ + {"event_type": "content_provided", "supersedes": content_event, + "exposure": {"source_exposure_id": exposure["exposure_id"], "boundary": "host_context"}}, + {"event_type": "evidence_registered", "supersedes": evidence_event["result"]["event_id"], + "evidence": {"evidence_id": "replacement", "kind": "test"}}, + ]: + with pytest.raises(KnowledgeTraceError, match="immutable"): + recorder.record_events(trace_id, [event]) + recorder.record_events(trace_id, [assessment(exposure)]) + report = recorder.report(trace_id) + assert report["evidence"][0]["evidence_id"] == "test-1" + assert report["assessments"][0]["evidence_ids"] == ["test-1"] + + +def test_host_entity_ids_cannot_retarget_existing_report_links(tmp_path): + recorder, trace_id = session(tmp_path) + first = exposed(recorder, trace_id) + second = recorder.provide(trace_id, step_id="other", document_id="repos/demo/other.md", + content=BODY, boundary="provider_response")["result"] + receipt = {"event_type": "content_provided", "step_id": "defaults", "exposure": { + "source_exposure_id": first["exposure_id"], "exposure_id": "host-1", "boundary": "host_context"}} + declaration = assessment(first, assessment_id="assessment-1", verification="unresolved", evidence_ids=[]) + recorder.record_events(trace_id, [receipt, declaration]) + with pytest.raises(KnowledgeTraceError, match="exposure_id conflicts"): + recorder.record_events(trace_id, [{**receipt, "exposure": { + **receipt["exposure"], "source_exposure_id": second["exposure_id"]}}]) + with pytest.raises(KnowledgeTraceError, match="assessment_id conflicts"): + recorder.record_events(trace_id, [assessment(first, assessment_id="assessment-1", + verification="unresolved", evidence_ids=[], rationale="A different assertion")]) + report = recorder.report(trace_id) + assert next(e for e in report["exposures"] if e["exposure_id"] == "host-1")["document_id"] == DOC + assert len(report["assessments"]) == 1 + + +def test_full_usage_batch_retry_is_idempotent_after_normalization(tmp_path): + recorder, trace_id = session(tmp_path) + first = exposed(recorder, trace_id) + events = [ + {"event_type": "content_provided", "exposure": { + "source_exposure_id": first["exposure_id"], "boundary": "host_context"}}, + {"event_type": "evidence_registered", "evidence": { + "evidence_id": "test-1", "kind": "test", "source_revision": "head-1", "content": "one passed"}}, + assessment(first), + ] + original = recorder.record_events(trace_id, events, idempotency_key="whole-batch") + retried = recorder.record_events(trace_id, events, idempotency_key="whole-batch") + assert [e["id"] for e in retried["events"]] == [e["id"] for e in original["events"]] + assert len(recorder.report(trace_id)["assessments"]) == 1 + + +@pytest.mark.parametrize("bad_event", [ + {"event_type": "gap_recorded", "step_id": ["bad"], "gap": {"reason": "invalid"}}, + {"event_type": "gap_recorded", "step_id": "bad\nstep", "gap": {"reason": "invalid"}}, +]) +def test_invalid_steps_cannot_poison_persisted_reports(tmp_path, bad_event): + recorder, trace_id = session(tmp_path) + with pytest.raises(KnowledgeTraceError, match="step_id"): + recorder.record_events(trace_id, [bad_event]) + assert recorder.report(trace_id)["counts"]["events"] == 1 + + +def test_engine_moa_step_labels_are_opaque_and_preserved(tmp_path): + recorder, trace_id = session(tmp_path) + step_id = "agent.answer_issue#moa/provider/model:reasoning" + recorder.retrieve(trace_id, step_id=step_id, entries=[{"path": DOC}], mechanism="routing") + recorder.provide(trace_id, step_id=step_id, document_id=DOC, content=BODY, + boundary="model_request", representation="full_document") + report = recorder.report(trace_id) + assert report["steps"][0]["step_id"] == step_id + assert report["retrievals"][0]["step_id"] == step_id + assert report["exposures"][0]["step_id"] == step_id + + +def test_rule_parser_ignores_mentions_and_fenced_examples(): + content = "DEMO-2b appears here\n```md\n### DEMO-3c — example\nbody\n```\n" + BODY + assert [s["rule_id"] for s in extract_rule_sections(content)] == ["DEMO-1a"] + assert not extract_rule_sections("### Rule DEMO-4d\nbody\n") + nested = "## DIFF-2af — parent\nParent.\n### DIFF-4aa — child\nChild.\n## Next\nAfter.\n" + sections = extract_rule_sections(nested) + assert [s["rule_id"] for s in sections] == ["DIFF-2af", "DIFF-4aa"] + assert all(nested[s["end"]:].startswith("## Next") for s in sections) + + +@pytest.mark.parametrize("path", ["../general/rules.md", "/general/rules.md", "repos/other/rules.md", "general/../rules.md", "general\\rules.md"]) +def test_repository_scope_cannot_be_escaped(tmp_path, path): + recorder, trace_id = session(tmp_path) + with pytest.raises(KnowledgeTraceError): + recorder.provide(trace_id, step_id="read", document_id=path, content=BODY, boundary="provider_response") + + +def test_navigation_shared_general_and_repository_namespace_are_distinct(tmp_path): + recorder, trace_id = session(tmp_path) + for path in ("AGENTS.md", "README.md", "repos/_index.md", "general/review/rules.md"): + recorder.provide(trace_id, step_id="read", document_id=path, content=BODY, boundary="provider_response") + assert len(recorder.report(trace_id)["references"]) == 8 + + +def test_host_receipt_preserves_exact_source_and_rejects_identity_override(tmp_path): + recorder, trace_id = session(tmp_path) + exposure = exposed(recorder, trace_id) + receipt = {"event_type": "content_provided", "step_id": "prompt", "exposure": { + "boundary": "host_context", "source_exposure_id": exposure["exposure_id"]}} + result = recorder.record_events(trace_id, [receipt])["events"][0]["result"]["exposure"] + assert result["payload_ref"] == exposure["exposure"]["payload_ref"] + assert result["reference"] == exposure["exposure"]["reference"] + assert result["boundary"] == "host_context" and result["reported_by"] == "host" + for key, value in (("reference", {}), ("payload_ref", "bad"), ("rule_refs", []), ("document_id", "repos/other/rules.md"), ("included_ranges", [[0, 1]])): + receipt["exposure"][key] = value + with pytest.raises(KnowledgeTraceError, match="cannot override"): + recorder.record_events(trace_id, [receipt]) + del receipt["exposure"][key] + + +def test_invalid_assessments_and_missing_evidence_are_refused(tmp_path): + recorder, trace_id = session(tmp_path) + exposure = exposed(recorder, trace_id) + for changes in ({"knowledge_ref_id": "unknown"}, {"exposure_ids": ["unknown"]}, {"evidence_ids": []}, + {"evidence_ids": ["unknown"]}, {"rationale": ""}, {"check": ""}): + with pytest.raises(KnowledgeTraceError): + recorder.record_events(trace_id, [assessment(exposure, **changes)]) + assert not recorder.report(trace_id)["assessments"] + + +def test_stale_evidence_is_visible_and_weakens_capture_status(tmp_path): + recorder, trace_id = session(tmp_path) + exposure = exposed(recorder, trace_id) + recorder.record_events(trace_id, [{"event_type": "evidence_registered", "evidence": { + "evidence_id": "test-1", "source_revision": "old-head", "kind": "test"}}, assessment(exposure)]) + recorder.finish_session(trace_id) + report = recorder.report(trace_id) + assert report["evidence"][0]["revision_status"] == "stale" + assert report["trace_status"] == "partial" + assert any(g["reason"] == "assessment_evidence_revision_stale" for g in report["gaps"]) + + +def test_snapshot_and_session_identity_are_immutable(tmp_path): + recorder, trace_id = session(tmp_path, trace_id="session-fixed") + with pytest.raises(KnowledgeTraceError, match="immutable"): + recorder.start_session("other", "direct", trace_id=trace_id) + with pytest.raises(KnowledgeTraceError, match="snapshot"): + exposed(recorder, trace_id, snapshot="snap-2") + + +def test_duplicate_delivery_batches_are_idempotent_across_instances(tmp_path): + recorder, trace_id = session(tmp_path) + exposure = exposed(recorder, trace_id) + events = [{"event_type": "content_provided", "step_id": "prompt", "exposure": { + "boundary": "host_context", "source_exposure_id": exposure["exposure_id"]}}] + first = recorder.record_events(trace_id, events, idempotency_key="host-1") + second = KnowledgeTrace(tmp_path, mode="required").record_events(trace_id, events, idempotency_key="host-1") + assert first == second + assert recorder.report(trace_id)["counts"]["exposures"] == 2 + events[0]["step_id"] = "different" + with pytest.raises(KnowledgeTraceError, match="conflicts"): + recorder.record_events(trace_id, events, idempotency_key="host-1") + + +def test_duplicate_evidence_is_idempotent_and_batches_link_assessments(tmp_path): + recorder, trace_id = session(tmp_path) + exposure = exposed(recorder, trace_id) + events = [{"event_type": "evidence_registered", "evidence": { + "evidence_id": "test-1", "kind": "test", "source_revision": "head-1", "content": "passed"}}, assessment(exposure)] + first = recorder.record_events(trace_id, events, idempotency_key="batch") + second = KnowledgeTrace(tmp_path, mode="required").record_events(trace_id, events, idempotency_key="batch") + assert first == second + + +def test_wall_clock_does_not_define_session_causality(tmp_path): + recorder = KnowledgeTrace(tmp_path, mode="required") + times = iter([100, 100, 99, 98]) + recorder.store._clock = lambda: next(times) + trace_id = recorder.start_session("demo", "strict")["trace_id"] + exposed(recorder, trace_id) + recorder.record_events(trace_id, [{"event_type": "gap_recorded", "gap": {"reason": "native_reads_unknown"}}]) + recorder.finish_session(trace_id) + report = recorder.report(trace_id) + assert [e["result"]["session_seq"] for e in report["events"]] == [1, 2, 3, 4] + assert report["status"] == "complete" + + +def _write_duplicate(root, trace_id): + return KnowledgeTrace(root, mode="required").record_events(trace_id, [{"event_type": "gap_recorded", + "step_id": "x", "gap": {"reason": "native_file_reads_unknown"}}], idempotency_key="parallel")["events"][0]["id"] + + +def test_cross_process_duplicate_append_is_serialized(tmp_path): + _, trace_id = session(tmp_path) + with concurrent.futures.ProcessPoolExecutor(max_workers=3) as pool: + ids = list(pool.map(_write_duplicate, [str(tmp_path)] * 6, [trace_id] * 6)) + assert len(set(ids)) == 1 + assert KnowledgeTrace(tmp_path).report(trace_id)["counts"]["events"] == 2 + + +def test_index_rebuild_and_missing_blobs_preserve_honest_report(tmp_path): + recorder, trace_id = session(tmp_path) + exposed(recorder, trace_id) + before = recorder.report(trace_id) + (tmp_path / "index.db").unlink() + other = KnowledgeTrace(tmp_path, mode="required") + assert other.report(trace_id) == before + for path in (tmp_path / "blobs").rglob("*.gz"): + path.unlink() + report = other.report(trace_id) + assert report["exposures"][0]["content_available"] is False + assert report["gaps"][0]["reason"] == "archived_content_unavailable" + + +def test_redaction_reaches_metadata_and_content(tmp_path, monkeypatch): + secret = "private-value-123456789" + monkeypatch.setenv("TEST_API_KEY", secret) + recorder, trace_id = session(tmp_path) + recorder.provide(trace_id, step_id="read", document_id=DOC, content=BODY + secret, boundary="provider_response") + recorder.record_events(trace_id, [{"event_type": "gap_recorded", "gap": {"reason": secret}}]) + raw = b"".join(p.read_bytes() for p in (tmp_path / "records").glob("*.jsonl")) + raw += b"".join(gzip.decompress(p.read_bytes()) for p in (tmp_path / "blobs").rglob("*.gz")) + assert secret.encode() not in raw + assert b"[REDACTED]" in raw + + +def test_off_storage_failures_required_policy_and_report_paging(tmp_path): + off_root = tmp_path / "off" + off = KnowledgeTrace(off_root, mode="off") + assert off.start_session("demo", "direct")["status"] == "off" + assert not off_root.exists() + bad_root = tmp_path / "file" + bad_root.write_text("not a directory") + assert KnowledgeTrace(bad_root, mode="best_effort").start_session("demo", "direct")["status"] == "degraded" + with pytest.raises(KnowledgeTraceError, match="required"): + KnowledgeTrace(bad_root, mode="required").start_session("demo", "direct") + recorder, trace_id = session(tmp_path / "good") + exposed(recorder, trace_id) + first = recorder.report(trace_id, limit=1) + assert first["counts"]["events"] == 2 and len(first["events"]) == 1 + assert first["next_cursor"] == "1" + second = recorder.report(trace_id, cursor=first["next_cursor"], limit=1) + assert second["next_cursor"] is None + assert second["events"][0]["result"]["event_type"] == "content_provided" + for limit in (0, -1, 1001, True): + with pytest.raises(KnowledgeTraceError): + recorder.report(trace_id, limit=limit) diff --git a/test/test_knowledge_trace_adaptive.py b/test/test_knowledge_trace_adaptive.py new file mode 100644 index 00000000..bed9268e --- /dev/null +++ b/test/test_knowledge_trace_adaptive.py @@ -0,0 +1,49 @@ +"""Adaptive capture must observe the budgeted packet, never source references.""" + +import hashlib +import json + +import pytest + +from infermatrix_copilot.knowledge_trace.adaptive import packet_fragments +from infermatrix_copilot.knowledge_trace.references import KnowledgeTraceError + + +def packet(*fragments): + content = "".join("\n\n" + json.dumps(fragment).replace("<", "\\u003c") + + "\n\n" for fragment in fragments) + return {"model_content": content, + "model_content_sha256": hashlib.sha256(content.encode()).hexdigest(), + "documents": [{"path": "repos/other/secret.md", "content": "not delivered"}]} + + +def test_only_delivered_fragments_preserve_provenance_and_wire(): + source = packet({"path": "repos/demo/rules.md", "content": "规则 ", + "offset": 40, "partial": True, "repo_id": "demo", "source_pins": ["a" * 40]}) + rows = packet_fragments(source) + assert len(rows) == 1 + assert rows[0]["content"] == "规则 " + assert rows[0]["offset"] == 40 + assert rows[0]["truncated"] is True + assert rows[0]["source_pins"] == ["a" * 40] + assert rows[0]["wire_content"] == source["model_content"] + + +@pytest.mark.parametrize("metadata", [{}, {"line": 10}, {"offset": 3, "continuation": True}]) +def test_unknown_coordinates_are_not_invented(metadata): + assert packet_fragments(packet({"path": "general/rules.md", "content": "text", **metadata}))[0]["offset"] is None + + +def test_metadata_and_empty_budget_do_not_create_exposures(): + assert packet_fragments({"model_content": "", "status": "no_new_context", + "documents": [{"path": "general/rules.md"}]}) == [] + + +def test_hash_mismatch_and_malformed_content_fail_explicitly(): + source = packet({"path": "general/rules.md", "content": "text"}) + source["model_content"] += "unframed" + with pytest.raises(KnowledgeTraceError, match="hash mismatch"): + packet_fragments(source) + source.pop("model_content_sha256") + with pytest.raises(KnowledgeTraceError, match="unframed"): + packet_fragments(source) diff --git a/test/test_knowledge_trace_adaptive_direct.py b/test/test_knowledge_trace_adaptive_direct.py new file mode 100644 index 00000000..b7064fe8 --- /dev/null +++ b/test/test_knowledge_trace_adaptive_direct.py @@ -0,0 +1,193 @@ +"""Adaptive traces observe delivered packets without spending context twice.""" + +import json +from dataclasses import replace + +import pytest + +from infermatrix_copilot.kb_service.repo_spec import RepoSpec +from infermatrix_copilot.knowledge_context import ContextBudget, ContextError +from infermatrix_copilot.knowledge_trace import KnowledgeTraceError +from infermatrix_copilot.knowledge_trace.adaptive import packet_fragments +from infermatrix_copilot.knowledge_view import build_manifest +from infermatrix_copilot.sdk.v1 import ChangedPath, DirectClient, DirectReviewRequest, RepositoryRef + +PIN = "a" * 40 + + +def configured_client(tmp_path, monkeypatch, *, allowed=(), mode="required"): + root = tmp_path / "snapshot" / "knowledge" + docs = {"AGENTS.md": "Knowledge navigation", "general/_index.md": "General navigation", + "general/review/guides/simplification-audit.md": "Read the current source before reporting findings."} + for name in ("r", "other"): + docs[f"repos/{name}/_index.md"] = "Repository navigation" + docs[f"repos/{name}/doc.md"] = ( + f"---\ntitle: API\ntype: guide\nfeature: api\nsources: [acme/{name}@{PIN}:src/api.py]\n---\n\n" + + "## RULE-1 API contract\nPreserve zero values. 真实的源码解释。\n" * 300) + for path, text in docs.items(): + target = root / path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(text) + link = {"repo_id": "other", "status": "confirmed", "source_pin": PIN, "catalog_hash": "b" * 64} + specs = {name: RepoSpec(name, source_pin=PIN, catalog_hash="b" * 64, policy_hash="c" * 64, + dependencies=(link,) if name == "r" else ()).to_dict() + for name in ("r", "other")} + (root / "_repositories.yaml").write_text(json.dumps({"schema_version": 1, "repos": specs})) + (root.parent / "MANIFEST.json").write_text(json.dumps(build_manifest(root, "trace-adaptive"))) + monkeypatch.setenv("KNOWLEDGE_ROOT", str(root.parent)) + client = DirectClient(knowledge_context_path=tmp_path / "context.sqlite", trace_root=tmp_path / "trace", + trace_mode=mode, allowed_knowledge_repositories=allowed) + request = DirectReviewRequest("adaptive-review", RepositoryRef("r"), 0, PIN, "API", "API contract", + (ChangedPath("src/api.py"),)) + return client, request + + +def tiny_budget(): + return ContextBudget(initial_tokens=1000, max_tokens=4000, model_context_tokens=8000, + source_reserve_tokens=2000, output_reserve_tokens=1000) + + +def captured_text(client, report): + return [client.knowledge_trace.recorder.store.blob(e["payload_ref"]) for e in report["exposures"]] + + +def test_adaptive_sdk_captures_only_post_budget_text_and_resumes_trace(tmp_path, monkeypatch): + client, request = configured_client(tmp_path, monkeypatch) + opened = client.open_knowledge_context(request, budget=tiny_budget()) + session = opened["session_id"] + first = client.read_knowledge_context(session, "repos/r/doc.md") + fragments = packet_fragments(first) + assert first["truncated"] and len(first["model_content"].encode()) <= 1000 + report = client.knowledge_trace.get_report(first["knowledge_trace"]["trace_id"], repository="r") + assert captured_text(client, report) == [fragment["content"] for fragment in fragments] + assert report["exposures"][0]["included_ranges"] == [[0, len(fragments[0]["content"])]] + assert report["retrievals"][0]["entries"][0]["source_pins"] == [PIN] + assert all("excerpt" not in document and "content" not in document for document in first["documents"]) + assert opened["knowledge_trace"]["trace_id"] == first["knowledge_trace"]["trace_id"] + + restarted = DirectClient(knowledge_context_path=tmp_path / "context.sqlite", trace_root=tmp_path / "trace", + trace_mode="required") + retry = restarted.read_knowledge_context(session, "repos/r/doc.md") + assert retry["model_content"] == first["model_content"] + assert retry["consumed_tokens"] == first["consumed_tokens"] + assert retry["knowledge_trace"]["trace_id"] == first["knowledge_trace"]["trace_id"] + restarted.expand_knowledge_context(session, target_tokens=4000, reason="Read remaining contract") + continuation = restarted.read_knowledge_context(session, "repos/r/doc.md") + assert continuation["documents"][0]["continuation"] + report = restarted.knowledge_trace.get_report(continuation["knowledge_trace"]["trace_id"]) + assert captured_text(restarted, report)[-1] == packet_fragments(continuation)[0]["content"] + assert report["exposures"][-1]["range_status"] == "unknown" + assert report["retrievals"][-1]["entries"][0]["continuation"] is True + + +def test_adaptive_plan_search_related_packets_use_same_trace(tmp_path, monkeypatch): + client, request = configured_client(tmp_path, monkeypatch) + plan = client.plan_adaptive(request, budget=ContextBudget(initial_tokens=4000, max_tokens=12000)) + trace_id = plan["knowledge_trace"]["trace_id"] + report = client.knowledge_trace.get_report(trace_id) + assert report["session"]["review_context_id"] == "" + assert report["session"]["instrumentation"]["adaptive_context_session_id"] == plan["session_id"] + assert captured_text(client, report) == [fragment["content"] for fragment in packet_fragments(plan)] + assert report["retrievals"][0]["mechanism"] == "adaptive_review" + session = plan["session_id"] + search = client.search_knowledge_context(session, "Preserve zero") + related = client.related_knowledge_context(session, ["src/api.py"], query="API") + assert search["knowledge_trace"]["trace_id"] == related["knowledge_trace"]["trace_id"] == trace_id + report = client.knowledge_trace.get_report(trace_id) + assert {row["mechanism"] for row in report["retrievals"]} == { + "adaptive_review", "adaptive_search", "adaptive_related"} + for exposure in search["knowledge_trace"]["exposures"]: + matching = next(row for row in report["exposures"] if row["exposure_id"] == exposure["exposure_id"]) + assert matching["range_status"] == "unknown" + + +def test_authorized_crossrepo_delivery_records_gap_and_does_not_grant_access(tmp_path, monkeypatch): + client, request = configured_client(tmp_path, monkeypatch, allowed=("other",)) + opened = client.open_knowledge_context(request, budget=tiny_budget()) + packet = client.read_knowledge_context(opened["session_id"], "repos/other/doc.md", repository="other") + assert packet["model_content"] and packet["knowledge_trace"]["status"] == "ready" + report = client.knowledge_trace.get_report(packet["knowledge_trace"]["trace_id"]) + assert not report["exposures"] + assert any(g["reason"] == "context_cross_repository_unobserved" for g in report["gaps"]) + denied = DirectClient(knowledge_context_path=tmp_path / "denied.sqlite", trace_root=tmp_path / "trace", + trace_mode="required") + session = denied.open_knowledge_context(replace(request, review_id="denied"))["session_id"] + with pytest.raises(ContextError, match="authorized"): + denied.read_knowledge_context(session, "repos/other/doc.md", repository="other") + + +def test_adaptive_hash_validation_and_failed_capture_are_explicit(tmp_path, monkeypatch): + client, request = configured_client(tmp_path, monkeypatch) + session = client.open_knowledge_context(request)["session_id"] + with pytest.raises(KnowledgeTraceError, match="hash mismatch"): + client._trace_adaptive(session, {"model_content": "", "model_content_sha256": "0" * 64}) + trace_id = "kt-adaptive-" + session + assert client.knowledge_trace.is_degraded(trace_id) + restarted = DirectClient(knowledge_context_path=tmp_path / "context.sqlite", trace_root=tmp_path / "trace") + assert restarted.knowledge_trace.get_report(trace_id)["trace_status"] == "partial" + + +def test_mcp_adaptive_context_tools_keep_trace_through_provider_restart(tmp_path, monkeypatch): + from test_knowledge_trace_direct import fake_mcp + + _client, _request = configured_client(tmp_path, monkeypatch) + monkeypatch.setenv("KNOWLEDGE_TRACE_ROOT", str(tmp_path / "mcp-trace")) + # Keep the runtime context ledger in a deterministic location across tool registries. + import infermatrix_copilot.sdk.v1.direct as direct_module + def make_client(**kwargs): + return DirectClient(**{**kwargs, "knowledge_context_path": tmp_path / "mcp-context.sqlite"}) + monkeypatch.setattr(direct_module, "DirectClient", make_client) + tools = fake_mcp(monkeypatch) + opened = tools["open_knowledge_context"](repo="r", source_pin=PIN, review_id="mcp-context") + first = tools["search_knowledge_context"](opened["session_id"], "Preserve zero") + trace_id = first["knowledge_trace"]["trace_id"] + tools = fake_mcp(monkeypatch) + retry = tools["search_knowledge_context"](opened["session_id"], "Preserve zero") + assert retry["model_content"] == first["model_content"] + assert retry["knowledge_trace"]["trace_id"] == trace_id + report = tools["get_knowledge_trace"](trace_id, repo="r") + assert report["exposures"] + assert "error" in tools["get_knowledge_trace"](trace_id, repo="other") + plan = tools["review"]("7", repo="r", knowledge_profile="adaptive", task_id="mcp-review", + expected_head_sha=PIN, title="API", body="API contract", changed_files=["src/api.py"]) + assert plan["knowledge_trace"]["trace_id"] != trace_id + report = tools["get_knowledge_trace"](plan["knowledge_trace"]["trace_id"], repo="r") + assert len(report["exposures"]) == len(packet_fragments(plan)) + assert all("content" not in row and "excerpt" not in row for row in plan["documents"]) + + +# Reuse the signed-policy fixture to verify trace metadata participates in +# containment's exact-payload provenance, including after a client restart. +from test_kb_adaptive_containment import adaptive as containment_fixture + + +@pytest.mark.parametrize("operation", ["legacy_read", "read", "search", "related"]) +def test_final_traced_sdk_payload_has_exact_containment_issuance(containment_fixture, operation): + from infermatrix_copilot.knowledge_service.containment import digest, knowledge_usage_record + from test_kb_adaptive_containment import FOLLOWUP + + c = containment_fixture + client = DirectClient(knowledge_context_path=c.context_path, knowledge_maintenance=c.cfg, + trace_root=c.tmp_path / "trace", trace_mode="required") + if operation == "legacy_read": + plan = client.plan(c.request) + packet = plan.to_dict() + context_id = plan.review_context_id + followup = client.read_document(FOLLOWUP, review_context_id=context_id).to_dict() + else: + packet = client.plan_adaptive(c.request) + context_id = packet["review_context_id"] + restarted = DirectClient(knowledge_context_path=c.context_path, knowledge_maintenance=c.cfg, + trace_root=c.tmp_path / "trace", trace_mode="required") + session = packet["session_id"] + if operation == "read": + followup = restarted.read_knowledge_context(session, FOLLOWUP) + elif operation == "search": + followup = restarted.search_knowledge_context(session, "UniqueFollowup values") + else: + followup = restarted.related_knowledge_context(session, ["src/followup.py"], query="UniqueFollowup") + for delivered in (packet, followup): + assert delivered["knowledge_trace"]["trace_id"] + receipt = knowledge_usage_record(delivered, injected=True, config=c.cfg) + assert receipt["context_sha256"] == digest(delivered) + assert client._issued_contexts[context_id].knowledge_usage["context_sha256"] == digest(packet) diff --git a/test/test_knowledge_trace_adaptive_strict.py b/test/test_knowledge_trace_adaptive_strict.py new file mode 100644 index 00000000..59fc1124 --- /dev/null +++ b/test/test_knowledge_trace_adaptive_strict.py @@ -0,0 +1,186 @@ +"""Adaptive trace capture follows the budgeted bytes through final injection.""" + +import asyncio +import hashlib +import json + +import pytest +import yaml + +from infermatrix_copilot.engine.agent_runtime.knowledge import _repo_docs_tool +from infermatrix_copilot.engine.agent_runtime.knowledge_trace import StepKnowledgeTrace, get_knowledge_trace +from infermatrix_copilot.engine.agent_runtime.runner import run_agent_step +from infermatrix_copilot.engine.step import StepContext +from infermatrix_copilot.kb_service.repo_spec import RepoSpec +from infermatrix_copilot.knowledge_context import ContextError +from infermatrix_copilot.knowledge_trace import KnowledgeTrace +from infermatrix_copilot.knowledge_view import build_manifest +from infermatrix_copilot.llm import Block, Reply + + +PIN = "a" * 40 + + +@pytest.fixture +def adaptive_context(settings, trace, tmp_path, monkeypatch): + monkeypatch.setenv("KNOWLEDGE_TRACE_MODE", "required") + monkeypatch.delenv("KNOWLEDGE_TRACE_ROOT", raising=False) + root = tmp_path / "snapshot" / "knowledge" + (root / "general").mkdir(parents=True) + (root / "AGENTS.md").write_text("Knowledge navigation") + (root / "general/briefing.md").write_text("## TRACE-1 Preserve values\nCheck and \\\"quoted\\\" values.\n") + specs = {} + for name in ("r", "other"): + (root / "repos" / name).mkdir(parents=True) + (root / "repos" / name / "_index.md").write_text("Repository navigation") + (root / "repos" / name / "doc.md").write_text( + f"---\ntitle: API\ntype: guide\nfeature: api\nsources: [acme/{name}@{PIN}:src/api.py]\n---\n\n" + "## TRACE-2 Source contract\nCheck and \\\"quoted\\\" values.\n" + "more context 中文. " * 500) + specs[name] = RepoSpec(name, aliases=(f"git.example/subgroup/{name}",), source_pin=PIN, + catalog_hash="b" * 64, policy_hash="c" * 64, + dependencies=({"repo_id": "other", "status": "confirmed", "source_pin": PIN, + "catalog_hash": "b" * 64},) if name == "r" else ()) + (root / "_repositories.yaml").write_text(yaml.safe_dump({"schema_version": 1, + "repos": {key: spec.to_dict() for key, spec in specs.items()}})) + manifest = build_manifest(root, "adaptive-trace") + (root.parent / "MANIFEST.json").write_text(json.dumps(manifest)) + monkeypatch.setenv("KNOWLEDGE_ROOT", str(root.parent)) + settings.knowledge_dir = root + settings.knowledge_general_docs = ["general/briefing.md"] + settings.allowed_knowledge_repositories = ("other",) + run_dir = tmp_path / "run" + run_dir.mkdir() + return StepContext(settings=settings, trace=trace, run_dir=run_dir, + params={"knowledge_context_profile": "adaptive", "knowledge_context_budget": { + "initial_tokens": 1400, "max_tokens": 5000}}, + state={"task_spec": {"kind": "pr_review", "repo": "git.example/subgroup/r", + "expected_head_sha": PIN}}) + + +def report(ctx): + return KnowledgeTrace(ctx.state["knowledge_trace_root"]).report(ctx.state["knowledge_trace_id"]) + + +@pytest.mark.parametrize("wire", ["model_content", "tool_packet"]) +def test_adaptive_strict_traces_only_budgeted_fragments_at_request(adaptive_context, wire): + ctx = adaptive_context + recorder = StepKnowledgeTrace(get_knowledge_trace(ctx), "review#adaptive") + tools = _repo_docs_tool(ctx, None, recorder) + packet = json.loads(tools["doc_read"].handler("repos/r/doc.md")) + assert packet["truncated"] + assert packet["consumed_tokens"] <= 1400 + assert "content" not in packet["documents"][0] + assert report(ctx)["session"]["repository"] == "r" + assert report(ctx)["exposures"] == [] + # A model request which omitted the packet must not count as delivery. + recorder.observe([{"content": "source evidence only"}]) + assert report(ctx)["exposures"] == [] + text = packet["model_content"] if wire == "model_content" else json.dumps(packet, ensure_ascii=False) + recorder.observe([{"content": text}]) + exposed = report(ctx)["exposures"] + assert len(exposed) == 1 + assert exposed[0]["boundary"] == "model_request" + assert exposed[0]["truncated"] is True + payload = recorder.trace.service.store.blob(exposed[0]["payload_ref"]) + assert "" in payload + assert payload in (ctx.settings.knowledge_dir / "repos/r/doc.md").read_text() + assert exposed[0]["included_ranges"] == [[0, len(payload)]] + + +def test_adaptive_strict_continuation_keeps_range_unknown(adaptive_context): + ctx = adaptive_context + recorder = StepKnowledgeTrace(get_knowledge_trace(ctx), "review") + tools = _repo_docs_tool(ctx, None, recorder) + packet = json.loads(tools["doc_read"].handler("repos/r/doc.md")) + recorder.observe([{"content": packet["model_content"]}]) + tools["doc_context_expand"].handler(target_tokens=5000, reason="Read the rest of the source contract") + continued = json.loads(tools["doc_read"].handler("repos/r/doc.md")) + assert continued["documents"][0]["continuation"] + recorder.observe([{"content": continued["model_content"]}]) + exposed = report(ctx)["exposures"] + assert len(exposed) == 2 + assert exposed[1]["range_status"] == "unknown" + assert exposed[1]["included_ranges"] == [] + + +@pytest.mark.parametrize("tool", ["doc_search", "doc_related"]) +def test_adaptive_search_and_related_keep_only_returned_prose(adaptive_context, tool): + ctx = adaptive_context + recorder = StepKnowledgeTrace(get_knowledge_trace(ctx), "review") + tools = _repo_docs_tool(ctx, None, recorder) + args = {"query": "Source contract"} if tool == "doc_search" else {"changed_files": ["src/api.py"]} + text = tools[tool].handler(**args) + packet = json.loads(text) + assert packet["model_content"] + recorder.observe([{"content": text}]) + records = report(ctx) + assert records["exposures"] + assert records["retrievals"][-1]["mechanism"] == tool + assert all(e["range_status"] == "unknown" for e in records["exposures"]) + for exposure in records["exposures"]: + payload = recorder.trace.service.store.blob(exposure["payload_ref"]) + assert json.dumps(payload, ensure_ascii=False)[1:-1].replace("<", "\\u003c") in packet["model_content"] + + +def test_adaptive_no_new_context_does_not_invent_exposure(adaptive_context): + ctx = adaptive_context + recorder = StepKnowledgeTrace(get_knowledge_trace(ctx), "review") + tools = _repo_docs_tool(ctx, None, recorder) + first = json.loads(tools["doc_search"].handler(query="Preserve values")) + recorder.observe([{"content": first["model_content"]}]) + count = len(report(ctx)["exposures"]) + second = json.loads(tools["doc_search"].handler(query="preserve VALUES")) + assert second["status"] == "no_new_context" + assert second["consumed_tokens"] == first["consumed_tokens"] + assert second["model_content"] == "" + recorder.observe([{"content": json.dumps(second)}]) + assert len(report(ctx)["exposures"]) == count + assert report(ctx)["retrievals"][-1]["status"] == "no_new_context" + + +def test_adaptive_strict_briefing_is_captured_by_final_model_call(adaptive_context): + ctx = adaptive_context + + class LLM: + available = True + + def create(self, **kwargs): + assert "Preserve values" in kwargs["messages"][0]["content"] + return Reply(blocks=[Block(type="text", text='{"status":"success","summary":"checked"}')]) + + ctx.llm = LLM() + result, _ = asyncio.run(run_agent_step(ctx, step_name="review", purpose="Inspect", evidence={})) + assert result.ok + records = report(ctx) + assert records["session"]["knowledge_snapshot"] == "adaptive-trace" + exposed = records["exposures"] + assert len(exposed) == 1 + assert exposed[0]["document_id"] == "general/briefing.md" + assert exposed[0]["boundary"] == "model_request" + + +def test_adaptive_crossrepo_permission_is_not_granted_by_tracer(adaptive_context): + ctx = adaptive_context + recorder = StepKnowledgeTrace(get_knowledge_trace(ctx), "review") + ctx.settings.allowed_knowledge_repositories = () + denied = _repo_docs_tool(ctx, None, recorder) + with pytest.raises(ContextError, match="authorized"): + denied["doc_read"].handler("repos/other/doc.md", repository="other") + ctx.settings.allowed_knowledge_repositories = ("other",) + allowed = _repo_docs_tool(ctx, None, recorder) + packet = json.loads(allowed["doc_read"].handler("repos/other/doc.md", repository="other")) + assert packet["documents"][0]["repo_id"] == "other" + recorder.observe([{"content": packet["model_content"]}]) + records = report(ctx) + assert records["exposures"] == [] + assert any(g["reason"] == "context_cross_repository_unobserved" for g in records["gaps"]) + + +def test_adaptive_corrupt_packet_is_reported_without_legacy_fallback(adaptive_context): + recorder = StepKnowledgeTrace(get_knowledge_trace(adaptive_context), "review") + packet = {"model_content": "substituted", "model_content_sha256": hashlib.sha256(b"original").hexdigest(), + "documents": [{"path": "repos/r/doc.md"}], "status": "ready"} + recorder.adaptive_packet(packet, "doc_read") + records = report(adaptive_context) + assert records["exposures"] == [] + assert any(g["reason"] == "adaptive_packet_invalid" for g in records["gaps"]) diff --git a/test/test_knowledge_trace_cli.py b/test/test_knowledge_trace_cli.py new file mode 100644 index 00000000..162d9470 --- /dev/null +++ b/test/test_knowledge_trace_cli.py @@ -0,0 +1,71 @@ +"""User-facing knowledge trace inspection works without a model or KB checkout.""" + +import json + +from infermatrix_copilot.cli.entry import main +from infermatrix_copilot.knowledge_trace import KnowledgeTrace + + +def _session(tmp_path): + root = tmp_path / "traces" + recorder = KnowledgeTrace(root=root, mode="required") + session = recorder.start_session(repository="demo", mode="direct", source_revision="a" * 40) + trace_id = session["trace_id"] + recorder.retrieve(trace_id, step_id="defaults", entries=[], mechanism="search", query="zero") + recorder.finish_session(trace_id) + return root, trace_id + + +def test_show_and_export_are_readable_with_collection_disabled(tmp_path, monkeypatch, capsys): + root, trace_id = _session(tmp_path) + monkeypatch.setenv("KNOWLEDGE_TRACE_MODE", "off") + assert main(["knowledge-trace", "show", trace_id, "--root", str(root), "--json"]) == 0 + report = json.loads(capsys.readouterr().out) + assert report["session"]["repository"] == "demo" + assert report["retrievals"][0]["step_id"] == "defaults" + destination = tmp_path / "exports" / "知识追踪.md" + assert main(["knowledge-trace", "--root", str(root), "export", trace_id, + "--out", str(destination), "--format", "markdown"]) == 0 + content = destination.read_text(encoding="utf-8") + assert "defaults" in content + assert "demo" in content + + +def test_query_and_json_stdout_export(tmp_path, capsys): + root, trace_id = _session(tmp_path) + assert main(["knowledge-trace", "query", "--root", str(root), "--repo", "demo"]) == 0 + records = json.loads(capsys.readouterr().out) + assert len(records) == 3 and {r["trace_id"] for r in records} == {trace_id} + assert [r["event_type"] for r in records] == [ + "session_started", "retrieval_completed", "session_finished"] + assert main(["knowledge-trace", "query", "--root", str(root), "--repo", "other"]) == 0 + assert json.loads(capsys.readouterr().out) == [] + assert main(["knowledge-trace", "export", trace_id, "--root", str(root), + "--format", "json", "--out", "-"]) == 0 + assert json.loads(capsys.readouterr().out)["session"]["repository"] == "demo" + + +def test_invalid_session_and_invalid_limit_have_actionable_errors(tmp_path, capsys): + root, trace_id = _session(tmp_path) + assert main(["knowledge-trace", "show", "missing", "--root", str(root)]) == 2 + assert "knowledge-trace:" in capsys.readouterr().err + assert main(["knowledge-trace", "query", "--repo", "demo", "--root", str(root), + "--limit", "0"]) == 2 + assert "limit" in capsys.readouterr().err.lower() + + +def test_export_collects_all_event_pages(tmp_path, monkeypatch, capsys): + import infermatrix_copilot.knowledge_trace as module + + root, trace_id = _session(tmp_path) + + class TinyPages(KnowledgeTrace): + def report(self, trace_id, cursor=None, limit=None): + return super().report(trace_id, cursor=cursor, limit=1) + + monkeypatch.setattr(module, "KnowledgeTrace", TinyPages) + assert main(["knowledge-trace", "export", trace_id, "--root", str(root), + "--format", "json", "--out", "-"]) == 0 + report = json.loads(capsys.readouterr().out) + assert len(report["events"]) == report["counts"]["events"] == 3 + assert report["next_cursor"] is None diff --git a/test/test_knowledge_trace_direct.py b/test/test_knowledge_trace_direct.py new file mode 100644 index 00000000..9f89f71c --- /dev/null +++ b/test/test_knowledge_trace_direct.py @@ -0,0 +1,313 @@ +"""Knowledge traces distinguish provider payloads from host-declared usage.""" + +from __future__ import annotations + +import sys +from dataclasses import replace +from pathlib import Path +from types import ModuleType, SimpleNamespace + +import pytest + +from infermatrix_copilot.sdk.v1 import ( + ChangedPath, DirectClient, DirectCompletionRequest, DirectReviewRequest, + InvalidRequestError, KnowledgeTraceClient, RepositoryRef, +) +from infermatrix_copilot.knowledge_view import KnowledgeView +from infermatrix_copilot.thin_mcp_server import build_mcp + +HEAD = "a" * 40 + + +def test_shipped_review_workflow_carries_trace_identity_and_explicit_usage(): + from infermatrix_copilot.upgrade.workflows import get_workflow + + content = get_workflow("imreview")["content"] + assert "pinned `expected_head_sha`" in content + assert "`trace_id` and `repo` to" in content + assert "record_knowledge_usage" in content + assert '"source_exposure_id"' in content + assert '"evidence_registered"' in content + assert '"assessment_recorded"' in content + assert '`status="interrupted"`' in content + + +def request(): + return DirectReviewRequest( + review_id="trace-attempt", repository=RepositoryRef("vllm-omni"), pr_number=7, + expected_head_sha=HEAD, title="Fix runtime scheduling", + body="Correct the affected runtime path.", + changed_paths=(ChangedPath("vllm_omni/core/sched/scheduler.py", "modified"),), + ) + + +def test_sdk_records_every_final_document_excerpt_and_embedded_map(tmp_path): + client = DirectClient(trace_root=tmp_path) + plan = client.plan(request()) + assert plan.knowledge_trace["status"] == "ready" + trace_id = plan.knowledge_trace["trace_id"] + report = client.knowledge_trace.get_report(trace_id, repository="vllm-omni") + expected = [plan.knowledge_entry, *plan.document_maps, *plan.mandatory_review_guides, + *(route.document for route in plan.knowledge_routes)] + for document in expected: + assert any(exposure["document_id"] == document.document_id + and exposure["representation"] in {"excerpt", "full_document"} + and exposure["reference"]["document_sha256"] == document.sha256 + for exposure in report["exposures"]) + quick_map = next(exposure for exposure in report["exposures"] if exposure["representation"] == "quick_map") + assert quick_map["rule_refs"] + assert {ref["coverage"] for ref in quick_map["rule_refs"]} == {"reference_only"} + assert {e["boundary"] for e in report["exposures"]} == {"provider_response"} + assert report["assessments"] == [] + assert report["session"]["source_revision"] == HEAD + + +def test_sdk_related_document_refs_are_recorded_beside_selected_prose(tmp_path): + client = DirectClient(trace_root=tmp_path) + plan = client.plan(replace(request(), repository=RepositoryRef("jiuwenswarm"), + title="Model platform configuration", body="models API", + changed_paths=(ChangedPath("jiuwenswarm/common/model_catalog.py"),))) + report = client.knowledge_trace.get_report(plan.knowledge_trace["trace_id"]) + for document in plan.related_knowledge["documents"]: + representations = {e["representation"] for e in report["exposures"] if e["document_id"] == document["path"]} + assert "related_excerpt" in representations + assert representations & {"full_document", "excerpt"} + + +def test_sdk_host_receipts_and_assessments_are_explicit_and_scope_checked(tmp_path): + trace = KnowledgeTraceClient(root=tmp_path) + session = trace.start_session("vllm-omni", source_revision=HEAD) + trace_id = session["trace_id"] + exposure = trace.recorder.provide(trace_id, step_id="review", document_id="repos/vllm-omni/rules.md", + content="## TEST-1 Zero values\nPreserve valid zero values.\n", boundary="provider_response", representation="full_document")["result"] + events = [ + {"event_type": "content_provided", "step_id": "defaults", "exposure": { + "source_exposure_id": exposure["exposure_id"], "boundary": "host_context"}}, + {"event_type": "evidence_registered", "step_id": "defaults", "evidence": { + "evidence_id": "zero-test", "path": "test/test_defaults.py", "source_revision": HEAD, + "evidence_origin": "captured", "content": "zero-value test passed"}}, + {"event_type": "assessment_recorded", "step_id": "defaults", "assessment": { + "knowledge_ref_id": exposure["rule_refs"][0]["knowledge_ref_id"], + "exposure_ids": [exposure["exposure_id"]], "applicability": "applicable", + "verification": "supported", "check": "Check the zero-value default path.", + "rationale": "The request explicitly accepts zero.", "evidence_ids": ["zero-test"]}}, + ] + trace.record_events(trace_id, events, repository="vllm-omni", idempotency_key="defaults") + report = trace.get_report(trace_id) + assert len(report["assessments"]) == 1 + assert report["evidence"][0]["evidence_origin"] == "host_reported" + assert any(e["boundary"] == "host_context" for e in report["exposures"]) + with pytest.raises(InvalidRequestError, match="different repository"): + trace.record_events(trace_id, [], repository="afd-plugin") + with pytest.raises(InvalidRequestError, match="host_context"): + trace.record_events(trace_id, [{"event_type": "content_provided", "exposure": {"boundary": "model_request"}}]) + + +def fake_mcp(monkeypatch): + fast_module = ModuleType("mcp.server.fastmcp") + class FakeMCP: + def __init__(self, *_args, **_kwargs): + self.tools = {} + def tool(self, **_kwargs): + def register(fn): + self.tools[fn.__name__] = fn + return fn + return register + fast_module.FastMCP = FakeMCP + annotations = ModuleType("mcp.types") + annotations.ToolAnnotations = lambda **kwargs: SimpleNamespace(**kwargs) + for name, module in {"mcp": ModuleType("mcp"), "mcp.server": ModuleType("mcp.server"), + "mcp.server.fastmcp": fast_module, "mcp.types": annotations}.items(): + monkeypatch.setitem(sys.modules, name, module) + import tempfile + + return build_mcp(core=SimpleNamespace(settings=SimpleNamespace( + run_root=Path(tempfile.mkdtemp()), allowed_knowledge_repositories=()))).tools + + +def knowledge_tree(tmp_path, name, text): + root = tmp_path / name + for path, content in {"AGENTS.md": "navigation", "general/_index.md": "general", + "repos/vllm-omni/_index.md": "repository", "repos/vllm-omni/rules.md": text, + "repos/afd-plugin/_index.md": "other", "repos/afd-plugin/rules.md": "other private rules"}.items(): + target = root / path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(content, encoding="utf-8") + return KnowledgeView(root=root.resolve(), snapshot=name) + + +def test_mcp_context_reads_stay_on_original_view_and_refuse_cross_repo(tmp_path, monkeypatch): + monkeypatch.setenv("KNOWLEDGE_TRACE_ROOT", str(tmp_path / "trace")) + old = knowledge_tree(tmp_path, "old", "## TEST-1\nold rule\n") + new = knowledge_tree(tmp_path, "new", "## TEST-1\nnew rule\n") + active = [old] + monkeypatch.setattr(KnowledgeView, "current", lambda: active[0]) + tools = fake_mcp(monkeypatch) + trace_id = tools["start_knowledge_trace"](repo="vllm-omni", source_revision=HEAD)["trace_id"] + active[0] = new + result = tools["doc_read"]("repos/vllm-omni/rules.md", trace_id=trace_id) + assert "old rule" in result["content"] + assert result["knowledge_trace"]["exposure_id"] + search = tools["doc_search"]("old rule", trace_id=trace_id) + assert search["matches"] + assert "error" in tools["get_knowledge_trace"](trace_id, repo="afd-plugin") + assert "error" in tools["doc_read"]("repos/afd-plugin/rules.md", trace_id=trace_id) + report = tools["get_knowledge_trace"](trace_id) + assert {e["reference"]["snapshot_id"] for e in report["exposures"]} == {"old"} + + +def test_trace_off_does_not_write_and_sdk_still_completes(tmp_path): + trace_root = tmp_path / "should-not-exist" + client = DirectClient(trace_root=trace_root, trace_mode="off") + plan = client.plan(request()) + assert plan.knowledge_trace == {} + result = client.validate(DirectCompletionRequest(review_context_id=plan.review_context_id, + expected_head_sha=HEAD, evidence_head_sha=HEAD, subtraction_signal="none", existing_feedback_status="checked")) + assert result.review_complete + assert not trace_root.exists() + + +def test_quick_map_references_require_existing_heading_ids(): + from infermatrix_copilot.sdk.v1.direct import _quick_map_rule_ids + source = "## TEST-1aa Rule\nbody\n## TEST-1ab Rule\nbody\n" + content = "| Zero values | TEST-1aa/1ab and UNKNOWN-9 | source.py |" + assert _quick_map_rule_ids(content, source) == ["TEST-1aa", "TEST-1ab"] + + +def test_mcp_review_records_only_embedded_payloads_with_specific_map_refs(tmp_path, monkeypatch): + monkeypatch.setenv("KNOWLEDGE_TRACE_ROOT", str(tmp_path / "traces")) + tools = fake_mcp(monkeypatch) + result = tools["review"]("7", title=request().title, body=request().body, + changed_files=[request().changed_paths[0].path], expected_head_sha=HEAD) + assert result["knowledge_trace"]["status"] == "ready" + report = tools["get_knowledge_trace"](result["knowledge_trace"]["trace_id"]) + assert report["exposures"] + assert {e["representation"] for e in report["exposures"]} <= {"quick_map", "related_excerpt"} + quick_map = next(e for e in report["exposures"] if e["representation"] == "quick_map") + assert quick_map["rule_refs"] + assert {r["coverage"] for r in quick_map["rule_refs"]} == {"reference_only"} + assert not any(e["document_id"].endswith("simplification-audit.md") for e in report["exposures"]) + assert any(entry["document_id"].endswith("simplification-audit.md") + for retrieval in report["retrievals"] for entry in retrieval["entries"]) + + +def test_retrying_identical_sdk_plan_keeps_original_attempt_trace(tmp_path): + client = DirectClient(trace_root=tmp_path) + first = client.plan(request()) + second = client.plan(request()) + trace_id = first.knowledge_trace["trace_id"] + assert second.knowledge_trace["trace_id"] == trace_id + assert first.review_context_id == second.review_context_id + page = client.read_document(first.knowledge_routes[0].document.document_id, + review_context_id=first.review_context_id, trace_id=trace_id) + assert page.knowledge_trace["status"] == "ready" + decision = client.validate(DirectCompletionRequest(review_context_id=first.review_context_id, + expected_head_sha=HEAD, evidence_head_sha=HEAD, subtraction_signal="none", existing_feedback_status="checked")) + assert decision.review_complete + assert client.knowledge_trace.get_report(trace_id)["session"]["status"] == "complete" + assert client.plan(request()).knowledge_trace["trace_id"] == trace_id + third = client.plan(replace(request(), review_id="another-attempt")) + assert third.knowledge_trace["trace_id"] != trace_id + + +def test_sdk_utf8_page_exposure_preserves_original_byte_range(tmp_path, monkeypatch): + view = knowledge_tree(tmp_path, "unicode", "你") + client = DirectClient(trace_root=tmp_path / "traces") + monkeypatch.setattr(client, "_view", lambda: view) + trace_id = client.knowledge_trace.start_session("vllm-omni", knowledge_snapshot=view.public_snapshot)["trace_id"] + context_id = "sha256:" + "b" * 64 + client._remember_context(context_id, HEAD, view=view, repository="vllm-omni", trace_id=trace_id) + page = client.read_document("repos/vllm-omni/rules.md", max_bytes=1, review_context_id=context_id) + assert page.next_offset == 1 + assert len(page.content.encode("utf-8")) == 3 # Existing byte-page replacement behavior. + report = client.knowledge_trace.get_report(trace_id) + exposure = report["exposures"][0] + assert exposure["included_ranges"] == [[0, 1]] + assert exposure["offset_unit"] == "utf8_bytes" + + +@pytest.mark.parametrize("operation,fail_call", [("provide", 1), ("retrieve", 2)]) +def test_sdk_missed_observation_remains_degraded_after_storage_recovers(tmp_path, monkeypatch, operation, fail_call): + client = DirectClient(trace_root=tmp_path) + original = getattr(client.knowledge_trace.recorder, operation) + calls = 0 + def transient_failure(*args, **kwargs): + nonlocal calls + calls += 1 + if calls == fail_call: + return {"status": "degraded", "error": "PermissionError"} + return original(*args, **kwargs) + monkeypatch.setattr(client.knowledge_trace.recorder, operation, transient_failure) + plan = client.plan(request()) + assert plan.knowledge_trace["status"] == "degraded" + trace_id = plan.knowledge_trace["trace_id"] + decision = client.validate(DirectCompletionRequest(review_context_id=plan.review_context_id, + expected_head_sha=HEAD, evidence_head_sha=HEAD, subtraction_signal="none", existing_feedback_status="checked")) + assert decision.review_complete + report = client.knowledge_trace.get_report(trace_id) + assert report["trace_status"] == "partial" + assert report["session"]["status"] == "degraded" + assert any(g["reason"] == "knowledge_capture_degraded" for g in report["gaps"]) + # The marker survives constructing a new host facade, and degraded sessions + # remain repository scoped. + new_host = KnowledgeTraceClient(root=tmp_path) + assert new_host.get_report(trace_id)["trace_status"] == "partial" + with pytest.raises(InvalidRequestError, match="different repository"): + new_host.get_report(trace_id, repository="afd-plugin") + + +def test_sdk_later_read_failure_is_sticky_in_context_and_report(tmp_path, monkeypatch): + client = DirectClient(trace_root=tmp_path) + plan = client.plan(request()) + original = client.knowledge_trace.recorder.provide + monkeypatch.setattr(client.knowledge_trace.recorder, "provide", lambda *args, **kwargs: {"status": "degraded"}) + page = client.read_document(plan.knowledge_routes[0].document.document_id, + review_context_id=plan.review_context_id) + assert page.knowledge_trace["status"] == "degraded" + monkeypatch.setattr(client.knowledge_trace.recorder, "provide", original) + assert client.plan(request()).knowledge_trace["status"] == "degraded" + client.validate(DirectCompletionRequest(review_context_id=plan.review_context_id, + expected_head_sha=HEAD, evidence_head_sha=HEAD, subtraction_signal="none", existing_feedback_status="checked")) + report = client.knowledge_trace.get_report(plan.knowledge_trace["trace_id"]) + assert report["session"]["status"] == "degraded" + assert report["trace_status"] == "partial" + + +def test_mcp_missed_retrieval_stays_degraded_at_completion(tmp_path, monkeypatch): + from infermatrix_copilot.knowledge_trace import KnowledgeTrace + monkeypatch.setenv("KNOWLEDGE_TRACE_ROOT", str(tmp_path / "traces")) + original = KnowledgeTrace.retrieve + calls = 0 + def transient_failure(self, *args, **kwargs): + nonlocal calls + calls += 1 + if calls == 2: # An auxiliary retrieval previously ignored its result. + return {"status": "degraded", "error": "PermissionError"} + return original(self, *args, **kwargs) + monkeypatch.setattr(KnowledgeTrace, "retrieve", transient_failure) + tools = fake_mcp(monkeypatch) + result = tools["review"]("7", title=request().title, body=request().body, + changed_files=[request().changed_paths[0].path], expected_head_sha=HEAD) + assert result["knowledge_trace"]["status"] == "degraded" + trace_id = result["knowledge_trace"]["trace_id"] + tools["validate_direct_review"](subtraction_signal="none", evidence_head_sha=HEAD, + existing_feedback_status="checked", trace_id=trace_id) + report = tools["get_knowledge_trace"](trace_id) + assert report["session"]["status"] == "degraded" + assert report["trace_status"] == "partial" + assert any(g["reason"] == "knowledge_capture_degraded" for g in report["gaps"]) + + +def test_scoped_failed_reads_are_reported_without_rejected_path_details(tmp_path, monkeypatch): + monkeypatch.setenv("KNOWLEDGE_TRACE_ROOT", str(tmp_path / "trace")) + view = knowledge_tree(tmp_path, "missing-doc", "rules") + monkeypatch.setattr(KnowledgeView, "current", lambda: view) + tools = fake_mcp(monkeypatch) + trace_id = tools["start_knowledge_trace"]()["trace_id"] + assert "error" in tools["doc_read"]("repos/vllm-omni/missing.md", trace_id=trace_id) + report = tools["get_knowledge_trace"](trace_id) + assert any(g["reason"] == "document_read_failed" for g in report["gaps"]) + assert "missing.md" not in str(report["gaps"]) + before = len(report["gaps"]) + assert "error" in tools["doc_read"]("repos/afd-plugin/missing.md", repo="afd-plugin", trace_id=trace_id) + assert len(tools["get_knowledge_trace"](trace_id)["gaps"]) == before diff --git a/test/test_knowledge_trace_strict.py b/test/test_knowledge_trace_strict.py new file mode 100644 index 00000000..7393c260 --- /dev/null +++ b/test/test_knowledge_trace_strict.py @@ -0,0 +1,272 @@ +"""Knowledge retrieval is observable separately from actual model delivery.""" + +import asyncio +import copy +import json +from types import SimpleNamespace + +import pytest + +from infermatrix_copilot.adapters.base import render_briefing_docs +from infermatrix_copilot.engine.agent_runtime.knowledge import _repo_docs_tool +from infermatrix_copilot.engine.agent_runtime.knowledge_trace import ( + StepKnowledgeTrace, finish_knowledge_trace, get_knowledge_trace, +) +from infermatrix_copilot.engine.agent_runtime.runner import run_agent_step +from infermatrix_copilot.engine.step import StepContext +from infermatrix_copilot.knowledge_trace import KnowledgeTrace +from infermatrix_copilot.llm import Block, Reply + + +@pytest.fixture +def context(settings, trace, tmp_path, git_repo, monkeypatch): + monkeypatch.setenv("KNOWLEDGE_TRACE_MODE", "best_effort") + monkeypatch.delenv("KNOWLEDGE_TRACE_ROOT", raising=False) + root = tmp_path / "knowledge" + (root / "general").mkdir(parents=True) + (root / "general/rules.md").write_text( + "## TEST-1: Preserve zero\nUse an explicit None check for a numeric default.\n", + encoding="utf-8") + settings.knowledge_dir = root + settings.knowledge_general_docs = ["general/rules.md"] + return StepContext(settings=settings, trace=trace, run_dir=tmp_path / "run", + params={}, state={"task_spec": {"kind": "pr_review", "repo": "widget"}, + "repo_path": str(git_repo)}) + + +def report(ctx): + return KnowledgeTrace(ctx.state["knowledge_trace_root"]).report(ctx.state["knowledge_trace_id"]) + + +def test_briefing_cap_records_only_final_included_fragments(context): + root = context.settings.knowledge_dir + (root / "general/long.md").write_text("A" * 60, encoding="utf-8") + (root / "general/after.md").write_text("must not be provided", encoding="utf-8") + recorder = StepKnowledgeTrace(get_knowledge_trace(context), "review#lens") + rendered = render_briefing_docs(root, ["general/long.md", "general/after.md"], + header="header", cap=40, on_document=recorder.briefing_document) + before = report(context) + assert len(before["retrievals"]) == 2 + assert before["exposures"] == [] + recorder.observe([{"content": rendered}]) + after = report(context) + assert len(after["exposures"]) == 1 + exposed = after["exposures"][0] + assert exposed["document_id"] == "general/long.md" + assert exposed["truncated"] is True + assert exposed["included_ranges"] == [[0, 32]] + assert recorder.trace.service.store.blob(exposed["payload_ref"]) == "A" * 32 + + +def test_document_tool_return_is_not_delivery_until_request(context): + recorder = StepKnowledgeTrace(get_knowledge_trace(context), "review") + tools = _repo_docs_tool(context, None, recorder) + text = tools["doc_read"].handler(path="general/rules.md") + assert report(context)["exposures"] == [] + recorder.observe([{"role": "user", "content": [{"type": "tool_result", + "tool_use_id": "read1", "content": text}]}]) + exposure = report(context)["exposures"][0] + assert exposure["representation"] == "full_document" + assert exposure["boundary"] == "model_request" + assert exposure["rule_refs"][0]["coverage"] == "body_complete" + # Repeated requests carry prior results without inventing a second read. + recorder.observe([{"content": text}]) + assert len(report(context)["exposures"]) == 1 + + +def test_search_hit_uses_visible_later_line_offset(context): + path = context.settings.knowledge_dir / "general/search.md" + content = "# Topic\nfirst line\nneedle occurs here\n" + path.write_text(content, encoding="utf-8") + recorder = StepKnowledgeTrace(get_knowledge_trace(context), "search") + tools = _repo_docs_tool(context, None, recorder) + text = tools["doc_search"].handler(query="needle") + recorder.observe([{"content": text}]) + exposure = report(context)["exposures"][0] + start = len("# Topic\nfirst line\n") + assert exposure["included_ranges"] == [[start, start + len("needle occurs here")]] + + +def test_disjoint_related_excerpt_marks_unknown_source_range(context): + content = "first selected section\n\nomitted source\n\nlast selected section" + path = context.settings.knowledge_dir / "general/disjoint.md" + path.write_text(content, encoding="utf-8") + recorder = StepKnowledgeTrace(get_knowledge_trace(context), "related") + excerpt = "first selected section\n\nlast selected section" + result = {"documents": [{"path": "general/disjoint.md", "content": excerpt, + "more_available": True}], "status": "ready"} + encoded = json.dumps(result, ensure_ascii=False) + recorder.tool_result("doc_related", result, encoded) + recorder.observe([{"content": encoded}]) + exposure = report(context)["exposures"][0] + assert exposure["included_ranges"] == [] + assert exposure["range_status"] == "unknown" + + +def test_dirty_source_evidence_does_not_claim_session_head_binding(context): + from pathlib import Path + + recorder = StepKnowledgeTrace(get_knowledge_trace(context), "source") + (Path(context.state["repo_path"]) / "mod_a.py").write_text("A = 0\n") + text = (context.settings.knowledge_dir / "general/rules.md").read_text() + recorder.queue("general/rules.md", text, representation="full_document") + recorder.observe([{"content": text}, {"content": [{"type": "tool_result", + "tool_use_id": "source1", "content": "A = 0\n"}]}]) + recorder.assessments({"knowledge_assessments": [{ + "document_id": "general/rules.md", "rule_id": "TEST-1", + "applicability": "applicable", "verification": "unresolved", + "rationale": "Inspect the dirty source", "check": "Check zero value", "evidence_ids": ["tool.source1"]}]}) + evidence = report(context)["evidence"][0] + assert evidence["workspace_state"] == "dirty" + assert evidence["revision_status"] == "unknown" + assert evidence["source_revision"] == "" + assert evidence["commit_verified"] is False + + +def test_strict_agent_reports_check_with_captured_source_evidence(context): + class SourceCheckLLM: + available = True + + def __init__(self): + self.calls = [] + self.ref = None + + def create(self, **kwargs): + self.calls.append(copy.deepcopy(kwargs["messages"])) + if len(self.calls) == 1: + return Reply(blocks=[Block(type="tool_use", id="kb1", name="doc_read", + input={"path": "general/rules.md"})]) + if len(self.calls) == 2: + note = kwargs["messages"][-1]["content"][-1]["text"] + refs = json.loads(note.split("\n", 1)[1]) + self.ref = next(ref for ref in refs if ref["rule_id"] == "TEST-1") + return Reply(blocks=[Block(type="tool_use", id="source1", name="read_file", + input={"path": "mod_a.py"})]) + return Reply(blocks=[Block(type="text", text=json.dumps({ + "status": "success", "summary": "Checked default handling", "findings": ["No numeric default exists in mod_a.py."], + "knowledge_assessments": [{**self.ref, "applicability": "applicable", + "verification": "unresolved", "rationale": "The source was inspected; this module has no numeric default.", + "check": "Inspect the source for a truthiness default.", + "evidence_ids": ["tool.source1"], "decision_ids": ["finding:0"]}]}))]) + + llm = SourceCheckLLM() + context.llm = llm + result, _ = asyncio.run(run_agent_step(context, step_name="review#defaults", + purpose="Check default handling", evidence={"source": "mod_a.py"})) + assert result.ok + records = report(context) + assert len(llm.calls) == 3 + assert {e["boundary"] for e in records["exposures"]} == {"model_request"} + assert len(records["assessments"]) == 1 + assert records["assessments"][0]["reference_validation"] == "valid" + assert records["assessments"][0]["decision_refs"] == [{ + "decision_id": "finding:0", "finding": "No numeric default exists in mod_a.py."}] + assert records["evidence"][0]["evidence_origin"] == "captured" + assert records["evidence"][0]["revision_status"] == "matched" + blob = get_knowledge_trace(context).service.store.blob(records["evidence"][0]["payload_ref"]) + assert "A = 1" in blob + finish_knowledge_trace(context.state, "complete") + artifact = json.loads((context.run_dir / "KNOWLEDGE_TRACE.json").read_text()) + assert artifact["trace_root"] == context.state["knowledge_trace_root"] + assert artifact["session"]["run_id"] == context.run_dir.name + assert (context.run_dir / "KNOWLEDGE_TRACE.md").is_file() + markdown = (context.run_dir / "KNOWLEDGE_TRACE.md").read_text() + assert "finding:0: No numeric default exists in mod_a.py." in markdown + + +def test_harness_records_handoff_without_claiming_model_request(context, monkeypatch): + from infermatrix_copilot.agent_loop import AgentOutcome + import infermatrix_copilot.providers as providers + + monkeypatch.setattr(type(context.settings), "tier_target", lambda *_: SimpleNamespace( + kind="harness", model="test", host="local", source="test")) + monkeypatch.setattr(providers, "run_harness_step", lambda *_, **__: AgentOutcome( + '{"status":"success","summary":"ok"}', iterations=1, tool_calls=0)) + context.llm = SimpleNamespace(available=True) + result, _ = asyncio.run(run_agent_step(context, step_name="review", purpose="check", evidence={})) + assert result.ok + captured = report(context) + assert {x["boundary"] for x in captured["exposures"]} == {"host_context"} + assert any(g["reason"] == "harness_model_requests_unobserved" for g in captured["gaps"]) + assert captured["instrumentation"]["native_file_reads"] == "unknown" + + +def test_repeated_runtime_step_keeps_input_evidence_per_invocation(context, monkeypatch): + """Foreach/retry calls may reuse a step name and input evidence names.""" + monkeypatch.setenv("KNOWLEDGE_TRACE_MODE", "required") + + class InputCheckLLM: + available = True + + def create(self, **kwargs): + return Reply(blocks=[Block(type="text", text=json.dumps({ + "status": "success", "summary": "Checked the supplied source", + "knowledge_assessments": [{ + "document_id": "general/rules.md", "rule_id": "TEST-1", + "applicability": "applicable", "verification": "supported", + "rationale": "Checked the numeric default in the supplied source.", + "check": "Inspect whether the default preserves zero.", + "evidence_ids": ["input.source"]}]}))]) + + context.llm = InputCheckLLM() + sources = ["return value or 1", "return 1 if value is None else value"] + for source in sources: + result, _ = asyncio.run(run_agent_step( + context, step_name="review#defaults", purpose="Check defaults", + evidence={"source": source})) + assert result.ok + + records = report(context) + assert context.state["knowledge_trace_status"] == "running" + assert len(records["assessments"]) == 2 + assert len(records["evidence"]) == 2 + assert len({a["invocation_id"] for a in records["assessments"]}) == 2 + evidence = {e["evidence_id"]: e for e in records["evidence"]} + store = get_knowledge_trace(context).service.store + for assessment, source in zip(records["assessments"], sources): + captured = evidence[assessment["evidence_ids"][0]] + assert captured["invocation_id"] == assessment["invocation_id"] + assert store.blob(captured["payload_ref"]) == source + finish_knowledge_trace(context.state, "complete") + artifact = json.loads((context.run_dir / "KNOWLEDGE_TRACE.json").read_text()) + assert artifact["status"] == "complete" + assert artifact["counts"]["assessments"] == 2 + + +def test_off_does_not_create_trace_files(context, monkeypatch): + monkeypatch.setenv("KNOWLEDGE_TRACE_MODE", "off") + assert get_knowledge_trace(context) is None + finish_knowledge_trace(context.state, "complete") + assert not (context.run_dir / "knowledge_trace").exists() + assert "knowledge_trace_id" not in context.state + + +def test_storage_degradation_is_visible(context, monkeypatch): + monkeypatch.setattr(KnowledgeTrace, "start_session", lambda *_, **__: { + "status": "degraded", "error": "PermissionError"}) + recorder = get_knowledge_trace(context) + assert recorder.error + assert context.state["knowledge_trace_status"] == "degraded" + assert list(context.trace.events("knowledge_trace_error")) + + +def test_required_storage_failure_blocks_before_model_call(context, monkeypatch): + monkeypatch.setenv("KNOWLEDGE_TRACE_MODE", "required") + monkeypatch.setattr(KnowledgeTrace, "start_session", lambda *_, **__: { + "status": "degraded", "error": "PermissionError"}) + context.llm = SimpleNamespace(available=True) + result, _ = asyncio.run(run_agent_step(context, step_name="review", purpose="check", evidence={})) + assert not result.ok + assert "required knowledge trace failed" in result.summary + + +def test_separate_execution_attempts_preserve_interrupted_session(context): + first = get_knowledge_trace(context) + first.finish("interrupted") + other_state = {k: v for k, v in context.state.items() if not k.startswith("_knowledge")} + second_ctx = StepContext(settings=context.settings, trace=context.trace, + run_dir=context.run_dir, params={}, state=other_state) + second = get_knowledge_trace(second_ctx) + assert first.trace_id != second.trace_id + assert first.service.report(first.trace_id)["status"] == "interrupted" + assert second.service.report(second.trace_id)["status"] == "running" diff --git a/test/test_repo_vocabulary.py b/test/test_repo_vocabulary.py index 9af08952..c22b3138 100644 --- a/test/test_repo_vocabulary.py +++ b/test/test_repo_vocabulary.py @@ -89,7 +89,7 @@ # thin MCP server: tool defaults + instructions prose. The routing table # that once shared this entry (52 in direct_routing.py) moved to # knowledge/repos//_routes.yaml. - "thin_mcp_server.py": 8, + "thin_mcp_server.py": 2, # tool defaults share the canonical fallback constant } @@ -140,4 +140,4 @@ def test_vocabulary_inventory_is_not_stale(): assert count == ceiling, ( f"{rel}: count {count} != ceiling {ceiling} — ceilings track " "the EXACT current debt; lower the ceiling when the count " - "drops (never leave headroom)") \ No newline at end of file + "drops (never leave headroom)") diff --git a/test/test_thin_mcp_server.py b/test/test_thin_mcp_server.py index d279c319..c2a2dd60 100644 --- a/test/test_thin_mcp_server.py +++ b/test/test_thin_mcp_server.py @@ -143,6 +143,10 @@ def test_direct_entrypoints_do_not_resolve_repo(monkeypatch): "knowledge_feedback_report", "knowledge_feedback_sync", "knowledge_agreement", + "start_knowledge_trace", + "record_knowledge_usage", + "finish_knowledge_trace", + "get_knowledge_trace", } review = mcp.tools["review"](