From 08a1b8e9acd6cf2cda95f588c386f3ef8f42294e Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Thu, 13 Aug 2026 17:23:17 +0800 Subject: [PATCH 1/2] docs(rfc): define long-horizon workloads --- ...oads_and_long_horizon_memory_evaluation.md | 298 ++++++++++++++++++ ...oads_and_long_horizon_memory_evaluation.md | 271 ++++++++++++++++ zensical.toml | 2 + 3 files changed, 571 insertions(+) create mode 100644 docs/en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md create mode 100644 docs/zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md diff --git a/docs/en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md b/docs/en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md new file mode 100644 index 000000000..54d54a074 --- /dev/null +++ b/docs/en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md @@ -0,0 +1,298 @@ +- Proposal Name: `unified_workloads_and_long_horizon_memory_evaluation` +- Start Date: 2026-08-13 +- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- Related RFCs: [RFC 0081](0081_end_to_end_evaluation_architecture.md) + +# Summary + +PowerContext represents built-in end-to-end samples and long-horizon tasks as workloads. Each workload selects a +pinned task, chooses an execution adapter, sets a budget, and declares how to evaluate the Memory produced during the +run. + +Workloads share one catalog, replay envelope, Memory evaluator, report format, and `acceptance` command. The current +implementation uses Bub as its execution adapter because Bub exposes model calls, tools, context +injection, capture, and checkpoints. The architecture can accept another adapter later without changing those common +contracts. + +A source-native task reward remains diagnostic. It does not decide whether PowerContext collected grounded, +recallable Memory. + +# Motivation + +RFC 0081 defines the broader end-to-end evaluation architecture, but it leaves local deterministic samples, +model-backed agent runs, and long-horizon tasks on separate command and artifact paths. That separation duplicates +selection, execution setup, evidence handling, and reporting. + +A unified workload contract provides stable answers to these questions: + +- Which pinned task and revision ran? +- Which execution adapter and runtime configuration drove it? +- Did execution start from an isolated, empty PowerContext scope? +- What evidence was captured during execution? +- Did the run create grounded Memory and recall it afterward? +- Can the same evidence be rescored without rerunning the task? + +Bub is a useful initial adapter because both deterministic tool flows and model-backed agent flows can traverse its +real ACP, command, tool, hook, and plugin boundaries. Deterministic execution does not need to pretend to be a model +run, while long-horizon execution can expose the model loop and capture policy. + +## Scope + +This RFC covers: + +- one Pydantic workload manifest and catalog; +- selection by workload ID or category; +- an isolated PowerContext scope for every workload; +- Harbor-backed Bub execution for repository and registry tasks; +- normalized replay evidence, Memory evaluation, and report rendering; +- deterministic acceptance across SQLite and OceanBase; +- opt-in model-backed and long-horizon workloads; and +- offline rescoring from committed evidence contracts. + +The complete LoCoMo benchmark and the separate SWE-Pro evaluation remain outside this catalog. This RFC does not +migrate them or change their native inputs, scoring, results, or operational commands. + +# Guide-level explanation + +## Workload manifest + +The workload manifest is both a catalog entry and an execution contract. Pydantic validates the manifest and runtime +settings before execution. + +```yaml +schema: powercontext.e2e-task/v1 +id: project-database-decision +categories: + - acceptance + - sample +dataset: + path: e2e/bub/harbor-tasks + task_id: project-database-decision + checksum: +execution: + type: bub + model: false + max_steps: 10 + max_tokens: 4096 +evaluation: + expected_memory: + - OceanBase + probes: + - id: database-decision + query: Which project decision selected multi-node persistent storage? + expected_context: + - OceanBase +``` + +`dataset` may identify a repository-maintained Harbor task or a versioned registry task. Repository tasks and +registry tasks use the same execution and evidence path. + +`execution.type` selects the adapter. The current contract implements `bub`. `execution.model` only declares whether +the workload needs a model: + +- `false` keeps the Bub execution deterministic and does not pass a model into the agent environment; +- `true` requires the runtime to resolve a model before the Harbor Job starts. + +Model identity, provider, endpoint, and authentication are runtime concerns. They do not belong in a portable +workload manifest. Replay evidence records the resolved model identity when a model is used, but never credentials. +The harness does not duplicate those settings: its Client consumes `POWERCONTEXT_CLIENT_*`, the Bub adapter forwards +native `BUB_*` values when a model is required, and the integration consumes `POWERCONTEXT_BUB_*`. Harbor receives an +`AgentConfig` directly and does not require model provider keys. + +Adapter package versions and timeouts follow the same ownership rule. The adapter runtime pins Bub and the ACP +server and records their resolved versions in replay evidence. Harbor task definitions own agent and environment timeouts; +Bub owns `BUB_MODEL_TIMEOUT_SECONDS`. The workload manifest does not override either timeout domain. + +## Selection and command surface + +Workloads have stable IDs. Categories are selection metadata. The `acceptance` command selects one or more IDs, +categories, or the default `acceptance` category: + +```bash +powercontext-e2e acceptance --output e2e/bub/results + +powercontext-e2e acceptance \ + --id locomo-support-group \ + --id project-database-decision \ + --output e2e/bub/results + +powercontext-e2e acceptance \ + --category long-horizon \ + --output e2e/bub/results +``` + +Both selectors use repeatable command options. Environment aliases and comma-separated selector syntax are not part +of the contract. + +Long-horizon and live workloads remain acceptance evaluations. Their categories control selection; they do not +introduce separate execution modes or commands. + +SQLite and OceanBase are runtime database variants, not execution adapters or workload categories. Required CI runs +the same deterministic `acceptance` workloads against both databases. + +## Current Bub adapter + +The current adapter enters every workload through a Harbor Job and Harbor's ACP runner. Harbor owns the task +environment and agent lifecycle. Bub runs through its supported installation path with the PowerContext integration. + +Deterministic workloads use `model: false` and execute Bub commands such as `powercontext.remember` and +`powercontext.context`. They verify the Harbor-to-ACP-to-Bub tool path without invoking a model. Model-backed +workloads use `model: true` and additionally exercise Bub's model, context-injection, trajectory-capture, and +checkpoint hooks. + +Both forms produce the same replay envelope and pass through the same Memory evaluator. A deterministic workload is +still a Bub workload because it traverses the Bub adapter; determinism is a property of model use, not adapter +identity. + +## Shared execution flow + +The harness performs common work before and after adapter execution: + +```text +manifest and task provenance + -> validated workload and isolated PowerContext scope + -> execution adapter + -> normalized replay evidence + -> Memory evaluation + -> report rendering +``` + +The harness records the pre-execution Memory baseline, invokes the adapter, records final Memory, and runs the +declared recall probes. A failed workload still writes the evidence collected before the failure. + +## Input and instruction boundary + +The pinned task owns execution input. Harbor tasks own agent-visible instructions. The workload manifest references +those inputs without copying them. Replay evidence records the resolved instruction identity and, where safe, its +content. + +Evaluation probes remain separate from execution input. They run after the task and cannot provide hints to the +agent or task verifier. + +## Memory acceptance + +Memory acceptance uses observable evidence: + +- the resolved task checksum and execution adapter match the manifest; +- required native execution evidence was recorded; +- eligible events were captured when capture is required; +- the run created Memory and completed required checkpoints or flushes; +- new Memory cites Sources captured during execution; and +- declared recall probes receive usable prepared context. + +A deterministic workload may require fixed Memory fragments. A long-horizon task normally measures capture +coverage, grounding, and recall instead of requiring a fixed task answer. + +Native task rewards, verifier results, duration, and model usage remain labels, scores, or metrics. A task may fail +its native grader and still pass Memory acceptance. + +# Reference-level explanation + +## Workload and adapter contracts + +The manifest is the harness-level workload abstraction. The adapter owns execution-specific settings and translates +the pinned task into normalized evidence. Dataset adapters only produce standard task layouts and pin upstream +provenance; they do not run workloads, evaluate Memory, or render reports. + +Dependencies flow in one direction: + +```text +manifest and task provenance + -> execution adapter + -> replay evidence + -> Memory evaluation + -> report rendering +``` + +The evaluator reads replay evidence and cannot control the adapter. Report rendering reads the evaluation result and +does not recalculate acceptance. + +## Evidence contract + +Every workload writes one artifact directory: + +| Artifact | Purpose | +| --- | --- | +| `replay.json` | Workload identity, adapter, task provenance, runtime observations, Memory snapshots, probes, and native evidence references. | +| `eval-report.json` | Assertions, scores, labels, metrics, and reasons. | +| `report.md` | A human-readable projection of the evaluation result. | + +The replay records the dataset checksum, the workload's single `execution.type`, resolved model identity when present, +database identity, resolved instructions, and PowerContext scope state. The common envelope supports offline rescoring. Adapter-native +evidence remains typed within that envelope; the current Bub adapter records ACP summaries, captured events, +checkpoints, tool observations, and trajectory artifacts. + +Final artifact sinks remove configured secrets. Native task artifacts may contain task content and require review +before publication. + +## Adapter extension + +If a future evaluation cannot be represented faithfully by Bub, `execution` can become a discriminated union with an +additional adapter. For example: + +```yaml +execution: + type: basic +``` + +```yaml +execution: + type: codex +``` + +These examples reserve no implementation and assign no benchmark to either adapter. A new adapter must define its +typed execution settings and native evidence while reusing workload identity, selection, the replay envelope, +Memory evaluation, artifact layout, and reporting. Migrating an existing benchmark requires separate scope and +validation against its native semantics. + +## Compatibility + +The public command remains `acceptance`. Existing SQLite and OceanBase CI jobs continue to invoke +`make harness-compose-acceptance` and evaluate the same default acceptance category. ID and category selectors extend +that command without introducing a generic `run` command. + +The current LoCoMo benchmark and SWE-Pro evaluation keep their existing commands, artifacts, and result contracts. +The LoCoMo-derived built-in workload remains a pinned sample and does not claim a complete benchmark result. + +# Drawbacks + +The unified replay envelope must preserve adapter-native evidence without reducing it to untyped dictionaries. +Long-horizon runs can consume paid model capacity, require privileged containers, and produce large artifacts. The +required database matrix therefore covers deterministic workloads, while model-backed categories remain explicit +opt-in evaluations. + +# Rationale and alternatives + +Separate harnesses for deterministic samples, live agent runs, and long-horizon tasks would duplicate selection, +evidence, evaluation, and reporting. The shared contracts keep those responsibilities in one place while the +adapter isolates execution semantics. + +Putting model names or authentication methods in manifests would make workloads depend on one operator environment. +A boolean requirement preserves the deterministic boundary while runtime settings select the available model and +credentials. + +Using a source-native reward as Memory acceptance would answer whether the task was solved, not whether PowerContext +collected useful Memory. The native result remains available without replacing the Memory evaluator. + +# Non-goals + +This RFC does not replace RFC 0081, define a leaderboard, require registry publication, or introduce a new agent +protocol. It does not standardize private adapter internals or replace source-native graders. It does not migrate, +rewrite, or retire the current LoCoMo benchmark or SWE-Pro evaluation. It does not implement another execution +adapter. + +# Acceptance criteria + +The proposal is complete when: + +- one Pydantic manifest represents deterministic, model-backed, and long-horizon workloads; +- `execution.type: bub` selects the current adapter and `execution.model` declares only whether a model is required; +- component-native runtime settings select model identity, provider, endpoint, and credentials without a harness + mapping layer; +- one `acceptance` command selects one or more workload IDs and categories; +- repository and registry Harbor tasks use the same execution and provenance contracts; +- SQLite and OceanBase run the same deterministic acceptance category in required CI; +- model-backed Bub workloads record native evidence through Harbor and ACP; +- long-horizon Memory acceptance remains independent of native task reward; +- every replay identifies its adapter and supports offline rescoring; and +- the current LoCoMo benchmark and SWE-Pro evaluation remain unchanged. diff --git a/docs/zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md b/docs/zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md new file mode 100644 index 000000000..27d879466 --- /dev/null +++ b/docs/zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md @@ -0,0 +1,271 @@ +- Proposal Name: `unified_workloads_and_long_horizon_memory_evaluation` +- Start Date: 2026-08-13 +- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- Related RFCs: [RFC 0081](0081_end_to_end_evaluation_architecture.md) + +# 摘要 + +PowerContext 将内置端到端样例和长程任务表示为 workload。每个 workload 选择固定的 task、指定 execution adapter、设置预算， +并声明如何评估本次运行产生的 Memory。 + +所有 workload 共用 catalog、replay envelope、Memory evaluator、report 格式和 `acceptance` 命令。当前实现使用 +Bub adapter，因为 Bub 的 model call、tool、context injection、capture 与 checkpoint 都可以观察。以后可以增加其他 adapter， +而不改变这些公共 contract。 + +任务原生 reward 只用于诊断，不决定 PowerContext 是否采集到有依据、可召回的 Memory。 + +# 动机 + +RFC 0081 定义了更广泛的端到端评估架构，但本地确定性样例、使用 model 的 agent 运行和长程任务仍然使用不同的命令与 +artifact 路径。这种分离会重复实现 selection、execution setup、evidence handling 与 reporting。 + +统一的 workload contract 应稳定回答以下问题： + +- 运行了哪个固定 task 与 revision？ +- 哪个 execution adapter 与运行时配置驱动了任务？ +- execution 是否从隔离且为空的 PowerContext scope 开始？ +- execution 期间采集了哪些 evidence？ +- 本次运行是否创建了有依据的 Memory，并在之后成功召回？ +- 是否可以在不重新运行任务的情况下使用相同 evidence 重新评分？ + +Bub 适合作为首个 adapter，因为确定性 tool flow 和使用 model 的 agent flow 都可以经过真实的 ACP、command、tool、hook 与 +plugin 边界。确定性执行不需要伪装成 model run，长程执行则可以暴露 model loop 与 capture policy。 + +## 范围 + +本 RFC 包括： + +- 一套 Pydantic workload manifest 与 catalog； +- 按 workload ID 或 category 选择； +- 每个 workload 使用隔离的 PowerContext scope； +- 通过 Harbor 运行仓库内与 registry-backed 的 Bub task； +- 标准化 replay evidence、Memory evaluation 与 report rendering； +- 在 SQLite 与 OceanBase 上运行确定性 acceptance； +- 显式选择使用 model 的 workload 与 long-horizon workload； +- 根据固定的 evidence contract 离线重新评分。 + +完整 LoCoMo benchmark 与独立的 SWE-Pro evaluation 不进入该 catalog。本 RFC 不迁移它们，也不改变其原生输入、评分、结果 +或运行命令。 + +# 使用方式 + +## Workload manifest + +Workload manifest 同时是 catalog entry 与执行契约。Manifest 和运行时配置都在 execution 前经过 Pydantic 校验。 + +```yaml +schema: powercontext.e2e-task/v1 +id: project-database-decision +categories: + - acceptance + - sample +dataset: + path: e2e/bub/harbor-tasks + task_id: project-database-decision + checksum: +execution: + type: bub + model: false + max_steps: 10 + max_tokens: 4096 +evaluation: + expected_memory: + - OceanBase + probes: + - id: database-decision + query: Which project decision selected multi-node persistent storage? + expected_context: + - OceanBase +``` + +`dataset` 可以指向仓库自行维护的 Harbor task，也可以指向带版本的 registry task。两者使用相同的 execution 与 evidence +路径。 + +`execution.type` 选择 adapter，当前 contract 实现 `bub`。`execution.model` 只声明 workload 是否需要 model： + +- `false` 保持 Bub execution 确定性，不向 agent environment 传入 model； +- `true` 要求运行时在 Harbor Job 启动前解析出 model。 + +Model identity、provider、endpoint 与 authentication 都属于运行时配置，不应进入可移植的 workload manifest。使用 model +时，replay evidence 会记录最终解析出的 model identity，但不会记录 credential。 +Harness 不复制这些 setting：Client 消费 `POWERCONTEXT_CLIENT_*`，Bub adapter 在需要 model 时原样转发 `BUB_*`，integration +消费 `POWERCONTEXT_BUB_*`。Harbor 直接接收 `AgentConfig`，不需要 model provider key。 + +Adapter package version 与 timeout 遵循相同的所有权规则。Adapter runtime 固定 Bub 与 ACP server 版本，并把解析后的版本写入 +replay evidence。Harbor task definition 拥有 agent 与 environment timeout；Bub 拥有 `BUB_MODEL_TIMEOUT_SECONDS`。Workload manifest +不覆盖这两个 timeout domain。 + +## Selection 与命令入口 + +Workload 使用稳定 ID，category 只是选择元数据。`acceptance` 命令可以选择一个或多个 ID、category，默认选择 +`acceptance` category： + +```bash +powercontext-e2e acceptance --output e2e/bub/results + +powercontext-e2e acceptance \ + --id locomo-support-group \ + --id project-database-decision \ + --output e2e/bub/results + +powercontext-e2e acceptance \ + --category long-horizon \ + --output e2e/bub/results +``` + +两类 selector 都只使用可重复的 command option；contract 不提供 environment alias 或逗号分隔语法。 + +Long-horizon 与 live workload 仍然是 acceptance evaluation。Category 控制选择，不需要引入新的 execution mode 或命令。 + +SQLite 与 OceanBase 是运行时 database variant，不是 execution adapter，也不是 workload category。Required CI 在两个数据库上 +运行相同的确定性 `acceptance` workload。 + +## 当前 Bub adapter + +当前 adapter 通过 Harbor Job 与 Harbor ACP runner 进入每个 workload。Harbor 管理 task environment 与 agent lifecycle， +Bub 通过受支持的安装路径运行，并加载 PowerContext integration。 + +确定性 workload 使用 `model: false`，执行 `powercontext.remember` 与 `powercontext.context` 等 Bub command。它们在不调用 +model 的情况下验证 Harbor-to-ACP-to-Bub tool path。使用 model 的 workload 设置 `model: true`，并额外覆盖 Bub model、 +context injection、trajectory capture 与 checkpoint hook。 + +两种形式生成相同的 replay envelope，并使用相同的 Memory evaluator。确定性 workload 仍然属于 Bub adapter，因为它经过 +Bub adapter；确定性描述是否使用 model，而不是 adapter identity。 + +## 共享执行流程 + +Harness 在 adapter execution 前后完成公共工作： + +```text +manifest and task provenance + -> validated workload and isolated PowerContext scope + -> execution adapter + -> normalized replay evidence + -> Memory evaluation + -> report rendering +``` + +Harness 记录 execution 前的 Memory baseline，随后调用 adapter，记录最终 Memory，并运行声明的 recall probe。Workload +中途失败时，已经采集的 evidence 仍会写入 artifact。 + +## 输入与指令边界 + +固定的 task 拥有 execution input，Harbor task 拥有 agent 可见的 instruction。Workload manifest 只引用这些输入，不复制 +内容。Replay evidence 记录最终解析出的 instruction identity，并在安全时记录其内容。 + +Evaluation probe 与 execution input 相互独立。Probe 在任务结束后运行，不能向 agent 或 task verifier 提供提示。 + +## Memory acceptance + +Memory acceptance 使用可观察的 evidence： + +- 最终解析出的 task checksum 与 execution adapter 符合 manifest； +- 记录了所需的原生 execution evidence； +- 要求 capture 时采集了符合条件的 event； +- 本次运行创建了 Memory，并完成要求的 checkpoint 或 flush； +- 新建 Memory 引用了 execution 期间采集的 Source； +- 声明的 recall probe 能获得可用的 prepared context。 + +确定性 workload 可以要求固定的 Memory 片段。长程任务通常评估 capture coverage、grounding 与 recall，不要求固定的任务答案。 + +任务原生 reward、verifier result、运行时长与 model usage 保留为 label、score 或 metric。任务可以没有通过原生 grader，同时 +通过 Memory acceptance。 + +# 设计 + +## Workload 与 adapter contract + +Manifest 是 harness 层的 workload 抽象。Adapter 拥有 execution-specific setting，并把固定 task 转换为标准化 evidence。 +Dataset adapter 只生成标准 task layout 并固定 upstream provenance；它不运行 workload、不评估 Memory，也不渲染 report。 + +依赖保持单向： + +```text +manifest and task provenance + -> execution adapter + -> replay evidence + -> Memory evaluation + -> report rendering +``` + +Evaluator 只读取 replay evidence，不能控制 adapter。Report renderer 只读取 evaluation result，不重新计算 acceptance。 + +## Evidence contract + +每个 workload 生成一个 artifact 目录： + +| Artifact | 用途 | +| --- | --- | +| `replay.json` | Workload identity、adapter、task provenance、runtime observation、Memory snapshot、probe 与原生 evidence reference。 | +| `eval-report.json` | Assertion、score、label、metric 与判断理由。 | +| `report.md` | Evaluation result 的可读表示。 | + +Replay 记录 dataset checksum、workload 唯一的 `execution.type`、存在时最终解析出的 model identity、database identity、最终 instruction 与 +PowerContext scope state。公共 envelope 支持离线重新评分。Adapter 原生 evidence 在 envelope 中保留类型信息；当前 Bub +adapter 记录 ACP summary、captured event、checkpoint、tool observation 与 trajectory artifact。 + +最终 artifact sink 会移除已配置的 secret。原生 task artifact 可能包含任务内容，发布前需要检查。 + +## Adapter 扩展 + +如果未来的评估无法由 Bub 忠实表达，可以将 `execution` 扩展成包含新 adapter 的 discriminated union。例如： + +```yaml +execution: + type: basic +``` + +```yaml +execution: + type: codex +``` + +这些示例不预留实现，也不把任何 benchmark 分配给其中一个 adapter。新 adapter 必须定义自己的 typed execution setting 与 +原生 evidence，同时复用 workload identity、selection、replay envelope、Memory evaluation、artifact layout 与 +reporting。迁移现有 benchmark 需要单独确定范围，并验证其原生语义。 + +## 兼容性 + +公开命令保持为 `acceptance`。现有 SQLite 与 OceanBase CI job 继续调用 `make harness-compose-acceptance`，并评估相同的默认 +acceptance category。ID 与 category selector 扩展该命令，不引入通用 `run` 命令。 + +当前 LoCoMo benchmark 与 SWE-Pro evaluation 保留现有 command、artifact 与 result contract。LoCoMo 衍生的内置 workload +保持为固定 sample，不代表完整 benchmark 结果。 + +# 代价 + +公共 replay envelope 必须保留 adapter 原生 evidence，不能把它压成无类型 dictionary。长程运行可能消耗付费模型额度、 +需要 privileged container，并产生较大的 artifact。因此 required database matrix 只覆盖确定性 workload，使用 model 的 +category 保持为显式选择的 evaluation。 + +# 理由与替代方案 + +为确定性样例、live agent run 与长程任务维护独立 harness，会重复 selection、evidence、evaluation 与 reporting。 +公共 contract 将这些职责放在一处，adapter 则隔离 execution semantics。 + +把 model name 或 authentication method 放入 manifest，会让 workload 依赖某个 operator environment。布尔 requirement 可以 +保留确定性边界，同时由运行时配置选择可用的 model 与 credential。 + +直接用任务原生 reward 作为 Memory acceptance，只能回答任务是否完成，不能回答 PowerContext 是否采集到有效 Memory。 +原生结果会保留，但不会取代 Memory evaluator。 + +# 非目标 + +本 RFC 不替代 RFC 0081，不定义 leaderboard，不要求发布到 registry，也不引入新的 agent protocol。它不统一 adapter 私有 +实现，也不替代任务原生 grader。本 RFC 不迁移、重写或移除当前 LoCoMo benchmark 与 SWE-Pro evaluation，也不实现其他 +execution adapter。 + +# 验收条件 + +满足以下条件时，本提案完成： + +- 一套 Pydantic manifest 可以表示确定性、使用 model 与 long-horizon workload； +- `execution.type: bub` 选择当前 adapter，`execution.model` 只声明是否需要 model； +- 各组件的原生运行时配置选择 model identity、provider、endpoint 与 credential，不增加 harness mapping layer； +- 一个 `acceptance` 命令可以选择一个或多个 workload ID 与 category； +- 仓库内与 registry Harbor task 使用相同的 execution 与 provenance contract； +- SQLite 与 OceanBase 在 required CI 中运行相同的确定性 acceptance category； +- 使用 model 的 Bub workload 通过 Harbor 与 ACP 记录原生 evidence； +- long-horizon Memory acceptance 与任务原生 reward 保持独立； +- 每个 replay 都标识 adapter，并支持离线重新评分； +- 当前 LoCoMo benchmark 与 SWE-Pro evaluation 保持不变。 diff --git a/zensical.toml b/zensical.toml index c499c780d..8ec7263e5 100644 --- a/zensical.toml +++ b/zensical.toml @@ -46,6 +46,7 @@ nav = [ { "0051 Experience and Skill Artifact Families" = "en/rfcs/0051_experience_skill_artifact_families.md" }, { "0080 Memory Search Reranking" = "en/rfcs/0080_memory_search_reranking.md" }, { "0081 End-to-end Evaluation Architecture" = "en/rfcs/0081_end_to_end_evaluation_architecture.md" }, + { "RFC Proposal: Unified Workloads and Long-horizon Memory Evaluation" = "en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "0072 Scoped Statistics and Usage" = "en/rfcs/0072_scoped_statistics_and_usage.md" }, ] }, ] }, @@ -86,6 +87,7 @@ nav = [ { "0051 Experience 与 Skill Artifact Family" = "zh/rfcs/0051_experience_skill_artifact_families.md" }, { "0080 Memory 搜索 Rerank" = "zh/rfcs/0080_memory_search_reranking.md" }, { "0081 端到端评估架构" = "zh/rfcs/0081_end_to_end_evaluation_architecture.md" }, + { "RFC 提案：统一工作负载与长程 Memory 评估" = "zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "0072 Scoped Statistics 与 Usage" = "zh/rfcs/0072_scoped_statistics_and_usage.md" }, ] }, ] }, From 2a1434b72ce762b9032df448cbe5c9dfe39018d7 Mon Sep 17 00:00:00 2001 From: Chojan Shang Date: Thu, 13 Aug 2026 19:21:09 +0800 Subject: [PATCH 2/2] docs(rfc): assign RFC 1229 and align format --- ...ads_and_long_horizon_memory_evaluation.md} | 64 +++++++++++------ ...ads_and_long_horizon_memory_evaluation.md} | 68 ++++++++++++------- zensical.toml | 4 +- 3 files changed, 90 insertions(+), 46 deletions(-) rename docs/en/rfcs/{0000_unified_workloads_and_long_horizon_memory_evaluation.md => 1229_unified_workloads_and_long_horizon_memory_evaluation.md} (89%) rename docs/zh/rfcs/{0000_unified_workloads_and_long_horizon_memory_evaluation.md => 1229_unified_workloads_and_long_horizon_memory_evaluation.md} (89%) diff --git a/docs/en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md b/docs/en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md similarity index 89% rename from docs/en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md rename to docs/en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md index 54d54a074..58a841dd8 100644 --- a/docs/en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md +++ b/docs/en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md @@ -1,7 +1,7 @@ - Proposal Name: `unified_workloads_and_long_horizon_memory_evaluation` - Start Date: 2026-08-13 -- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) -- Related RFCs: [RFC 0081](0081_end_to_end_evaluation_architecture.md) +- RFC PR: [oceanbase/powercontext#1229](https://github.com/oceanbase/powercontext/pull/1229) +- Related RFC: [RFC 0081: End-to-End Evaluation Architecture](0081_end_to_end_evaluation_architecture.md) # Summary @@ -254,6 +254,29 @@ that command without introducing a generic `run` command. The current LoCoMo benchmark and SWE-Pro evaluation keep their existing commands, artifacts, and result contracts. The LoCoMo-derived built-in workload remains a pinned sample and does not claim a complete benchmark result. +## Non-goals + +This RFC does not replace RFC 0081, define a leaderboard, require registry publication, or introduce a new agent +protocol. It does not standardize private adapter internals or replace source-native graders. It does not migrate, +rewrite, or retire the current LoCoMo benchmark or SWE-Pro evaluation. It does not implement another execution +adapter. + +## Acceptance criteria + +The proposal is complete when: + +- one Pydantic manifest represents deterministic, model-backed, and long-horizon workloads; +- `execution.type: bub` selects the current adapter and `execution.model` declares only whether a model is required; +- component-native runtime settings select model identity, provider, endpoint, and credentials without a harness + mapping layer; +- one `acceptance` command selects one or more workload IDs and categories; +- repository and registry Harbor tasks use the same execution and provenance contracts; +- SQLite and OceanBase run the same deterministic acceptance category in required CI; +- model-backed Bub workloads record native evidence through Harbor and ACP; +- long-horizon Memory acceptance remains independent of native task reward; +- every replay identifies its adapter and supports offline rescoring; and +- the current LoCoMo benchmark and SWE-Pro evaluation remain unchanged. + # Drawbacks The unified replay envelope must preserve adapter-native evidence without reducing it to untyped dictionaries. @@ -274,25 +297,26 @@ credentials. Using a source-native reward as Memory acceptance would answer whether the task was solved, not whether PowerContext collected useful Memory. The native result remains available without replacing the Memory evaluator. -# Non-goals +# Prior art -This RFC does not replace RFC 0081, define a leaderboard, require registry publication, or introduce a new agent -protocol. It does not standardize private adapter internals or replace source-native graders. It does not migrate, -rewrite, or retire the current LoCoMo benchmark or SWE-Pro evaluation. It does not implement another execution -adapter. +RFC 0081 separates runtime integration, workload execution, evidence collection, evaluation, and reporting. This +proposal keeps those boundaries and gives the built-in acceptance scenarios a shared workload and artifact contract. -# Acceptance criteria +Harbor provides pinned task environments, agent lifecycle management, and native task verification. Bub provides the +first observable execution adapter. The replay envelope keeps Harbor and Bub evidence typed while Memory acceptance +remains independent of the native task score. -The proposal is complete when: +# Unresolved questions -- one Pydantic manifest represents deterministic, model-backed, and long-horizon workloads; -- `execution.type: bub` selects the current adapter and `execution.model` declares only whether a model is required; -- component-native runtime settings select model identity, provider, endpoint, and credentials without a harness - mapping layer; -- one `acceptance` command selects one or more workload IDs and categories; -- repository and registry Harbor tasks use the same execution and provenance contracts; -- SQLite and OceanBase run the same deterministic acceptance category in required CI; -- model-backed Bub workloads record native evidence through Harbor and ACP; -- long-horizon Memory acceptance remains independent of native task reward; -- every replay identifies its adapter and supports offline rescoring; and -- the current LoCoMo benchmark and SWE-Pro evaluation remain unchanged. +None. Adding another execution adapter, migrating an existing benchmark, and publishing model-backed artifacts each +require separate review. + +# Future possibilities + +The workload contract can add a `basic` or `codex` execution variant when a workload cannot be represented faithfully +through Bub. Such an adapter would reuse selection, replay, evaluation, and reporting rather than create another +harness. + +The complete LoCoMo benchmark or SWE-Pro evaluation may move to the catalog after its native scoring and artifact +contracts have been validated against this workload model. Registry publication and shared artifact retention can be +considered separately once their privacy and operational requirements are defined. diff --git a/docs/zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md b/docs/zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md similarity index 89% rename from docs/zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md rename to docs/zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md index 27d879466..872893407 100644 --- a/docs/zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md +++ b/docs/zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md @@ -1,9 +1,9 @@ - Proposal Name: `unified_workloads_and_long_horizon_memory_evaluation` - Start Date: 2026-08-13 -- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) -- Related RFCs: [RFC 0081](0081_end_to_end_evaluation_architecture.md) +- RFC PR: [oceanbase/powercontext#1229](https://github.com/oceanbase/powercontext/pull/1229) +- Related RFC: [RFC 0081：端到端评估架构](0081_end_to_end_evaluation_architecture.md) -# 摘要 +# Summary PowerContext 将内置端到端样例和长程任务表示为 workload。每个 workload 选择固定的 task、指定 execution adapter、设置预算， 并声明如何评估本次运行产生的 Memory。 @@ -14,7 +14,7 @@ Bub adapter，因为 Bub 的 model call、tool、context injection、capture 与 任务原生 reward 只用于诊断，不决定 PowerContext 是否采集到有依据、可召回的 Memory。 -# 动机 +# Motivation RFC 0081 定义了更广泛的端到端评估架构，但本地确定性样例、使用 model 的 agent 运行和长程任务仍然使用不同的命令与 artifact 路径。这种分离会重复实现 selection、execution setup、evidence handling 与 reporting。 @@ -47,7 +47,7 @@ plugin 边界。确定性执行不需要伪装成 model run，长程执行则可 完整 LoCoMo benchmark 与独立的 SWE-Pro evaluation 不进入该 catalog。本 RFC 不迁移它们，也不改变其原生输入、评分、结果 或运行命令。 -# 使用方式 +# Guide-level explanation ## Workload manifest @@ -171,7 +171,7 @@ Memory acceptance 使用可观察的 evidence： 任务原生 reward、verifier result、运行时长与 model usage 保留为 label、score 或 metric。任务可以没有通过原生 grader，同时 通过 Memory acceptance。 -# 设计 +# Reference-level explanation ## Workload 与 adapter contract @@ -232,13 +232,34 @@ acceptance category。ID 与 category selector 扩展该命令，不引入通用 当前 LoCoMo benchmark 与 SWE-Pro evaluation 保留现有 command、artifact 与 result contract。LoCoMo 衍生的内置 workload 保持为固定 sample，不代表完整 benchmark 结果。 -# 代价 +## Non-goals + +本 RFC 不替代 RFC 0081，不定义 leaderboard，不要求发布到 registry，也不引入新的 agent protocol。它不统一 adapter 私有 +实现，也不替代任务原生 grader。本 RFC 不迁移、重写或移除当前 LoCoMo benchmark 与 SWE-Pro evaluation，也不实现其他 +execution adapter。 + +## Acceptance criteria + +满足以下条件时，本提案完成： + +- 一套 Pydantic manifest 可以表示确定性、使用 model 与 long-horizon workload； +- `execution.type: bub` 选择当前 adapter，`execution.model` 只声明是否需要 model； +- 各组件的原生运行时配置选择 model identity、provider、endpoint 与 credential，不增加 harness mapping layer； +- 一个 `acceptance` 命令可以选择一个或多个 workload ID 与 category； +- 仓库内与 registry Harbor task 使用相同的 execution 与 provenance contract； +- SQLite 与 OceanBase 在 required CI 中运行相同的确定性 acceptance category； +- 使用 model 的 Bub workload 通过 Harbor 与 ACP 记录原生 evidence； +- long-horizon Memory acceptance 与任务原生 reward 保持独立； +- 每个 replay 都标识 adapter，并支持离线重新评分； +- 当前 LoCoMo benchmark 与 SWE-Pro evaluation 保持不变。 + +# Drawbacks 公共 replay envelope 必须保留 adapter 原生 evidence，不能把它压成无类型 dictionary。长程运行可能消耗付费模型额度、 需要 privileged container，并产生较大的 artifact。因此 required database matrix 只覆盖确定性 workload，使用 model 的 category 保持为显式选择的 evaluation。 -# 理由与替代方案 +# Rationale and alternatives 为确定性样例、live agent run 与长程任务维护独立 harness，会重复 selection、evidence、evaluation 与 reporting。 公共 contract 将这些职责放在一处，adapter 则隔离 execution semantics。 @@ -249,23 +270,22 @@ category 保持为显式选择的 evaluation。 直接用任务原生 reward 作为 Memory acceptance，只能回答任务是否完成，不能回答 PowerContext 是否采集到有效 Memory。 原生结果会保留，但不会取代 Memory evaluator。 -# 非目标 +# Prior art -本 RFC 不替代 RFC 0081，不定义 leaderboard，不要求发布到 registry，也不引入新的 agent protocol。它不统一 adapter 私有 -实现，也不替代任务原生 grader。本 RFC 不迁移、重写或移除当前 LoCoMo benchmark 与 SWE-Pro evaluation，也不实现其他 -execution adapter。 +RFC 0081 将 runtime integration、workload execution、evidence collection、evaluation 与 reporting 分开。本提案保持这些 +边界，并为内置 acceptance scenario 定义公共的 workload 与 artifact contract。 -# 验收条件 +Harbor 提供固定的 task environment、agent lifecycle management 与任务原生验证。Bub 是首个可观察的 execution adapter。 +Replay envelope 保留 Harbor 与 Bub evidence 的类型，同时让 Memory acceptance 与任务原生 score 保持独立。 -满足以下条件时，本提案完成： +# Unresolved questions -- 一套 Pydantic manifest 可以表示确定性、使用 model 与 long-horizon workload； -- `execution.type: bub` 选择当前 adapter，`execution.model` 只声明是否需要 model； -- 各组件的原生运行时配置选择 model identity、provider、endpoint 与 credential，不增加 harness mapping layer； -- 一个 `acceptance` 命令可以选择一个或多个 workload ID 与 category； -- 仓库内与 registry Harbor task 使用相同的 execution 与 provenance contract； -- SQLite 与 OceanBase 在 required CI 中运行相同的确定性 acceptance category； -- 使用 model 的 Bub workload 通过 Harbor 与 ACP 记录原生 evidence； -- long-horizon Memory acceptance 与任务原生 reward 保持独立； -- 每个 replay 都标识 adapter，并支持离线重新评分； -- 当前 LoCoMo benchmark 与 SWE-Pro evaluation 保持不变。 +无。增加其他 execution adapter、迁移现有 benchmark、发布使用 model 的 artifact 都需要单独评审。 + +# Future possibilities + +当 Bub 无法忠实表示某个 workload 时，workload contract 可以增加 `basic` 或 `codex` execution variant。新 adapter 复用 +selection、replay、evaluation 与 reporting，不创建另一套 harness。 + +完整 LoCoMo benchmark 或 SWE-Pro evaluation 可以在其原生 scoring 与 artifact contract 经过验证后迁入 catalog。Registry +发布与公共 artifact retention 可以在隐私和运行要求明确后单独评审。 diff --git a/zensical.toml b/zensical.toml index 8ec7263e5..c99c85ca3 100644 --- a/zensical.toml +++ b/zensical.toml @@ -46,7 +46,7 @@ nav = [ { "0051 Experience and Skill Artifact Families" = "en/rfcs/0051_experience_skill_artifact_families.md" }, { "0080 Memory Search Reranking" = "en/rfcs/0080_memory_search_reranking.md" }, { "0081 End-to-end Evaluation Architecture" = "en/rfcs/0081_end_to_end_evaluation_architecture.md" }, - { "RFC Proposal: Unified Workloads and Long-horizon Memory Evaluation" = "en/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md" }, + { "1229 Unified Workloads and Long-Horizon Memory Evaluation" = "en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "0072 Scoped Statistics and Usage" = "en/rfcs/0072_scoped_statistics_and_usage.md" }, ] }, ] }, @@ -87,7 +87,7 @@ nav = [ { "0051 Experience 与 Skill Artifact Family" = "zh/rfcs/0051_experience_skill_artifact_families.md" }, { "0080 Memory 搜索 Rerank" = "zh/rfcs/0080_memory_search_reranking.md" }, { "0081 端到端评估架构" = "zh/rfcs/0081_end_to_end_evaluation_architecture.md" }, - { "RFC 提案：统一工作负载与长程 Memory 评估" = "zh/rfcs/0000_unified_workloads_and_long_horizon_memory_evaluation.md" }, + { "1229 统一工作负载与长程 Memory 评估" = "zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, { "0072 Scoped Statistics 与 Usage" = "zh/rfcs/0072_scoped_statistics_and_usage.md" }, ] }, ] },