Repository navigation
DEC-019 完整 live 回归(47/48 · exit 0)+ 修好 sut-harness - #83
Merged
Merged
Conversation
- run 20261018-182724-e6212c1d:被测 fastapi-rag-agent@150a963 + 隔离语料库(125 行) → 47/48 · 0.9792 · exit 0 · judge 39,697 token · 0 重试/0 解析失败 - 关键判据:rule_hit_judge_fail=0、soft_miss_judge_pass=0 ⇒ α 未把正确回答判 fail - 唯一失败 id=20 是真实被测失败:召回 [55,64,8],未召回含「两者结果互补」的 doc 60 - 过程错误:两轮无效(judge key 401 · 我方语料筛选漏 16 篇 seed),judge 合计 ≈83,000 token - 落点:报告/DEC-019 §7/失败回灌/复盘/证据归档(3 个 spans.jsonl)
- 被测已从 api/ 改为 app/,官方镜像 = fastapi-rag-agent:latest(旧 my-fixed-name-api 已过期) - 源码只读挂 /src → 容器内复制到可写 /app:直接只读挂 /app 会被 /app/.mem0/qdrant/.lock 挡住(实测 OSError [Errno 30] Read-only file system) - 新增可选覆盖 EVAL_SUT_DB / EVAL_SUT_SESSION_TOKEN_LIMIT(隔离库/会话预算) - 保留能力探针与 DEC-016 被测自证;去掉旧 /deps 卷与启动时 pip install - 验证:单测 11 passed(stub docker/curl 契约)+ 真机端到端 exit 0、探针通过 - 决策记录:DEC-023 §6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
为什么
DEC-019的 α(判分器提示词收窄到「仅拒答语境」)改完后,唯一欠着的验证是完整 live 回归 —— 离线calibrate测不到「判分器变严后在真实链路里会不会误伤」。跑完回归后发现旧
tools/sut-harness/run_sut.sh已与被测新结构不匹配(api/→app/、旧镜像过期),同一任务内一并修好,保证这轮回归可复现。结果
20261018-182724-e6212c1d:47/48 · 0.9792 · exit 0 · judge 39,697 token · 0 重试 / 0 解析失败rule_hit_judge_fail = 0、soft_miss_judge_pass = 0⇒ 未观测到误伤id=20是真实被测失败:召回[55,64,8],未召回含「两者结果互补」的 doc 60;被测如实拒答,judge 判 fail 正确环境修复(第二个 commit)
run_sut.sh适配app/+ 官方镜像fastapi-rag-agent:latest/src→ 容器内复制到可写/app:直接只读挂/app会被/app/.mem0/qdrant/.lock挡住(实测OSError [Errno 30] Read-only file system)EVAL_SUT_DB/EVAL_SUT_SESSION_TOKEN_LIMIT覆盖代价与边界(如实)
rag_db已有 345 行,评测语料仅约 125 行 ⇒ 使用副本库rag_eval_20261018(原库一行未动);决策见DEC-023docs/复盘/2026-10-18-语料筛选漏掉seed金标语料.mdf2dad78到150a963,不能与历史 48/48 基线直接比较验证
pytest:590 passed / 2 skippedtools/check_evidence_coverage.py:✅ 通过EVAL_SUT_VERSION=150a963)落点
DEC-019 §7·DEC-023·docs/reports/DEC-019-live回归-2026-10-18.md·eval/cases/fastapi-rag-2026-10-18.json· 复盘 · 3 份spans.jsonl证据归档 ·tools/sut-harness/