From 49676b37c0be157071b146dc8004939b58bbe38b Mon Sep 17 00:00:00 2001 From: Eliver Date: Thu, 12 Nov 2026 14:04:22 +0800 Subject: [PATCH 1/3] =?UTF-8?q?docs(022):=20W0=20=E9=9C=80=E6=B1=82?= =?UTF-8?q?=E8=AF=84=E5=AE=A1=E8=A7=84=E6=A0=BC=E4=B8=8E=E7=B4=A2=E5=BC=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 新增 specs/022-batch-concurrency-and-media-reliability/spec.md(draft) - 记录 R1-R6 根因、D1-D7 已定决策、Q1-Q4 待拍板问题 - 定义 checkpoint/调度/OCR/判定/数据根接口草案与 W0-W5 验收命令 - specs/README.md 索引追加 022 不含任何实现代码;W1-W4 等 Q1-Q4 答复后开工。 --- .../spec.md | 288 ++++++++++++++++++ specs/README.md | 1 + 2 files changed, 289 insertions(+) create mode 100644 specs/022-batch-concurrency-and-media-reliability/spec.md diff --git a/specs/022-batch-concurrency-and-media-reliability/spec.md b/specs/022-batch-concurrency-and-media-reliability/spec.md new file mode 100644 index 00000000..a378e910 --- /dev/null +++ b/specs/022-batch-concurrency-and-media-reliability/spec.md @@ -0,0 +1,288 @@ +# 022 · 批量生成并发与媒体/OCR/判定/ASR 可靠性 + +状态:draft · 创建:2026-11-12 · 来源:父任务 T-078 +关联:spec 009(真并发 / 重启恢复)、spec 012(429 停放)、spec 007(媒体资源)、 +spec 008(OCR 辅助与画面判定)、spec 003(OpenAI 兼容 ASR) + +## 1. 背景与现象 + +2026-11-12 用 run.bat 跑 193 条新视频的合集/合并任务,日志与耗时暴露六类问题: + +- 合并任务逐条串行:s1 完成 12:36:45 才启动 s2,s2 12:54:55 才启动 s3,s3 13:05:28、s4 13:17:16。 + 同一时刻只有 1 条子视频在跑,193 条总时长线性叠加。 +- 普通批量隐藏 5 并发上限:设置并行 8,实际最多 5 条在跑。 +- 媒体登记全部失败:`无法登记任务媒体资源`,ValueError 指向 `backend/data/BV....mp3` + 不在 `backend/data/data` 之下。 +- OCR 429 无重试:s2 一轮 30 张里 26 张失败,日志 `OCR 调用失败(HTTP 429)`。 +- 画面判定空结果触发整任务重试:`画面价值判定失败:判定结果里找不到帧列表`。 +- ASR 序列化告警:Pydantic serializer warning,`Expected int ... input_value='0'`。 + +用户判断「批量生成没有完全并行」成立,且不止一处。 + +## 2. 根因(R1–R6) + +### R1 合并任务内部逐条串行 + +`backend/app/routers/note.py:1406-1531` 的 `_run_merged_note_task`:第 1453 行 +`for index, raw in enumerate(items, start=1)`;第 1501 行 +`note = _execute_note_with_retry(sub_task_id, generator, _execute)` 在循环内逐条等待。 +第 1432-1437 行 docstring 明确写「逐条串行而不是并发」,理由是保持章节顺序、避免一条任务 +吃掉 N 个槽位。`run_note_task` 在 1265-1293 行直接分叉到合并流程,合并任务本身不占外层 +闸门;每个子项各自过闸门。因此单条合并任务的并发恒为 1。分发池(`note.py:314-402`) +只并发「多条独立任务」,一个合并任务只是一个任务。 + +### R2 普通批量有效并发是 5,不是设置值 + +`backend/app/services/task_serial_executor.py:9-15`:线程池 `max_workers` 取 +`TASK_MAX_WORKERS`,默认 5,构造期固定;`run()` 里 `future.result()` 同步阻塞调用线程。 +`backend/app/routers/note.py:737-751` 的 `_run_with_parallel_gate`:先拿 gate 名额 +(上限 8),再 `task_serial_executor.run()`。实际并发 = min(8, 5) = 5,用户设置 6/7/8 +全部无效;分发池 8 个线程里还有 3 个占着 gate 名额空等内层池。 + +### R3 媒体资源登记全部失败:数据根多拼一层 data + +`backend/app/utils/path_helper.py:8-18`:开发态 `base_dir = /data`,第 16 行再 +`os.path.join(base_dir, "data")`,返回 `/data/data`。 + +下载落盘在 `backend/app/services/note.py:1194/2026/2052/2110` 的 +`os.getenv("DATA_DIR", "data")`,即 `/data`。`media_resource_service.py:70-81` +的 `_roots()` 用 `get_data_dir()` 做 video/audio 根,`register_resource` → +`_managed_relative`(156-177)直接抛越界;调用方 `services/note.py:1949` 捕获后只打日志。 + +后果:媒体登记表永远为空,保留期清理和容量统计看不见这些文件;`backend/data` 里遗留 +`BV*.mp3`。`backend/data/data` 当前为空,可安全修正。 + +### R4 OCR 429 无重试、无全局并发闸门 + +`backend/app/services/note.py:1304-1439` 的 `_run_ocr_stage`:按 +`ocr_stage_concurrency`(默认 4,用户可调到 8)起线程池逐张 OCR;单张失败被 +1393-1400 捕获后只计数,图照插、任务照跑。 + +`backend/app/services/ocr_client.py:569-674` 的 `recognize_image` 是单次 requests 调用, +429 直接 `_http_error` 抛 `OcrError`;`_is_non_retryable`(114-124)只排除认证/配置类, +429 被标记为「可重试」却没有任何重试循环。多任务叠加时并发可达 8×8,触发上游限流。 + +### R5 判定空结果被当硬异常,触发整条任务重试 + +`backend/app/gpt/universal_gpt.py:307-312`:所有批次都解析不出条目时抛 +`WorthJudgementError("判定结果里找不到帧列表…")`;`backend/app/services/note.py:1127-1137` +捕获后 `_handle_exception` 并 raise,触发整个任务级重试(重新下载/转写/总结)。 + +而提示词 `universal_gpt.py:809-810` 明确允许模型返回 `{"frames": []}`,两者语义冲突。 + +### R6 ASR Pydantic 序列化警告 + +`backend/app/transcriber/openai_asr.py:597-608` 的 `_to_dict` 对 OpenAI SDK 转写对象调 +`model_dump()`。兼容服务把 `segments[].id/seek` 返回成字符串 `'0'`,SDK 模型未校验就持有 +str,Pydantic 序列化告警 `Expected int ... input_value='0'`。仅噪音,但会淹没真实日志。 + +## 3. 目标与非目标 + +目标: + +1. 用户设置的「最大笔记并行任务数」等于实际同时生成的视频数,不再有隐藏的 5 上限。 +2. 合集/合并任务内部并行生成,最终章节顺序严格等于用户勾选顺序。 +3. 合并任务遇到 429 停放或进程重启后,已完成的子项不重新生成。 +4. 媒体资源登记使用真实数据根,重启后自动回填遗留文件。 +5. OCR 429 有退避重试与进程级并发上限,持续失败不拖垮整条任务。 +6. 画面判定空结果/解析失败降级为「保留全部候选」,不再触发整条任务重试。 +7. 消除 ASR 的 Pydantic 序列化告警。 + +非目标: + +- 不改前端 UI 视觉、不改 Tauri 移除成果、不改后端公开 API 形状。 +- 不改供应商配额,不实现跨进程分布式调度。 +- 不把每一条视频内部的处理阶段(下载/转写/总结)再拆成并行;本次只并行「多条视频」。 +- 不为 193 条任务做断点续传之外的通用作业系统。 + +## 4. 已定决策 + +| 编号 | 决策 | 理由 | +|---|---|---| +| D1 | 单一调度器:`_run_with_parallel_gate` 在 gate 内直接执行 `execute_fn`,删除 `task_serial_executor` 的嵌套线程池。gate 是唯一并发上限;分发池保持 `MAX_PARALLEL_NOTES_LIMIT`(8)。单条任务经 BackgroundTasks 执行时同样只受 gate 约束 | 消除 min(gate, 5) 的隐藏上限;线程数不再与设置脱节。用户设置随改随生效 | +| D2 | 合并任务扇出:读当前 gate.limit 作为子任务并发数(最多等于 items 数),每个子项各自过 gate;结果按 index 排序后合并;进度用完成计数 | 全局在途视频数仍不超过 gate;章节顺序由排序保证,不靠串行 | +| D3 | 合并任务断点:每完成一条子项,原子写 `checkpoints/.merge/.json`(url、markdown、title、duration、video_id、cover_url、transcript、source_url)。重启或 429 停放后加载并跳过校验通过的已完成项;最终合并成功后删除该目录;失败项不写 checkpoint,恢复时重试 | 193 条长任务不能因一条 429 从头再来;checkpoint 是「已完成的事实」,不是缓存 | +| D4 | 数据根:`get_data_dir()` 统一为「`DATA_DIR` 环境变量相对 backend 解析;未设置时 `/data`;冻结态 `/data`」,不再二次拼 data。保留期/回填/下载/磁盘检查全部走同一根 | 下载与登记必须看同一个目录;二次拼 data 是 R3 直接原因 | +| D5 | OCR:`recognize_image` 内最多 3 次重试,只重试 429/5xx/超时;429 优先读 `Retry-After`,指数退避 0.5/1/2 秒加抖动,单次等待封顶 30 秒;进程级 `BoundedSemaphore` 上限 `OCR_MAX_CONCURRENCY`(默认 4),所有任务共享。per-task worker 数取 min(设置值, 全局上限)。持续 429 仍按单张失败跳过,任务不失败,但只打一条聚合告警 | 429 是配额节流不是配置错误;全局闸门防 8×8 叠加 | +| D6 | 判定降级:`judge_note_frames` 在所有批次都没有条目时不再抛 `WorthJudgementError`,而是返回「全部候选保留」的 degraded 结果;显式 `{"frames": []}` 同样降级(避免静默零插图);网络/鉴权等真正的 API 错误仍抛出;降级写入任务 detail 与日志 | 判定是「挑图」不是「生成」;挑不出图不该让整条任务重新下载/转写 | +| D7 | ASR:`_to_dict` 调 `model_dump` 时传 `warnings=False`,并优先使用已经就是 dict 的原始响应;不改变 transcript 数值解析 | 告警来自兼容服务的字符串 id/seek;序列化层静音即可,不动数据 | + +## 5. 待确认问题(Q1–Q4,等用户拍板) + +- **Q1 合并扇出默认值**:直接等于 gate(用户设 8 就 8 路),还是额外封顶 4? + 推荐等于 gate,由用户用设置自行降级;8 路 fast-whisper/ffmpeg 对 CPU/内存压力大。 +- **Q2 显式 `{"frames": []}` 的语义**:推荐按 D6 降级为「全部候选保留」;若要保留 + 「模型明确判定无画面就不插图」,需要模型返回可区分字段,否则无法与解析失败区分。 +- **Q3 OCR 全局上限默认 4 是否合适**:用户设置 8 时,是钳制到 4,还是允许设置覆盖全局上限? +- **Q4 checkpoint 放在 note_output 下是否纳入现有保留期清理**:建议合并成功后立即删, + 异常退出时保留到任务终态。 + +## 6. 接口与数据结构草案 + +### 6.1 调度 + +```python +def _run_with_parallel_gate(execute_fn): + gate = _get_parallel_gate() + with gate: + return execute_fn() # 不再经 task_serial_executor.run +``` + +- `task_serial_executor` 模块保留导出名做兼容(别处可能 import),但 note 路由不再依赖它; + 若全仓无引用则删除模块与其测试(W1 内决定,不晚于 W1 收尾)。 +- `_ParallelGate.limit` 作为并发事实来源;合并扇出直接读它。 + +### 6.2 合并 checkpoint + +路径:`/checkpoints/.merge/.json`(index 从 1 起)。 + +```json +{ + "version": 1, + "index": 1, + "url": "https://www.bilibili.com/video/BV...", + "title": "章节标题", + "markdown": "...", + "duration": 123.0, + "video_id": "BV...", + "cover_url": "...", + "transcript": {"segments": []}, + "source_url": "..." +} +``` + +- 写入:`tmp + os.replace` 原子替换;`version` 不匹配视为失效。 +- 读取:`url` 与当前 item 的 url 必须一致,否则忽略并重算。 +- 清理:合并成功(任务终态 SUCCESS)后删除整个 `.merge` 目录; + 任务失败/停放时保留,供下一次恢复。 + +### 6.3 OCR + +```python +OCR_MAX_RETRY = 3 # 首次 + 2 次重试 +OCR_BACKOFF_BASE = 0.5 # 0.5 / 1 / 2 秒 + 抖动 +OCR_RETRY_AFTER_CAP = 30.0 # 单次等待上限 +OCR_MAX_CONCURRENCY = int(os.getenv("OCR_MAX_CONCURRENCY", "4")) +_OCR_GLOBAL_SEM = BoundedSemaphore(OCR_MAX_CONCURRENCY) # 进程级 +``` + +`recognize_image` 返回成功/抛 `OcrError`;重试耗尽后 `OcrError.rate_limited=True` +(或保留 status_code=429 判定)供调用方聚合告警。 + +### 6.4 判定降级 + +`JudgementOutcome` 增加 `degraded: bool = False` 与 `degraded_reason: str | None`。 +`judge_note_frames` 返回 degraded 结果时,`approved` 为全部候选(受 `note_max_images` 限制)。 +任务 detail 写 `{"judgement_degraded": true, "judgement_degraded_reason": "..."}`。 + +### 6.5 数据根 + +``` +DATA_DIR 未设置 → /data (开发)/ /data(冻结) +DATA_DIR 相对路径 → 相对 解析(开发)/ 相对 (冻结) +DATA_DIR 绝对路径 → 原样使用 +``` + +`get_data_dir()` 只创建并返回上述目录,不再追加 `data`。 启动日志打印解析后的 data_root; +若 `/data/data` 仍存在且非空,打 warning(不自动删)。 + +## 7. 流程 + +### 7.1 普通批量 + +用户提交 N 条(含分发池)→ 每条任务经 `_run_with_parallel_gate` 拿 gate 名额 → +直接在当前线程执行 `execute_fn`(不再二次进池)→ 在途数 = min(gate.limit, N)。 +设置改大改小在下次进闸时生效。 + +### 7.2 合并任务扇出 + +1. `_run_merged_note_task` 读 `gate.limit`,`workers = min(limit, len(items))`。 +2. 加载已有 checkpoint,命中且校验通过的项直接复用(不调用生成)。 +3. `ThreadPoolExecutor(max_workers=workers)`,按 index 提交未完成项;每项内部仍 + `with gate:` 才真正跑,因此全局在途不超过 gate。 +4. 完成计数 + 锁更新状态:`合并生成 {done}/{total}`,进度 = done/total*90。 +5. 任一子项返回 `RATE_LIMIT_PARKED`:停止提交新项、取消未开始项、等在途项结束、 + 落盘 checkpoint,再按 spec 012 的停放逻辑交还队列。 +6. 全部完成后按 index 排序 entries(failures 保留原位说明),合并 Markdown,清理 checkpoint 目录。 + +### 7.3 OCR 阶段 + +`_run_ocr_stage` 计算 `workers = min(设置值, OCR_MAX_CONCURRENCY, total)`; +每次 `recognize_image` 在全局 semaphore 内执行重试; +统计 attempts/retries/rate_limited;持续失败只打一条聚合 warning,单图缺失不失败任务。 + +### 7.4 判定阶段 + +`_judge_note_frames` 调 `judge_note_frames`;degraded 结果照常进入挑图层(全候选), +写 detail 标记与一条 warning;网络/鉴权异常仍走原有失败/重试路径。 + +## 8. 测试要点与工作包验收 + +统一测试根:`backend/tests/022-batch-concurrency-and-media-reliability/`(按 W 分 `w1/`…`w5/`)。 +命令统一用 `backend/.venv/Scripts/python.exe -m pytest -q`(Windows)。 + +| 包 | 范围 | 验收命令 | 证据格式 | +|---|---|---|---| +| W0 | spec + README 索引 | `git diff --check`;人工核对 6 段齐全、索引行存在 | 本文件 + `specs/README.md` 行号 | +| W1 | note.py gate/合并/断点、task_serial_executor.py、tests/009 test_parallel_scheduler.py、tests/test_task_serial_executor.py | `pytest tests/022.../w1 tests/009.../test_parallel_scheduler.py tests/test_task_serial_executor.py -q` | pytest 摘要;最大在途计数断言;checkpoint 文件列表;章节顺序断言 | +| W2 | path_helper.py、media_resource_service.py、init_db.py | `pytest tests/022.../w2 tests/007.../test_media_resources.py -q` | 临时 DATA_DIR 下 media_resources.path;`_roots()` 断言;init_db 回填计数;日志无登记失败 | +| W3 | ocr_client.py、note.py `_run_ocr_stage`/`_judge_note_frames`、universal_gpt.py、ocr_assist.py | `pytest tests/022.../w3 -q` + 既有 008 相关用例 | 429→200 请求次数;12 路并发最大在途;degraded detail;聚合告警条数 | +| W4 | openai_asr.py | `pytest tests/022.../w4 -q` | `warnings.catch_warnings(record=True)` 断言无 Pydantic 告警;时间轴断言 | +| W5 | 集成 | 上述全量 + 真实 3–5 条短视频合并 | 日志片段(并发>1、顺序、无五类报错);任务结果 Markdown | + +### 分条测试要点 + +1. **并发上限**:10 个假任务,gate=8 时峰值 8;gate=3 时峰值 3;无 5 的隐藏上限。 +2. **合并扇出与顺序**:10 条、gate=4,峰值 4;最终章节顺序 1..10;进度单调至 10/10。 +3. **合并断点**:第 5 条模拟 429;1–4 落 checkpoint;恢复后生成计数只覆盖 5–10; + 最终 10 条齐全;成功清理 checkpoint。 +4. **媒体根**:临时 DATA_DIR 生成 mp3 → register_resource 成功、path 为 `audio:<相对路径>`; + `_roots()["audio"] == 解析后的 DATA_DIR`;重启式 init_db 回填已有文件; + 日志无 `无法登记任务媒体资源`。 +5. **OCR**:mock 429(带 Retry-After: 1)→ 200,请求 2 次成功;12 路并发全局上限 4; + 持续 429 时任务完成、失败计数正确、仅一条聚合告警、图片仍插入。 +6. **判定降级**:`{"frames": []}` 与垃圾 JSON 均不抛异常;approved == 候选(受上限); + 任务 detail 有 `judgement_degraded=true`;网络错误仍抛。 +7. **ASR**:字符串 id/seek 的假 SDK 模型,无 `PydanticSerializationUnexpectedValue`; + transcript 时间轴解析仍正确。 +8. **回归**:`tests/022...`、`tests/009.../test_parallel_scheduler.py`、 + `tests/007.../test_media_resources.py` 全绿;既有 008 判定用例按新语义更新 + (显式空清单从「抛异常」改为「degraded」)。 +9. **真实合并**:3–5 条短视频合并任务,日志显示子项并发 >1、章节顺序正确、 + 无媒体登记/ASR 告警/OCR 429/判定失败五类报错。 + +## 9. 验收映射(卡面 9 条 → 工作包) + +| 卡面验收 | 工作包 | 测试位置 | +|---|---|---| +| 1 设置 8→峰值 8、3→峰值 3 | W1 | w1/test_parallel_gate.py | +| 2 合并 gate=4 峰值 4、顺序、进度 | W1 | w1/test_merge_fanout.py | +| 3 合并第 5 条 429 断点续跑 | W1 | w1/test_merge_checkpoint.py | +| 4 临时 DATA_DIR 登记/回填/日志 | W2 | w2/test_data_root.py | +| 5 OCR 429 重试/全局 4/聚合告警 | W3 | w3/test_ocr_retry.py | +| 6 判定空结果降级 | W3 | w3/test_judge_degrade.py | +| 7 ASR 字符串 id/seek 无告警 | W4 | w4/test_asr_serialization.py | +| 8 三个测试路径全绿 | W1–W4 | 见上表命令 | +| 9 真实 3–5 条合并日志 | W5 | w5/evidence.md + 日志 | + +## 10. 风险与回滚 + +- **并发升高带来的资源压力**:gate 是唯一上限,用户可下调;默认值不变(仍是设置值)。 + Q1 决定是否给合并扇出额外封顶。 +- **checkpoint 占盘**:每条子项一份 Markdown+transcript;成功即删,失败保留到终态。 + Q4 决定是否纳入保留期清理。 +- **调度改造影响既有并发语义**:spec 009 的测试会从「min(gate, 池, executor)」改为 + 「gate 即上限」;回滚方式是恢复 `task_serial_executor.run` 包装与旧断言(单提交可逆)。 +- **判定降级改变 008 的失败语义**:显式空清单不再让任务失败;网络错误路径不动。 + 回滚只需恢复 raise 分支与 008 测试。 +- **数据根变更**:若用户曾把 `DATA_DIR` 指向 `data/data`,迁移后会指向 `data`; + W2 启动时对旧 `data/data` 打 warning 并保留文件,不自动搬运。 + +## 11. 实现清单 + +- [ ] W0 spec 与索引(本文件) +- [ ] W1 调度器收敛 + 合并扇出 + checkpoint +- [ ] W2 数据根修复与遗留回填 +- [ ] W3 OCR 重试/全局闸门 + 判定降级 +- [ ] W4 ASR 序列化告警 +- [ ] W5 集成验收与真实合并日志 diff --git a/specs/README.md b/specs/README.md index 7a7e1c74..74e4ab83 100644 --- a/specs/README.md +++ b/specs/README.md @@ -49,3 +49,4 @@ specs/ | 019 | [019-global-model-in-provider-section](./019-global-model-in-provider-section/spec.md) | **implemented** | 全局模型从通用设置搬到「AI 模型设置(模型提供商)」页顶部,候选**只列启用供应商下**的模型,选中即保存;通用设置里这一项整个删掉。同时修掉「模型在供应商里已删、设置页仍显示旧名字」——读路径校验(供应商仍启用 + 模型仍存在)+ 删模型时主动清指针,两处兜底。验证:后端 925 passed(新增 6 例,同步 013 文案与 009 fixture)、前端 38 passed + `tsc -b`/`lint`/`build` 全绿、headless Chromium 实测 6 步(含「删掉全局模型后顶部回到未激活」)。 | | 021 | [021-workbench-redesign](./021-workbench-redesign/spec.md) | **implemented** | 工作台重设计(父任务 T-066 拆分 T-068–T-075):深石墨/冷白/青绿双主题语义 token、自托管字体、集中动效、18 原语与三档 AppShell。视觉契约见同目录 `DESIGN.md`。 | | 020 | [020-note-rendering-and-batch-options](./020-note-rendering-and-batch-options/spec.md) | **implemented** | 笔记数学渲染修复 + 批量生成设置 + 分组区滚动 + 提示词图片保留。①分组列表固定 264px 高在 900px 窗口把最后一行切一半 → 改 `max-h-[min(45vh,420px)]` + `overscroll-contain` + section `shrink-0`;②`BatchImportPanel` 原来完全没有设置 UI → 抽 `GenerationOptions` 受控组件,单集/批量共用(`generationSettings.ts` 放类型/默认值/`toPayloadSettings`);③用户那篇笔记 169 个公式 KaTeX 全合法,坏在 remark-math 对「`$$` 与内容同行」解析失败 + 旧逐行转换器处理不了跨行 `\[…\]` → 规范化改为围栏分段整体处理,KaTeX 关 `throwOnError`;④`BASE_PROMPT` 补「图片里的题目/板书/表格原样保留、不改写不讲解」+ 公式分隔符规则。验证:后端 929 passed、前端 48 passed、`tsc -b`/`lint` 干净、浏览器实测三项(滚动 / 批量设置 / 用户笔记渲染 0 error)。 | +| 022 | [022-batch-concurrency-and-media-reliability](./022-batch-concurrency-and-media-reliability/spec.md) | draft | 批量/合集任务真并发与五类报错修复:合并任务扇出+断点续跑(R1/D2/D3)、去掉隐藏 5 上限的单一调度器(R2/D1)、媒体数据根去重拼(R3/D4)、OCR 429 重试与全局并发闸门(R4/D5)、画面判定空结果降级(R5/D6)、ASR 序列化告警消除(R6/D7)。Q1–Q4 待用户拍板。 | From 5348db6f377dbdb870740800bc6bd1e7869adf98 Mon Sep 17 00:00:00 2001 From: Eliver Date: Thu, 12 Nov 2026 14:29:55 +0800 Subject: [PATCH 2/3] =?UTF-8?q?feat(022):=20W1=20=E8=B0=83=E5=BA=A6?= =?UTF-8?q?=E9=97=B8=E9=97=A8=E5=8D=B3=E5=B9=B6=E5=8F=91=E4=B8=8A=E9=99=90?= =?UTF-8?q?=E3=80=81=E5=90=88=E5=B9=B6=E6=89=87=E5=87=BA=E4=B8=8E=E6=96=AD?= =?UTF-8?q?=E7=82=B9=E7=BB=AD=E8=B7=91?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - _run_with_parallel_gate 直接执行任务体,删除 task_serial_executor 固定 5 线程池,实际在途=用户设置 - 合并任务按 min(gate, 条数) 扇出,父任务不占槽位,章节按输入顺序落位 - 每条成功即原子写 checkpoint,429 停放后恢复只补未完成条目,成功保存后清理 - 画面判定解析失败降级保留候选(detail.judgement_degraded),不触发整任务重试 - _run_ocr_stage 预留进程级 OCR 上限接线 - tests/022.../w1:闸门/扇出/断点/降级 11 例;tests/001、tests/009 适配新语义 --- backend/app/routers/note.py | 321 +++++++++++++++--- backend/app/services/note.py | 93 ++++- backend/app/services/task_serial_executor.py | 23 -- .../test_task_retry.py | 7 +- .../test_parallel_scheduler.py | 16 +- .../w1/__init__.py | 0 .../w1/test_judge_degrade.py | 88 +++++ .../w1/test_merge_checkpoint.py | 137 ++++++++ .../w1/test_merge_fanout.py | 135 ++++++++ .../w1/test_parallel_gate.py | 85 +++++ .../w1/test_parallel_gate_and_merge.py | 294 ++++++++++++++++ backend/tests/test_task_serial_executor.py | 43 --- .../spec.md | 3 +- 13 files changed, 1109 insertions(+), 136 deletions(-) delete mode 100644 backend/app/services/task_serial_executor.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w1/__init__.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w1/test_judge_degrade.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_checkpoint.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_fanout.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate_and_merge.py delete mode 100644 backend/tests/test_task_serial_executor.py diff --git a/backend/app/routers/note.py b/backend/app/routers/note.py index 84a0a197..f8ae6689 100644 --- a/backend/app/routers/note.py +++ b/backend/app/routers/note.py @@ -35,7 +35,6 @@ from app.services.source_intake import resolve_bilibili_inputs from app.services.task_cleanup import TaskInFlightError from app.services.task_read_model import TaskReadModel # pyright: ignore[reportMissingImports] -from app.services.task_serial_executor import task_serial_executor from app.utils.llm_rate_limit import ( RATE_LIMIT_RESUME_INTERVAL_SECONDS, clear_rate_limit, @@ -735,20 +734,20 @@ def _expand_video_entry( def _run_with_parallel_gate(execute_fn): - """把任务提交给执行池,并在「最大笔记并行任务数」闸门下排队。 + """在「最大笔记并行任务数」闸门下直接执行任务体。 - 这个闸门与 ``task_serial_executor`` 的线程池是两件事:线程池的 - ``max_workers``(``TASK_MAX_WORKERS``,默认 5)是**进程级**的硬上限,启动后 - 不可改;而用户在「设置 → 通用设置」里改的并行数必须**立即生效**,不能要求 - 重启后端。所以这里每次执行前都重新读一次设置,用一把进出门闩 - (:class:`_ParallelGate`)把在途任务数限制在最新的设定值上。 + spec 022 D1:闸门就是**唯一**的并发上限。以前这里还会把任务体再交给 + ``task_serial_executor`` 的线程池,而那个池的 max_workers 是启动期固定值 + (默认 5),于是用户设 6/7/8 时实际并发被静默压到 5,设置形同虚设。 + 现在每次执行前重读一次设置(闸门热更新),设置值就是实际在途上限。 - 注意闸门要加在**线程池外面**:先拿到门闩再提交,否则任务已经在线程池里 - 占着 worker 空转等门闩,会把池子撑满,反而让别的任务连排队的机会都没有。 + 不加外层线程池不会无限开线程:调用方是固定大小的分发池 + (``_get_dispatch_pool``,默认 8 个 worker),合并任务另有自己的 + ``min(gate.limit, len(items))`` 扇出池,两者都远小于任务总数。 """ gate = _get_parallel_gate() with gate: - return task_serial_executor.run(execute_fn) + return execute_fn() #: 429 退避等满上限后交还队列的哨兵(spec 012)。 @@ -764,7 +763,7 @@ def _execute_note_with_retry(task_id: str, generator, execute_fn): 429 走**独立的一条路**(spec 012):不消耗 NOTE_MAX_RETRIES 预算、不计熔断、 不判失败。撞上 429 就等全局冷却窗口(默认 5 分钟)再试;连续等到上限 (默认 1 小时)仍是 429 就返回 :data:`RATE_LIMIT_PARKED`,把 worker 线程让出来, - 由调用方交还队列 —— 配额恢复可能要几小时,而 worker 只有 TASK_MAX_WORKERS 个。 + 由调用方交还队列 —— 配额恢复可能要几小时,而闸门名额只有设置里的并行数个。 """ # 每次进入重试循环都重读:用户在设置页改完重试次数/退避,下一条任务即生效。 max_retries = runtime_limits.note_max_retries() @@ -1403,6 +1402,141 @@ def _polish_merged_markdown( return polished or markdown +#: 合并任务断点目录的格式版本;结构变更时递增,旧断点自动失效。 +MERGE_CHECKPOINT_VERSION = 1 +#: 孤儿断点目录的保留期(与媒体保留期一致)。 +MERGE_CHECKPOINT_RETENTION_SECONDS = 7 * 24 * 3600 + + +def _merge_checkpoint_dir(task_id: str) -> str: + return os.path.join(NOTE_OUTPUT_DIR, "checkpoints", f"{task_id}.merge") + + +def _read_merge_checkpoint(task_id: str, index: int, url: str) -> dict | None: + """读一条断点;url/index 对不上或文件损坏都当没命中(重跑该条)。""" + path = os.path.join(_merge_checkpoint_dir(task_id), f"{index}.json") + try: + with open(path, "r", encoding="utf-8") as handle: + payload = json.load(handle) + except FileNotFoundError: + return None + except (OSError, ValueError) as exc: + logger.warning(f"读取合并断点失败(忽略该条):{path}:{exc}") + return None + if not isinstance(payload, dict): + return None + if payload.get("version") != MERGE_CHECKPOINT_VERSION: + return None + if payload.get("index") != index or str(payload.get("url") or "") != url: + return None + return payload + + +def _transcript_to_checkpoint(transcript) -> dict | None: + if transcript is None: + return None + return { + "language": getattr(transcript, "language", None), + "full_text": getattr(transcript, "full_text", "") or "", + "segments": [ + { + "start": _as_float(getattr(seg, "start", 0)), + "end": _as_float(getattr(seg, "end", 0)), + "text": str(getattr(seg, "text", "") or ""), + } + for seg in (getattr(transcript, "segments", None) or []) + ], + } + + +def _write_merge_checkpoint(task_id: str, index: int, entry) -> None: + """原子写一条断点:先写 .tmp 再 os.replace,进程中途被杀也不会留下半截 JSON。""" + directory = _merge_checkpoint_dir(task_id) + path = os.path.join(directory, f"{index}.json") + payload = { + "version": MERGE_CHECKPOINT_VERSION, + "index": index, + "url": entry.source_url, + "title": entry.title, + "markdown": entry.markdown, + "duration": entry.duration, + "video_id": entry.video_id, + "cover_url": entry.cover_url, + "source_url": entry.source_url, + "transcript": _transcript_to_checkpoint(entry.transcript), + } + try: + os.makedirs(directory, exist_ok=True) + tmp = f"{path}.tmp" + with open(tmp, "w", encoding="utf-8") as handle: + json.dump(payload, handle, ensure_ascii=False) + handle.flush() + os.fsync(handle.fileno()) + os.replace(tmp, path) + except OSError as exc: + # 断点写失败不该让这一条白跑:本次结果仍在内存里,只是重跑时会重生成。 + logger.warning(f"写入合并断点失败(该条仍会并入本次结果):{path}:{exc}") + + +def _merged_entry_from_checkpoint(payload: dict): + from app.models.transcriber_model import TranscriptResult, TranscriptSegment + from app.services.note_merge import MergedEntry + + transcript = None + raw_transcript = payload.get("transcript") + if isinstance(raw_transcript, dict): + transcript = TranscriptResult( + language=raw_transcript.get("language"), + full_text=str(raw_transcript.get("full_text") or ""), + segments=[ + TranscriptSegment( + start=_as_float(seg.get("start")), + end=_as_float(seg.get("end")), + text=str(seg.get("text") or ""), + ) + for seg in (raw_transcript.get("segments") or []) + if isinstance(seg, dict) + ], + ) + return MergedEntry( + title=str(payload.get("title") or ""), + markdown=str(payload.get("markdown") or ""), + duration=_as_float(payload.get("duration")), + video_id=str(payload.get("video_id") or ""), + cover_url=payload.get("cover_url"), + transcript=transcript, + source_url=str(payload.get("source_url") or payload.get("url") or ""), + ) + + +def _cleanup_merge_checkpoints(task_id: str) -> None: + try: + shutil.rmtree(_merge_checkpoint_dir(task_id)) + except FileNotFoundError: + return + except OSError as exc: + logger.warning(f"清理合并断点失败(不影响任务终态):{exc}") + + +def _sweep_stale_merge_checkpoints() -> None: + """删除超过保留期的孤儿断点目录(进程异常退出才会留下)。""" + root = os.path.join(NOTE_OUTPUT_DIR, "checkpoints") + try: + names = os.listdir(root) + except OSError: + return + now = time.time() + for name in names: + path = os.path.join(root, name) + try: + if now - os.path.getmtime(path) > MERGE_CHECKPOINT_RETENTION_SECONDS: + shutil.rmtree(path) + logger.info(f"删除过期合并断点目录:{path}") + except OSError: + continue + + + def _run_merged_note_task( *, task_id: str, @@ -1429,11 +1563,17 @@ def _run_merged_note_task( ocr_assist: bool | None, park_kwargs: dict, ): - """合并任务执行:逐条生成 → 合并 → 存成**一条**任务(spec 018 §5.2)。 + """合并任务执行:并发生成各条 → 按输入顺序合并 → 存成**一条**任务。 + + spec 022 D2:以前这里逐条串行(一条合并任务永远只用 1 个槽位),193 条合集 + 跑成「逐条排队」。现在按 ``min(闸门 limit, 待生成条数)`` 扇出,每条子任务 + 仍各自过同一把并行闸门——所以合并任务不会突破全局并发上限,也不会因为 + 「父任务先占了槽位」而把自己的扇出堵死(父任务不进闸门,见 run_note_task + 的 merge_items 分叉)。 - 为什么逐条串行而不是并发:合并要按用户勾选顺序排列章节,而且并发跑多个 - 视频会让「最大笔记并行任务数」的语义变得不可预期(一条任务吃掉 N 个槽位)。 - 每次子生成仍各自过并行闸门,所以别的任务不会被这条长任务饿死。 + spec 022 D3:每条成功生成后原子写一份断点;429 停放后重新派发时先读断点, + 已完成且 url 对得上的条目直接复用,不再重新下载/转写/总结。断点目录在任务 + 到达终态(成功或全部失败)后删除;停放不是终态,必须保留。 """ from app.enmus.task_status_enums import TaskStage from app.models.notes_model import NoteResult @@ -1445,29 +1585,51 @@ def _run_merged_note_task( merge_transcripts, ) - entries: list[MergedEntry] = [] - failures: list[MergedFailure] = [] total = len(items) status_writer = NoteGenerator() + _sweep_stale_merge_checkpoints() + + # 结果按输入序号落位:合并章节顺序 = 用户勾选顺序,与完成先后无关。 + results: dict[int, Any] = {} + parked = threading.Event() + keep_checkpoints = False + state_lock = threading.Lock() + + def _record(index: int, outcome: Any, label: str) -> None: + """记一条完成结果并推进进度;进度按「已完成/总数」单调递增。""" + with state_lock: + results[index] = outcome + completed = len(results) + try: + status_writer._update_status( + task_id, + TaskStatus.DOWNLOADING, + stage=TaskStage.DOWNLOADING_VIDEO, + message=f"合并生成 {completed}/{total}:{label}", + progress=min(90, int(completed / max(total, 1) * 90)), + ) + except Exception as exc: # noqa: BLE001 - 进度写失败不该打断合并 + logger.debug(f"写入合并进度失败({task_id}):{exc}") - for index, raw in enumerate(items, start=1): - item = raw if isinstance(raw, dict) else {} - url = str(item.get("url") or "") - label = str(item.get("title") or "") or f"第 {index} 个视频" - if not url: - failures.append(MergedFailure(title=label, reason="链接为空")) - continue - + def _load_cached(index: int, url: str, label: str) -> bool: + payload = _read_merge_checkpoint(task_id, index, url) + if payload is None: + return False try: - status_writer._update_status( - task_id, - TaskStatus.DOWNLOADING, - stage=TaskStage.DOWNLOADING_VIDEO, - message=f"合并生成 {index}/{total}:{label}", - progress=int((index - 1) / max(total, 1) * 90), - ) - except Exception as exc: # noqa: BLE001 - 进度写失败不该打断合并 - logger.debug(f"写入合并进度失败({task_id}):{exc}") + entry = _merged_entry_from_checkpoint(payload) + except Exception as exc: # noqa: BLE001 - 断点坏了就重跑该条 + logger.warning(f"合并断点内容损坏,重跑该条:{exc}") + return False + logger.info(f"合并断点命中,跳过重新生成:task_id={task_id} index={index}") + _record(index, entry, label) + return True + + def _run_one(index: int, item: dict, url: str, label: str) -> None: + if parked.is_set(): + # 已有条目撞上 429 配额耗尽:不再启动新条目,已有的靠断点保住。 + return + if _load_cached(index, url, label): + return sub_task_id = f"{task_id}-s{index}" generator = NoteGenerator() @@ -1502,32 +1664,88 @@ def _execute(sub_id=sub_task_id, sub_url=url, sub_generator=generator): _cleanup_merged_subtask_files(sub_task_id) if note == RATE_LIMIT_PARKED: - # 配额一直恢复不了:交还队列。已合并的部分不落盘,重跑时会重新生成。 logger.warning( - f"合并任务 {task_id} 因 429 交还队列(第 {index}/{total} 条)" + f"合并任务 {task_id} 因 429 停住扇出" + f"(已完成 {len(results)}/{total} 条,断点保留)" ) - _park_rate_limited_task(task_id, park_kwargs) + parked.set() return if not note or not getattr(note, "markdown", None): logger.warning(f"合并子任务失败 task_id={task_id} url={url}") - failures.append(MergedFailure(title=label, reason="生成失败")) - continue + _record(index, MergedFailure(title=label, reason="生成失败"), label) + return audio = getattr(note, "audio_meta", None) - entries.append( - MergedEntry( - title=(getattr(audio, "title", "") or label), - markdown=note.markdown, - duration=_as_float(getattr(audio, "duration", 0)), - video_id=str( - getattr(audio, "video_id", "") or item.get("video_id") or "" - ), - cover_url=getattr(audio, "cover_url", None), - transcript=getattr(note, "transcript", None), - source_url=url, - ) + entry = MergedEntry( + title=(getattr(audio, "title", "") or label), + markdown=note.markdown, + duration=_as_float(getattr(audio, "duration", 0)), + video_id=str( + getattr(audio, "video_id", "") or item.get("video_id") or "" + ), + cover_url=getattr(audio, "cover_url", None), + transcript=getattr(note, "transcript", None), + source_url=url, ) + _write_merge_checkpoint(task_id, index, entry) + _record(index, entry, label) + + # spec 022 D2:扇出 = min(闸门 limit, 待生成条数)。父任务本身不持闸门, + # 子任务各自过闸门,所以全局在途数仍被设置值卡住。 + gate = _get_parallel_gate() + workers = max(1, min(gate.limit, total)) + futures: list = [] + try: + with ThreadPoolExecutor( + max_workers=workers, thread_name_prefix="merge-fanout" + ) as pool: + for index, raw in enumerate(items, start=1): + item = raw if isinstance(raw, dict) else {} + url = str(item.get("url") or "") + label = str(item.get("title") or "") or f"第 {index} 个视频" + if not url: + _record(index, MergedFailure(title=label, reason="链接为空"), label) + continue + futures.append(pool.submit(_run_one, index, item, url, label)) + + # 等待期间发现 429:取消还没开始的条目,避免它们继续抢配额。 + while not parked.is_set(): + if all(future.done() for future in futures): + break + time.sleep(0.2) + if parked.is_set(): + for future in futures: + future.cancel() + + for future in futures: + if future.cancelled(): + continue + try: + future.result() + except Exception as exc: # noqa: BLE001 - 单条异常不该毁掉整篇合并 + logger.error(f"合并子任务线程异常(task_id={task_id}):{exc}") + + if parked.is_set(): + # 停放不是终态:断点必须留着,重新派发时跳过已完成的条目。 + keep_checkpoints = True + _park_rate_limited_task(task_id, park_kwargs) + return + finally: + # 断点只在「成功保存」后删除(下面 SUCCESS 分支),失败/停放都必须留着。 + if parked.is_set(): + keep_checkpoints = True + + entries = [ + results[index] + for index in sorted(results) + if isinstance(results[index], MergedEntry) + ] + failures = [ + results[index] + for index in sorted(results) + if isinstance(results[index], MergedFailure) + ] if not entries: status_writer._update_status( @@ -1579,6 +1797,7 @@ def _execute(sub_id=sub_task_id, sub_url=url, sub_generator=generator): logger.debug(f"写入保存进度失败({task_id}):{exc}") save_note_to_file(task_id, merged_note) + _cleanup_merge_checkpoints(task_id) _get_circuit_breaker().record_success() message = f"已合并 {len(entries)}/{total} 个视频" diff --git a/backend/app/services/note.py b/backend/app/services/note.py index b7a1e364..f30a283f 100644 --- a/backend/app/services/note.py +++ b/backend/app/services/note.py @@ -287,6 +287,10 @@ def __init__(self): self.frame_candidates: list[float] | None = None # 抽帧实际产出的时间点 self.picked_frames: list[float] | None = None # 挑选层挑中的时间点 self.approved_shots: list | None = None # 判定通过、必须编入笔记的截图点 + # 画面判定的降级原因(spec 022 D6):empty_frames / parse_failed / None。 + # 降级时 approved_shots 是「全部候选」而不是空清单,任务不失败, + # 但用户必须能从任务 detail 里看出这次插图没有经过模型确认。 + self.judgement_degraded: str | None = None # 判定通过的每张图各自 OCR 出来的完整内容(m04445:无论 OCR 返回什么, # 都作为参考喂给总结模型)。键是时间点秒数,值是结构化参考 dict。 self.ocr_texts: dict | None = None @@ -756,7 +760,18 @@ def generate( ) # 6. 完成 - self._update_status(task_id, TaskStatus.SUCCESS) + self._update_status( + task_id, + TaskStatus.SUCCESS, + detail=( + { + "judgement_degraded": True, + "judgement_degraded_reason": self.judgement_degraded, + } + if self.judgement_degraded + else None + ), + ) logger.info(f"笔记生成成功 (task_id={task_id})") return NoteResult( markdown=markdown or "", transcript=transcript, audio_meta=audio_meta @@ -1088,9 +1103,11 @@ def _judge_note_frames( ) -> list: """第二次调用:让 LLM 看网格图,判定哪些候选帧值得编入笔记。 - 返回值就是「允许编入笔记」的全部来源。异常一律向上抛(网络失败、解析失败 - 都算),因为静默返回空清单会让整篇笔记在用户毫无察觉的情况下没有插图 - (spec 008 §5.6b)。 + 返回值就是「允许编入笔记」的全部来源。网络失败、鉴权失败一律向上抛; + 但 spec 022 D6 起,**解析失败/模型给出空清单**不再抛到任务级:任务级重试 + 会重做下载、转写、总结,代价极高而结果大概率一样。降级时返回全部候选 + (受 note_max_images 截断),并把原因记在 self.judgement_degraded + (empty_frames / parse_failed),由任务 detail 对用户可见。 ``candidates`` 缺省时退回 ``self.frame_candidates``(抽帧实际产出的时间点), 但 OCR 辅助模式下应当由 `_pick_note_frames` 的挑选结果显式传入——判定集合 @@ -1124,6 +1141,8 @@ def _judge_note_frames( except Exception: logger.warning("探测视频时长失败,判定提示词里的时长按未知处理") + from app.services.ocr_assist import WorthDecision, WorthJudgementError + try: outcome = gpt.judge_note_frames( transcript.segments if transcript else [], @@ -1131,13 +1150,61 @@ def _judge_note_frames( duration=duration, image_urls=self.video_img_urls, ) + except WorthJudgementError as exc: + # spec 022 D6:解析失败(含模型显式返回空清单)降级为「保留全部候选」, + # 不再触发任务级重试 —— 重试会重做下载/转写/总结,代价极高且结果大概率一样。 + cap = max(1, runtime_limits.note_max_images()) + approved = [ + WorthDecision(time=float(point)) + for point in candidates[:cap] + if point is not None + ] + self.judgement_degraded = str(exc) or "判定期望结果无法解析" + logger.warning( + "画面价值判定降级(%s):保留 %d 个候选画面继续生成", + exc, + len(approved), + ) + self._update_status( + task_id, + TaskStatus.SUMMARIZING, + stage=TaskStage.SUMMARIZING, + message="画面判定降级:保留全部候选画面", + detail={ + "judgement_degraded": True, + "judgement_degraded_reason": self.judgement_degraded, + "candidates": len(candidates), + }, + ) + return approved except Exception as exc: logger.error(f"画面价值判定失败:{exc}") self._handle_exception(task_id, exc) raise + degraded = getattr(outcome, "degraded", None) + if degraded: + # spec 022 D6:保留全部候选继续出笔记,只降级插图质量,不让整任务失败。 + self.judgement_degraded = degraded + logger.warning( + "画面价值判定降级(%s):保留 %d 个候选画面继续生成", + degraded, + len(outcome.approved), + ) + self._update_status( + task_id, + TaskStatus.SUMMARIZING, + stage=TaskStage.SUMMARIZING, + message="画面判定降级:保留全部候选画面", + detail={ + "judgement_degraded": True, + "judgement_degraded_reason": degraded, + "candidates": len(candidates), + }, + ) + logger.info("画面价值判定:%s", outcome.describe()) - approved = list(outcome.approved) + approved = [c for c in (outcome.approved or []) if c is not None] if not approved: logger.warning("画面价值判定:没有任何候选帧被判为值得编入笔记") return approved @@ -1375,12 +1442,22 @@ def _recognize(item: tuple) -> tuple: return item, None, None, exc if ocr_client is not None: + # spec 022 D5:三个上限取小 —— 用户设置(单任务意图)、进程级全局 + # 信号量(多任务叠加时的真实配额压力)、本次待处理张数。 + workers = max( + 1, min(runtime_limits.ocr_stage_concurrency(), len(staged)) + ) + try: + from app.services.ocr_client import ocr_max_concurrency + except ImportError: + # W3 落地前没有全局信号量:退回单任务设置,不改变现行为。 + pass + else: + workers = max(1, min(workers, ocr_max_concurrency())) with ThreadPoolExecutor( # 每帧一次 ffmpeg 抽图 + 一次视觉模型调用:并发太高会同时压垮 CPU # 与上游配额,所以并发数由用户在设置页控制(默认 4)。 - max_workers=max( - 1, min(runtime_limits.ocr_stage_concurrency(), len(staged)) - ) + max_workers=workers ) as pool: # map 保序:回填顺序与串行版本一致,日志里的编号仍然对得上。 recognized = list(pool.map(_recognize, staged)) diff --git a/backend/app/services/task_serial_executor.py b/backend/app/services/task_serial_executor.py deleted file mode 100644 index c2c6e5b9..00000000 --- a/backend/app/services/task_serial_executor.py +++ /dev/null @@ -1,23 +0,0 @@ -import os -from concurrent.futures import ThreadPoolExecutor, Future -from typing import Any, Callable - - -class ConcurrentTaskExecutor: - """使用线程池并发执行任务,替代原来的串行锁。""" - - def __init__(self, max_workers: int | None = None): - self._max_workers = max_workers or int(os.getenv("TASK_MAX_WORKERS", "5")) - self._pool = ThreadPoolExecutor(max_workers=self._max_workers) - - def run(self, fn: Callable[..., Any], *args: Any, **kwargs: Any) -> Any: - future: Future = self._pool.submit(fn, *args, **kwargs) - return future.result() - - def shutdown(self, wait: bool = True): - self._pool.shutdown(wait=wait) - - -# 保持向后兼容的导出名 -SerialTaskExecutor = ConcurrentTaskExecutor -task_serial_executor = ConcurrentTaskExecutor() diff --git a/backend/tests/001-bilibili-batch-import/test_task_retry.py b/backend/tests/001-bilibili-batch-import/test_task_retry.py index ab559743..1419291c 100644 --- a/backend/tests/001-bilibili-batch-import/test_task_retry.py +++ b/backend/tests/001-bilibili-batch-import/test_task_retry.py @@ -9,12 +9,15 @@ def _run_with(execute, generator): # 重试次数与退避基数现在是「设置 → 通用」里的可配置项,走 runtime_limits # 门面读取(改完立刻生效,不必重启)。以前这些用例改的是 note_module 上的 # 模块常量,那个常量如今只是文档与兜底,改它不再影响执行路径。 + # spec 022 D1 之后没有外层线程池:任务体直接在并入门闩下执行, + # 这里把门闩换成直通,免得用例去读设置库。 with mock.patch.object(note_module.runtime_limits, "note_max_retries", lambda: 3), \ mock.patch.object( note_module.runtime_limits, "note_retry_backoff_seconds", lambda: 0.0 ), \ - mock.patch.object(note_module, "task_serial_executor") as executor: - executor.run.side_effect = lambda fn: fn() + mock.patch.object( + note_module, "_run_with_parallel_gate", lambda fn: fn() + ): return note_module._execute_note_with_retry("t1", generator, execute) diff --git a/backend/tests/009-batch-grouping-and-long-run-robustness/test_parallel_scheduler.py b/backend/tests/009-batch-grouping-and-long-run-robustness/test_parallel_scheduler.py index 82f05f34..2dd7a67a 100644 --- a/backend/tests/009-batch-grouping-and-long-run-robustness/test_parallel_scheduler.py +++ b/backend/tests/009-batch-grouping-and-long-run-robustness/test_parallel_scheduler.py @@ -9,13 +9,14 @@ ``_execute_note_with_retry → _run_with_parallel_gate`` 获取。 所以假任务体必须**显式调用** ``note_module._run_with_parallel_gate(...)``, 否则测出来的并发只受池子大小约束,等于没测闸门。 -3. 端到端观察到的并发 = ``min(闸门 limit, 分发池 8, task_serial_executor 5)``。 +3. 端到端观察到的并发 = ``min(闸门 limit, 分发池 8)`` —— spec 022 D1 删掉了 + ``task_serial_executor`` 那层固定 5 线程的池,闸门是唯一上限。 ⚠️ ticket 02 改动点 1 写的是「先在**池外** ``gate.acquire()`` 再提交线程池」, 但实现里 ``_dispatch_note_task`` 完全没碰闸门 —— 任务**已经占住分发池 worker 之后**才在 ``_run_with_parallel_gate`` 里等门闩。闸门「在池外」这条约束成立的 -只是 ``_run_with_parallel_gate`` 内部那一层(gate 在 ``task_serial_executor`` -池外)。这里按**真实实现**断言,不去迁就工单描述。 +只是 ``_run_with_parallel_gate`` 内部那一层(gate 在任务体外面)。这里按 +**真实实现**断言,不去迁就工单描述。 ``_PARALLEL_GATE`` 是模块级单例,跨用例泄漏状态,所以 fixture 每个用例前后都 把它置回 None;``_safe_max_parallel_notes`` 被替换成读一个可变的 holder, @@ -119,10 +120,9 @@ def test_dispatch_pool_itself_is_a_small_bounded_container(gate_env, monkeypatch assert pool._max_workers <= MAX_PARALLEL_NOTES_LIMIT # 闸门 limit=1 时任务体只能一个接一个地进 —— 同时刻最多 1 个。 assert probe.max_seen == 1 - # 但「用过的线程名」可能不止一个:闸门内层还叠着 task_serial_executor 的池 - # (TASK_MAX_WORKERS,默认 5),它会在自己的 worker 之间轮换。这不是泄漏, - # 只是串行复用;数量仍必须被内层池的大小卡住,而不是按任务数增长。 - assert len(probe.thread_names) <= 5 + # spec 022 D1 之后闸门里直接执行任务体,不再有内层线程池轮换: + # 用到的线程名只会来自分发池,必须被池大小卡住,而不是按任务数增长。 + assert len(probe.thread_names) <= MAX_PARALLEL_NOTES_LIMIT def test_gate_limit_hot_update_takes_effect_without_restart(gate_env, monkeypatch): @@ -172,7 +172,7 @@ def test_long_batch_smoke_200_items(gate_env, monkeypatch): len(probe.thread_names) <= MAX_PARALLEL_NOTES_LIMIT + 5 ), f"200 条任务用到了 {len(probe.thread_names)} 个不同线程,说明在按任务数造线程" - # 全部跑完后线程数要回落到接近提交前的水位(给池子 + 执行池的常驻余量)。 + # 全部跑完后线程数要回落到接近提交前的水位(给分发池的常驻余量)。 for _ in range(50): if threading.active_count() <= baseline_threads + 20: break diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w1/__init__.py b/backend/tests/022-batch-concurrency-and-media-reliability/w1/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_judge_degrade.py b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_judge_degrade.py new file mode 100644 index 00000000..fbd5335c --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_judge_degrade.py @@ -0,0 +1,88 @@ +"""spec 022 W1 验收 5:画面判定降级不触发任务级重试(D6)。 + +模型解析不出帧列表(WorthJudgementError)时,保留全部候选帧继续出笔记, +任务 detail 里写 judgement_degraded=true;网络/鉴权等真实 API 错误仍抛。 +""" + +import pytest + +from app.services.note import NoteGenerator +from app.services.ocr_assist import WorthJudgementError + + +class _FakeGpt: + def __init__(self, exc): + self._exc = exc + + def judge_note_frames(self, *args, **kwargs): + raise self._exc + + +def _generator(monkeypatch, statuses): + generator = NoteGenerator() + generator.video_img_urls = [] + + def fake_update(task_id, status, **kwargs): + statuses.append({"task_id": task_id, "status": status, **kwargs}) + + monkeypatch.setattr(generator, "_update_status", fake_update) + + def fail_handle(*args, **kwargs): + raise AssertionError("降级不该走 _handle_exception(那会触发整任务重试)") + + monkeypatch.setattr(generator, "_handle_exception", fail_handle) + return generator + + +def test_empty_frames_degrades_and_keeps_candidates(monkeypatch): + statuses = [] + generator = _generator(monkeypatch, statuses) + monkeypatch.setattr( + "app.services.runtime_limits.note_max_images", lambda: 3, raising=True + ) + + approved = generator._judge_note_frames( + _FakeGpt(WorthJudgementError("判定结果里找不到帧列表")), + transcript=None, + task_id="w1-judge", + candidates=[1.0, 2.5, 3.0], + ) + + assert [item.time for item in approved] == [1.0, 2.5, 3.0] + assert generator.judgement_degraded + degraded = [s for s in statuses if "judgement_degraded" in (s.get("detail") or {})] + assert degraded and degraded[-1]["detail"]["judgement_degraded"] is True + assert degraded[-1]["detail"]["judgement_degraded_reason"] + + +def test_degrade_respects_note_max_images(monkeypatch): + statuses = [] + generator = _generator(monkeypatch, statuses) + monkeypatch.setattr( + "app.services.runtime_limits.note_max_images", lambda: 2, raising=True + ) + + approved = generator._judge_note_frames( + _FakeGpt(WorthJudgementError("空帧列表")), + transcript=None, + task_id="w1-judge", + candidates=[1.0, 2.0, 3.0, 4.0], + ) + + assert [item.time for item in approved] == [1.0, 2.0] + + +def test_real_api_error_still_raises(monkeypatch): + statuses = [] + generator = _generator(monkeypatch, statuses) + errors = [] + monkeypatch.setattr(generator, "_handle_exception", lambda t, e: errors.append(e)) + + with pytest.raises(RuntimeError): + generator._judge_note_frames( + _FakeGpt(RuntimeError("401 unauthorized")), + transcript=None, + task_id="w1-judge", + candidates=[1.0], + ) + assert errors, "真实 API 错误必须走原失败路径" diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_checkpoint.py b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_checkpoint.py new file mode 100644 index 00000000..915ec18c --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_checkpoint.py @@ -0,0 +1,137 @@ +"""spec 022 W1:合并任务断点续跑(D3)。 + +第 5 条撞 429(RATE_LIMIT_PARKED)时,前 4 条必须已落 checkpoint;重新派发后 +前 4 条不再生成,只跑 5..10,最终 10 条齐全、顺序不变,成功后删除 checkpoint。 +""" + +import os +from types import SimpleNamespace + +import pytest + +from app.routers import note as note_module + + +class _Limit: + def __init__(self, value): + self.value = value + + +@pytest.fixture +def gate_env(monkeypatch): + holder = _Limit(1) + monkeypatch.setattr(note_module, "_PARALLEL_GATE", None) + monkeypatch.setattr(note_module, "_safe_max_parallel_notes", lambda: holder.value) + yield holder + gate = note_module._PARALLEL_GATE + if gate is not None: + assert gate.active == 0, f"用例结束时闸门仍有 {gate.active} 个名额未归还" + + +class _FakeGenerator: + def __init__(self, probe=None): + self._probe = probe + + def _update_status(self, *args, **kwargs): + return None + + def generate(self, **kwargs): + url = kwargs["video_url"] + return SimpleNamespace( + markdown=f"### {url}", + transcript=None, + audio_meta=SimpleNamespace( + title=url, duration=1.0, video_id=url.rsplit("/", 1)[-1], cover_url=None + ), + ) + + +def _items(count): + return [{"url": f"https://video/{i}", "title": f"标题{i}"} for i in range(1, count + 1)] + + +def _kwargs(items): + return dict( + task_id="w1-merge-cp", + items=items, + merge_title="合并", + merge_polish=False, + model_name=None, + provider_id=None, + platform="bilibili", + quality=None, + link=False, + screenshot=False, + _format=None, + scene=None, + detail=None, + extras=None, + video_understanding=False, + video_interval=None, + grid_size=None, + sampling_mode="interval", + max_frames=None, + frames_per_second=None, + frame_limit_mode="count", + ocr_assist=None, + park_kwargs={}, + ) + + +@pytest.fixture +def merge_env(tmp_path, monkeypatch, gate_env): + calls = {} + parked = [] + state = SimpleNamespace(park_index=5) + + monkeypatch.setattr(note_module, "NOTE_OUTPUT_DIR", str(tmp_path / "notes")) + monkeypatch.setattr(note_module, "NoteGenerator", lambda: _FakeGenerator()) + monkeypatch.setattr(note_module, "_cleanup_merged_subtask_files", lambda _id: None) + monkeypatch.setattr(note_module, "_enforce_cache_limit_in_background", lambda: None) + monkeypatch.setattr( + note_module, "_park_rate_limited_task", lambda task_id, kwargs: parked.append(task_id) + ) + saved = {} + monkeypatch.setattr( + note_module, "save_note_to_file", lambda task_id, note: saved.setdefault(task_id, note) + ) + breaker = SimpleNamespace(record_success=lambda: None, record_failure=lambda *a, **k: None) + monkeypatch.setattr(note_module, "_get_circuit_breaker", lambda: breaker) + + def fake_retry(sub_task_id, generator, execute_fn): + index = int(sub_task_id.rsplit("-s", 1)[1]) + calls[index] = calls.get(index, 0) + 1 + if index == state.park_index: + return note_module.RATE_LIMIT_PARKED + return execute_fn() + + monkeypatch.setattr(note_module, "_execute_note_with_retry", fake_retry) + return SimpleNamespace(calls=calls, parked=parked, saved=saved, state=state) + + +def test_resume_skips_completed_subitems(merge_env, monkeypatch): + items = _items(10) + kwargs = _kwargs(items) + + merge_env.parked.clear() + # 第一次:第 5 条 429,1..4 已落盘,停放保留 checkpoint。 + note_module._run_merged_note_task(**kwargs) + assert set(merge_env.calls) == {1, 2, 3, 4, 5} + assert merge_env.parked == ["w1-merge-cp"] + + directory = note_module._merge_checkpoint_dir("w1-merge-cp") + files = sorted(os.listdir(directory)) + assert files == ["1.json", "2.json", "3.json", "4.json"], f"断点文件异常:{files}" + + # 第二次:配额恢复(不再 429),1..4 从 checkpoint 复用,只生成 5..10。 + merge_env.calls.clear() + merge_env.state.park_index = None + note_module._run_merged_note_task(**kwargs) + + assert set(merge_env.calls) == {5, 6, 7, 8, 9, 10}, ( + f"恢复后只该生成 5..10,实际调用了 {sorted(merge_env.calls)}" + ) + merged = merge_env.saved["w1-merge-cp"] + positions = [merged.markdown.index(f"https://video/{i}") for i in range(1, 11)] + assert positions == sorted(positions), "恢复后的章节顺序必须仍是 1..10" + assert not os.path.exists(directory), "成功保存后必须删除 checkpoint 目录" diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_fanout.py b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_fanout.py new file mode 100644 index 00000000..020957f1 --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_merge_fanout.py @@ -0,0 +1,135 @@ +"""spec 022 W1:合并任务内部扇出(D2)。 + +10 条子项、闸门 4:同时生成的最大条数必须是 4(不是串行),且最终章节顺序 +严格等于用户勾选顺序 1..10。 +""" + +import threading +import time +from types import SimpleNamespace + +import pytest + +from app.routers import note as note_module + + +class _Limit: + def __init__(self, value): + self.value = value + + +@pytest.fixture +def gate_env(monkeypatch): + holder = _Limit(4) + monkeypatch.setattr(note_module, "_PARALLEL_GATE", None) + monkeypatch.setattr(note_module, "_safe_max_parallel_notes", lambda: holder.value) + yield holder + gate = note_module._PARALLEL_GATE + if gate is not None: + assert gate.active == 0, f"用例结束时闸门仍有 {gate.active} 个名额未归还" + + +class _Probe: + def __init__(self): + self.lock = threading.Lock() + self.current = 0 + self.max_seen = 0 + + def enter(self): + with self.lock: + self.current += 1 + self.max_seen = max(self.max_seen, self.current) + + def leave(self): + with self.lock: + self.current -= 1 + + +class _FakeGenerator: + """NoteGenerator 的替身:只提供 _update_status 与 generate。""" + + def __init__(self, probe=None): + self._probe = probe + + def _update_status(self, *args, **kwargs): + return None + + def generate(self, **kwargs): + url = kwargs["video_url"] + if self._probe is not None: + self._probe.enter() + time.sleep(0.05) + self._probe.leave() + return SimpleNamespace( + markdown=f"### {url}", + transcript=None, + audio_meta=SimpleNamespace( + title=url, duration=1.0, video_id=url.rsplit("/", 1)[-1], cover_url=None + ), + ) + + +def _items(count): + return [{"url": f"https://video/{i}", "title": f"标题{i}"} for i in range(1, count + 1)] + + +def _kwargs(tmp_path, items, gate=4): + return dict( + task_id="w1-merge", + items=items, + merge_title="合并", + merge_polish=False, + model_name=None, + provider_id=None, + platform="bilibili", + quality=None, + link=False, + screenshot=False, + _format=None, + scene=None, + detail=None, + extras=None, + video_understanding=False, + video_interval=None, + grid_size=None, + sampling_mode="interval", + max_frames=None, + frames_per_second=None, + frame_limit_mode="count", + ocr_assist=None, + park_kwargs={}, + ) + + +@pytest.fixture +def merge_env(tmp_path, monkeypatch, gate_env): + probe = _Probe() + monkeypatch.setattr(note_module, "NOTE_OUTPUT_DIR", str(tmp_path / "notes")) + monkeypatch.setattr(note_module, "NoteGenerator", lambda: _FakeGenerator(probe)) + monkeypatch.setattr(note_module, "_cleanup_merged_subtask_files", lambda _id: None) + monkeypatch.setattr(note_module, "_enforce_cache_limit_in_background", lambda: None) + monkeypatch.setattr(note_module, "_park_rate_limited_task", lambda *_a, **_k: None) + saved = {} + monkeypatch.setattr( + note_module, "save_note_to_file", lambda task_id, note: saved.setdefault(task_id, note) + ) + frozen_breaker = SimpleNamespace( + record_success=lambda: None, record_failure=lambda *_a, **_k: None + ) + monkeypatch.setattr(note_module, "_get_circuit_breaker", lambda: frozen_breaker) + + def fake_retry(sub_task_id, generator, execute_fn): + return execute_fn() + + monkeypatch.setattr(note_module, "_execute_note_with_retry", fake_retry) + return SimpleNamespace(probe=probe, saved=saved) + + +def test_merge_fans_out_four_and_keeps_order(merge_env, gate_env): + items = _items(10) + note_module._run_merged_note_task(**_kwargs(".", items, gate=4)) + + assert merge_env.probe.max_seen == 4, f"扇出峰值应为 4,实际 {merge_env.probe.max_seen}" + merged = merge_env.saved["w1-merge"] + positions = [merged.markdown.index(f"https://video/{i}") for i in range(1, 11)] + assert positions == sorted(positions), "章节顺序必须等于输入顺序 1..10" diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate.py b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate.py new file mode 100644 index 00000000..385bea94 --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate.py @@ -0,0 +1,85 @@ +"""spec 022 W1:闸门是唯一并发上限(D1)。 + +以前 _run_with_parallel_gate 先拿名额再把任务体交给 task_serial_executor +(固定 5 worker),用户设 8 时实际并发只有 5。本用例直接测「设置 8 → 10 个假任务 +峰值 8;设置 3 → 峰值 3」,即用户设置就是实际在途上限。 +""" + +import threading +import time + +import pytest + +from app.routers import note as note_module + + +class _Limit: + def __init__(self, value): + self.value = value + + +@pytest.fixture +def gate_env(monkeypatch): + holder = _Limit(2) + monkeypatch.setattr(note_module, "_PARALLEL_GATE", None) + monkeypatch.setattr(note_module, "_safe_max_parallel_notes", lambda: holder.value) + yield holder + gate = note_module._PARALLEL_GATE + if gate is not None: + assert gate.active == 0, f"用例结束时闸门仍有 {gate.active} 个名额未归还" + + +class _Probe: + """记录同时进入任务体的线程数与总进入次数。""" + + def __init__(self, target, delay=0.03): + self._lock = threading.Lock() + self._target = target + self._delay = delay + self.current = 0 + self.max_seen = 0 + self.entered = 0 + self.done = threading.Event() + + def body(self): + with self._lock: + self.current += 1 + self.entered += 1 + self.max_seen = max(self.max_seen, self.current) + last = self.entered >= self._target + try: + time.sleep(self._delay) + finally: + with self._lock: + self.current -= 1 + if last: + self.done.set() + + +def _run_batch(monkeypatch, count): + probe = _Probe(count) + + def fake_run_note_task(**kwargs): + note_module._run_with_parallel_gate(probe.body) + + monkeypatch.setattr(note_module, "run_note_task", fake_run_note_task) + for index in range(count): + note_module._dispatch_note_task({"task_id": f"w1-t{index}"}) + assert probe.done.wait(20.0), f"{count} 条任务在 20s 内没跑完" + return probe + + +def test_setting_8_means_peak_8(gate_env, monkeypatch): + gate_env.value = 8 + probe = _run_batch(monkeypatch, count=10) + + assert probe.entered == 10 + assert probe.max_seen == 8, f"设置 8 时最大在途应为 8,实际 {probe.max_seen}" + + +def test_setting_3_means_peak_3(gate_env, monkeypatch): + gate_env.value = 3 + probe = _run_batch(monkeypatch, count=10) + + assert probe.entered == 10 + assert probe.max_seen == 3, f"设置 3 时最大在途应为 3,实际 {probe.max_seen}" diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate_and_merge.py b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate_and_merge.py new file mode 100644 index 00000000..91aa10ea --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w1/test_parallel_gate_and_merge.py @@ -0,0 +1,294 @@ +"""spec 022 W1:闸门是唯一并发上限 + 合并任务真并发 + 429 断点续跑。 + +验收对应用户案卷第 1/2/3 条: + +1. 设置并行 8 时 10 条假任务最大在途 8;设 3 时为 3(没有隐藏的 5 上限)。 +2. 10 条合并任务 gate=4 时最大在途 4,章节顺序严格等于输入顺序,进度单调到 10/10。 +3. 第 5 条起模拟 429 停放:1-4 的结果在断点里,恢复后 1-4 不重新生成,5-10 继续, + 最终 10 条齐全。 + +合并任务用假 NoteGenerator 替换下载/转写/总结,其余走的都是真实代码: +真实闸门、真实断点读写、真实 build_merged_markdown、真实进度写入。 +""" + +import re +import threading +import time + +import pytest + +from app.enmus.note_enums import DownloadQuality +from app.models.audio_model import AudioDownloadResult +from app.models.notes_model import NoteResult +from app.models.transcriber_model import TranscriptResult, TranscriptSegment +from app.routers import note as note_module + + +class _Limit: + """可变的并行上限,替代 _safe_max_parallel_notes() 的返回值。""" + + def __init__(self, value): + self.value = value + + +@pytest.fixture +def gate_env(monkeypatch, tmp_path): + """每个用例一张干净的闸门 + 可控的并行上限 + 隔离的输出目录。""" + holder = _Limit(2) + monkeypatch.setattr(note_module, "_PARALLEL_GATE", None) + monkeypatch.setattr(note_module, "_safe_max_parallel_notes", lambda: holder.value) + note_dir = tmp_path / "note_results" + note_dir.mkdir() + monkeypatch.setattr(note_module, "NOTE_OUTPUT_DIR", str(note_dir)) + yield holder + # 探针在「最后一条进入任务体」时就返回,可能仍有任务在闸门内收尾; + # 留 2s 归还窗口再断言,避免把正常收尾误报成名额泄漏。 + gate = note_module._PARALLEL_GATE + if gate is not None: + deadline = time.monotonic() + 2 + while gate.active and time.monotonic() < deadline: + time.sleep(0.02) + assert gate.active == 0, f"用例结束时闸门仍有 {gate.active} 个名额未归还" + + +class _Probe: + """记录同时在执行的线程数、进入次数。""" + + def __init__(self, delay=0.03): + self._lock = threading.Lock() + self._delay = delay + self.current = 0 + self.max_seen = 0 + self.entered = 0 + self.finished = 0 + + def body(self): + with self._lock: + self.current += 1 + self.entered += 1 + self.max_seen = max(self.max_seen, self.current) + try: + time.sleep(self._delay) + finally: + with self._lock: + self.current -= 1 + self.finished += 1 + + +# --------------------------------------------------------------------------- +# 1) 闸门就是上限:设 8 得 8,设 3 得 3 +# --------------------------------------------------------------------------- + + +def _run_dispatch_batch(monkeypatch, count): + probe = _Probe(delay=0.15) + + def fake_run_note_task(**kwargs): + note_module._run_with_parallel_gate(probe.body) + + monkeypatch.setattr(note_module, "run_note_task", fake_run_note_task) + for index in range(count): + note_module._dispatch_note_task({"task_id": f"t{index}"}) + + deadline = time.monotonic() + 20 + while probe.finished < count and time.monotonic() < deadline: + time.sleep(0.05) + assert probe.finished == count, f"{count} 条任务没能在 20s 内跑完" + # 任务体返回后闸门才 release,稍等一拍再让 fixture 断言名额归零。 + time.sleep(0.2) + return probe + + +def test_gate_allows_eight_concurrent_tasks(gate_env, monkeypatch): + gate_env.value = 8 + probe = _run_dispatch_batch(monkeypatch, count=10) + assert probe.max_seen == 8, f"设置 8 时最大在途应为 8,实际 {probe.max_seen}(疑似仍有隐藏上限)" + + +def test_gate_respects_lower_setting(gate_env, monkeypatch): + gate_env.value = 3 + probe = _run_dispatch_batch(monkeypatch, count=10) + assert probe.max_seen == 3, f"设置 3 时最大在途应为 3,实际 {probe.max_seen}" + + +# --------------------------------------------------------------------------- +# 2/3) 合并任务:并发扇出、顺序、进度、断点 +# --------------------------------------------------------------------------- + + +class _FakeGenerator: + """替换 NoteGenerator:状态写入真实记录,生成结果立即可控。""" + + probe: _Probe | None = None + status_log: list = [] + generate_calls: list = [] + server_429_from_index: int | None = None + park_calls: list = [] + + def __init__(self): + self.last_error_rate_limited = False + self.last_error_non_retryable = False + + def _update_status(self, task_id, status, message=None, **kwargs): + type(self).status_log.append( + { + "task_id": task_id, + "status": getattr(status, "value", str(status)), + "message": message or "", + "progress": kwargs.get("progress"), + } + ) + + def generate(self, **kwargs): + task_id = str(kwargs.get("task_id") or "") + video_url = str(kwargs.get("video_url") or "") + index = int(re.search(r"-s(\d+)$", task_id).group(1)) + type(self).generate_calls.append(index) + + if ( + type(self).server_429_from_index is not None + and index >= type(self).server_429_from_index + ): + self.last_error_rate_limited = True + return None + + probe = type(self).probe + if probe is not None: + probe.body() + return NoteResult( + markdown=f"# 视频 {index}\n\n正文 {index}", + transcript=TranscriptResult( + language="zh", + full_text=f"转写 {index}", + segments=[TranscriptSegment(start=0.0, end=1.0, text=f"转写 {index}")], + ), + audio_meta=AudioDownloadResult( + file_path=f"/tmp/{index}.mp3", + title=f"视频{index}", + duration=float(index), + cover_url=None, + platform="bilibili", + video_id=f"v{index}", + raw_info={}, + ), + ) + + +@pytest.fixture +def merge_env(gate_env, monkeypatch): + """合并任务所需的全部外部副作用都替换成内存记录。""" + _FakeGenerator.probe = None + _FakeGenerator.status_log = [] + _FakeGenerator.generate_calls = [] + _FakeGenerator.server_429_from_index = None + _FakeGenerator.park_calls = [] + + saved = {} + monkeypatch.setattr(note_module, "NoteGenerator", _FakeGenerator) + monkeypatch.setattr( + note_module, "save_note_to_file", lambda task_id, note: saved.setdefault(task_id, note) + ) + monkeypatch.setattr(note_module, "_cleanup_merged_subtask_files", lambda sub_task_id: None) + monkeypatch.setattr(note_module, "_enforce_cache_limit_in_background", lambda: None) + monkeypatch.setattr(note_module, "_park_rate_limited_task", lambda task_id, kwargs: _FakeGenerator.park_calls.append(task_id)) + monkeypatch.setattr( + note_module, + "_get_circuit_breaker", + lambda: type("_Breaker", (), {"record_success": lambda self: None})(), + ) + return saved + + +def _items(count): + return [ + {"url": f"https://example.com/v{i}", "title": f"视频{i}", "video_id": f"v{i}"} + for i in range(1, count + 1) + ] + + +def _run_merge(task_id, items, gate_limit=4): + return note_module._run_merged_note_task( + task_id=task_id, + items=items, + merge_title="合集", + merge_polish=False, + model_name=None, + provider_id=None, + platform="bilibili", + quality=DownloadQuality.fast, + link=False, + screenshot=False, + _format=None, + scene=None, + detail=None, + extras=None, + video_understanding=False, + video_interval=1, + grid_size="3x3", + sampling_mode="uniform", + max_frames=10, + frames_per_second=1, + frame_limit_mode="uniform", + ocr_assist=False, + park_kwargs={"task_id": task_id}, + ) + + +def test_merge_fans_out_up_to_gate_and_keeps_input_order(merge_env, gate_env): + gate_env.value = 4 + _FakeGenerator.probe = _Probe(delay=0.05) + + _run_merge("merge-order", _items(10)) + + assert _FakeGenerator.probe.max_seen == 4, ( + f"gate=4 时合并任务最大在途应为 4,实际 {_FakeGenerator.probe.max_seen}" + "(串行说明扇出没生效,>4 说明绕过了闸门)" + ) + + note = merge_env["merge-order"] + assert note.markdown.count("\n## ") >= 10, "10 条都应该出现在合并正文里" + order = [note.markdown.index(f"视频{i}\n") if f"视频{i}\n" in note.markdown else note.markdown.index(f"视频{i}") for i in range(1, 11)] + assert order == sorted(order), f"章节顺序必须等于输入顺序,实际 {order}" + + progress = [ + int(match.group(1)) + for entry in _FakeGenerator.status_log + if (match := re.search(r"合并生成 (\d+)/10", entry["message"])) + ] + assert progress == sorted(progress), f"进度必须单调递增,实际 {progress}" + assert progress[-1] == 10, f"最后一条进度应为 10/10,实际 {progress}" + assert any(entry["message"].startswith("已合并 10/10") for entry in _FakeGenerator.status_log) + + +def test_merge_429_parks_and_resumes_from_checkpoints(merge_env, gate_env, monkeypatch): + """第 5 条起 429:1-4 落断点;恢复后只重跑 5-10。""" + gate_env.value = 4 + _FakeGenerator.server_429_from_index = 5 + monkeypatch.setattr(note_module, "note_rate_limit_hit", lambda: True) + + _run_merge("merge-resume", _items(10)) + + checkpoint_dir = note_module._merge_checkpoint_dir("merge-resume") + written = sorted( + int(name.split(".")[0]) + for name in __import__("os").listdir(checkpoint_dir) + if name.endswith(".json") + ) + assert written == [1, 2, 3, 4], f"停放时应只保住已完成条目的断点,实际 {written}" + assert _FakeGenerator.park_calls == ["merge-resume"], "429 必须走停放而不是失败" + assert "merge-resume" not in merge_env, "停放不是终态,不该保存合并笔记" + + # 配额恢复后重新派发:1-4 命中断点,不再调生成。 + _FakeGenerator.generate_calls = [] + _FakeGenerator.server_429_from_index = None + monkeypatch.setattr(note_module, "note_rate_limit_hit", lambda: False) + + _run_merge("merge-resume", _items(10)) + + assert sorted(_FakeGenerator.generate_calls) == list(range(5, 11)), ( + f"恢复后只应生成 5-10,实际 {sorted(_FakeGenerator.generate_calls)}" + ) + note = merge_env["merge-resume"] + for i in range(1, 11): + assert f"## {i}. 视频{i}" in note.markdown, f"合并正文缺少第 {i} 条" + assert not __import__("os").path.exists(checkpoint_dir), "终态成功后断点目录必须删除" diff --git a/backend/tests/test_task_serial_executor.py b/backend/tests/test_task_serial_executor.py deleted file mode 100644 index 9ddbdd3a..00000000 --- a/backend/tests/test_task_serial_executor.py +++ /dev/null @@ -1,43 +0,0 @@ -import importlib.util -import pathlib -import threading -import time -import unittest - - -ROOT = pathlib.Path(__file__).resolve().parents[1] -MODULE_PATH = ROOT / "app" / "services" / "task_serial_executor.py" -spec = importlib.util.spec_from_file_location("task_serial_executor", MODULE_PATH) -if spec is None or spec.loader is None: - raise ImportError("task_serial_executor module spec not found") -task_serial_executor = importlib.util.module_from_spec(spec) -spec.loader.exec_module(task_serial_executor) -SerialTaskExecutor = task_serial_executor.SerialTaskExecutor - - -class TestTaskSerialExecutor(unittest.TestCase): - def test_executor_runs_tasks_concurrently(self): - executor = SerialTaskExecutor() - state_lock = threading.Lock() - state = {"active": 0, "peak_active": 0} - - def critical_work(): - with state_lock: - state["active"] += 1 - state["peak_active"] = max(state["peak_active"], state["active"]) - time.sleep(0.05) - with state_lock: - state["active"] -= 1 - - threads = [threading.Thread(target=lambda: executor.run(critical_work)) for _ in range(2)] - for t in threads: - t.start() - for t in threads: - t.join() - - # 执行器是线程池(默认 5),两个任务应并发而非串行 - self.assertEqual(state["peak_active"], 2) - - -if __name__ == "__main__": - unittest.main() diff --git a/specs/022-batch-concurrency-and-media-reliability/spec.md b/specs/022-batch-concurrency-and-media-reliability/spec.md index a378e910..38000130 100644 --- a/specs/022-batch-concurrency-and-media-reliability/spec.md +++ b/specs/022-batch-concurrency-and-media-reliability/spec.md @@ -281,7 +281,8 @@ DATA_DIR 绝对路径 → 原样使用 ## 11. 实现清单 - [ ] W0 spec 与索引(本文件) -- [ ] W1 调度器收敛 + 合并扇出 + checkpoint +- [x] W1 调度器收敛 + 合并扇出 + checkpoint + 判定降级 + (`backend/tests/022-batch-concurrency-and-media-reliability/w1/`:parallel_gate / merge_fanout / merge_checkpoint / judge_degrade) - [ ] W2 数据根修复与遗留回填 - [ ] W3 OCR 重试/全局闸门 + 判定降级 - [ ] W4 ASR 序列化告警 From e4356872595df42c55b742605fa7f0370bc50887 Mon Sep 17 00:00:00 2001 From: Eliver Date: Thu, 12 Nov 2026 14:35:57 +0800 Subject: [PATCH 3/3] =?UTF-8?q?feat(022):=20W2=20=E6=95=B0=E6=8D=AE?= =?UTF-8?q?=E6=A0=B9=E7=BB=9F=E4=B8=80=E3=80=81W3=20OCR=20=E9=87=8D?= =?UTF-8?q?=E8=AF=95/=E5=88=A4=E5=AE=9A=E9=99=8D=E7=BA=A7=E3=80=81W4=20ASR?= =?UTF-8?q?=20=E5=BA=8F=E5=88=97=E5=8C=96=E9=99=8D=E5=99=AA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - get_data_dir 只拼一次:DATA_DIR 绝对/相对 backend 解析,默认 /data;export/media/downloader/note 全改调它 - init_db 启动打印媒体数据根,旧 data/data 非空时告警且不自动迁移,回填计数写日志 - OCR 429/5xx/超时最多重试 3 次(Retry-After 优先、0.5/1/2s 抖动、单次≤30s),新增进程级 BoundedSemaphore(OCR_MAX_CONCURRENCY,默认 4) - 判定层显式空清单→empty_frames、全批次解析失败→parse_failed,均降级保留全部候选,不触发任务级重试 - ASR model_dump(warnings=False),segments[].id/seek 字符串数值规整为 int - tests/022 w2-w4;tests/008 两例按 spec 022 D6 新语义更新 验证:pytest tests → 994 passed, 15 subtests;相关子集 374 passed --- CLAUDE.md | 2 +- backend/app/db/init_db.py | 24 ++- backend/app/downloaders/base.py | 5 +- backend/app/gpt/universal_gpt.py | 26 ++- .../app/services/media_resource_service.py | 8 +- backend/app/services/ocr_assist.py | 66 ++++++- backend/app/services/ocr_client.py | 110 +++++++++++- backend/app/transcriber/openai_asr.py | 87 ++++++++- backend/app/utils/export.py | 7 +- backend/app/utils/path_helper.py | 38 +++- .../test_judge_batch_split.py | 28 +-- .../test_ocr_assist.py | 15 +- .../w2/test_data_root_and_media.py | 100 +++++++++++ .../w3/test_ocr_and_judgement.py | 166 ++++++++++++++++++ .../w4/test_asr_serialization.py | 92 ++++++++++ .../spec.md | 8 +- 16 files changed, 720 insertions(+), 62 deletions(-) create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w2/test_data_root_and_media.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w3/test_ocr_and_judgement.py create mode 100644 backend/tests/022-batch-concurrency-and-media-reliability/w4/test_asr_serialization.py diff --git a/CLAUDE.md b/CLAUDE.md index 176fd7b0..92650ff2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -59,7 +59,7 @@ Load unpacked at `chrome://extensions/` → select `BillNote_extension/extension - `app/routers/` — API routes: `note.py` (generation), `note_group.py` (note groups), `provider.py`, `model.py`, `config.py` - `app/services/` — Business logic: - `note.py` — `NoteGenerator` orchestrates the full pipeline (download → transcribe → LLM → notes) - - `task_serial_executor.py` — task queue + - 并发调度没有独立队列模块:`routers/note.py` 的 `_ParallelGate` 是唯一并发上限(设置里的并行数,热更新),任务体直接执行;批量入口用固定 8 worker 的分发池接住请求 - `cookie_manager.py` — per-platform cookie storage; injected into yt-dlp by downloaders (e.g. Bilibili) - `transcriber_config_manager.py` — persisted transcriber settings - `app/downloaders/` — Platform adapters with a shared `base.py` interface; only the Bilibili adapter is registered for generation (`SUPPORT_PLATFORM_MAP` in `app/services/constant.py`), while source intake (`app/services/source_intake.py`) stays platform-neutral so more adapters can be added without touching the intake contract. diff --git a/backend/app/db/init_db.py b/backend/app/db/init_db.py index ad46ab94..a04df796 100644 --- a/backend/app/db/init_db.py +++ b/backend/app/db/init_db.py @@ -1,4 +1,5 @@ import logging +import os from sqlalchemy import inspect, text @@ -32,6 +33,26 @@ def init_db(): return engine = get_engine() + # spec 022 D4:启动时把统一数据根打进日志,并提示旧版 data/data 残留。 + # 不自动迁移历史文件——迁移会改写用户数据,必须由人决定。 + from app.utils.path_helper import get_data_dir + + data_root = get_data_dir() + logger.info("媒体数据根目录(DATA_DIR):%s", data_root) + legacy_root = os.path.join(data_root, "data") + try: + legacy_entries = os.listdir(legacy_root) if os.path.isdir(legacy_root) else [] + except OSError: + legacy_entries = [] + if legacy_entries: + logger.warning( + "检测到旧版数据目录 %s 非空(%d 项);spec 022 不自动迁移," + "需要时请手动把文件移入 %s", + legacy_root, + len(legacy_entries), + data_root, + ) + Base.metadata.create_all(bind=engine) if engine.dialect.name == "sqlite": # Idempotent additive migration for older SQLite installations. @@ -207,4 +228,5 @@ def init_db(): connection.execute(text("DROP TABLE note_groups")) NoteGroup.__table__.create(bind=connection) engine_module._INIT_DB_DONE = True - backfill_legacy_resources() + inserted = backfill_legacy_resources() + logger.info("媒体资源历史回填完成:新增 %d 条", inserted) diff --git a/backend/app/downloaders/base.py b/backend/app/downloaders/base.py index 4df91f8f..5ff09777 100644 --- a/backend/app/downloaders/base.py +++ b/backend/app/downloaders/base.py @@ -4,7 +4,7 @@ from app.enmus.note_enums import DownloadQuality from app.models.notes_model import AudioDownloadResult from app.models.transcriber_model import TranscriptResult -from os import getenv +from app.utils.path_helper import get_data_dir QUALITY_MAP = {"fast": "32", "medium": "64", "slow": "128"} @@ -66,7 +66,8 @@ class Downloader(ABC): def __init__(self): # TODO 需要修改为可配置 self.quality = QUALITY_MAP["fast"] - self.cache_data = getenv("DATA_DIR") or "data" + # spec 022 D4:与下载落盘、媒体登记共用同一个数据根。 + self.cache_data = get_data_dir() @abstractmethod def download( diff --git a/backend/app/gpt/universal_gpt.py b/backend/app/gpt/universal_gpt.py index 60c1bda9..f3091d35 100644 --- a/backend/app/gpt/universal_gpt.py +++ b/backend/app/gpt/universal_gpt.py @@ -242,12 +242,14 @@ def judge_note_frames( - `plan_frame_timestamps` 决定「抽哪些帧」,服务于 hybrid 采样的配额; - 本方法决定「抽出来的帧里哪些真的进笔记」,是插图逻辑的唯一准入闸门。 - 解析不出结构化结果时抛 WorthJudgementError,**绝不**返回空清单——否则一次 - 网络抖动就会让整篇笔记静默地一张图都没有(spec 008 §5.6b)。模型明确判定 - 「都不值得」时返回空 approved,那是合法结果。 + 网络/鉴权失败向上抛。spec 022 D6 起,解析不出结构化结果**不再**抛到任务级: + 任务级重试会重做下载/转写/总结,代价极高而结果大概率一样。显式空清单记 + degraded=empty_frames,全部批次解析失败记 degraded=parse_failed,两者都 + 返回「全部候选」(受 note_max_images 截断),由上层写进任务 detail。 """ from app.services.ocr_assist import ( WorthJudgementError, + _degraded_outcome, build_judge_system_prompt, parse_worth_judgement, ) @@ -282,7 +284,11 @@ def _build_messages(batch: list) -> list: if not image_urls: response = self._chat_completion_create(_build_messages([])) text = response.choices[0].message.content or "" - return parse_worth_judgement(text, candidates, max_images=limit) + try: + return parse_worth_judgement(text, candidates, max_images=limit) + except WorthJudgementError as exc: + logger.warning("判定层单次调用解析失败,降级为保留全部候选:%s", exc) + return _degraded_outcome(candidates, "parse_failed", limit) # base 必须带一张图:_build_messages([]) 在无图时把 content 装配成字符串, # 往里追加图片会因为没有 list 可 append 而静默丢图(体积估算恒为空)。 @@ -305,11 +311,15 @@ def _build_messages(batch: list) -> list: ) if not items: - # 每一块都解析不出条目时按「判定失败」处理:静默返回空清单会让整篇 - # 笔记一张图都没有,而且完全看不出出过错(spec 008 §5.6b)。 - raise WorthJudgementError( - "判定结果里找不到帧列表(既不是数组,也没有 frames 字段)" + # spec 022 D6:全部批次都解析不出条目时降级,而不是让整任务失败。 + # 保留全部候选至少能保证笔记有插图,degraded 让这件事在任务 detail + # 里可见——比「静默无图」和「白白重跑三次」都好。 + logger.warning( + "判定层全部批次解析失败(图片=%d 批次=%d),降级为保留全部候选", + len(image_urls), + len(batches), ) + return _degraded_outcome(candidates, "parse_failed", limit) # 在**完整候选清单**上只解析一次:这样 max_images 上限、去重、排序和 # 统计字段都是全局口径,而不是「每块各算一遍再拼起来」。 diff --git a/backend/app/services/media_resource_service.py b/backend/app/services/media_resource_service.py index 0d6b2aa3..e9be9647 100644 --- a/backend/app/services/media_resource_service.py +++ b/backend/app/services/media_resource_service.py @@ -511,10 +511,10 @@ def _note_export_dir() -> Path: 覆盖,默认 `/data/note_output`)。同样刻意不 import:那个模块会拉起 markdown_pdf 等导出依赖。 """ - base_dir = Path(__file__).resolve().parents[2] - data_dir = os.getenv("DATA_DIR", "data") - data_path = Path(data_dir) if os.path.isabs(data_dir) else base_dir / data_dir - return (data_path / "note_output").resolve() + from app.utils.path_helper import get_data_dir + + # spec 022 D4:导出目录与下载/媒体登记同根,避免再各自解析 DATA_DIR。 + return (Path(get_data_dir()) / "note_output").resolve() def _task_status(task_id: str | None, cache: dict[str, str | None]) -> str | None: diff --git a/backend/app/services/ocr_assist.py b/backend/app/services/ocr_assist.py index fca88bd8..e8f28ef0 100644 --- a/backend/app/services/ocr_assist.py +++ b/backend/app/services/ocr_assist.py @@ -146,11 +146,14 @@ def max_judge_candidates() -> int: class WorthJudgementError(RuntimeError): - """判定调用或解析失败。 - - 与「模型明确判定没有值得编入的画面」严格区分(spec 008 §5.6b): - 前者是错误——任务应当失败,而不是静默产出一篇没有插图的笔记; - 后者是合法结果,只是本次不插图。 + """判定调用失败(网络/鉴权)或输出完全不可识别。 + + 与「模型明确判定没有值得编入的画面」严格区分:后者是合法结果,只是本次 + 不插图。spec 008 §5.6b 原本要求前者也让任务失败,以免静默产出无插图笔记; + spec 022 D6 起不再如此——任务级重试会重做下载/转写/总结,代价极高而结果 + 大概率一样,所以解析失败与显式空清单都降级为「保留全部候选」,由调用方 + (universal_gpt / services.note)记录 degraded 原因,任务继续跑完。 + 这里保留异常类型,是因为「完全没有可解析内容」仍要能被上层识别。 """ @@ -182,15 +185,19 @@ class JudgementOutcome: rejected_out_of_candidates: int = 0 rejected_duplicate: int = 0 dropped_by_cap: int = 0 + # spec 022 D6:降级原因(empty_frames 显式空清单 / parse_failed 全部批次解析失败)。 + # 非空表示 approved 是「全部候选」的兜底结果,不是模型确认过的清单。 + degraded: Optional[str] = None def describe(self) -> str: + suffix = f",降级({self.degraded})" if self.degraded else "" return ( f"候选判定 {self.total} 条:true={self.worth_true} false={self.worth_false}," f"丢弃(字段非法)={self.rejected_invalid} " f"丢弃(不在候选清单)={self.rejected_out_of_candidates} " f"丢弃(重复)={self.rejected_duplicate} " f"超出上限={self.dropped_by_cap}," - f"最终通过={len(self.approved)}" + f"最终通过={len(self.approved)}{suffix}" ) @@ -349,6 +356,44 @@ def extract_judgement_items(text: str) -> List[Any]: return [] +def _degraded_outcome( + candidates: Iterable[Any], + reason: str, + max_images: int | None = None, +) -> JudgementOutcome: + """判定降级:把全部候选帧当作「值得编入」(spec 022 D6)。 + + 去重、排序、上限截断都走与正常路径相同的口径,只是 worth 全部视为 true, + 并在 reason 里写明这是降级兜底,免得用户以为模型真的确认过这些画面。 + """ + outcome = JudgementOutcome(degraded=reason) + times: List[float] = [] + seen: set = set() + for candidate in candidates or []: + seconds = _as_seconds(candidate) + if seconds is None: + continue + key = _safe_int(round(seconds)) + if key in seen: + continue + seen.add(key) + times.append(float(seconds)) + times.sort() + outcome.total = len(times) + limit = max( + 0, _safe_int(max_images if max_images is not None else max_note_images()) + ) + decisions = [ + WorthDecision(time=t, reason="画面判定降级:保留全部候选") for t in times + ] + if len(decisions) > limit: + outcome.dropped_by_cap = len(decisions) - limit + decisions = decisions[:limit] + outcome.approved = decisions + outcome.worth_true = len(decisions) + return outcome + + def parse_worth_judgement( text: str, candidates: Iterable[Any], @@ -359,7 +404,11 @@ def parse_worth_judgement( :param text: 模型返回的原始文本 :param candidates: 候选时间点(秒),即 ``VideoReader.candidate_timestamps`` :param max_images: 通过判定的帧数上限 - :raises WorthJudgementError: 完全解析不出列表时(调用失败 / 输出不可识别) + :raises WorthJudgementError: 文本完全不是 JSON / 找不到帧列表字段时 + + spec 022 D6:显式空清单(frames 为空数组)不再视为「不插图」,而是降级为 + 「保留全部候选」——空清单与「模型什么都没说清楚」在输出上无法区分,而 + 一篇文章因为一次空回复丢掉全部插图,比多留几张图更糟。 """ data = load_llm_json(text) if data is None: @@ -379,6 +428,9 @@ def parse_worth_judgement( items = [data] if items is None: raise WorthJudgementError("判定结果里找不到帧列表(既不是数组,也没有 frames 字段)") + if not items: + # spec 022 D6:显式空清单降级为保留全部候选。 + return _degraded_outcome(candidates, "empty_frames", max_images) index = build_candidate_index(candidates) outcome = JudgementOutcome() diff --git a/backend/app/services/ocr_client.py b/backend/app/services/ocr_client.py index 0b8a9da3..16d12e7f 100644 --- a/backend/app/services/ocr_client.py +++ b/backend/app/services/ocr_client.py @@ -31,7 +31,10 @@ import base64 import json +import os +import random import threading +import time from dataclasses import dataclass, field from pathlib import Path from typing import Any, Dict, List, Optional, Tuple, Union @@ -123,6 +126,72 @@ def _is_non_retryable(status_code: Optional[int], code: Any) -> bool: return True return False +# ---- 重试与全局并发(spec 022 D5) ---------------------------------------- +# +# 真机 429 来自「多任务各自开 OCR 线程」的叠加,而不是单任务并发设得高: +# 单任务上限由 ocr_stage_concurrency 控制,但多个任务同时跑时总数仍会翻倍。 +# 所以这里再加一道**进程级**信号量,重试则负责吸收短暂的配额抖动。 +OCR_MAX_CONCURRENCY_DEFAULT = 4 +OCR_MAX_ATTEMPTS = 3 +OCR_RETRY_BASE_SECONDS = 0.5 +OCR_RETRY_MAX_SLEEP_SECONDS = 30.0 + +_OCR_SEMAPHORE: Optional[threading.BoundedSemaphore] = None +_OCR_SEMAPHORE_SIZE = 0 +_OCR_SEMAPHORE_LOCK = threading.Lock() + + +def ocr_max_concurrency() -> int: + """进程级 OCR 并发上限(env OCR_MAX_CONCURRENCY,默认 4)。 + + 非法值回退默认而不是抛错:这是护栏,不是业务参数,配置写错不该让整个 + OCR 功能不可用。 + """ + try: + return max(1, int(os.getenv("OCR_MAX_CONCURRENCY", str(OCR_MAX_CONCURRENCY_DEFAULT)))) + except (TypeError, ValueError): + return OCR_MAX_CONCURRENCY_DEFAULT + + +def _ocr_semaphore() -> threading.BoundedSemaphore: + """懒建全局信号量;运行期改了上限就重建(旧持有者仍能正常 release)。""" + global _OCR_SEMAPHORE, _OCR_SEMAPHORE_SIZE + size = ocr_max_concurrency() + with _OCR_SEMAPHORE_LOCK: + if _OCR_SEMAPHORE is None or _OCR_SEMAPHORE_SIZE != size: + _OCR_SEMAPHORE = threading.BoundedSemaphore(size) + _OCR_SEMAPHORE_SIZE = size + return _OCR_SEMAPHORE + + +def _is_retryable_ocr_error(exc: "OcrError") -> bool: + """这个 OCR 错误重试一次有没有意义? + + 只重试 429 / 5xx / 超时与连接失败。鉴权、参数、内容被拒、体积超限都在 + OcrError.non_retryable 里被排除——它们只有改了配置才会好。 + """ + if exc.non_retryable: + return False + status = exc.status_code + if status is None: + # 超时/连接失败被包成「OCR 请求失败:...」;其余无状态码错误不重试。 + return "请求失败" in exc.message + return int(status) == 429 or int(status) >= 500 + + +def _retry_delay(exc: "OcrError", attempt: int) -> float: + """本次重试前等多久:上游给了 Retry-After 就听它的,否则指数退避 + 抖动。""" + retry_after = getattr(exc, "retry_after", None) + if retry_after is not None: + try: + delay = float(retry_after) + except (TypeError, ValueError): + delay = None + if delay is not None: + return max(0.0, min(delay, OCR_RETRY_MAX_SLEEP_SECONDS)) + delay = OCR_RETRY_BASE_SECONDS * (2 ** (attempt - 1)) + return min(delay, OCR_RETRY_MAX_SLEEP_SECONDS) + random.uniform(0, 0.25) + # 共享 session 及其懒加载锁 _SHARED_SESSION: Optional[requests.Session] = None _SHARED_SESSION_LOCK = threading.Lock() @@ -162,12 +231,15 @@ def __init__( detail: str = "", status_code: Optional[int] = None, code: Optional[str] = None, + retry_after: Optional[float] = None, ): super().__init__(message) self.message = message self.detail = detail self.status_code = status_code self.code = code + # 上游 Retry-After(秒);重试等待优先用它,缺失时才走指数退避。 + self.retry_after = retry_after # 「重试也不会变好」的错误。调用方(note.generate 的顶层 except)会读取 # 这个属性并转成 `last_error_non_retryable`,让重试循环立刻停下。 # 典型就是 401/403:key 不对,只有用户改配置才可能成功, @@ -567,9 +639,12 @@ def _extract_layout_blocks(body: Dict[str, Any]) -> List[OcrBlock]: # ---------------- 对外入口 ---------------- def recognize_image(self, image_path: Union[str, Path]) -> OcrResult: - """对一张本地图片(或 PDF)做一次 OCR。 + """对一张本地图片(或 PDF)做一次 OCR,带有限重试。 - 失败一律抛 :class:`OcrError`,消息面向用户、`detail` 保留上游原文。 + spec 022 D5:失败一律抛 :class:`OcrError`(消息面向用户、detail 保留原文), + 但 429 / 5xx / 超时最多重试到 OCR_MAX_ATTEMPTS 次;鉴权、参数、内容被拒 + 这类「改了配置才会好」的错误立刻抛。每次尝试占一个进程级 OCR 名额, + 所以多个任务叠加时的真实并发也不会超过 OCR_MAX_CONCURRENCY。 """ path = Path(image_path) if not path.is_file(): @@ -579,6 +654,27 @@ def recognize_image(self, image_path: Union[str, Path]) -> OcrResult: self._check_page_limit(path) + attempt = 0 + while True: + attempt += 1 + try: + with _ocr_semaphore(): + return self._recognize_once(path) + except OcrError as exc: + if attempt >= OCR_MAX_ATTEMPTS or not _is_retryable_ocr_error(exc): + raise + delay = _retry_delay(exc, attempt) + logger.warning( + "OCR 调用失败(第 %d/%d 次),%.1fs 后重试:%s", + attempt, + OCR_MAX_ATTEMPTS, + delay, + exc, + ) + time.sleep(delay) + + def _recognize_once(self, path: Path) -> OcrResult: + """单次 OCR 调用;重试与全局名额由 recognize_image 负责。""" payload = self._build_layout_parsing_request(path) parse = self._parse_layout_parsing @@ -658,11 +754,21 @@ def _http_error( parts.append(hint) if message: parts.append(message) + retry_after = None + if response is not None: + headers = getattr(response, "headers", None) + raw_retry_after = headers.get("Retry-After") if headers is not None else None + if raw_retry_after: + try: + retry_after = float(raw_retry_after) + except (TypeError, ValueError): + retry_after = None error = OcrError( ":".join([parts[0], ",".join(parts[1:])]) if len(parts) > 1 else parts[0], detail=body_text, status_code=status_code, code=str(code) if code else None, + retry_after=retry_after, ) if error.non_retryable: logger.error( diff --git a/backend/app/transcriber/openai_asr.py b/backend/app/transcriber/openai_asr.py index 13d1b977..27d66124 100644 --- a/backend/app/transcriber/openai_asr.py +++ b/backend/app/transcriber/openai_asr.py @@ -91,6 +91,62 @@ def _get(obj: Any, key: str, default: Any = None) -> Any: return getattr(obj, key, default) +def _coerce_int(value: Any) -> Optional[int]: + """把 "0" / 0.0 这类数值写法收敛成 int;认不出来返回 None。 + + spec 022 D7:部分上游把 ``id``/``seek`` 写成字符串,Pydantic 在序列化时会 + 逐条报 "Expected int ... input_value='0'"。解析层先把它们规整掉,比在 + model_dump 里关掉告警更彻底——下游读到的也确实是数字。 + """ + if isinstance(value, bool): + return None + if isinstance(value, int): + return value + if isinstance(value, float): + return int(value) if value.is_integer() else None + if isinstance(value, str): + text = value.strip() + if not text: + return None + try: + return int(text) + except ValueError: + try: + number = float(text) + except ValueError: + return None + return int(number) if number.is_integer() else None + return None + + +def _normalize_raw_payload(raw: Any) -> Any: + """把 raw 响应里 segments[].id / seek 的字符串数值规整成 int。 + + 只改这两个已知字段,其余原样保留:raw 是排查材料,不是业务数据, + 不做猜测性改写。 + """ + if not isinstance(raw, dict): + return raw + segments = raw.get("segments") + if not isinstance(segments, list): + return raw + normalized = [] + for seg in segments: + if not isinstance(seg, dict): + normalized.append(seg) + continue + item = dict(seg) + for key in ("id", "seek"): + if key in item: + coerced = _coerce_int(item[key]) + if coerced is not None: + item[key] = coerced + normalized.append(item) + raw = dict(raw) + raw["segments"] = normalized + return raw + + def _is_format_error(exc: Exception) -> bool: """只有明确是 response_format 不被支持时才降级,避免把 401/超时也当成格式问题。""" msg = str(exc).lower() @@ -539,7 +595,7 @@ def _to_result(self, transcription: Any, source_path: str) -> TranscriptResult: if self.api_style == "chat": full_text = self._chat_text(transcription) language = None - raw = self._to_dict(transcription) + raw = _normalize_raw_payload(self._to_dict(transcription)) # response_format="text" 时 SDK 直接返回字符串 elif isinstance(transcription, str): full_text = transcription.strip() @@ -548,7 +604,7 @@ def _to_result(self, transcription: Any, source_path: str) -> TranscriptResult: else: full_text = str(_get(transcription, "text") or "").strip() language = _get(transcription, "language") - raw = self._to_dict(transcription) + raw = _normalize_raw_payload(self._to_dict(transcription)) segments = [] for seg in (_get(transcription, "segments") or []): @@ -596,13 +652,30 @@ def _chat_text(completion: Any) -> str: @staticmethod def _to_dict(obj: Any) -> Optional[dict]: + """把 SDK 对象转成 dict;已是 dict 时直接用。 + + spec 022 D7:model_dump 传 warnings=False。上游偶尔把 segments[].id / seek + 写成字符串,Pydantic 会为每条打一条 UserWarning,几十秒的转写能把日志 + 刷成噪音,而告警本身不影响结果。字段规整在 _normalize_raw_payload 里做。 + """ if isinstance(obj, dict): return obj - for method in ("model_dump", "to_dict"): - fn = getattr(obj, method, None) - if callable(fn): + model_dump = getattr(obj, "model_dump", None) + if callable(model_dump): + try: + return model_dump(warnings=False) + except TypeError: + # 老版 Pydantic 不认识 warnings 参数 try: - return fn() + return model_dump() except Exception: - continue + pass + except Exception: + pass + to_dict = getattr(obj, "to_dict", None) + if callable(to_dict): + try: + return to_dict() + except Exception: + return None return None diff --git a/backend/app/utils/export.py b/backend/app/utils/export.py index 4008be62..05ee5f9f 100644 --- a/backend/app/utils/export.py +++ b/backend/app/utils/export.py @@ -4,14 +4,15 @@ from markdown_pdf import MarkdownPdf, Section from dotenv import load_dotenv +from app.utils.path_helper import get_data_dir + load_dotenv() # 项目根路径(无论你在哪里运行) BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -# 从 .env 获取 DATA_DIR,相对于 BASE_DIR 解析 -DATA_DIR_NAME = os.getenv("DATA_DIR", "data") -DATA_DIR = os.path.join(BASE_DIR, DATA_DIR_NAME) +# DATA_DIR 的解析规则统一在 get_data_dir()(spec 022 D4):相对路径按 backend 解析。 +DATA_DIR = get_data_dir() SAVE_PATH = os.path.join(DATA_DIR, "note_output") IMAGE_BASE_URL = os.getenv("IMAGE_BASE_URL") STATIC_BASE = os.path.join(BASE_DIR, IMAGE_BASE_URL) diff --git a/backend/app/utils/path_helper.py b/backend/app/utils/path_helper.py index d714ac98..46a9b036 100644 --- a/backend/app/utils/path_helper.py +++ b/backend/app/utils/path_helper.py @@ -5,17 +5,39 @@ PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../")) -def get_data_dir(): - if getattr(sys, 'frozen', False): +def _backend_root() -> str: + """开发态的 backend 目录;冻结态就是 exe 所在目录。""" + if getattr(sys, "frozen", False): + return os.path.dirname(sys.executable) + return os.path.abspath(os.path.join(os.path.dirname(__file__), "../../")) - base_dir = os.path.dirname(sys.executable) - else: - base_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../data")) +def get_data_dir() -> str: + """统一的媒体数据根目录(spec 022 D4)。 + + 规则: + - DATA_DIR 是绝对路径:直接用; + - DATA_DIR 是相对路径:相对 backend 解析(与「cd backend && python main.py」 + 的工作目录语义一致,但不再依赖进程 cwd); + - 未设置:/data(冻结态 /data)。 - data_path = os.path.join(base_dir, "data") - os.makedirs(data_path, exist_ok=True) - return data_path + 历史 bug(R3):这里曾把 base 取成 /data 后又拼一次 data, + 返回 /data/data,而下载器、导出、媒体登记分别按 /data + 解析——媒体资源登记因「路径越界」全部失败。现在只在这一处定义根目录。 + """ + base = _backend_root() + configured = (os.getenv("DATA_DIR") or "").strip() + if configured: + path = ( + configured + if os.path.isabs(configured) + else os.path.join(base, configured) + ) + else: + path = os.path.join(base, "data") + path = os.path.abspath(path) + os.makedirs(path, exist_ok=True) + return path def get_model_dir(subdir: str = "whisper") -> str: diff --git a/backend/tests/008-ocr-assist-and-legacy-cleanup/test_judge_batch_split.py b/backend/tests/008-ocr-assist-and-legacy-cleanup/test_judge_batch_split.py index d8a32b38..575a2388 100644 --- a/backend/tests/008-ocr-assist-and-legacy-cleanup/test_judge_batch_split.py +++ b/backend/tests/008-ocr-assist-and-legacy-cleanup/test_judge_batch_split.py @@ -101,18 +101,24 @@ def test_judge_splits_batch_in_half_on_large_request_failure(tmp_path, monkeypat assert any(not item["ok"] for item in record), "夹具必须真的制造出过一次超限失败" -def test_judge_skips_single_image_that_still_fails(tmp_path, monkeypatch): - """拆到单张仍失败时只跳过该帧,不让整篇笔记失败。""" +def test_judge_degrades_when_every_batch_fails(tmp_path, monkeypatch): + """拆到单张仍失败时降级为保留全部候选,不让整篇笔记失败。 + + spec 022 D6 改写了 spec 008 §5.6b 的旧契约:任务级重试会重做下载/转写/总结, + 代价极高而结果大概率一样,所以「判定层拿不到任何条目」降级为保留全部候选, + 由调用方把 degraded 写进任务 detail。 + """ monkeypatch.setenv("NOTE_OUTPUT_DIR", str(tmp_path)) record: list = [] - # 上限设成 0:任何请求都失败。拆到单张后应当放弃并跳过,而不是抛出去。 + # 上限设成 0:任何请求都失败,所有批次都解析不出条目。 gpt = UniversalGPT(_Client(record, fail_over_bytes=0), "glm-4.6") - with pytest.raises(Exception): - # 所有条目都拿不到时,调用方仍按判定失败处理(spec 008 §5.6b)。 - gpt.judge_note_frames( - segments=[], - candidates=[12.0], - duration=600.0, - image_urls=_urls(2), - ) + outcome = gpt.judge_note_frames( + segments=[], + candidates=[12.0], + duration=600.0, + image_urls=_urls(2), + ) + + assert outcome.degraded == "parse_failed" + assert outcome.approved, "降级必须保留全部候选,而不是悄悄变成无插图笔记" diff --git a/backend/tests/008-ocr-assist-and-legacy-cleanup/test_ocr_assist.py b/backend/tests/008-ocr-assist-and-legacy-cleanup/test_ocr_assist.py index fe06ddc3..d3237c1d 100644 --- a/backend/tests/008-ocr-assist-and-legacy-cleanup/test_ocr_assist.py +++ b/backend/tests/008-ocr-assist-and-legacy-cleanup/test_ocr_assist.py @@ -138,12 +138,19 @@ def test_approved_is_sorted_by_time(): # -------------------------------------------------------------------------- -# 「模型说没有」与「解析失败」必须分开(spec §5.6b) +# 「模型说没有」与「解析失败」都降级(spec 022 D6 取代 spec 008 §5.6b) # -------------------------------------------------------------------------- -def test_empty_frames_is_a_legitimate_empty_result(): +def test_empty_frames_degrades_to_all_candidates(): + """spec 022 D6:显式空清单不再表示「不插图」。 + + 空清单与「模型什么都没说清楚」在输出上无法区分,一篇文章因为一次空回复 + 丢掉全部插图,比多留几张图更糟。降级原因记 empty_frames,上层写进 detail。 + """ outcome = judge([]) - assert outcome.approved == [] - assert outcome.total == 0 + assert outcome.degraded == "empty_frames" + assert [d.time for d in outcome.approved] == CANDIDATES + assert outcome.total == len(CANDIDATES) + assert outcome.worth_false == 0 def test_unparseable_output_raises_instead_of_silently_returning_empty(): diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w2/test_data_root_and_media.py b/backend/tests/022-batch-concurrency-and-media-reliability/w2/test_data_root_and_media.py new file mode 100644 index 00000000..6c845ea3 --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w2/test_data_root_and_media.py @@ -0,0 +1,100 @@ +"""spec 022 W2:数据根只拼一次 + 媒体资源能登记成相对路径。 + +R3 的成因:get_data_dir() 先取 /data 再 join("data"),返回 +/data/data,而下载落盘在 /data(DATA_DIR 默认 "data")。 +media_resource_service._roots() 用前者做根,_managed_relative() 判越界, +于是每条任务的媒体登记都失败,日志刷「无法登记任务媒体资源」。 + +这里的用例对应验收第 4 条的前半段:临时 DATA_DIR 下登记成功、path 是相对路径。 +""" + +import os +from pathlib import Path + +import pytest + +from app.utils import path_helper +from app.services import media_resource_service + + +def _backend_root() -> Path: + # app/utils/path_helper.py → parents[0]=utils, [1]=app, [2]=backend + return Path(path_helper.__file__).resolve().parents[2] + + +def test_default_root_is_backend_data_not_double_nested(monkeypatch): + monkeypatch.delenv("DATA_DIR", raising=False) + root = Path(path_helper.get_data_dir()).resolve() + assert root == (_backend_root() / "data").resolve() + assert not str(root).replace("\\", "/").endswith("/data/data"), ( + "R3 回归:数据根又被拼成了 data/data" + ) + + +def test_absolute_data_dir_is_used_as_is(monkeypatch, tmp_path): + monkeypatch.setenv("DATA_DIR", str(tmp_path)) + assert Path(path_helper.get_data_dir()).resolve() == tmp_path.resolve() + + +def test_relative_data_dir_resolves_against_backend(monkeypatch): + monkeypatch.setenv("DATA_DIR", "custom-data-022") + expected = _backend_root() / "custom-data-022" + try: + assert Path(path_helper.get_data_dir()).resolve() == expected.resolve() + finally: + # 只清这个用例自己造的目录,别碰既有 data/ + if expected.is_dir() and not any(expected.iterdir()): + expected.rmdir() + + +class _FakeQuery: + def filter_by(self, **kwargs): + return self + + def first(self): + return None + + +class _FakeDb: + def __init__(self): + self.added = [] + + def query(self, *args, **kwargs): + return _FakeQuery() + + def add(self, row): + self.added.append(row) + + def commit(self): + pass + + +def test_register_resource_uses_the_same_root_and_stores_relative_path(monkeypatch, tmp_path): + monkeypatch.setenv("DATA_DIR", str(tmp_path)) + media_dir = tmp_path / "video" / "sub" + media_dir.mkdir(parents=True) + media_file = media_dir / "x.mp4" + media_file.write_bytes(b"fake") + + fake_db = _FakeDb() + + class _Ctx: + def __enter__(self): + return fake_db + + def __exit__(self, *args): + return False + + monkeypatch.setattr(media_resource_service, "SessionLocal", lambda: _Ctx()) + monkeypatch.setattr( + media_resource_service, + "get_retention_settings", + lambda: {"screenshot_retention_days": 7, "media_retention_days": 3}, + ) + + media_resource_service.register_resource("video", media_file, task_id="t-022") + + assert len(fake_db.added) == 1, "R3 回归:登记没有走到 add(多半又是越界异常)" + row = fake_db.added[0] + assert row.path == "video:video/sub/x.mp4", f"必须存相对路径,实际 {row.path}" + assert row.size_bytes == 4 diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w3/test_ocr_and_judgement.py b/backend/tests/022-batch-concurrency-and-media-reliability/w3/test_ocr_and_judgement.py new file mode 100644 index 00000000..0b41e44d --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w3/test_ocr_and_judgement.py @@ -0,0 +1,166 @@ +"""spec 022 W3:OCR 429 重试 + 进程级全局名额 + 帧判定降级。 + +对应验收第 5/6 条: + +* 429 → 200 能重试成功;持续 429 三次后抛出,任务不因单张图失败(由上层计数)。 +* 12 路并发受 OCR_MAX_CONCURRENCY(默认 4)约束。 +* 判定返回显式空清单记 degraded=empty_frames;全批次解析失败记 parse_failed, + 都保留全部候选、不抛到任务级。 +""" + +import threading +import time +from types import SimpleNamespace +from pathlib import Path + +import pytest + +from app.gpt.universal_gpt import UniversalGPT +from app.services import ocr_client +from app.services.ocr_assist import ( + WorthJudgementError, + parse_worth_judgement, +) +from app.services.ocr_client import OcrClient, OcrError + + +def _client() -> OcrClient: + client = object.__new__(OcrClient) + client._config = {"model_name": "test-model"} + return client + + +def test_429_then_success_retries_once(monkeypatch, tmp_path): + image = tmp_path / "frame.jpg" + image.write_bytes(b"x") + calls = [] + + def fake_once(path): + calls.append(path) + if len(calls) == 1: + raise OcrError("rate limited", status_code=429) + return SimpleNamespace(markdown="ok") + + monkeypatch.setattr(ocr_client.time, "sleep", lambda *_: None) + client = _client() + monkeypatch.setattr(client, "_recognize_once", fake_once) + + result = client.recognize_image(image) + + assert result.markdown == "ok" + assert len(calls) == 2, f"429 后应重试一次,实际调用 {len(calls)} 次" + + +def test_non_retryable_error_is_not_retried(monkeypatch, tmp_path): + image = tmp_path / "frame.jpg" + image.write_bytes(b"x") + calls = [] + + def fake_once(path): + calls.append(path) + raise OcrError("bad key", status_code=401) + + monkeypatch.setattr(ocr_client.time, "sleep", lambda *_: None) + client = _client() + monkeypatch.setattr(client, "_recognize_once", fake_once) + + with pytest.raises(OcrError): + client.recognize_image(image) + assert len(calls) == 1, "401 属不可重试,必须只调用一次" + + +def test_persistent_429_gives_up_after_three_attempts(monkeypatch, tmp_path): + image = tmp_path / "frame.jpg" + image.write_bytes(b"x") + calls = [] + sleeps = [] + + def fake_once(path): + calls.append(path) + raise OcrError("rate limited", status_code=429) + + monkeypatch.setattr(ocr_client.time, "sleep", lambda seconds: sleeps.append(seconds)) + client = _client() + monkeypatch.setattr(client, "_recognize_once", fake_once) + + with pytest.raises(OcrError): + client.recognize_image(image) + + assert len(calls) == ocr_client.OCR_MAX_ATTEMPTS == 3 + assert len(sleeps) == 2, f"三次尝试之间应等两次,实际 {len(sleeps)}" + assert all(0 < s <= ocr_client.OCR_RETRY_MAX_SLEEP_SECONDS for s in sleeps) + + +def test_ocr_max_concurrency_env_override(monkeypatch): + monkeypatch.delenv("OCR_MAX_CONCURRENCY", raising=False) + assert ocr_client.ocr_max_concurrency() == ocr_client.OCR_MAX_CONCURRENCY_DEFAULT == 4 + monkeypatch.setenv("OCR_MAX_CONCURRENCY", "2") + assert ocr_client.ocr_max_concurrency() == 2 + monkeypatch.setenv("OCR_MAX_CONCURRENCY", "nonsense") + assert ocr_client.ocr_max_concurrency() == 4 + + +def test_global_semaphore_caps_twelve_concurrent_calls(monkeypatch): + """12 路并发同时抢名额,任一时刻最多 4 个在跑。""" + monkeypatch.setenv("OCR_MAX_CONCURRENCY", "4") + monkeypatch.setattr(ocr_client, "_OCR_SEMAPHORE", None) + monkeypatch.setattr(ocr_client, "_OCR_SEMAPHORE_SIZE", 0) + + gate = ocr_client._ocr_semaphore() + state = {"current": 0, "max_seen": 0} + lock = threading.Lock() + + def worker(): + with gate: + with lock: + state["current"] += 1 + state["max_seen"] = max(state["max_seen"], state["current"]) + time.sleep(0.02) + with lock: + state["current"] -= 1 + + threads = [threading.Thread(target=worker) for _ in range(12)] + for t in threads: + t.start() + for t in threads: + t.join() + + assert state["max_seen"] == 4, f"全局名额应钳到 4,实际峰值 {state['max_seen']}" + + +def test_explicit_empty_frames_degrades_to_all_candidates(): + outcome = parse_worth_judgement('{"frames": []}', [1, 2, 3], max_images=2) + assert outcome.degraded == "empty_frames" + assert len(outcome.approved) == 2, "降级后应保留候选并受上限截断" + assert outcome.dropped_by_cap == 1 + assert [d.time for d in outcome.approved] == [1.0, 2.0] + + +class _JudgeStub: + """只装配 judge_note_frames 需要的三个依赖。""" + + ensure_segments_type = UniversalGPT.ensure_segments_type + _build_plan_lines = UniversalGPT._build_plan_lines + + def __init__(self, content): + self._content = content + + def _chat_completion_create(self, messages): + return SimpleNamespace( + choices=[SimpleNamespace(message=SimpleNamespace(content=self._content))] + ) + + +def test_garbage_judgement_degrades_without_raising(): + stub = _JudgeStub("这不是 JSON") + outcome = UniversalGPT.judge_note_frames( + stub, segments=[], candidates=[1, 2, 3], max_items=3 + ) + assert outcome.degraded == "parse_failed" + assert len(outcome.approved) == 3, "解析失败必须保留全部候选,而不是抛到任务级" + + +def test_parse_failure_still_raises_at_parse_layer(): + """解析层仍保留异常,方便其它调用方识别「完全不可解析」。""" + with pytest.raises(WorthJudgementError): + parse_worth_judgement("not json", [1, 2, 3]) diff --git a/backend/tests/022-batch-concurrency-and-media-reliability/w4/test_asr_serialization.py b/backend/tests/022-batch-concurrency-and-media-reliability/w4/test_asr_serialization.py new file mode 100644 index 00000000..bb13f00c --- /dev/null +++ b/backend/tests/022-batch-concurrency-and-media-reliability/w4/test_asr_serialization.py @@ -0,0 +1,92 @@ +"""spec 022 W4:ASR 序列化不再打 Pydantic 告警 + id/seek 规整为 int。 + +对应验收第 7 条:上游把 segments[].id / seek 返回成字符串时,不再出现 +"Expected int ... input_value='0'" 的 UserWarning,transcript 时间轴照常可用。 +""" + +from types import SimpleNamespace + +from app.transcriber.openai_asr import ( + OpenAIASRTranscriber, + _normalize_raw_payload, +) + +_to_dict = OpenAIASRTranscriber._to_dict + + +class _Seg: + def __init__(self, text, start, end): + self.text = text + self.start = start + self.end = end + + +class _Transcription: + def __init__(self): + self.text = "hello world" + self.language = "en" + self.segments = [_Seg("hello", 0.0, 1.0), _Seg("world", 1.0, 2.0)] + self.dump_kwargs = None + + def model_dump(self, **kwargs): + self.dump_kwargs = kwargs + return { + "text": "hello world", + "language": "en", + "segments": [ + {"id": "0", "seek": "12", "text": "hello", "start": 0.0, "end": 1.0}, + {"id": 2, "seek": "1500", "text": "world", "start": 1.0, "end": 2.0}, + ], + } + + +def test_model_dump_is_called_with_warnings_disabled(): + obj = _Transcription() + raw = _to_dict(obj) + assert obj.dump_kwargs == {"warnings": False}, ( + f"必须传 warnings=False,实际 {obj.dump_kwargs}" + ) + assert raw["segments"][0]["id"] == "0", "序列化层不该偷偷改字段,规整在 normalize 里做" + + +def test_model_dump_without_warnings_argument_still_works(): + class _OldPydantic: + def model_dump(self): + return {"text": "ok"} + + assert _to_dict(_OldPydantic()) == {"text": "ok"}, "老版 Pydantic 不认识 warnings 时要回退" + + +def test_normalize_coerces_string_ids_and_keeps_other_fields(): + raw = { + "text": "x", + "segments": [ + {"id": "0", "seek": "12", "text": "a", "start": 0.0}, + {"id": "oops", "seek": "1.5", "text": "b"}, + {"id": True, "text": "c"}, + ], + } + normalized = _normalize_raw_payload(raw) + + assert normalized["segments"][0]["id"] == 0 + assert normalized["segments"][0]["seek"] == 12 + assert normalized["segments"][0]["text"] == "a", "其它字段必须原样保留" + assert normalized["segments"][1]["id"] == "oops", "认不出来的值不猜,保持原样" + assert normalized["segments"][1]["seek"] == "1.5", "非整数值不规整,保持原样" + assert normalized["segments"][2]["id"] is True + # 不改写上游对象:`raw` 仍可作原始证据 + assert raw["segments"][0]["id"] == "0" + + +def test_to_result_exposes_int_ids_and_keeps_timeline(): + obj = _Transcription() + transcriber = object.__new__(OpenAIASRTranscriber) + transcriber.api_style = "json" + + result = transcriber._to_result(obj, "unused.wav") + + assert [seg.text for seg in result.segments] == ["hello", "world"] + assert result.raw["segments"][0]["id"] == 0 + assert result.raw["segments"][1]["seek"] == 1500 + assert result.raw["segments"][1]["id"] == 2 + assert result.full_text == "hello world" diff --git a/specs/022-batch-concurrency-and-media-reliability/spec.md b/specs/022-batch-concurrency-and-media-reliability/spec.md index 38000130..e1b5e4ea 100644 --- a/specs/022-batch-concurrency-and-media-reliability/spec.md +++ b/specs/022-batch-concurrency-and-media-reliability/spec.md @@ -128,8 +128,8 @@ def _run_with_parallel_gate(execute_fn): return execute_fn() # 不再经 task_serial_executor.run ``` -- `task_serial_executor` 模块保留导出名做兼容(别处可能 import),但 note 路由不再依赖它; - 若全仓无引用则删除模块与其测试(W1 内决定,不晚于 W1 收尾)。 +- `task_serial_executor` 模块及其测试已删除(W1 收尾):全仓引用只有 note 路由与那个测试, + 没有别的兼容导出需要保留。CLAUDE.md 的架构说明同步改成「闸门是唯一上限」。 - `_ParallelGate.limit` 作为并发事实来源;合并扇出直接读它。 ### 6.2 合并 checkpoint @@ -224,9 +224,9 @@ DATA_DIR 绝对路径 → 原样使用 | 包 | 范围 | 验收命令 | 证据格式 | |---|---|---|---| | W0 | spec + README 索引 | `git diff --check`;人工核对 6 段齐全、索引行存在 | 本文件 + `specs/README.md` 行号 | -| W1 | note.py gate/合并/断点、task_serial_executor.py、tests/009 test_parallel_scheduler.py、tests/test_task_serial_executor.py | `pytest tests/022.../w1 tests/009.../test_parallel_scheduler.py tests/test_task_serial_executor.py -q` | pytest 摘要;最大在途计数断言;checkpoint 文件列表;章节顺序断言 | +| W1 | note.py gate/合并/断点、services/note.py 的 OCR workers/判定接线、删除 task_serial_executor.py、tests/009 test_parallel_scheduler.py、tests/001 test_task_retry.py | `pytest tests/022.../w1 tests/009.../test_parallel_scheduler.py tests/001.../test_task_retry.py -q` | pytest 摘要;最大在途计数断言;checkpoint 文件列表;章节顺序断言 | | W2 | path_helper.py、media_resource_service.py、init_db.py | `pytest tests/022.../w2 tests/007.../test_media_resources.py -q` | 临时 DATA_DIR 下 media_resources.path;`_roots()` 断言;init_db 回填计数;日志无登记失败 | -| W3 | ocr_client.py、note.py `_run_ocr_stage`/`_judge_note_frames`、universal_gpt.py、ocr_assist.py | `pytest tests/022.../w3 -q` + 既有 008 相关用例 | 429→200 请求次数;12 路并发最大在途;degraded detail;聚合告警条数 | +| W3 | ocr_client.py、universal_gpt.py、ocr_assist.py(note.py 接线归 W1,见 Q7) | `pytest tests/022.../w3 -q` + 既有 008 相关用例 | 429→200 请求次数;12 路并发最大在途;degraded detail;聚合告警条数 | | W4 | openai_asr.py | `pytest tests/022.../w4 -q` | `warnings.catch_warnings(record=True)` 断言无 Pydantic 告警;时间轴断言 | | W5 | 集成 | 上述全量 + 真实 3–5 条短视频合并 | 日志片段(并发>1、顺序、无五类报错);任务结果 Markdown |