feat(worker): 调度配置提取到 DB + 网页端实时热重载 - #62
Open
Color2333 wants to merge 1 commit into
Open
Conversation
将 worker 的 4 个 cron 调度(topic_dispatch / cs_feed_dispatch / daily_brief / weekly_graph)从硬编码/env 提取到 DB 单例表 WorkerScheduleConfig, 网页端修改后 worker 轮询线程 30s 内检测 updated_at 变化并热重载 APScheduler job, 无需重启容器。 核心改动: - 新表 WorkerScheduleConfig(model + alembic 迁移 + repository) - 后端 GET/PUT /settings/worker-schedule 端点 - worker scheduler 提升到模块级 + get_scheduler() 访问器 - _read_schedule_config / _register_all_jobs / reload_worker_schedule / _config_poll_loop(30s 轮询 + daemon 线程,镜像 IdleProcessor 模式) - _sync_idle_processor 按配置开关 start/stop 闲时处理器 - worker 写回 last_applied_at 供前端显示已生效状态 - 前端新增 Worker / 调度 tab(3 个 cron 输入 + idle 开关 + 同步状态指示器) 附带修复: - brief_service.py publish(): 删除 if not recipient 块里局部 import 的 session_scope,用模块级 import(第 18 行已有)。修复 recipient 已传入时 session_scope 未绑定导致 generated_content 写入失败的 bug - auto_deploy.sh: docker compose up -d 后加 docker image prune -f + builder prune -f,防磁盘满再次搞崩 postgres 测试:73 passed + 2 skipped(新增 4 个 WorkerScheduleConfigRepository 测试); 前端 tsc + vite build 通过。 部署:合 PR 后需在 backend 容器手动 alembic upgrade head 建新表
🔍 OpenCode PR Review Required这是一个受保护的分支,merge 前需要进行 code review。 请运行以下命令进行 OpenCode review: 或者在 PR 页面评论 This is an automated reminder from PR Review Gate. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
用户反馈:网页端不能及时控制 worker。worker 启动时读一次 DB/env 配置注册 APScheduler job,之后固定。网页端改完 cron 必须重启 worker 容器才生效。其中
topic_dispatch(每整点)/cs_feed_dispatch(:05) 是硬编码在apps/worker/main.py,weekly_graph读 envWEEKLY_CRON(@lru_cache冻结),连 DB 都没写。daily_brief虽在 DB 里但同样只启动时读一次。另:今日 04:00 UTC 每日简报因磁盘满 → postgres 进 recovery mode 而失败(已手动补发,根因另案处理)。
方案:DB 可配置 + 轮询热重载(30s 延迟)
沿用现有三个模式,不引入新基础设施(不加 worker HTTP 端口、不加消息队列):
DailyReportConfig单例表 → 新增WorkerScheduleConfig单例表IdleProcessor模式)→ 新增_config_poll_loop架构
改动清单(13 文件,3 新建)
后端
packages/storage/models.py— 新增WorkerScheduleConfig单例表(5 配置字段 + last_applied_at)infra/migrations/versions/a7b8c9d0e1f2_add_worker_schedule_config.py— 新建 alembic 迁移(down_revision = f6a7b8c9d0e1)packages/storage/repositories/worker_schedule.py— 新建WorkerScheduleConfigRepository(get_config / update_config / update_last_applied_at)packages/storage/repositories/__init__.py— 导出WorkerScheduleConfigRepositoryapps/api/routers/settings.py— 新增WorkerScheduleConfigUpdatepydantic +GET/PUT /settings/worker-schedule端点Worker 调度重构(核心)
apps/worker/main.py:scheduler从run_worker()局部变量提升到模块级 +get_scheduler()访问器(镜像get_idle_processor)_read_schedule_config()— 从 DB 读所有 cron + idle 开关,DB 不可用回退默认值_register_all_jobs(scheduler, cfg)— 统一注册 4 个 job(replace_existing=True 等价 reschedule)reload_worker_schedule()— 重读 DB → reschedule + sync idle → 写 last_applied_at_config_poll_loop(stop)— daemon 线程每 30s 查WorkerScheduleConfig.updated_at+DailyReportConfig.updated_at,变化即调 reload_sync_idle_processor(enabled)— 按配置开关 start/stop,避免无谓循环run_worker()重写:模块级 scheduler → 初始注册 → 轮询线程 → start附带修复(同一 PR)
packages/ai/brief_service.py:600— 删除if not recipient:块里局部from packages.storage.db import session_scope,用模块级 import(第 18 行已有)。修recipient已传入时session_scope未绑定导致generated_content写入失败的 bug(今日补发简报时触发)scripts/auto_deploy.sh—docker compose up -d后加docker image prune -f+builder prune -f,防磁盘满再次搞崩 postgres前端
frontend/src/types/index.ts— 新增WorkerScheduleConfiginterfacefrontend/src/services/api.ts— 新增workerScheduleApi(getConfig / updateConfig)+ import/re-exportfrontend/src/pages/Settings.tsx— 注册第 5 个 tabworker(Clock 图标)frontend/src/components/settings/WorkerSettingsTab.tsx— 新建 tab 组件:测试
tests/test_repositories.py— 新增TestWorkerScheduleConfigRepository(4 个测试:单例创建 / 单例复用 / 部分更新 / last_applied_at 写回)验证
pytest:73 passed + 2 skipped(新增 4 个测试)tsc --noEmit:无错误vite build:构建成功ruff check+ruff format:通过(pre-commit 全绿)不在本次范围
用户选了"只先把调度参数提取出来",以下暂不做:
worker_retry_max/paper_concurrency/cost_guard/skim_score_threshold/user_timezone等 env-only 运行参数batch_size/check_interval(只加 on/off 开关)部署步骤
docker compose exec backend alembic upgrade head建新表(auto_deploy.sh 不自动跑 alembic)验证场景
topic_dispatch_cron为*/10 * * * *→ 30s 内 worker 日志出现🔄 worker 调度已热重载+last_applied_at更新0 * * * *→ 同步生效idle_processor_enabled→ 闲时线程停止;开 → 重启