feat(observability): worker 心跳告警闭环 + topic last_error 面板 - #52
Merged
Conversation
此前 worker 故障只记日志:心跳过期只触发 docker restart(无人工告警),
topic 抓取失败静默(last_error 被 API 用 CollectionAction.created_at
覆盖掩盖)。运维只能登服务器看日志发现问题。
后端:
- apps/worker/main.py:心跳改写共享卷 /app/data/worker_heartbeat.json
(pm_data 卷,backend 可读)+ 新增 heartbeat_alert_job(每10min自检):
心跳过期/主题 last_error 非空 → 发告警邮件给 notify_default_to,
带 1h 去重防刷屏(模块级 _last_alerts)。注册到 scheduler(*/10)。
- scripts/worker_healthcheck.py:改读共享卷心跳(与 worker 写入对齐)。
- apps/api/routers/topics.py:list_topics/_topic_dict 修复 last_run_at/
last_error 掩盖——改读 TopicSubscription 真实抓取状态(PR1 存的),
CollectionAction.created_at 改作 last_action_at 单独字段。加 ?failed=true
过滤(只返回 last_error 非空的 topic,供面板)。
- apps/api/routers/system.py:新增 /system/worker 端点(读共享卷心跳返
{ts,error,age_seconds,is_stale});system_status 增补 worker_heartbeat +
topic_errors 计数。
前端:
- types/index.ts:Topic 加 last_error/last_action_at;新增 WorkerHeartbeat
类型;SystemStatus 加 worker_heartbeat/topic_errors。
- services/api.ts:systemApi.worker + topicApi.list(failed) + WorkerHeartbeat 导入。
- pages/Operations.tsx:加 Worker 心跳面板(绿/红时效判定)+ 主题抓取错误
面板(列表 last_error + 重新抓取按钮复用 topicApi.fetch)。mount 时加载。
测试:新增 TestWorkerAlertDedup(去重+无收件人静默)、TestTopicLastErrorExposed
(last_error 持久化+成功清空),全套 61 passed。tsc --noEmit 通过。
🔍 OpenCode PR Review Required这是一个受保护的分支,merge 前需要进行 code review。 请运行以下命令进行 OpenCode review: 或者在 PR 页面评论 This is an automated reminder from PR Review Gate. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
此前 worker 故障只记日志无人知:心跳过期只触发 docker restart(无人工告警),topic 抓取失败静默(
last_error被 API 用CollectionAction.created_at覆盖掩盖)。运维只能登服务器看日志发现问题。修复
后端
apps/worker/main.py:心跳改写共享卷/app/data/worker_heartbeat.json(pm_data 卷,backend 可读)+ 新增heartbeat_alert_job(每 10min 自检):心跳过期 / 主题 last_error 非空 → 发告警邮件给notify_default_to,带 1h 去重防刷屏(模块级_last_alerts)。注册到 scheduler(*/10)。scripts/worker_healthcheck.py:改读共享卷心跳(与 worker 写入对齐)。apps/api/routers/topics.py:list_topics/_topic_dict修复last_run_at/last_error掩盖——改读TopicSubscription真实抓取状态(PR1 存的),CollectionAction.created_at改作last_action_at单独字段。加?failed=true过滤。apps/api/routers/system.py:新增/system/worker端点(读共享卷心跳返{ts,error,age_seconds,is_stale});system_status增补worker_heartbeat+topic_errors计数。前端
types/index.ts:Topic加last_error/last_action_at;新增WorkerHeartbeat类型;SystemStatus加worker_heartbeat/topic_errors。services/api.ts:systemApi.worker+topicApi.list(failed)+WorkerHeartbeat导入。pages/Operations.tsx:加 Worker 心跳面板(绿/红时效判定)+ 主题抓取错误面板(列表last_error+ 重新抓取按钮复用topicApi.fetch)。mount 时加载。验证
部署影响
/tmp/worker_heartbeat改到/app/data/worker_heartbeat.json(共享卷),worker healthcheck 同步改,docker-compose 无需改(已用python -m scripts.worker_healthcheck)heartbeat_alert(每 10min)notify_default_to)不做(诚实标注)