feat(cs_feed): 论文关联 PaperTopic + 触发引用 auto_link - #51
Merged
Conversation
此前 cs_feed 按分类抓取论文后只 upsert_paper,无 PaperTopic 关联、
无 _bg_auto_link。后果:cs_feed 论文进不了主题侧边栏文件夹、topic
图谱视图、topic 统计;归入"未分类";推荐排序 20% 劣势;日报归
uncategorized 桶;论文间引用关系只靠后续 sync 补。
修复:
- 每分类建一个 disabled topic(name=csfeed:{code}, query=cat:{code},
enabled=False)。enabled=False 防 topic_dispatch 重复抓取同一分类
(topic_dispatch 只选 enabled=True),仅 cs_feed_dispatch 负责。
disabled topic 仍出现在侧边栏(list_topics 不过滤 enabled,
get_folder_stats INNER JOIN PaperTopic 统计有论文的 topic)。
- CSFeedOrchestrator.run() 抓取循环加 _link_cs_papers_to_topic,
按 category_code upsert_topic(幂等)+ link_to_topic(uq_paper_topic
兜底幂等)。
- 抓取后补 _trigger_auto_link,复用 paper_pipelines._auto_link_pool
有界线程池,与 ingest_arxiv 对齐。
- 手动 fetch_category 端点同步加 link + auto_link,保持两入口一致。
- 两个 helper 设为 staticmethod(不依赖实例状态,路由可直接调)。
测试:新增 TestCSFeedTopicLink 3 项(建 disabled topic + 关联、
幂等无重复行、分类隔离),全套 58 passed。
🔍 OpenCode PR Review Required这是一个受保护的分支,merge 前需要进行 code review。 请运行以下命令进行 OpenCode review: 或者在 PR 页面评论 This is an automated reminder from PR Review Gate. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
cs_feed 按分类抓取论文后只
upsert_paper,无 PaperTopic 关联、无_bg_auto_link。后果:修复
每分类建一个 disabled topic + 关联 + auto_link
name=csfeed:{category_code}(如csfeed:cs.AI),query=cat:{category_code},enabled=Falsetopic_dispatch_job只选enabled=True的 topic(main.pylist_topics(enabled_only=True)),所以 cs_feed topic 永不被 topic_dispatch 抓取,仅cs_feed_dispatch负责 → 零重复抓取list_topics(enabled_only=False)默认显示全部;get_folder_stats.by_topicINNER JOIN PaperTopic 统计,不过滤 enabled、只显示有论文的 topic → cs_feed 抓取后该 topic 即可见改动
packages/ai/cs_feed_orchestrator.pyrun():抓取循环upsert_paper后调_link_cs_papers_to_topic(按 category_codeupsert_topic幂等 +link_to_topicuq_paper_topic 兜底幂等);抓取后补_trigger_auto_link(复用paper_pipelines._auto_link_pool有界线程池,与 ingest_arxiv 对齐)apps/api/routers/cs_feeds.py手动fetch_category:同步加 link + auto_link,保持两入口一致@staticmethod(不依赖实例状态,路由可直接调,免实例化 TokenBucket)验证
部署影响
不做(诚实标注)