Skip to content

[PR] Blog pipeline: every post gets its own bilingual GitHub Discussion automatically - #5414

Merged
ranxianglei merged 5 commits into
masterfrom
2026-11-01_blog-pipeline
Nov 2, 2026
Merged

ranxianglei merged 5 commits into
masterfrom
2026-11-01_blog-pipeline

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What this closes the loop

Owner decision (2026-11-01): the discussion area must be created automatically (never manually), mirror the article + reference it, and carry both languages. This PR is the machinery that does exactly that, end to end:

author pushes article (+ zh twin)          author adds a line to website/articles.json
              └──────────── same PR, two things ────────────┘
                                    │ push to master
                                    ▼
        Actions (this repo, no external service) creates ONE Discussion:
        title  = "<English title> · <中文标题>"
        body   = reference lines → ## English → ## 中文版 → invisible marker
                                    │
                                    ▼
        write-back commit records  "discussion": <n>  in articles.json
                                    │
        ≤ 1 h later: running instances serve the article WITH a comment
        button linking straight to /discussions/<n>   (#5394 feeds the
        number through, #5396 renders the exact link; both pending human merge)

Pieces

  • scripts/blog-discussions.mjs — zero-dependency Node runner. Idempotency is three-layered so a crash between "thread created" and "number committed" can never silently produce a second thread: recorded numbers are re-verified before any creation, recent discussions are scanned for an invisible <!-- bili-blog:<slug> --> marker before creating, and the concurrency group serializes runs. A push consisting only of our own [bili-blog-pipeline] write-backs exits early (no trigger loop). Talks to api.github.com only — raw.githubusercontent.com is unreachable behind the GFW while this host is not, which matters because Actions itself must run from here.
  • .github/workflows/blog-discussions.yml — fires on pushes touching website/blog/posts/** or website/articles.json; cancel-in-progress: false (cancelling mid-write would orphan a thread).
  • website/articles.json — first real manifest: both current posts with explicit slugs (package.json already listed this file in the npm files array, so the slot was reserved for months).
  • bundled feed dedupe — a manifest entry pointing at a file that also sits in the flat posts directory used to render twice under one id; the manifest entry now wins (pinned facet). Needed precisely because our normal state is overlapping slugs.

Honest limits

  • The script has not yet executed against the live GitHub API from CI (it will run for the first time when this merges and the next blog push happens); any GraphQL shape mismatch fails the run loudly and the next qualifying push retries — nothing half-created survives invisibly.
  • One manual step remains, forever: deleting a post's Discussion after the article is removed (irreversible act stays with a human). Article updates need no action at all — the mirror is deliberately frozen at creation time.
  • Owner one-time prerequisite: Settings → Actions → Workflow permissions = Read and write (the token needs discussions:write).

Tests

scripts/blog-discussions.mjs passes node --check; workflow YAML parses; full local blog suite green (web-blogfeed 12/12 incl. new dedupe facet, web-blog-route + i18n + refresh 25/25); tsc clean.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-01_blog-pipeline (8a9c844)

Latest build — install this exact version

npm install -g billion-context@0.0.7780-pr.5414.372dc18

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
8a9c844 billion-context@0.0.7780-pr.5414.372dc18 build
99357cd billion-context@0.0.7775-pr.5414.3b2df75 build
1dd6a13 billion-context@0.0.7758-pr.5414.ade704e build
23ea4e9 billion-context@0.0.7745-pr.5414.7a4c5f5 build
11aefd5 billion-context@0.0.7669-pr.5414.dabb8ee build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5414.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title Blog pipeline: every post gets its own bilingual GitHub Discussion automatically [PR] Blog pipeline: every post gets its own bilingual GitHub Discussion automatically Nov 1, 2026

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: opencode/vllm-qwen/qwen3.8-27b)

Verdict: REQUEST_CHANGES (posted as a COMMENT review because GitHub refuses a change-request event on a PR from its own authoring identity — treat as blocking; do NOT merge until Finding 1 is resolved).

One load-bearing step cannot work against this repo's actual branch protection; every other facet reviewed clean.

Finding 1 (blocker): the manifest write-back commit will be rejected by master's branch protection

scripts/blog-discussions.mjs:218 (writeBackManifest) commits website/articles.json back to master via the Contents API using GITHUB_TOKEN, and .github/workflows/blog-discussions.yml grants it contents: write. Master currently enforces required status checks (strict, all CI test lanes as contexts). A brand-new commit has no passing check runs at push time, and GITHUB_TOKEN direct pushes are not exempt from required status checks (it is not an admin identity, and enforce_admins is off) — so the recording PUT will be rejected on every run.

Evidence, not speculation:

  • Across master's entire history, zero commits authored by github-actions[bot] sit on the first-parent chain — all 72 bot release-bump commits entered via PR merges. Direct pushes by the Actions token have never landed here.
  • This repo's own release infrastructure codes the same expectation: .github/workflows/release-manual.yml:215 attempts git push origin HEAD:master and carries an explicit fallback — ::warning::direct push to master was blocked (branch protection). Fallback: release prepared on a branch; a PR is needed to publish.

Impact if merged as-is: thread creation succeeds (the GraphQL mutation is unaffected), but the recording step fails → a red workflow on every qualifying blog push. The three-layer idempotency keeps duplicate threads out (good), but articles.json never gains numbers, so the dependent #5394/#5396 discussion-button feature is dead on arrival until someone edits the manifest by hand.

Suggested fix (pick one; owner decides posture):

  • (a) Mirror the release-manual pattern: attempt the direct write-back, and on rejection fall back to pushing a [bili-blog-pipeline] branch + opening a PR for the manifest update. One human merge closes the loop today; if protection is later relaxed for the Actions app, the direct path starts working automatically without code changes.
  • (b) A repo-scoped fine-grained PAT stored as a secret, used only for the write-back commit (an admin identity bypasses checks while enforce_admins is off).
  • (c) Owner relaxes master protection for the Actions app (ruleset bypass) — largest blast radius, your call.

Related body note: the "one-time prerequisite" (Settings → Actions → Workflow permissions = Read and write) is likely unnecessary — the workflow-level permissions: block (contents: write, discussions: write) already overrides the repository default token setting per GitHub's docs. Please verify on first live run and correct the body either way.

Verified clean

  • GraphQL shapes, verified live read-only (no mutation executed): repository id ✓; discussion categories include lowercase general (the script's "General" ?? "general" lookup lands correctly) ✓; discussions connection returns newest-first with bodyText (marker-scan ordering holds) ✓; CreateDiscussionInput accepts exactly repositoryId/categoryId/title/body as sent ✓.
  • Idempotency: walked every crash point — created-before-recorded (marker scan recovers), lost response after create (same), concurrent runs (concurrency group serializes), recorded thread deleted (recreated + self-healing record), unverifiable state (aborts rather than duplicates). No duplicate-thread path found.
  • Re-trigger loop guard (thisPushIsOnlyOurOwnWriteBack): correct for plain pushes, squash merges (commit message = PR title), and merge commits (payload.commits = the PR's own commits, none prefixed).
  • src/web/blogfeed.ts dedupe: module read in full; the manifest-wins filter is directionally right and the shipped manifest slugs align with the date-stripped flat-dir ids, so the two bundled posts collapse to one row each in the real tree. The new test builds a genuine overlap fixture (same file in flat dir + manifest entry) and asserts both the single-row outcome and the winner kind — both directions pinned, real regression teeth.
  • Behavior-drift inventory (intended, disclosed here): the two bundled posts' feed rows flip kind from post → article, so the panel shows the article-type label (src/web/client.ts:985) — cosmetic, content bytes unchanged. Both posts carry explicit frontmatter dates, so no sort/date drift despite the article path lacking the filename-date fallback. Remote-overlay merge (#4906) stays consistent (per-id Map, article wins on both paths). package.json files already ships website/articles.json.
  • Conventions & security: branch naming, single clean feat: commit, no version bump, @v4 action pins matching existing workflows, no new config/env surface (standard GITHUB_* vars only), least-privilege token permissions, script talks only to api.github.com with UA + 30 s timeouts, token header-only (nothing secret in logs or errors).

Minor (non-blocking — fix in passing or leave)

  • repoId() is re-queried per post inside the creation loop — hoist it out.
  • Write-back re-reads the sha immediately before the PUT but the content snapshot dates from run start (lost-update window ≈ run duration; serialized by the concurrency group, acceptable).
  • Two manifest entries sharing one slug would alias to a single thread (authoring error; degenerate, not dangerous).

Bottom line: the mechanism design is genuinely good and the web-side facet ships clean — but the write-back leg is provably dead on arrival against the current master protection, so REQUEST_CHANGES on Finding 1. Happy to re-review as soon as a fallback/PAT/ruleset decision lands.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: opencode/vllm-qwen/qwen3.8-27b)

Review 完成 → 已在 PR #5414 提交评审(判定 REQUEST_CHANGES):#5414 (review) —— 一个阻断项:「把讨论编号写回 articles.json」这步在 master 现行分支保护下必然被拒。查重:不重复(无其他 open PR/issue 做同一 blog-discussions 自动化;#5394/#5396 是下游依赖 PR,不同关注点)。GitHub 不允许对本人账号的 PR 发 change-request 事件,故以 COMMENT 评审形式发布、正文已显著标注阻断判定,合并权仍在人。

八问:

  1. 是什么问题:PR 核心闭环(每篇博客自动建双语 Discussion + 把编号写回 website/articles.json)的写回步在 master 现行保护下不可执行——合入后每次博客推送都会留一条红 CI,且 [PR] Blog refresh falls back to the GitHub contents API when raw.githubusercontent.com is unreachable #5394/[PR] Blog page: per-article GitHub discussion button (exact link from recorded data) + EN/中文 pin #5396 的评论按钮因拿不到编号永不生效。
  2. 什么原因:master 开启 strict required status checks(全部 CI test lane 为 contexts);全新 commit 在推送时刻不可能有已过检查,GITHUB_TOKEN 直推不受豁免(非 admin 身份,enforce_admins off)。实证:master 全史 first-parent 链上零个 github-actions[bot] 直推 commit(72 个 release 版本 commit 全部经 PR merge 进入);release-manual.yml:215 早已编码同款预期(直推被拒 → 回退 branch+PR)。
  3. 影响范围与严重程度:仅新功能面,不破坏既有功能;无数据丢失、无重复线程(三层幂等守住);严重度 = 新功能核心路径不可用 + 周期性红 run;手动补 manifest 可止血。
  4. 怎么修(作者/owner 三选一):(a) 镜像 release-manual——先试直推,失败自动回退 [bili-blog-pipeline] 分支 + 开 PR【推荐,与仓库既有先例一致】;(b) 仓库级 fine-grained PAT secret 仅用于写回 commit(admin 身份绕过检查);(c) owner 放宽 Actions app 保护(ruleset bypass,爆炸半径最大)。
  5. 回归审查:feed 去重 = 有意变更(两 bundled 文章由双渲染收敛单行,manifest 胜,已披露);kind 标签 post→article(src/web/client.ts:985,外观级);日期/排序 = 无漂移(两文均有显式 frontmatter date);remote overlay(Web Panel Blog: Automatically pulls new posts from GitHub master — no need to wait for a release. | Web 面板博客:自动从 GitHub master 拉取新文章,不用等发版 #4906)合并一致性 OK;kernel/wire/config 面未触。
  6. 是否破坏缓存:不涉及(feed 进程内缓存失效逻辑正常,测试经 _setBlogRootForTest 覆盖)。
  7. 是否改设计方向:否——纯增量 CI + 数据文件;blogfeed 去重属渲染路径 bugfix。
  8. 新配置引入:无——不加 BILI_/config 字段,仅标准 GITHUB_ env。另纠正 PR body 一处:声称需一次性改 repo Settings,实际 workflow 级 permissions: 块已覆盖(review 中已指出,请首跑验证)。

判定:REQUEST_CHANGES——Finding 1(写回通道)解决前不应合并;无 kernel 涉及,无需授权申请。修法落地后我复审。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 全仓 issue(open+closed)检索无同主题条目——这是落实你 2026-11-01 决策(讨论区自动建、镜像文章、双语)的新功能 PR,不是对已有条目的重做。

一句话判定:可合,但合前先收掉 F1(一处「代码 / 文档 / 日志三者自相矛盾」的小缺陷;happy path 和「绝不重复建线程」的安全性质都不受影响,属该修而非硬拦)。CI 全绿,diff 干净、每行都在目的之内。

我核了什么(既有行为清单 + 回归判定)

  • bundled feed 去重(src/web/blogfeed.ts):正确且真实承重。确认 post-id(文件名去日期前缀,如 meet-bili)与 article-id(manifest 显式 slug,同为 meet-bili)在随包发布时确实撞 id;package.json 的 files 同时打包了 website/articles.json 和 website/blog/posts,所以安装后的包里两者并存,不去重就会同一 id 渲染两行——是真 bug 不是理论问题。排序保持(posts→articles→sortItems,与改动前一致),不会误删不同 post(只有真正同 id 的重复才收敛、manifest 赢),与既有 remote-overlay「canonical 按 id 赢」语义一致。测试是真测试:fixture 里 2026-10-09_new.md 既是扁平 post 又被 manifest 指向,断言 length===1 && kind==="article" 有牙、还钉了非重叠项不受影响。✅
  • workflow:push 限定 master + 博客路径;permissions 声明 contents:write + discussions:write;concurrency group 串行 + cancel-in-progress:false(中途取消会孤儿线程,判断正确);node 22 支持 fetch/AbortSignal.timeout。✅
  • 防回写死循环:write-back 只改 articles.json,thisPushIsOnlyOurOwnWriteBack() 识别纯回写 push 提前退出;手动/本地跑(无 event path)绝不跳过。✅
  • fail-loud:任何错误 exit 1 → 红 job 可见 → 下次合格 push 从头重试;整批 write-back 要么全写要么不写,不会留半截记录或重复线程。✅
  • 仅走 api.github.com(raw host 被墙、本机不被墙)——满足「Actions 必须从这台机器跑」的约束。✅
  • CI:push head 全绿(test ubuntu/windows × node22/24 各 shard、static-gates、kernel/version/config-surface/docs-sync/lock-sync 等 guard、mysql-contract 均 success;real-client e2e 按各自 gate skipped)。✅
  • 无配置面变更:没有新增 env/config key,只用标准 GITHUB_* action 变量;不碰 kernel/wire/persistence。✅

发现

F1(建议合前处理)— verifyRecorded 返回值被丢弃:文档承诺的第 1 层幂等没接线 + 日志撒谎

脚本头注释把幂等第 1 层写成「recorded number 仍解析得到就跳过」(隐含解析不到就该重建),且 verifyRecorded 在 number 失效时打印「no longer resolves — will recreate」。但两处都没接上:

  • main() 里 if (hasRecorded(entry)) await verifyRecorded(...) —— 布尔返回值被丢弃;
  • 创建循环 if (hasRecorded(entry)) continue; —— 只要 manifest 里还有 number 就跳过。

后果:某帖的讨论被删、但 manifest 条目和 number 还在时,这条会永远跳过、永不重建,日志却谎称「will recreate」。(「网络不可验证」那条走 catch 抛错整体中止,是对的。)频率低(设计上删除通常伴随删文章),但这是安全路径上的契约/逻辑不一致 + 误导性日志(§7.3 诚实输出)。

推荐修法 A(让代码对齐它自己已写好的文档和日志,约 4 行):把验证结果接进创建门槛——只有「验证通过仍存在」才跳过,验证为「已删除」则进入创建:

const needsRecreate = new Set();
for (const { entry, index } of entries) {
  if (hasRecorded(entry) && !(await verifyRecorded(entry, slugOf(entry)))) needsRecreate.add(index);
}
// 创建循环里:
if (hasRecorded(entry) && !needsRecreate.has(index)) continue;

(catch 的「不可验证→中止」保持不变。)副作用需你知晓:这会让「故意删了讨论但保留文章」在下一次博客 push 时被自动复活——我认为正是想要的自愈,但也可能不是你想要的。若不想自动复活,退路 B 是保持跳过行为、把那句日志改成如实描述(「no longer resolves — keeping recorded value; 删掉 discussion 字段才会新建」),零行为变化。因为这是你自己的安全路径 PR、脚本本地又没有单测我无法替你实测三层幂等,我没擅自动它——你定 A/B,我就推一个 follow-up commit 到分支。

F2 / F3 / F4(可选,非拦)

  • F2:脚本三层幂等只有 node --check,没有单测钉住。「崩溃窗口不可能重复建线程」这个核心安全声明目前靠 code review 背书。建议后续加一个 mock global.fetch 的单测钉住第 1、2 层。
  • F3:findMarkerInRecentDiscussions 分页扫最近讨论找 marker,依赖 GitHub 默认 newest-first 排序来命中「崩溃窗口新线程」。建议显式加 orderBy:{field:CREATED_AT,direction:DESC} 让 2 页上限确定化。(主保险是 recorded-number 那层,优先级低。)
  • F4:repoId() 在逐条循环里每次调用(稳定查询、冗余),可提出去。微 nit。

行为变更披露(Deliverables 第 4 条)

唯一用户可见的行为变化就是去重本身:改动前——manifest 列出、同时又平铺在 posts 目录的文件,在安装包里会同一 id 渲染两行;改动后——渲染一行、manifest 条目赢。这是本 PR 的目的性修复,不是回归。Discussion 自动创建是新增能力(feat),受 workflow + 你一次性设置 Actions 权限(Read and write,token 需 discussions:write)约束,不改任何既有行为。

八问

  1. 是什么问题:给每篇博客文章自动建一条双语 GitHub Discussion(镜像文章 + 引用权威源 + 双语标题),并把线程号回写进 articles.json;顺带修掉「manifest 与平铺目录撞 id 导致同一篇文章渲染两行」的去重 bug。
  2. 什么原因导致的:去重 bug 根因是 collect() 直接 [...posts, ...articles] 拼接、未按 id 合并,而正常状态本就是「manifest 指向平铺文件」(重叠是常态);本 PR 用 articleIds 集合过滤掉撞 id 的 post,从源头消除。
  3. 影响范围与严重程度:去重影响所有装了包、且 manifest 与 posts 目录重叠的用户(即当前两篇帖子),体验瑕疵级(列表里同一篇出现两次);自动建 Discussion 是新能力,未启用前零影响。
  4. 怎么修复的:去重 = 集合过滤(manifest 赢);Discussion = Action 里零依赖 Node 脚本,三层幂等 + fail-loud + 防回写死循环,只在 api.github.com 上活动。
  5. 是否有回归问题:审过 collect()/collectPosts/collectArticles 的 id 推导与排序——去重只收敛真正同 id 项、不丢不同 post、排序不变;workflow/loop-guard/fail-loud 均核对无误。唯一待收口的是 F1(文档/日志承诺的重建没接线,现表现为永久跳过 + 谎报),不影响 happy path 与「绝不重复建线程」的安全性质。
  6. 是否破坏缓存:不涉及(无运行时缓存/重试/游标/去重键;articles.json 回写是 git 提交,非进程内缓存)。
  7. 是否改了设计方向:不改内核/压缩/存储方向;新增的是仓库侧内容管道(GitHub Action + 脚本),边界清晰、零依赖。
  8. 是否引入新配置:无新增 env/config key/默认值,仅用 Actions 标准变量(GITHUB_REPOSITORY/TOKEN/REF_NAME/EVENT_PATH)。前置一次性动作是你手动把 Settings→Actions 权限设为 Read and write。

收工判定:可合,先收 F1。 建议按 A 法接线(或明确选 B、由我改日志)后我推 follow-up commit;F2–F4 记为可选后续。其余质量、diff 整洁度、测试有效性(去重 facet 有真测试)都达标,CI 全绿。要我直接动 F1 的话说一声。

ranxianglei pushed a commit that referenced this pull request Nov 1, 2026
The pin was drafted against a mirror-side issue number (#5414); on GitHub that number is an unrelated open PR, while the D2-pin issue is #5413. Renames tests/issue5414-* -> tests/issue5413-* and updates the six in-file references so cross-references resolve on GitHub. Test content unchanged.
ranxianglei-devbox added 2 commits November 1, 2026 23:49
…ord thread numbers back into articles.json

- scripts/blog-discussions.mjs: zero-dependency runner. For each manifest entry without a recorded number it creates one Discussion (English section first, then Chinese; creation-time mirror plus canonical-source reference; invisible body marker), then commits all new numbers back into website/articles.json with a [bili-blog-pipeline] prefix that also guards against re-trigger loops. Three-layer idempotency (number re-verification, marker scan over recent discussions, serialized concurrency) makes crash-window duplicates impossible to hide. api.github.com only — the raw host is blocked by the GFW, this host is not.- .github/workflows/blog-discussions.yml: push-triggered on blog surface changes, contents+discussions write, never cancel-in-progress.- website/articles.json: first real manifest (meet-bili + frequent-compression, explicit slugs matching the flat files so ids stay stable).- bundled feed dedupe: a manifest entry pointing at a flat posts-dir file previously rendered twice under one id; the manifest entry now wins (pinned test).
… be verified against a real repo without any writes
@ranxianglei
ranxianglei force-pushed the 2026-11-01_blog-pipeline branch from 23ea4e9 to 1dd6a13 Compare November 1, 2026 15:49
ranxianglei-devbox added 3 commits November 2, 2026 00:35
…n — try direct push, fall back to a fixed branch + one PR; honest log for vanished threads; newest-first marker scan; resolve repo id once; pin the safety properties with mocked-fetch tests

(cherry picked from commit aebfd85c2d45c96d3d22ade2a40db47622abc2ab)
…rectly; numbers always land on the fixed branch as exactly one PR for a human merge
@ranxianglei
ranxianglei merged commit ee0dcc7 into master Nov 2, 2026
47 checks passed
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.995-master.ee0dcc7 (rolling master tag).

npm install -g billion-context@0.0.995-master.ee0dcc7  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei added a commit that referenced this pull request Nov 2, 2026
Automatic up-to-date update: master advanced via #5414 (blog pipeline,
6 files, zero overlap with this PR's 16 files) after head ef19bf8.
Merge tree built path-wise (no content conflicts possible).
ee0dcc7
ranxianglei pushed a commit that referenced this pull request Nov 2, 2026
…iscussion, not node)

The new blog pipeline (#5414) selected "{ node { number url } }" from the
createDiscussion mutation, but GitHub's CreateDiscussionPayload exposes
"discussion", not "node" — so the first master run of sync-discussions failed
with "Field 'node' doesn't exist on type 'CreateDiscussionPayload'". Align the
query and its result reads, and pin the selection in the unit test (positive +
negative) so the wrong field cannot silently ship again.
Louie0806 added a commit to Louie0806/billion-context that referenced this pull request Nov 2, 2026
… gate + ranxianglei#5414 blog pipeline

No overlap with this lane's 16 files. The three generated docs conflicted
because master regenerated them from a newer seed; resolved by taking
master's bytes and regenerating from the lane's seed (chat pack +212 lines),
which is the documented revert-then-keep flow for tools/gen-config-docs.mjs.

ranxianglei#5506 matters to this workspace historically: it fixes the issue2129
restart-measurement flake I reported as ranxianglei#5379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant