Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# 手动上线按钮:Actions → Deploy → Run workflow。
#
# 这个 workflow 跑在 GitHub 托管的 runner 上,不接触服务器:它只确认目标提交的 CI 已通过,
# 然后在 GitHub 上登记一条排队中的部署请求(environment: production,payload.source = manual)。
# seed-SER9 上的 ash-cd 每 30 秒拉取一次,看到请求就按标准流程部署(备份 → 构建 → 切换 → 冒烟 → 失败回滚),
# 并把结果写回这条部署记录;本 workflow 等到结果后以成功 / 失败结束。
# 选旧提交即可回滚;手动上线的版本不是 main 最新时,自动部署会暂停到 main 出现新提交为止。
name: Deploy

on:
workflow_dispatch:
inputs:
ref:
description: 要上线的分支、tag 或提交(默认 main 最新)
required: true
default: main
skip_ci_check:
description: 跳过「CI 已通过」检查(紧急修复,或回滚到建立 CI 之前的旧版本)
type: boolean
default: false

concurrency:
group: deploy-production
cancel-in-progress: false

permissions:
contents: read
checks: read
deployments: write

jobs:
deploy:
runs-on: ubuntu-latest
timeout-minutes: 30
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
REF: ${{ inputs.ref }}
FORCE: ${{ inputs.skip_ci_check }}
ACTOR: ${{ github.actor }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
steps:
- name: Resolve commit
id: resolve
run: |
sha=$(gh api "repos/$REPO/commits/$REF" --jq .sha) || { echo "::error::找不到 $REF"; exit 1; }
echo "sha=$sha" >> "$GITHUB_OUTPUT"
echo "### 上线 \`${sha:0:7}\`($REF)" >> "$GITHUB_STEP_SUMMARY"

- name: Require green CI
if: ${{ !inputs.skip_ci_check }}
env:
SHA: ${{ steps.resolve.outputs.sha }}
run: |
# 与 ash-cd 相同的判定:check、docker 两项最近一次运行都成功
for name in check docker; do
state=$(gh api "repos/$REPO/commits/$SHA/check-runs?check_name=$name&per_page=100" \
--jq '[.check_runs[]] | sort_by(.started_at) | last | if . == null then "missing" else "\(.status)/\(.conclusion)" end')
if [ "$state" != completed/success ]; then
echo "::error::${SHA:0:7} 的 CI 检查 $name 为 $state,不能上线(确需上线请勾选 skip_ci_check)"
exit 1
fi
done

- name: Request deployment
id: request
env:
SHA: ${{ steps.resolve.outputs.sha }}
run: |
id=$(gh api -X POST "repos/$REPO/deployments" --jq .id --input - <<JSON
{"ref": "$SHA", "environment": "production", "auto_merge": false, "required_contexts": [],
"production_environment": true, "description": "manual deploy by $ACTOR",
"payload": {"source": "manual", "force": $FORCE, "requested_by": "$ACTOR", "run": "$RUN_URL"}}
JSON
)
gh api -X POST "repos/$REPO/deployments/$id/statuses" \
-f state=queued -f description="等待 seed-SER9 上的 ash-cd 拉取" -f log_url="$RUN_URL" >/dev/null
echo "id=$id" >> "$GITHUB_OUTPUT"
echo "部署请求 #$id 已排队,ash-cd 约 30 秒内开始。"

- name: Wait for ash-cd
env:
ID: ${{ steps.request.outputs.id }}
run: |
deadline=$((SECONDS + 1200))
last=""
while [ "$SECONDS" -lt "$deadline" ]; do
status=$(gh api "repos/$REPO/deployments/$ID/statuses?per_page=1" --jq '.[0] | "\(.state)|\(.description)"')
[ "$status" != "$last" ] && echo "$(date -u +%T) ${status/|/ — }" && last=$status
case "${status%%|*}" in
success) echo "✅ ${status#*|}" >> "$GITHUB_STEP_SUMMARY"; exit 0 ;;
failure|error) echo "::error::${status#*|}"; echo "❌ ${status#*|}" >> "$GITHUB_STEP_SUMMARY"; exit 1 ;;
esac
sleep 15
done
gh api -X POST "repos/$REPO/deployments/$ID/statuses" \
-f state=error -f description="20 分钟内 ash-cd 没有处理这个请求(服务器或定时器是否在运行?)" >/dev/null
echo "::error::ash-cd 20 分钟内没有处理部署请求"
exit 1
108 changes: 88 additions & 20 deletions deploy/cd/ash-cd.sh
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
#!/usr/bin/env bash
# ash-cd — seed-SER9 上的拉取式持续部署。
#
# systemd 定时器每 2 分钟执行一次 `ash-cd poll`:main 出现新提交、且它的 CI 检查(check、docker)
# 全部通过时,按部署手册的顺序上线:备份 DB 与技能文件 → 构建 ash:<sha> → 切换容器 → 冒烟。
# 冒烟失败自动切回上一个版本。部署进度回写到 GitHub Deployments(environment: production)。
# systemd 定时器每 30 秒执行一次 `ash-cd poll`,两种触发方式:
# · 手动:GitHub 上 Actions → Deploy → Run workflow 登记的部署请求(优先处理)
# · 自动:main 出现新提交、且它的 CI 检查(check、docker)全部通过
# 部署按手册顺序:备份 DB 与技能文件 → 构建 ash:<sha> → 切换容器 → 冒烟;冒烟失败自动切回上一个版本。
# 进度回写到 GitHub Deployments(environment: production)。
# 手动上线的不是 main 最新提交时(例如回滚到旧版),自动部署暂停,直到 main 出现新提交。
#
# 用法:
# ash-cd poll 定时器入口;没有新的可部署提交时什么也不做
# ash-cd poll 定时器入口;没有请求、也没有新的可部署提交时什么也不做
# ash-cd deploy <sha> 立即部署某个提交(仍要求它的 CI 已通过;--force 跳过 CI 检查)
# ash-cd rollback [<tag>] 切回上一个(或指定的)已发布版本
# ash-cd status 当前版本、main 最新提交及其 CI 状态、最近的部署记录
Expand All @@ -27,7 +30,9 @@ LOCAL_URL="${ASH_CD_LOCAL_URL:-http://127.0.0.1:9444}"
CONTAINER="${ASH_CD_CONTAINER:-ash}" # 容器名
PROJECT="${ASH_CD_PROJECT:-ash}" # docker compose -p
IMAGE="${ASH_CD_IMAGE:-ash}" # 镜像仓库名,tag 为 7 位短 sha
GH_DEPLOYMENTS="${ASH_CD_GH_DEPLOYMENTS:-1}" # 0 = 不写 GitHub Deployments(演练用)
GH_DEPLOYMENTS="${ASH_CD_GH_DEPLOYMENTS:-1}" # 0 = 不读写 GitHub Deployments(演练用)
GH_ENV="${ASH_CD_GH_ENV:-production}" # GitHub Deployments 的 environment 名
REQUEST_WINDOW_MIN=25 # 只处理 25 分钟内登记的手动请求(Deploy workflow 等 20 分钟后自行判超时)
PUBLIC_URL="${ASH_CD_PUBLIC_URL:-https://ash.709970.xyz}"
REQUIRED_CHECKS=(check docker) # 与 .github/workflows/ci.yml 的 job 名一致
KEEP_IMAGES=10 # 磁盘紧张:只保留最近 10 个 ash 镜像(当前版本永远保留)
Expand Down Expand Up @@ -70,11 +75,11 @@ ci_state() {
}

# ——— GitHub Deployments(失败不影响部署本身)———
DEPLOY_ID=""
DEPLOY_ID="" # 手动请求时预先填好(沿用 Deploy workflow 登记的那条),否则由 gh_deployment 新建
gh_deployment() {
[ "$GH_DEPLOYMENTS" = 1 ] || return 0
[ "$GH_DEPLOYMENTS" = 1 ] && [ -z "$DEPLOY_ID" ] || return 0
DEPLOY_ID=$(gh api -X POST "repos/$REPO/deployments" --jq .id --input - 2>/dev/null <<JSON || true
{"ref": "$1", "environment": "production", "auto_merge": false, "required_contexts": [],
{"ref": "$1", "environment": "$GH_ENV", "auto_merge": false, "required_contexts": [],
"description": "ash-cd on $(hostname)", "production_environment": true}
JSON
)
Expand Down Expand Up @@ -181,21 +186,80 @@ with_lock() {
flock -n 9 || { log "另一个部署正在进行"; exit 0; }
}

cmd_poll() {
with_lock
sync_mirror
# ——— 跳过标记:失败 / 回滚掉 / 被手动上线暂停的提交,定时器不自动部署 ———
skip() { printf '%s %s\n' "$(date '+%F %T')" "$2" > "$STATE/skip-$1"; }
skipped() { [ -f "$STATE/skip-$1" ]; }
unskip() { rm -f "$STATE/skip-$1"; }

# 手动上线的不是 main 最新时,暂停自动部署,免得半分钟后又被 main 覆盖回去
hold_main_unless() {
local head; head=$(git -C "$MIRROR" rev-parse refs/heads/main)
[ "${head:0:7}" = "$1" ] && return 0
skip "${head:0:7}" "暂停:手动上线了 $1"
log "手动上线的 $1 不是 main 最新(${head:0:7}),自动部署暂停到 main 出现新提交"
}

# 输出排队中的手动部署请求:「id sha force」,最新的在前
queued_requests() {
[ "$GH_DEPLOYMENTS" = 1 ] || return 0
local since; since=$(date -u -d "-$REQUEST_WINDOW_MIN min" +%Y-%m-%dT%H:%M:%SZ)
local id sha force state
gh api "repos/$REPO/deployments?environment=$GH_ENV&per_page=20" \
--jq ".[] | select(.payload.source == \"manual\" and .created_at > \"$since\") | \"\(.id) \(.sha) \(.payload.force // false)\"" 2>/dev/null \
| while read -r id sha force; do
state=$(gh api "repos/$REPO/deployments/$id/statuses?per_page=1" --jq '.[0].state // "none"' 2>/dev/null) || continue
if [ "$state" = queued ]; then echo "$id $sha $force"; fi
done || true # 查询失败时当作没有请求,不中断本轮轮询
}

# 处理手动请求;有请求时返回 0(本轮不再走自动部署)
handle_requests() {
local requests id full force current
requests=$(queued_requests)
[ -n "$requests" ] || return 1
read -r id full force <<<"$(head -1 <<<"$requests")"
# 同时排了多个请求:只做最新的,其余标为被取代
tail -n +2 <<<"$requests" | while read -r old_id _ _; do
DEPLOY_ID=$old_id gh_status error "被更新的部署请求取代"
done
DEPLOY_ID=$id
log "收到手动部署请求 #$id:${full:0:7}$([ "$force" = true ] && echo '(跳过 CI 检查)')"
if ! git -C "$MIRROR" cat-file -e "$full^{commit}" 2>/dev/null; then
gh_status error "服务器的镜像仓库里找不到 ${full:0:7}"; log "找不到 ${full:0:7}"; return 0
fi
if [ "$force" != true ] && [ "$(ci_state "$full")" != success ]; then
gh_status failure "${full:0:7} 的 CI 没有全部通过"; log "${full:0:7} 的 CI 没有全部通过,拒绝"; return 0
fi
current=$(current_tag)
if [ "${full:0:7}" = "$current" ]; then
gh_status success "$IMAGE:$current 已在线上"; log "${full:0:7} 已在线上"
hold_main_unless "$current"
return 0
fi
unskip "${full:0:7}"
if deploy "$full"; then hold_main_unless "${full:0:7}"; else skip "${full:0:7}" "手动部署失败"; fi
return 0
}

auto_deploy() {
local head sha
head=$(git -C "$MIRROR" rev-parse refs/heads/main)
sha=${head:0:7}
[ "$sha" = "$(current_tag)" ] && return 0
[ -f "$STATE/failed-$sha" ] && return 0 # 失败过的提交不自动重试,修好后推新提交即可
skipped "$sha" && return 0 # 失败过 / 被回滚 / 被手动上线暂停:修好后推新提交即可
case "$(ci_state "$head")" in
success) deploy "$head" || touch "$STATE/failed-$sha" ;;
failure) log "main@$sha 的 CI 没通过,不部署"; touch "$STATE/failed-$sha" ;;
success) deploy "$head" || skip "$sha" "自动部署失败" ;;
failure) log "main@$sha 的 CI 没通过,不部署"; skip "$sha" "CI 未通过" ;;
*) log "main@$sha 等待 CI" ;;
esac
}

cmd_poll() {
with_lock
sync_mirror
handle_requests || auto_deploy
}

cmd_deploy() {
local ref=${1:-} force=${2:-}
[ -n "$ref" ] || die "用法:ash-cd deploy <sha> [--force]"
Expand All @@ -205,12 +269,13 @@ cmd_deploy() {
if [ "$force" != "--force" ]; then
[ "$(ci_state "$full")" = success ] || die "${full:0:7} 的 CI 没有全部通过(--force 可跳过)"
fi
rm -f "$STATE/failed-${full:0:7}"
deploy "$full"
unskip "${full:0:7}"
deploy "$full" && hold_main_unless "${full:0:7}"
}

cmd_rollback() {
with_lock
sync_mirror
local target=${1:-} current; current=$(current_tag)
if [ -z "$target" ]; then
target=$(grep -E ' deployed [0-9a-f]{7} ' "$HISTORY" 2>/dev/null | tail -1 | sed -E 's/.*\(from ([0-9a-f]+)\)/\1/')
Expand All @@ -220,16 +285,19 @@ cmd_rollback() {
log "回滚:$IMAGE:$current → $IMAGE:$target"
switch_to "$target"
wait_healthy || die "$IMAGE:$target 没有就绪"
touch "$STATE/failed-$current" # 防止定时器马上又把刚回滚掉的版本部署回去
skip "$current" "被回滚" # 防止定时器马上又把刚回滚掉的版本部署回去
hold_main_unless "$target"
printf '%s rollback %s (from %s)\n' "$(date '+%F %T')" "$target" "$current" >> "$HISTORY"
log "✅ 已回滚到 $IMAGE:$target;$IMAGE:$current 被标记为失败,main 出现新提交前不会自动重新部署"
log "✅ 已回滚到 $IMAGE:$target;main 出现新提交前不会自动部署"
}

cmd_status() {
sync_mirror
local head; head=$(git -C "$MIRROR" rev-parse refs/heads/main)
local head requests; head=$(git -C "$MIRROR" rev-parse refs/heads/main)
echo "线上版本:$IMAGE:$(current_tag)"
echo "main 最新:${head:0:7} CI:$(ci_state "$head")$([ -f "$STATE/failed-${head:0:7}" ] && echo ' (已标记失败,不会自动部署)')"
echo "main 最新:${head:0:7} CI:$(ci_state "$head")$(skipped "${head:0:7}" && echo " (不自动部署:$(cut -d' ' -f3- "$STATE/skip-${head:0:7}"))")"
requests=$(queued_requests)
echo "排队请求:$([ -n "$requests" ] && awk '{printf "#%s → %s ", $1, substr($2,1,7)}' <<<"$requests" || echo 无)"
echo "定时器: $(systemctl --user is-active ash-cd.timer 2>/dev/null || true)"
echo "最近部署:"
tail -5 "$HISTORY" 2>/dev/null | sed 's/^/ /' || echo " (无)"
Expand Down
6 changes: 3 additions & 3 deletions deploy/cd/ash-cd.timer
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
[Unit]
Description=Poll main for deployable commits every 2 minutes
Description=Poll main for deployable commits every 30 seconds

[Timer]
OnBootSec=3min
OnUnitActiveSec=2min
RandomizedDelaySec=15s
OnUnitActiveSec=30s
RandomizedDelaySec=5s

[Install]
WantedBy=timers.target
20 changes: 15 additions & 5 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@

```
分支 / worktree ──push──▶ PR ──CI: check + docker──▶ 合并到 main ──CI 再跑一遍──▶ ash-cd 自动上线
GitHub: Actions → Deploy → Run workflow ──▶ ash-cd 手动上线
```

- **CI**:`.github/workflows/ci.yml`,GitHub Actions 托管 runner。每个 PR、main 的每次推送都跑两个检查:
- `check`:`npm ci` → lint(error 为 0)→ UI 测试 → 构建 → 服务端测试
- `docker`:构建镜像并真正启动容器,验证 `/healthz`、首页、`/docs` 为 200,`/api/skills` 未登录为 401
- **CD**:seed-SER9 上的 `ash-cd`(`deploy/cd/`),systemd 用户定时器每 2 分钟轮询 main。main 最新提交的 `check`、`docker` **都通过**才部署:
- **CD**:seed-SER9 上的 `ash-cd`(`deploy/cd/`),systemd 用户定时器每 30 秒拉取一次,两种触发:
- **自动**:main 最新提交的 `check`、`docker` **都通过**就部署。
- **手动**:GitHub 上 Actions → **Deploy** → Run workflow,填分支 / tag / 提交(默认 `main`)。workflow 跑在 GitHub 托管的 runner 上,只检查 CI、登记一条排队的部署请求,然后等 ash-cd 回报结果;run 的成败就是部署的成败。手动请求优先于自动部署;同时排了多个只做最新的。
部署流程:
备份 DB 与技能文件 → 从自己的镜像仓库 `git archive` → `docker build -t ash:<sha>` → compose 只换 tag → 冒烟 → 失败自动切回上一版本。
冒烟项:本机 `/healthz`、首页、`/docs` 为 200,`/api/skills` 为 401,**容器里的技能数与部署前一致**,日志无迁移/未捕获错误;公网 `/healthz` 不通只告警(隧道是独立设施)。
部署进度回写到仓库的 Deployments(environment: `production`);成功后只保留最近 10 个 `ash` 镜像。
Expand All @@ -23,23 +27,29 @@

```bash
git push -u origin <分支> && gh pr create --fill # CI 在 PR 上跑
gh pr merge --squash --delete-branch # 绿了就合并;约 2 分钟 + CI 时间后自动上线
gh pr merge --squash --delete-branch # 绿了就合并;main 上 CI 通过后约 30 秒自动上线
gh workflow run deploy.yml -f ref=<分支|tag|sha> # 等同于在 GitHub 上点 Deploy;可加 -f skip_ci_check=true
~/services/skillhub-cd/bin/ash-cd status # 线上版本 / main 最新提交及 CI 状态 / 最近部署
journalctl --user -u ash-cd -f # 实时日志
```

### 出问题时

**回滚首选 GitHub 上的 Deploy 按钮**:`ref` 填要回到的旧提交(建立 CI 之前的版本没有检查结果,勾选 `skip_ci_check`)。
手动上线的不是 main 最新提交时,自动部署会**暂停到 main 出现新提交为止**,不会半分钟后又被覆盖回去;要恢复,再用 Deploy 上线 `main` 即可。

服务器上的等价命令:

```bash
ash-cd rollback # 撤销最近一次部署(切回它之前的版本),并把被撤下的提交标记为失败
ash-cd rollback <tag> # 切到指定版本(镜像还在时)
ash-cd deploy <sha> # 手动部署某个 CI 通过的提交;--force 跳过 CI 检查(紧急修复)
systemctl --user stop ash-cd.timer # 暂停自动部署;start 恢复
```

- 部署失败或被回滚的提交记在 `~/services/skillhub-cd/state/failed-<sha>`,**不会自动重试**;修复后推新提交即可。
- 部署失败、被回滚、或因手动上线而暂停的 main 提交记在 `~/services/skillhub-cd/state/skip-<sha>`(内容是原因),**不会自动部署**;推新提交、或用 Deploy 按钮明确上线它即可。
- 自动回滚只切代码。新版本做过不兼容的迁移时,还要用 `backup-before-<sha>.db` / `backup-files-before-<sha>.tgz` 恢复数据(见下文「回滚」)。
- 需要人工步骤的发布(例如「上线多用户鉴权」那样要先改 compose 环境变量、再认领数据):先 `systemctl --user stop ash-cd.timer`,按手动流程做完,再 `start`。
- 需要人工步骤的发布(例如「上线多用户鉴权」那样要先改 compose 环境变量、再认领数据):先 `systemctl --user stop ash-cd.timer`,按手动流程做完,再 `start`。定时器停着时 Deploy 按钮会在 20 分钟后以超时失败结束。

### 安装 / 更新 ash-cd(服务器上一次性)

Expand All @@ -49,7 +59,7 @@ deploy/cd/install.sh # 复制脚本到 ~/services/skillhub-cd/bin/ash-cd,

脚本是**复制**过去固定下来的:合并到 main 的 `deploy/cd/` 改动不会自动改变部署逻辑,要在服务器上重跑 `install.sh`。
前提:用户在 docker 组、`gh auth status` 已登录(用来读 CI 结果、写 Deployments)、`loginctl` linger 已开启。
在预发环境演练时,可以用 `ASH_CD_SERVICES` / `ASH_CD_CONTAINER` / `ASH_CD_PROJECT` / `ASH_CD_IMAGE` / `ASH_CD_LOCAL_URL` 指向另一套容器,`ASH_CD_GH_DEPLOYMENTS=0` 不写 GitHub。
在预发环境演练时,可以用 `ASH_CD_SERVICES` / `ASH_CD_CONTAINER` / `ASH_CD_PROJECT` / `ASH_CD_IMAGE` / `ASH_CD_LOCAL_URL` 指向另一套容器,`ASH_CD_GH_ENV=staging` 让部署记录与手动请求都落在 GitHub 的 staging 环境(与生产互不干扰),`ASH_CD_GH_DEPLOYMENTS=0` 则完全不读写 GitHub。

## 环境

Expand Down
Loading