fix: restore liveness and database readiness probes

[seonghobae]

Current authority — 2026-09-08

• protected root: develop@042b0c70531b229af3acbd0421a2f23098d848b3
• direct stacked base: fix(deploy): 기존 runtime 쌍의 조건부 복구 검증 #1588 codex/deploy-runtime-rollback@c943c0277f2a2449e51cca8e0417cdcafaf7122d
• exact head: 2c126127eb1a2e9bd77401fb940fac82e140cfc6
• lifecycle: Draft / mergeable / prerequisite + hosted admission pending
• effective delta against the direct stack parent: 7 files — AGENTS.md, backend/main.py, backend/tests/test_release_manifest_digests.py, backend/tests/test_runtime_health.py, docs/doctoring/runtime-image-boundary-verification.md, docs/product-technical-gap-baseline.md, k8s/backend-deployment.yaml

This PR remains a dependent deployment/readiness slice. It does not replace #126, #1586, or #1588, and their broader deployment/release authority is not transferred here.

RED → causal repair

The inherited runtime path lacked explicit process liveness and database readiness endpoints. The repair adds:

• /healthz: process-only liveness, no database/provider/customer-system access, Cache-Control: no-store;
• /readyz: SELECT 1 against both primary and readonly pools, sanitized 503 {"status":"unavailable"} on supported SQLAlchemy/native PostgreSQL/OS/timeout failures, and no raw exception detail in the response;
• cancellation propagation rather than converting cancellation into readiness failure;
• deterministic connection-lifecycle regressions for healthy, primary-failure, readonly-failure, acquisition-failure, timeout/OS/native PostgreSQL failure and cancellation paths;
• rendered backend deployment readiness wired to /readyz on port 8000 while explicitly not reusing dependency readiness as liveness.

A real isolated PostgreSQL check reproduced asyncpg.InvalidCatalogNameError before query execution when the selected database was absent. The native PostgresError boundary was therefore added rather than assuming every connection-establishment error is wrapped by SQLAlchemy. Doctoring records the isolation limits and the fact that both pools were pointed at one isolated server; this is not production-cluster or replica-readiness proof.

Current verification boundary

The exact current head has no repository-owned pull-request workflow runs. This is expected from the still-live base filters in Naruon's repo-local PR workflows: stacked feature-branch targets are not admitted until canonical prerequisite #1587 lands in protected develop. Commit statuses from review transports are not a substitute for hosted required checks, and there is currently no formal review submission on this exact head.

The focused local/isolated evidence in this lineage is development evidence only. It does not transfer predecessor GREEN to 2c126127..., does not prove real AKS endpoint removal/recovery, startup/liveness behavior, deployment serialization/rollback, or release eligibility.

Admission and succession

1. fix(ci): run repo-local validation on stacked PR bases #1587 must normally integrate into protected develop; no bypass or gate weakening.
2. fix(deploy): 기존 runtime 쌍의 조건부 복구 검증 #1588/fix(deploy): 게시 digest를 독립 runtime 배포까지 전달 #1586 and this descendant must be synchronized by ordinary non-force restack/retarget while preserving all valid deployment delta.
3. The unchanged restacked head must receive repository-owned Application/Bandit/Docker/Security/Semgrep/CodeQL plus every then-live central required workflow and reach terminal success.
4. Hosted isolated-PostgreSQL acceptance must re-establish dual-pool readiness and cleanup; actual cluster validation must establish endpoint removal/recovery and appropriate liveness/startup behavior before deployment completion is claimed.
5. Valid unresolved finding/thread count must be zero and a qualifying post-last-push independent approval must exist.
6. Only after protected integration may this readiness slice be considered for release succession; it is not sufficient evidence to close 릴리스 거버넌스와 CI/CD 배포 경로 정비 #126 or the parent deployment stack.

No self-approval, bypass/admin merge, force-push, destructive rebase, dummy/no-op requeue, predecessor evidence transfer, central workflow copy, synthetic status, or gate weakening.

[coderabbitai]

Warning

Review limit reached

Next included review available in 51 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: a89ca82d-596c-4ef6-817e-f26ec0e19bbb

📥 Commits

Reviewing files that changed from the base of the PR and between c943c02 and 2c12612.

📒 Files selected for processing (7)

• AGENTS.md
• backend/main.py
• backend/tests/test_release_manifest_digests.py
• backend/tests/test_runtime_health.py
• docs/doctoring/runtime-image-boundary-verification.md
• docs/product-technical-gap-baseline.md
• k8s/backend-deployment.yaml

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[seonghobae]

@coderabbitai review

Please review the current exact head, including the liveness/readiness contract, connection cleanup, sanitized failure response, and preserved AGENTS.md operating guidance.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '== Git state and effective diff =='
git rev-parse HEAD
git merge-base HEAD c943c0277f2a2449e51cca8e0417cdcafaf7122d
git diff --name-status c943c0277f2a2449e51cca8e0417cdcafaf7122d...HEAD
git diff --check c943c0277f2a2449e51cca8e0417cdcafaf7122d...HEAD
git diff -- AGENTS.md c943c0277f2a2449e51cca8e0417cdcafaf7122d...HEAD || true

echo '== Changed source and test diff =='
git diff --unified=80 c943c0277f2a2449e51cca8e0417cdcafaf7122d...HEAD -- backend/main.py backend/tests/test_runtime_health.py

echo '== Source and test outlines =='
ast-grep outline backend/main.py --items all
ast-grep outline backend/tests/test_runtime_health.py --items all

echo '== Health/database symbols and route registrations =='
rg -n -C 5 --glob 'backend/**/*.py' \
  '@(?:app|router)\.(?:get|api_route)\(\s*["'\'']/(?:healthz|readyz)|\b(?:healthz|readyz)\b|\b(?:engine|readonly_engine|read_only_engine)\.connect\s*\(|\bSQLAlchemyError\b' \
  backend || true

echo '== Repository instructions =='
fd -HI '^AGENTS\.md$' . -x sh -c 'echo "--- $1"; sed -n "1,260p" "$1"' sh {} \;

Length of output: 38802

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '== Commit and AGENTS.md equivalence =='
git show -s --format='HEAD=%H%nparents=%P%nsubject=%s' HEAD
git diff --exit-code c943c0277f2a2449e51cca8e0417cdcafaf7122d HEAD -- AGENTS.md \
  && echo 'AGENTS.md is identical to the stack parent.'
git diff --stat c943c0277f2a2449e51cca8e0417cdcafaf7122d...HEAD

echo '== Database session module outline =='
ast-grep outline backend/db/session.py --items all
echo '== Database session implementation =='
sed -n '1,280p' backend/db/session.py

echo '== Python/project test configuration candidates =='
fd -HI -t f '^(pyproject\.toml|pytest\.ini|setup\.cfg|tox\.ini|requirements.*\.txt)$' . \
  -x sh -c 'echo "--- $1"; sed -n "1,240p" "$1"' sh {} \;

echo '== Existing async engine connection patterns =='
rg -n -C 4 --glob 'backend/**/*.py' \
  '(?:async with .*\.connect\(\)|await .*\.connect\(\)|create_async_engine|readonly_engine)' backend

echo '== Test conventions for ASGITransport and SQLAlchemy connection failures =='
rg -n -C 5 --glob 'backend/tests/**/*.py' \
  '(?:ASGITransport|AsyncClient\(|OperationalError\(|SQLAlchemyError|asynccontextmanager)' backend/tests || true

echo '== Route/middleware ordering and full probe test with line numbers =='
nl -ba backend/main.py | sed -n '175,280p'
nl -ba backend/tests/test_runtime_health.py | sed -n '1,150p'

Length of output: 123469

---

@seonghobae Reviewed the current checked-out head: 7bffb35732831b065d033f659d9eace1cf8be999.

I found no actionable source-level findings.

• /healthz is process-only. It does not access either database engine.
• /readyz checks engine and readonly_engine with SELECT 1.
• Each acquired connection is inside an async with boundary.
• SQLAlchemy, OS, and timeout failures return the sanitized 503 {"status":"unavailable"} response.
• The probe responses set Cache-Control: no-store.
• The regression test covers healthy readiness, primary failure, readonly failure, liveness without database access, sanitized output, and connection cleanup.
• AGENTS.md is identical to the stack parent. The effective diff is limited to backend/main.py and backend/tests/test_runtime_health.py.

The reviewed head differs from the SHA in the PR description. The stated SHA is f0a957..., but the current checked-out head is 7bffb357....

I did not run tests or linters because the sandbox does not execute repository code. Hosted validation remains required before merge.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[seonghobae]

Visual inspection receipt: inspected the rendered AGENTS.md Phase 10 readiness paragraph in Microsoft Edge at exact head a5859c4, desktop viewport 1897x949. The new paragraph, inline driver names, and evidence path are readable with normal wrapping and no visible overlap or clipping. Screenshot was displayed inline in the working task; no downloadable screenshot artifact was produced. This is documentation-only visual evidence, not product UI, mobile, authenticated journey, or deployment validation. Current head has successful CodeRabbit and Devin status contexts but no approval reviews or required CI check-run evidence in the queried rollup. Protected integration remains pending; no merge or deployment was performed.

[seonghobae]

Regression verification for 706cd74: the native driver/OS/timeout cases now fail before connection acquisition for both primary and read-only pools. Existing SQLAlchemy query-failure coverage remains. The read-only acquisition-failure cases assert that the preceding primary connection was returned; primary acquisition failure asserts no connection was acquired. Command from backend/: python -m pytest --noconftest tests/test_runtime_health.py -q -W error. Result: 10 passed in 165.28s, followed by confirmed process exit 0 (session 18443). Ruff and git diff --check passed. Only this tested file was committed; normal push and matching remote head were verified. Current status contexts are not a substitute for required CI or approval; no protected merge/deployment claim.

[seonghobae]

Visual inspection: rendered doctoring/runtime-image-boundary-verification.md at exact 2c12612 in the in-app browser, 1265x712 desktop screenshot. Deployment probe wiring follow-up heading, both evidence/limitations paragraphs and Kubernetes citation are readable without visible overlap or clipping. Removed the empty references heading. Screenshot was displayed inline; no separate image artifact. This inspection covers that document section only, not the Gap/AGENTS changes, mobile UI, or live kubelet behavior. PR description now separates current evidence from the preserved historical predecessor description.

[seonghobae]

Exact-head combined verification: at clean checkout 2c12612 (matching current PR head), ran both backend/tests/test_runtime_health.py and backend/tests/test_release_manifest_digests.py together with pytest --noconftest -q -W error from backend. Result: 40 passed in 12.08s, process exit 0. This freshly verifies the combined probe response/cancellation contracts and release-manifest rendering contract at the current head, rather than combining predecessor results. Runtime tests use substituted database engines; this run is not a fresh real-PostgreSQL smoke, kubelet/EndpointSlice observation, registry publication, deployment, or protected-check acceptance. Those broader gates remain separate.

[seonghobae]

Deployment preflight evidence (read-only, 2026-09-08): local kubectl has no configured contexts. Repository Actions secret metadata and repository-accessible organization Actions secret metadata both report no AKS_KUBECONFIG entry; no secret values were retrieved. At head 2c12612, docker-publish.yml explicitly skips the AKS call when that secret is absent, and deploy.yml requires it. Therefore PyPI/crates publication credentials do not establish AKS deployment readiness. Local cached Naruon images also predate this head and were not treated as release artifacts. Actual cluster acceptance needs the intended cluster credential configured through the existing protected path, plus the still-required exact-head checks and published image digests. No cluster configuration, workload, or secret was changed.

[seonghobae]

@coderabbitai review

Please review current head 2c12612, particularly native asyncpg acquisition failures, cancellation propagation/connection return, and the rendered Kubernetes readiness probe. Fresh combined tests: 40 passed, exit 0 (receipt above). Current-head Actions runs API returns total_count=0; CodeRabbit/Devin SUCCESS statuses are not a substitute for required CI or formal current-head approval. Preserve the stacked base codex/deploy-runtime-rollback; missing stacked validation is tracked by #1587, not grounds to drop valid changes or bypass checks.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base='c943c0277f2a2449e51cca8e0417cdcafaf7122d'
head='2c126127eb1a2e9bd77401fb940fac82e140cfc6'

echo '== Revision and effective diff =='
printf 'checked_out='; git rev-parse HEAD
git show -s --format='requested_head=%H%nparents=%P%nsubject=%s' "$head"
printf 'merge_base='; git merge-base "$base" "$head"
git diff --name-status "$base...$head"
git diff --check "$base...$head"
echo '== AGENTS.md delta against stacked base =='
git diff --exit-code "$base" "$head" -- AGENTS.md && echo 'AGENTS.md is identical to the stacked base.'

echo '== Candidate files =='
fd -HI -t f . | rg '(^|/)(main\.py|session\.py|test_runtime_health\.py|test_release_manifest_digests\.py|AGENTS\.md|.*\.(ya?ml|json|j2|tmpl))$' | sort

echo '== Python outlines =='
ast-grep outline backend/main.py --items all
ast-grep outline backend/db/session.py --items all
ast-grep outline backend/tests/test_runtime_health.py --items all
ast-grep outline backend/tests/test_release_manifest_digests.py --items all

echo '== Probe, connection, cancellation, and asyncpg references =='
rg -n -C 8 --glob '*.py' \
  'healthz|readyz|readonly_engine|engine\.connect\s*\(|SELECT 1|CancelledError|asyncpg|TimeoutError|OSError|SQLAlchemyError|OperationalError' \
  backend

echo '== Changed Python source and tests =='
git diff --unified=100 "$base...$head" -- \
  backend/main.py \
  backend/db/session.py \
  backend/tests/test_runtime_health.py \
  backend/tests/test_release_manifest_digests.py

echo '== Deployment and workflow readiness-probe references =='
rg -n -C 8 --glob '*.{yaml,yml,json,j2,tmpl}' --glob '!**/.git/**' \
  'readinessProbe|livenessProbe|healthz|readyz|containerPort|AKS_KUBECONFIG' .

echo '== Relevant manifest and workflow diffs =='
git diff --unified=100 "$base...$head" -- \
  ':!backend/main.py' ':!backend/db/session.py' \
  ':!backend/tests/test_runtime_health.py' ':!backend/tests/test_release_manifest_digests.py'

echo '== Current GitHub PR metadata and check rollup =='
gh pr view 1597 --repo ContextualWisdomLab/naruon \
  --json number,isDraft,headRefName,headRefOid,baseRefName,mergeStateStatus,reviewDecision,reviews,statusCheckRollup \
  --jq '{number,isDraft,headRefName,headRefOid,baseRefName,mergeStateStatus,reviewDecision,reviews:[.reviews[]|{author:.author.login,state,commit_id}],statusCheckRollup:[.statusCheckRollup[]|{name,conclusion,status}]}'

Length of output: 180010

---

@seonghobae Reviewed pull request #1597 at exact head 2c126127eb1a2e9bd77401fb940fac82e140cfc6.

I found no actionable source-level findings.

• database_readiness catches PostgresError. This covers native asyncpg connection-establishment errors such as InvalidCatalogNameError.
• database_readiness does not catch CancelledError. Cancellation propagates to the caller.
• Each acquired database connection is inside async with database_engine.connect(). The connection returns when query execution fails or cancellation occurs.
• backend/tests/test_runtime_health.py covers primary and read-only acquisition failure, query failure, cancellation before acquisition, cancellation after acquisition, and sanitized 503 responses.
• k8s/backend-deployment.yaml configures readiness as GET /readyz on port 8000.
• backend/tests/test_release_manifest_digests.py renders the deployment manifest and verifies that readiness uses /readyz and liveness does not use /readyz.
• The Phase 10 readiness guidance remains in AGENTS.md. The file is intentionally changed from the stacked base by this PR.

GitHub reports base codex/deploy-runtime-rollback, no formal approval, and no usable current-head check-run evidence. The reported local test receipt does not replace those merge gates. The missing stacked validation remains tracked by #1587. No reviewed change requires dropping the valid stack or bypassing checks.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[seonghobae]

Current-head verification — 2026-09-08

• exact head: 2c126127eb1a2e9bd77401fb940fac82e140cfc6
• uv run --project backend --frozen python -m pytest -q backend/tests/test_release_manifest_digests.py: 26 passed
• git diff --check: passed
• The manifest evidence confirms /readyz wiring in the rendered release manifest; it does not prove kubelet behavior, startup timing, live deployment, or production recovery.

PR remains Draft with no qualifying review. No Ready toggle, merge bypass, or deployment was performed.