[PetrefiedThunder]

Summary

Diligence E07 (backend readiness): make /readiness fail promotion when essential background units are down, while keeping /health as a separate liveness probe.

Specs: /workspace/regengine-diligence/reliability-E07-E09-decision-2026-10-28.md and the E07 section of devops-plan-E07-E09-E13-2026-10-28.md.

No Railway deploy / redeploy / rollback (including 1a6d30d0). Codex will review.

Requirement → code → tests

Requirement	Code	Tests
Readiness 503 when ingestion worker reports error (startup failure)	server/main.py _readiness_checks gate: worker_essential_error	test_worker_start_failure_returns_503
Readiness 503 when worker dies after start (thread not alive; detected on next probe, well under 2 min)	existing thread is_alive() check + new gate	test_worker_dead_thread_returns_503
Embedded scheduler 503 only when SCHEDULER_EMBEDDED enables it	gate uses embedded_scheduler_enabled()	test_scheduler_start_failure_returns_503_when_embedded, test_scheduler_dead_thread_returns_503_when_embedded
Scheduler disabled / not embedded stays non-blocking	unchanged status reporting; gate skips when not enabled	test_scheduler_disabled_is_healthy, test_scheduler_*_when_not_embedded
Neo4j stays non-blocking	unchanged; not in essential gate	test_e07_neo4j_error_stays_non_blocking (+ existing neo4j tests)
Rate limiter stays non-blocking	unchanged; not in essential gate	test_e07_rate_limiter_error_alone_stays_non_blocking
Returns 200 after restart (healthy state again)	same gate	test_e07_readiness_recovers_to_200_after_restart
Liveness separate / unaffected	/health untouched; readiness-only gate	test_e07_liveness_unaffected_when_essential_unit_fails (+ existing stalled-probe liveness test)

Files

• server/main.py — readiness gate + comments
• tests/test_server_readiness_probe.py — contract updates + E07 cases

Overlaps avoided

Open red PRs were inventoried first. None touch server/main.py or tests/test_server_readiness_probe.py. Stayed off the main-red secrets baseline fix (#3703 / #3700).

Local verification

venv/bin/python -m pytest tests/test_server_readiness_probe.py -q --no-cov
# 33 passed

Notes / ambiguities

• “Within 2 minutes” after post-start death is satisfied by the existing per-probe thread.is_alive() / thread_alive checks (immediate on the next /readiness poll). No new heartbeat window was added.
• Staging kill-test + alert delivery remain Reliability-owned (out of this PR).

---

Note

Medium Risk
Changes deploy promotion behavior when ingestion or embedded scheduler is down; well-scoped to the readiness gate with broad test coverage, but misconfiguration could block healthy deploys or delay catching worker issues until promotion.

Overview
Implements diligence E07 by tightening /readiness (Railway’s deploy healthcheck) so broken background work cannot promote, while /health stays a separate liveness probe.

Essential units now return HTTP 503: ingestion worker startup failure or a dead worker thread always fails readiness. Embedded scheduler startup failure or a dead scheduler thread fails readiness only when SCHEDULER_EMBEDDED enables it via embedded_scheduler_enabled(); disabled / not_configured scheduler states stay non-blocking.

Still non-blocking alone: Neo4j and rate-limiter problems remain in the checks payload (200 degraded when Postgres/Redis/essential units are OK). Postgres and strict-runtime Redis rules are unchanged.

Tests in test_server_readiness_probe.py were updated for the new contract and add E07 cases (Neo4j/rate limiter alone, recovery after restart, liveness vs readiness).

^{Reviewed by Cursor Bugbot for commit 56c6238. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
regengine	Ready	Preview	Oct 28, 2026 12:02pm UTC

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-28T12:02:29.394300Z	56c6238	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: regengine/RegEngine
• Branch: claude/e07-backend-readiness
• File: .coderabbit.yaml
• Commit: 56c6238
• Layers: Repository YAML
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 107dcf55-c947-4c27-b0f2-7436ca8aacef

Commits

Reviewing files that changed from the base of the PR and between 8548c09 and 56c6238.

Files selected for processing: 2

• server/main.py
• tests/test_server_readiness_probe.py

Limit details: You’ve used all 10 included reviews currently available.

Walkthrough

The readiness status calculation now treats ingestion-worker errors as fatal and treats embedded-scheduler errors as fatal only when embedded scheduling is enabled. PostgreSQL remains required, and Redis remains required in strict-auth runtimes. Neo4j and rate-limiter errors remain non-blocking. Tests cover failure outcomes, recovery, and the separation between /readiness and /health.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: Minimal · up to 56c62

Readiness now blocks on essential background failures. No merge-blocking issue was identified.

Security architecture risk: Low · up to 56c62

The change strengthens readiness without expanding access or weakening authentication controls. Remaining uncertainty is operational: deployed probe wiring and actual restart recovery were not demonstrated, and thread liveness does not establish successful background execution.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The direct changed outcome is the readiness result for the serving application instance, potentially affecting traffic eligibility or promotion wherever that result is consumed. The gate grants no new tenant, data-store, credential, or infrastructure authority.

Security Findings and Attack Paths

• observed — The routed test ranges register routes on local FastAPI applications accessed through TestClient. They do not introduce production network entrypoints or expand attacker-controlled reachability.

Trust Boundaries and Controls

• observed — The readiness endpoint's public classification is pre-existing. The change adds failure conditions rather than relaxing access controls, and preserves Redis readiness enforcement for strict-auth runtimes.

Resilience and Maintainability Implications

• inferred — Blocking readiness on recorded startup failure or essential-thread death improves containment of instances unable to run background work. This guarantee remains bounded to recorded failure and liveness signals; it does not establish successful compliance-job execution.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly identifies the main change: readiness now fails for essential worker and scheduler errors. It is concise and specific.
Description check		The description covers the required sections and provides clear requirements, implementation details, tests, verification results, and scope notes. Template checkboxes are not marked, but the descript…
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description covers the required sections and provides clear requirements, implementation details, tests, verification results, and scope notes. Template checkboxes are not marked, but the description is otherwise complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files.

Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Usage-based review receipt

• Mode: Continue automatically
• Reviewed files: 2
• Waived: $0.50 (charged $0.00)
• View usage details

Note

This review exceeded your plan’s limits and used usage-based reviews—free during trial, billed after paid activation unless disabled. Manage usage-based reviews.

_{Comment @coderabbitai help to get the list of available commands.}

[PetrefiedThunder]

Codex review — RegEngine #3704 (E07 readiness)

Commit: 56c6238daf14db7d838d580fc8c942d4c7552821
Spec: reliability-E07-E09-decision-2026-10-28.md (E07)
Scope: server/main.py (_readiness_checks), tests/test_server_readiness_probe.py

Verdict

PASS — readiness gating matches the E07 essential-unit contract; /health is untouched and remains a separate liveness probe; focused tests are data-free and cover the listed behaviors.

Verify map

Requirement	Result
Worker startup failure → 503	Yes — worker_essential_error + test_worker_start_failure_returns_503
Worker dead thread → 503	Yes — thread.is_alive() + test_worker_dead_thread_returns_503
Scheduler error → 503 only when embedded	Yes — gate uses embedded_scheduler_enabled(); embedded vs not-embedded tests for start failure and dead thread
Neo4j / rate limiter non-blocking	Yes — informational only; test_e07_neo4j_error_stays_non_blocking, test_e07_rate_limiter_error_alone_stays_non_blocking
Recovers to 200 after healthy restart	Yes — test_e07_readiness_recovers_to_200_after_restart (failed then healthy app.state)
/health liveness unchanged	Yes — PR does not modify services/ingestion/app/routes_health_metrics.py (sole GET /health owner); no worker/scheduler refs there
Spec match	Yes — essential units fail readiness; disabled/not_configured/neo4j/rate limiter stay non-blocking; liveness kept separate (Railway restart-loop concern)
Thread liveness races	No unsafe race — local thread capture + is_alive() / thread_alive property; post-sample death is caught on the next probe (within the 2‑minute staging window)
Data-free tests	Yes — minimal FastAPI apps, monkeypatch, stubs; no live DB/Redis/Neo4j

Local checks

• pytest tests/test_server_readiness_probe.py -q → 33 passed
• Head re-verified: 56c6238daf14db7d838d580fc8c942d4c7552821

Notes (non-blocking)

1. P2 / nit: test_e07_liveness_unaffected_when_essential_unit_fails registers a fabricated /health that always returns alive. Source inspection confirms the real handler is unchanged and decoupled; mounting the real health router on the minimal app would harden the regression.
2. Out of scope for this PR: E07 staging acceptance (inject/kill units, alert timing, deploy non-promotion) remains Reliability-owned per the spec. Unit recovery is state-swap, not a process restart.
3. CI at review time still had pending/failing gate jobs (Head SHA Check Guard, PR Blocker Checklist); not treated as a code-contract fail.

No merge performed.

REVIEW: PASS codex 56c6238

[github-actions]

PR Blocker Checklist

Head SHA: 56c6238daf14db7d838d580fc8c942d4c7552821

Status: Ready

• All review threads resolved
• No reviews requesting changes
• No blocked review engines

[PetrefiedThunder]

REVIEW: PASS pi 56c6238

E07 readiness: worker error always 503; scheduler error 503 only when SCHEDULER_EMBEDDED enables it; neo4j/rate_limiter stay non-blocking; /health unchanged. Tests green (tests/test_server_readiness_probe.py). Scope matches diligence; no secrets/tenant issues.