Repository navigation
📍 Project Roadmap #17
Description
Tracking issue for the procurement-rag build. Phases follow docs/IMPLEMENTATION_PLAN.md; production-readiness items are tracked in the section below.
Build Phases
| Phase | Description | Issues | Status |
|---|---|---|---|
| 1 | Foundation & Ingestion | #18 #16 #1 #2 #3 #4 | ✅ Done |
| 2 | Retrieval | #5 #6 #7 #8 | ✅ Done |
| 3 | Pipeline & Webhooks | #9 #10 | ✅ Done |
| 4 | API & Frontend | #11 #12 | ✅ Done |
| 5 | Evaluation | #13 | ✅ Done |
| 6 | Infra & CI | #14 #15 | ✅ Done |
| — | Integration testing (Groups 1–7) | #21 | ✅ Done |
Production Readiness
| # | Description | Priority | Status |
|---|---|---|---|
| #28 | Auth / access control (Option B — ERPNext OAuth2 + JWT) | 🔴 Blocker | ✅ Done — PR #32 merged |
| #29 | End-to-end verification against live ERPNext | 🟡 High | ✅ Done — PR #35 merged |
| #30 | Production secrets & infra hardening (Langfuse keys, port bindings, nginx) | 🟡 High | ✅ Done — PR #36 merged |
| #31 | Wire OPENAI_API_KEY into GitHub Actions CI evaluate job |
🟢 Low | ✅ Done — secret added, confirmed green on run 28851595410 |
| #41 | Langfuse UI access on AWS (SSH tunnel, loopback-bound port) | 🟢 Low | ✅ Done — PR #42 merged |
| #43 | Fix ruff lint errors breaking CI (tests/test_integration.py) |
🟢 Low | ✅ Done — PR #46 merged |
| — | Qdrant dashboard UI access on AWS (SSH tunnel, mirrors #41) | 🟢 Low | ✅ Done — PR #44 merged |
| — | AWS EC2 deployment guide (Option B, single instance) | 🟢 Low | ✅ Done — PR #37 merged |
| — | OAuth sub/email userinfo lookup fix |
🟡 High | ✅ Done — PR #38 merged |
| — | Docker Compose restart policies (survive instance stop/reboot) | 🟢 Low | ✅ Done — PR #39 merged |
| — | Nginx domain config via envsubst templates (fixes recurring git pull-revert crash-loop) |
🟡 High | ✅ Done — PR #40 merged |
| — | Nginx default_server fix (stray base-image default.conf welcome page catching unmatched Host headers) |
🟢 Low | ✅ Done — commit a6c42ab |
| — | OAuth callback duplicate-request tolerance (ERPNext consent page double-click bug) | 🟡 High | ✅ Done — commit 5884a48 |
| — | App logging configuration (INFO progress logs were silently swallowed, no logging.basicConfig()) |
🟢 Low | ✅ Done — commit bc3c5f3 |
| #139 | Migrate the RAG stack to a fresh EC2 box + full green-field rehearsal of docs/CLIENT_DEPLOYMENT_RUNBOOK.md; decommission the old box |
🟡 High | ✅ Done — new t3.large (i-02f0a82e928d43a0b old box terminated, EIP moved, AMI kept per #143); Phase 5 validated (smoke tests, evaluate.py --no-judge 53/53, latency/cost baselined). Runbook + benchmark fixes on branches phase-141-runbook-fixes (#141) / phase-147-benchmarks-t3large (#147). Rehearsal findings → #144–#146, #148–#149. |
Known Issues
| # | Description | Severity | Status |
|---|---|---|---|
| #63 | SSRF + ERPNext credential leak via unvalidated file_url in erpnext_client.get_file_content |
🔴 High | Open — fix deferred |
| #64 | OAuth /auth/callback replay-cache skips code validation for 60s window |
🟡 Medium | Open — fix deferred |
| #65 | Hardcoded Postgres credentials in docker-compose.yml (langfuse/langfuse) |
🟢 Low | Open — fix deferred |
Future Enhancements
| # | Description | Area | Priority |
|---|---|---|---|
| #33 | Persist JWT across Streamlit page reloads (cookie/localStorage) | UX | 🟡 High |
| #34 | Conversational context — resolve pronoun references across turns | UX / Pipeline | 🟡 High |
| — | Mesh VPN (Tailscale) for team-wide Langfuse/Qdrant access, replacing per-session SSH tunnels — documented in docs/DEPLOYMENT.md § Future Enhancements, not implemented |
Observability | 🟢 Low |
| — | ✅ Step 17 — ERPNext Desk E2E test suite (Playwright) — browser-driven tests that catch webhook bugs REST-API integration tests miss (3 bugs found historically: webhooks not configured, not firing, on_update_after_submit silent). Spec in docs/IMPLEMENTATION_PLAN.md Step 17; strategy in docs/ARCHITECTURE.md § Testing Strategy. Done — PR #93 merged (5872a37). |
Testing | ✅ Done |
| — | Re-index submitted PO on post-submit edits (on_update_after_submit) — blocked by Frappe 15 platform limitation; see docs/DEPLOYMENT.md § Future Enhancements |
Ingestion | 🟢 Low |
| — | Evaluate switching the RAGAS evaluate CI job from per-push to a nightly schedule, to catch OpenAI-side model drift independent of code changes (discussed alongside #31, deferred) |
Evaluation / CI | 🟢 Low |
| #49 | Act on RAGAS results — threshold gating (fail CI on regression) + $GITHUB_STEP_SUMMARY visibility, using GitHub's native failed-run notification |
Evaluation / CI | 🟡 High |
| #50 | Production quality monitoring — Langfuse scoring (user feedback + LLM-as-judge); see docs/ARCHITECTURE.md § Observability |
Evaluation / Observability | 🟢 Low |
| #90 | Make LLM/embedding provider swappable (not just model version — that part is done, PR #61) — hand-rolled ChatModel/EmbeddingModel adapter seam; design documented in docs/ARCHITECTURE.md § Model Configuration. LangChain considered and rejected (dependency/version-risk not justified with no swap currently planned). (Split from #51, closed.) |
Pipeline / Config | 🟢 Low |
| #97 | Migrate lexical search from in-memory BM25 to Qdrant native sparse vectors (sparse vector field + idf modifier, native RRF via query_points/prefetch), replacing the manual BM25 index/fusion in retrieval/hybrid_search.py. Rationale for staying on Qdrant vs. Elasticsearch/OpenSearch in docs/adr/0001-qdrant-bm25-over-elasticsearch-opensearch.md. Not urgent — deferred until corpus size or the BM25 metadata-filter gap actually bites. |
Retrieval | 🟢 Low |
| #98 | Apply metadata filter to the BM25 leg before RRF fusion — today only Qdrant's leg is filtered (vector_store.py:85-102), so filter_conditions is a soft rank bias, not a hard constraint, since an unfiltered BM25 match can still surface via hybrid_search.py:49-77. Lightweight, standalone fix (~15 lines, filter ranked_indices using Qdrant's exact-match semantics) distinct from #97's larger migration, which would also subsume this. |
Retrieval | 🟢 Low |
| #101 | Evaluate switching EMBEDDING_MODEL from OpenAI's text-embedding-3-small to a local sentence-transformers model, mirroring the reranker's existing local-model pattern (retrieval/reranker.py). Motivated by per-token embedding cost, contract data leaving the network for embedding calls, and external-API latency in the ingest/query hot paths. First concrete consumer of #90's adapter seam, if picked up. |
Ingestion / Pipeline | 🟢 Low |
| #100 | Switch dependency management from flat requirements.txt + plain pip to uv — move deps into pyproject.toml [project.dependencies] with a committed uv.lock, swap ci.yml/evaluate.yml to astral-sh/setup-uv + uv sync. Dev-ergonomics/CI-speed improvement, not a correctness fix. |
Infra / CI | 🟢 Low |
Auth strategy documented in docs/DEPLOYMENT.md — Option B implemented and verified (PR #32).
E2e verification completed against local ERPNext (prag01.test) — PR #35 merged.
AWS EC2 deployment guide (Option B, single-instance topology) added to docs/DEPLOYMENT.md and verified end-to-end on a live instance (procurement-rag.22logic.com) — PR #37 merged. Along the way this also surfaced and fixed a latent OAuth login bug (PR #38) and added Docker Compose restart policies so the stack survives instance stop/reboot (PR #39).
Follow-up hardening after the initial AWS deploy: nginx's manual domain substitution was replaced with an envsubst-on-templates mechanism driven by FRONTEND_DOMAIN/API_DOMAIN in .env, immune to git pull reverting the config (PR #40); a default_server fix stopped stray bot traffic from hitting the base image's welcome page; a duplicate-OAuth-callback tolerance fix resolved a reproducible ERPNext consent-page double-click bug; and app logging was fixed so long-running operations like full ingest show live progress instead of appearing to hang.
#139 rehearsed the whole client-deployment runbook green-field on a new EC2 t3.large: fresh erp.22logic.com reinstall, fresh RAG stack, Phase 5 validation (login/403/401, webhook re-index both event types, evaluate.py --split test --no-judge 53/53, latency + cost baselined for the prod box), then the old box was decommissioned (terminated, EIP moved, final AMI kept — cleanup tracked in #143). The runbook drift and missing ops scripts it surfaced are fixed on phase-141-runbook-fixes (#141 — webhook table 4→5, scripts/setup_erpnext_webhooks.py, scripts/sample_ingest.py, service-key privilege note); docs/BENCHMARKS.md was refreshed from M1 numbers to the t3.large figures on phase-147-benchmarks-t3large (#147). Other findings became #144 (SSO logout), #145 (OAuth allowed_roles gate), #146 (fresh-deploy eval tooling), #148 (query-path latency), #149 (backup_all.sh Postgres role).
Observability access round: Langfuse (PR #42) and Qdrant (PR #44) dashboards are now reachable via loopback-bound ports + SSH tunnel, instead of SQL-only/API-only inspection. CI lint (PR #46) and the missing OPENAI_API_KEY secret (#31) were also fixed, which finally let the evaluate job run end-to-end for the first time — which is how #47 (evaluation has been silently non-functional) was discovered. #45 (aggregation queries failing for real suppliers) was found via manual testing against the live frontend.
#54 (false-negative "could not find relevant information" fallback on answerable questions) fixed — PR #56 merged. #52 (Purchase Invoice, Terms and Conditions, and PDF attachments never wired into ingestion) fixed — PR #57 merged.
#45 (aggregation/enumeration queries returning incomplete or zero results) closed as out of scope — this pipeline is a passage-grounded RAG (top-20 hybrid search reranked to a fixed top-5), not an aggregation/analytics engine; reliably supporting "total/list/count all matching X" queries would require a second, exact-fetch retrieval path bypassing top-k rerank entirely, a different architecture rather than a bug fix. Documented as a known limitation in docs/ARCHITECTURE.md.
#53 (Contract's document_type/document_name link wasn't captured during ingestion) fixed — PR #58 merged. Docs-only follow-up recording the #45 won't-fix decision — PR #59 merged. Known Issues is now empty.
#51 (model config scattered/hardcoded) partially addressed: OPENAI_MODEL/EMBEDDING_MODEL centralized into a new config.py, replacing every hardcoded gpt-4o/text-embedding-3-small literal — PR #61 merged (07614e2). Provider swapping (not just model-version swapping) was scoped further during review and deferred — LangChain's ChatOpenAI/OpenAIEmbeddings were considered and rejected as unnecessary dependency/version-risk surface for a swap that isn't happening yet; a hand-rolled ChatModel/EmbeddingModel adapter design is documented in docs/ARCHITECTURE.md § Model Configuration for when it's picked up. #51 closed and the remaining scope split into #90 for a single-purpose issue.
Full-repo documentation audit found and fixed 5 factual doc bugs + ~9 staleness items across README.md, CLAUDE.md, and docs/*.md — PR #62 merged (95ccd5a). Added docs/MODEL_PROVIDER_SWAP.md, a practical developer runbook for switching model provider with the codebase as it exists today.
Full codebase security review (not tied to a diff) found 3 findings, independently verified via a two-pass identification + false-positive-verification process: #63 (High, SSRF + credential leak, confidence 8/10), #64 (Medium, OAuth replay-cache, confidence 7/10), #65 (Low, hardcoded Postgres creds, confidence 6/10). All three opened as issues; fixes deferred pending prioritization.
ADR added recording why the lexical-search leg is a Qdrant-side in-memory BM25 index rather than a separate Elasticsearch/OpenSearch cluster — docs/adr/0001-qdrant-bm25-over-elasticsearch-opensearch.md (commit c6e637a, committed directly to main). #97 opened to track the ADR's named next step (migrating that lexical leg onto Qdrant's native sparse-vector + IDF support) if/when the in-memory index becomes a bottleneck.
#98 opened after a design-review discussion surfaced that HybridSearch.search()'s BM25 leg ignores filter_conditions entirely (by design, per its own docstring) while the Qdrant leg applies them — RRF fusion means filters end up as a soft rank bias rather than a hard constraint. Tracked as a lightweight standalone fix, separate from #97's larger sparse-vector migration which would also happen to close this gap.
#101 opened: evaluate switching embeddings from OpenAI to a local sentence-transformers model, given the reranker already runs one locally. #100 (previously an empty stub) fleshed out: switch dependency management from requirements.txt/pip to uv (pyproject.toml deps + uv.lock, uv sync in CI) for reproducibility and speed.
Activity
arunjoyt commentedon Jun 18, 2026
CLAUDE.md updated (fe5eb10) to encode the branch/PR/roadmap workflow: branch naming, no-direct-main rule, cross-referencing conventions, PR body template, and instructions to keep this issue in sync on every commit and phase completion.
docs(CLAUDE.md): update project state and add branch/PR/roadmap workflow
arunjoyt commentedon Jun 18, 2026
Groups 1 & 2 integration tests verified live against real OpenAI (text-embedding-3-small), ERPNext, and Qdrant — 14/14 passed. Groups 3 & 4 remain skipped pending Steps 10–12. Refs #19 #21.
arunjoyt commentedon Jun 18, 2026
Issue #19 closed — HyDE and GPT-4o live verification items moved to #21 Group 3. #21 is now the single integration testing tracker.
arunjoyt commentedon Jun 19, 2026
Phase 3 PR: #22 (Steps 10-11 — query rewriter + query pipeline)
arunjoyt commentedon Jun 19, 2026
Phase 3 live integration tests — 4/4 passed (2026-06-19)
Commit: 9a88d52 on branch
phase-3-pipelinehydeandstep_backstrategies produce answersFinding documented in
docs/ARCHITECTURE.md: metadata filters apply only to the Qdrant vector-search leg of hybrid search. BM25 is corpus-wide and cannot apply field conditions, so the fused result set may include documents outside the filter._parse_sourcesonly includes cited documents, limiting practical leakage. If strict supplier isolation is needed, a post-rerank filter must be added inquery_pipeline.py.arunjoyt commentedon Jun 19, 2026
Phase 4 PR: #23 (Steps 12-13 — FastAPI + Streamlit)
arunjoyt commentedon Jun 19, 2026
Phase 4 Group 4 API integration tests — 4/4 passed (2026-06-19)
Commit: fe08f03 on branch
phase-4-api-frontendarunjoyt commentedon Jun 19, 2026
Phase 4 live UI test — verified 2026-06-19
Commit: edf7896 on branch
phase-4-api-frontenduvicorn api.main:app --reloadstarts, collection created, BM25 built, reranker warmPOST /ingest/fullpopulatesprocurementcollection (12 points from ERPNext demo data)streamlit run frontend/app.py— UI renders with sidebar filters and chat inputPR #23 ready for merge.
arunjoyt commentedon Jun 19, 2026
Phase 4 complete ✅ — PR #23 merged (2026-06-19). Steps 12 and 13 shipped.
arunjoyt commentedon Jun 22, 2026
Phase 5 PR: #24 — RAGAS evaluation dataset + evaluate.py (Step 14) + Group 5 integration tests. All tests green.
arunjoyt commentedon Jun 23, 2026
Phase 6 PR: #25 (Steps 15-16 — Dockerfile, docker-compose, CI workflow + Langfuse wiring)
arunjoyt commentedon Jun 23, 2026
Phase 6 complete — Steps 15-16 shipped in PR #25 (merged 551cbcd). Dockerfile, docker-compose, docker-compose.frontend, CI workflow, and Langfuse startup wiring all landed on main. ✅
arunjoyt commentedon Jun 23, 2026
Group 6 + 7 integration tests written and pushed (PR #26). Live run pending — disk space issue during docker compose build. Resume steps documented in PR #26.
112 remaining items
arunjoyt commentedon Aug 31, 2026
#138 shipped —
4ff8f73onmain(merge2728278).The query path was making two identical
text-embedding-3-smallcalls per query (rewriter embedded the rewritten text and discarded the vector;HybridSearchre-embedded the same string). Now embedded once and threaded through — one round-trip removed (~0.15–0.2s/query).The remaining embed is traced as its own
embed_querygeneration onEMBEDDING_MODEL, sorewritespan latency is no longer polluted by the embed + its retries.rewriteis now skipped entirely forQUERY_REWRITE_STRATEGY=none.Verified live: one
/querytrace shows a singleembed_querygeneration (109 prompt / 0 completion tokens),rewritechat-only, grounded answer with citations. 237 tests pass, ruff clean. Docs updated (ARCHITECTURE observability + pipeline steps + diagram, BENCHMARKS, CLAUDE.md).arunjoyt commentedon Aug 31, 2026
#137 shipped —
d8f58b6onmain(merge4b2762e).Self-hosted Langfuse 2.x's bundled price table prices
gpt-4oat its stale mid-2024 launch rate ($5/$15 per 1M vs. current $2.50/$10), so everytotalCostin the UI/API was ~1.9x high forgpt-4ogenerations (token counts were always fine).Decision: per-project price override, not a Langfuse upgrade (v2→v3 is a major migration) and not "ignore the UI number".
scripts/langfuse_fix_model_prices.py— idempotent; adds a project-levelgpt-4oprice override viaPOST /api/public/models. Rates mirrorevaluate.py::_PRICE_PER_1M_TOKENS.--dry-run/--force.CLIENT_DEPLOYMENT_RUNBOOK.mdPhase 3 (run per deployment) + Phase 6 (re-run after a Langfuse upgrade / fresh restore). BENCHMARKS / ARCHITECTURE / DEPLOYMENT updated.Verified live: on a project with the stale managed price, the script creates the override and a synthetic 1000/100-token
gpt-4ogeneration then costs $0.0035 (current list) instead of $0.0078. Re-run is a no-op. 246 tests pass, ruff clean.arunjoyt commentedon Aug 31, 2026
#127 closed —
fe72f01onmain(merge6755e22).Per-client pipeline validation. Most of it shipped incrementally (candidate generator, guided-review path, reach-order re-tune guidance, env-overridable knobs). This close-out added
evaluation/evaluate.py --collectionand consolidated the workflow docs (CLIENT_DEPLOYMENT_RUNBOOK.mdPhase 5,PIPELINE_TUNING.md).Scope change: the separate
{client}_evalcollection from the original issue is dropped — post-#113 every per-client knob is query-time (nothing rebuilds the collection) and the staging RAG stack strategy already isolates validation from the client's users. Validation runs read-only against the staging (or pre-go-live prod) collection.250 tests pass, ruff clean.
arunjoyt commentedon Aug 31, 2026
#113 closed (merge
ae354a2, commit5886367, direct tomain) — the pipeline tuning pass concludes conventional defaults retained.QUERY_REWRITE_STRATEGY) and Step C (RERANK_TOP_N) swept on--split devwith 3× replicates + paired Wilcoxon. Neither knob showed an effect clearing the RAGAS judge's ~±0.05 run-to-run noise band. The 2026-08-29top_n=8lead did not reproduce on the shippedhydedefault (answer_relevancy Δ −0.011, p=0.23).hydeandRERANK_TOP_N=5stay as shipped. Retrieval-pool knobs stay at conventional defaults, owned by per-client validation (eval: per-client pipeline validation workflow (onboarding step) #127).OPENAI_MODELswap (gpt-4o → gpt-4o-mini) carved out to eval: benchmark generation-model swap (gpt-4o -> gpt-4o-mini) as a multi-client cost lever #132.PIPELINE_TUNING.mdSteps A/C Outcome callouts,ARCHITECTURE.md§ Evaluation.Phase 5 (Evaluation) tuning work is now wound down to the #132 model-swap benchmark; #113's checkbox ticks automatically.
arunjoyt commentedon Aug 31, 2026
#132 closed (not planned) — generation-model swap (gpt-4o → gpt-4o-mini) deferred, not benchmarked on the demo corpus.
Same rationale as #113/#129: the ~38-chunk reference corpus + ±0.05 judge noise can't produce a trustworthy quality delta, and a demo result wouldn't transfer to a real client's contract/question style.
gpt-4ostays the shipped default; a cheaper generation tier becomes a per-client onboarding option (#127/#135) — validategpt-4o-miniagainst the client's real questions during guided review, no code change needed (config.pyalready readsOPENAI_MODELfrom env).Phase 5 (Evaluation) is now fully wound down — no open eval issues remain.
arunjoyt commentedon Sep 8, 2026
Deployment-path cleanup from the #139 EC2 rehearsal — branches pushed, no PRs (per workflow):
phase-147-benchmarks-t3large(b638f8c) — BENCHMARKS.md refreshed for the prod t3.large box (Closes docs: refresh BENCHMARKS.md — committed numbers are M1-laptop and partly stale #147)phase-141-runbook-fixes(8d7cc7f) — CLIENT_DEPLOYMENT_RUNBOOK.md + 2 new ops scripts (Closes docs: CLIENT_DEPLOYMENT_RUNBOOK.md fixes found during the #139 EC2 rehearsal #141; notes for auth/docs: OAuth Client allowed_roles defaults to 'Desk User' and gates /authorize with a misleading error #145/eval/ops: Phase 5 eval + benchmark tooling misbehaves on a fresh deploy #146)Still open from the rehearsal: #144 (SSO logout), #145 (OAuth allowed_roles code side), #146 (eval tooling code side), #148 (query-path latency), #143 (AMI cleanup ~2026-09-21).
Merge branch 'phase-141-runbook-fixes'
Merge branch 'phase-147-benchmarks-t3large'
arunjoyt commentedon Sep 8, 2026
Merged to
main(no-PR workflow):setup_erpnext_webhooks.py/sample_ingest.pyfixes from the ops: migrate RAG-app stack to EC2 + post-deploy validation #139 rehearsal — mergeb17fa1c(commits8d7cc7f,13eeeda)e34f286(commits8285b2c,b638f8c)Both auto-closed. Remaining #139 rehearsal findings still open: #144, #145, #146, #148, #149.
arunjoyt commentedon Sep 8, 2026
Branch
phase-149-backup-postgres-user(commitb188d3f) — #149:backup_all.sh/restore_all.shnow readPOSTGRES_USERfrom.envinstead of only falling back to thelangfusedefault, so the Langfuse dump no longer silently fails on a deployment with a generated Postgres role. Unmerged, awaiting review.Merge branch 'phase-149-backup-postgres-user'
arunjoyt commentedon Sep 8, 2026
Merged to
main— #149 fixed in merged0862bf(commitb188d3f). Branch deleted.Merge branch 'phase-146-fresh-deploy-tooling'
arunjoyt commentedon Sep 8, 2026
Merged to
main— merge57d1be0(commita617ba2):evaluate.pyskips the golden-set link when the dataset isn't pushed (kills the scary per-questionERROR Internal error occurred…on fresh deploys);benchmark_from_langfuse.pyreadsLANGFUSE_*from the environment so it runs as-is in theappcontainer; runbook Phase 5 recipe simplified to one line.DEPLOYMENT.mddocuments the OAuth Clientallowed_roles=Desk Userdefault + the misleadingInvalid client_iderror. Optional code change n/a (the 400 is ERPNext's browser-facingauthorizeresponse, never seen by FastAPI).Both auto-closed. Remaining #139 findings: #144 (SSO logout), #148 (query-path latency).
arunjoyt commentedon Sep 8, 2026
(correction to the SHA in my previous comment: the implementation commit is
dc04f68, merged as57d1be0.)