A financial research agent for GOOGL SEC filings + US macroeconomic data, featuring hybrid retrieval, async task execution, code-verified computation, and cross-session research memory.
┌─────────────────────────────────────────────────────────┐
│ Chat Mode (synchronous) Task Mode (async) │
│ Gradio UI :7860 FastAPI :7878 │
└────────────┬────────────────────────┬────────────────────┘
│ │ POST /api/tasks
│ ┌────▼─────┐
│ │ tasks │ PostgreSQL
│ │ table │
│ └────┬─────┘
│ │ Worker polls
└────────────┬───────────┘
▼
┌────────────────────────┐
│ Memory Retrieval │ pgvector similarity
│ research_memory │ inject prior findings
└────────────┬───────────┘
▼
┌─────────────────────────────────────────────────────────┐
│ PER Loop │
│ │
│ Planner → Tool Use → structured sub-queries (JSON) │
│ (anti-repeat: already_searched) │
│ ↓ │
│ Executor → Pure SQL, no LLM call │
│ sec_chunks ── pgvector cosine ──┐ │
│ ── tsvector FTS ───┼── RRF fusion │
│ events ── pgvector cosine ──┤ │
│ ── tsvector FTS ───┘ │
│ macro_indicators ── exact SQL + keyword fallback │
│ ↓ │
│ Critic → Tool Use → sufficiency judgment │
│ ↓ │
│ Synthesizer → Step 1: Tool Use selects evidence IDs │
│ → Step 2: agentic loop writes answer │
│ LLM calls compute tool for math │
│ sandboxed Python executes inline │
│ → Step 3: citation validation [n] │
└────────────────────────┬────────────────────────────────┘
↓
┌───────────────────────┐
│ Report Writer │ structured markdown
│ Memory Extractor │ Tool Use → findings → pgvector
└───────────────────────┘
PER Loop: Plan → Execute → Critique → (refine up to 3×) → Synthesize
WITH semantic AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $vec) AS sem_rank
FROM sec_chunks WHERE fiscal_year = $year LIMIT 20
),
lexical AS (
SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank(content_tsv, query) DESC) AS lex_rank
FROM sec_chunks WHERE content_tsv @@ $query LIMIT 20
)
SELECT id, 1.0/(60+sem_rank) + 1.0/(60+lex_rank) AS rrf_score
FROM semantic FULL OUTER JOIN lexical USING (id)
ORDER BY rrf_score DESC LIMIT 12The Synthesizer uses an agentic loop so computation results flow directly into the generation stream — no regex parsing, no post-processing, no orphaned lines:
Agentic Answer Generation
LLM reads full retrieved context, writes answer citing [n] sources
When a derived metric is needed (CAGR, growth rate, basis points):
→ LLM calls the compute tool with self-contained Python
→ sandboxed Python executes (no import, 15s timeout)
→ result flows back inline — LLM continues writing naturally
Loop ends at end_turn (no more tool calls needed)
Citation Validation
All [n] references verified to exist in context
Out-of-range citations logged as warnings
Sources Panel Filtering (script, zero LLM cost)
Scans [n] citations in final answer → shows only referenced chunks
This replaces the prior <compute> regex approach — no tag parsing, no substitution pass, no risk of malformed extraction.
POST /api/tasks {"question": "..."} → {"task_id": "uuid", "status": "pending"}
Background worker:
1. Retrieve relevant memories (pgvector)
2. Run PER Loop
3. Write structured markdown report → tasks.report_md
4. Extract key findings → research_memory
GET /api/tasks/{id} → {"status": "completed", "report_md": "..."}
After each task, an LLM call extracts 2-4 key findings and stores them as vector embeddings. Future tasks retrieve relevant prior findings via similarity search and inject them into the planning context — giving the agent continuity across sessions.
The Planner classifies each question before decomposition. Fully out-of-domain questions (weather, general coding, chitchat) are marked in_scope=false and short-circuited with a rejection message — no retrieval, no wasted PER Loop iterations. Questions that are in-domain but currently unanswerable (future data, un-ingested tickers, cross-source comparisons) stay in the pipeline and are declined by the Synthesizer. Evaluated by Set F (5 out-of-domain questions); Set C guardrail questions (C02, C05) confirmed not falsely rejected.
| Source | Content | Size |
|---|---|---|
| SEC EDGAR | MAG7 10-K / 10-Q / 8-K (2019–2024) — GOOGL + MSFT + META + AMZN + AAPL + NVDA + TSLA | ~35,000 chunks |
| FRED | 12 US macro series (GDP, CPI, FEDFUNDS, UNRATE, etc.) | ~5,000 data points |
| Events | 30 hand-curated key events (Fed policy, earnings, antitrust) | 30 records |
| Price History | MAG7 daily OHLCV + P/E ratios (2015–present, auto weekly refresh) | ~90,000 rows |
| Earnings History | MAG7 quarterly EPS actual vs estimate + revenue (2015–present) | ~700 rows |
Evaluated on 4 factual SEC questions (Set A) using key-fact text matching against RRF-ranked chunks. No LLM calls required.
| K | Hit@K | MRR@K |
|---|---|---|
| 1 | 0.750 | 0.750 |
| 3 | 1.000 | 0.875 |
| 5 | 1.000 | 0.875 |
All 4 questions retrieve a relevant chunk within top-3. MRR@3 = 0.875 indicates the relevant chunk ranks first for 3 of 4 questions (A01 ranks 2nd).
| Strategy | Precision | Recall | Avg Tokens |
|---|---|---|---|
| Fixed 512/128 (selected) | 0.062 | 0.250 | 482 |
| Recursive | 0.016 | 0.250 | 516 |
| Semantic (threshold=0.75) | 0.000 | 0.375 | 198 |
Evaluated with 23 questions across four sets (A/B/C/D). Judge model: Gemini 2.5 Pro (separate from pipeline LLM).
| Version | faithfulness | answer_relevancy | context_precision | context_recall | ragas_score |
|---|---|---|---|---|---|
| v1 (baseline) | 0.618 | 1.000 | 0.174 | 0.471 | 0.566 |
| v11 (eval methodology) | 0.544 | 0.944 | 0.603 | 0.587 | 0.670 |
| v12 (section fix) | 0.667 | 0.972 | 0.688 | 0.651 | 0.741 |
| v13 (MAG7 + price/earnings) | 0.713 | 0.930 | 0.571 | 0.590 | 0.707 |
| v14 (monthly aggregation + eval fixes) | 0.710 | 0.952 | 0.622 | 0.490 | 0.694 |
| v13→v14 Δ | -0.003 | +0.022 | -0.027 | +0.065 | +0.024 |
| v1→v14 Δ | +0.092 | -0.048 | +0.448 | +0.019 | +0.128 |
Key improvements from v1 → v11 (eval methodology):
context_precision+0.429: upgraded from holistic estimate to Precision@K (per-chunk boolean + rank-weighted average)context_recall+0.116: upgraded from holistic score to atomic fact decompositionfaithfulness-0.073: Gemini 2.5 Pro is stricter than Flash Lite; Set B causal questions penalised — data ceiling, not a code issueanswer_relevancy: correctly-refusing-unanswerable questions (C03) now score 1.0 instead of 0.0- Eval pipeline:
run_eval.pynow callsper_loop.run()directly, restoringalready_searchedanti-repeat
Key improvements from v11 → v12 (section detection fix, Bug #18):
faithfulness+0.123: retrieved chunks now contain actual MD&A / Risk Factors content instead of TOC fragments — LLM can verify its claims against real substance- All four metrics improved: section boundaries now correctly partition 10-K body text (MD&A 2→30, Risk Factors 0→34, Financial Statements 2→72 chunks)
Key changes from v12 → v13 (MAG7 expansion + price/earnings data sources):
faithfulness+0.046: structured price/earnings data gives LLM verifiable numbers — hallucination rate dropscontext_precision-0.117 /context_recall-0.061: daily price rows (252/year) cause Precision@K mismatch — see Observation #19 indocs/failure_analysis.md- New Set D (5 questions) covers valuation, earnings beat/miss, macro-price correlation, and MAG7 competitor comparison
Key improvements from v13 → v14 (monthly aggregation + eval fixes):
context_recall+0.065: ground_truth D01/D03 updated with specific numbers (P/E 18x–38x mean 25x; FEDFUNDS 0.08%→4.33%, GOOGL -39%) — Judge now has concrete facts to verifyanswer_relevancy+0.022: compute tool description strengthened ("import FORBIDDEN") — LLM no longer generates brokenimport numpy as npcausing compute errorscontext_precision-0.027: monthly aggregation reduces row count but Judge still cautious about price rows — evaluation method ceiling, not retrieval quality
| Component | v1 | v11 | v12 | v13/v14 |
|---|---|---|---|---|
| Judge model | gemini-3.1-flash-lite-preview (shared with pipeline) | gemini-2.5-pro (dedicated, configurable via judge: in config.yaml) |
same as v11 | same as v11 |
| context_precision | holistic fraction estimate | Precision@K — per-chunk boolean + rank-weighted average | same as v11 | same as v11 |
| context_recall | holistic score | atomic fact decomposition — each ground-truth claim checked independently | same as v11 | same as v11 |
| context window for judge | 3 000 chars / 15 chunks | 10 000 chars / 25 chunks | same as v11 | same as v11 |
| Pipeline used during eval | reimplemented inline (missing already_searched) |
per_loop.run() directly |
same as v11 | same as v11 |
| answer_relevancy for refusals | penalises correct "I cannot answer" responses | 1.0 for correctly-identified unanswerable questions | same as v11 | same as v11 |
| Synthesizer background knowledge | soft "do not fabricate" warning | hard Rule 5: general knowledge does not exist for this answer | same as v11 | same as v11 |
| SEC chunk section accuracy | TOC-based (MD&A 2, Risk 0, FinStmt 2) | same as v1 | fixed (MD&A 30, Risk 34, FinStmt 72) | same as v12 |
| Data sources in eval | sec_chunks, events, macro_indicators | same as v1 | same as v1 | + price_history (monthly agg), earnings_history |
| Eval question sets | Set A (8) | + Set B (5), Set C (5) | same as v11 | + Set D (5) — valuation, earnings, correlation, MAG7 |
| price_history granularity | — | — | — | v13: daily (252 rows/yr) → v14: monthly auto-agg (12 rows/yr) |
| compute tool constraint | — | — | — | v14: import FORBIDDEN explicitly stated in tool description |
| Layer | Technology |
|---|---|
| LLM | Gemini (pluggable via LLMClient Protocol) |
| Embedding | Qwen3-Embedding-0.6B (local llama.cpp, F16 GGUF) / BGE-M3 |
| Reranker | qwen3-rerank (DashScope) / BGE-Reranker-v2-m3 |
| Vector DB | PostgreSQL 17 + pgvector (HNSW index) |
| Full-text | PostgreSQL tsvector (GIN index) |
| Agent | Pure Python (no LangChain) |
| Task Queue | PostgreSQL + asyncio worker (SELECT FOR UPDATE SKIP LOCKED) |
| API | FastAPI |
| UI | Gradio |
| Evaluation | Custom LLM-as-Judge (Precision@K + atomic recall) + chunk ablation |
| Observability | Langfuse (LLM tracing: token / cost / latency), fail-safe & optional |
| Quality | ruff (lint) + GitHub Actions CI (pytest on mock LLM) |
- Python 3.11+
- Docker (for PostgreSQL)
- llama.cpp +
Qwen3-Embedding-0.6B-f16.gguf(local embedding server, no API key) - API keys: Gemini, DashScope, FRED
# 1. Clone and install
git clone https://github.com/sangowu/macrolens
cd macrolens
uv sync
# 2. Start PostgreSQL
docker run -d --name macrolens-pg \
-e POSTGRES_USER=macrolens \
-e POSTGRES_PASSWORD=macrolens \
-e POSTGRES_DB=macrolens \
-p 5433:5432 pgvector/pgvector:pg17
# 3. Configure
cp .env.example .env # fill in API keys
# 4. Initialize DB
uv run python -c "
import psycopg
conn = psycopg.connect('postgresql://macrolens:macrolens@localhost:5433/macrolens')
conn.autocommit = True
for f in ['migrations/001_init.sql', 'migrations/002_tasks_memory.sql']:
conn.cursor().execute(open(f).read())
print('DB ready')
"
# 5. Start local embedding server (llama.cpp, Qwen3-Embedding-0.6B, F16)
# Download GGUF: Qwen/Qwen3-Embedding-0.6B-GGUF (ModelScope or HuggingFace)
llama-server -m Qwen3-Embedding-0.6B-f16.gguf --embedding --pooling last \
-ngl 99 -c 2048 -b 2048 -ub 2048 --host 127.0.0.1 --port 8081
# 6. Ingest data
uv run ingestion/ingest_sec.py --ingest-only
uv run ingestion/ingest_fred.py
uv run ingestion/ingest_events.py
# 7. Launch (three terminals)
uv run ui/app.py # Gradio UI :7860
uv run uvicorn api.tasks:app --port 7878 # Task API :7878
uv run worker/task_worker.py --verbose # Workeruv run agent/per_loop.py "How did Fed rate hikes in 2022 affect Google's advertising revenue?"
uv run agent/per_loop.py --max-iter 3 --verbose "What are Google's main risk factors in 2023?"uv run eval/run_eval.py --sets A B C --output eval/results_v10.csv
uv run eval/compare_versions.py eval/results_v1.csv eval/results_v10.csv
uv run eval/compare_versions.py eval/results_v1.csv eval/results_v10.csv --plot
uv run eval/chunk_ablation.py --files 3macrolens/
├── agent/
│ ├── planner.py # LLM decomposes question → sub-queries
│ ├── executor.py # Hybrid retrieval (RRF SQL)
│ ├── critic.py # Sufficiency judge
│ ├── synthesizer.py # Citation-grounded answer + <compute> blocks
│ ├── per_loop.py # PER Loop orchestration
│ ├── report_writer.py # Formats markdown research report
│ ├── memory.py # Research memory: extract findings + retrieve
│ └── tools/
│ └── code_executor.py # Sandboxed Python execution
├── api/
│ └── tasks.py # FastAPI: POST/GET /api/tasks
├── worker/
│ └── task_worker.py # Async polling worker
├── ingestion/
│ ├── ingest_sec.py # SEC EDGAR → sec_chunks
│ ├── ingest_fred.py # FRED API → macro_indicators
│ ├── ingest_events.py # events.json → events
│ └── chunkers.py # Fixed / Recursive / Semantic chunkers
├── models/
│ ├── base.py # EmbeddingBackend / RerankerBackend Protocol
│ ├── factory.py # Backend factory (local / remote / online)
│ ├── embedding/ # local_bge, local_qwen, remote, online
│ └── reranker/ # local, remote, online (DashScope / Cohere)
├── eval/
│ ├── run_eval.py # RAGAS evaluation runner
│ ├── chunk_ablation.py
│ ├── questions.py # Evaluation question sets (A/B/C)
│ └── metrics.py # context_precision / context_recall
├── ui/
│ └── app.py # Gradio UI (Chat tab + Analysis Task tab)
├── migrations/
│ ├── 001_init.sql # Core schema (sec_chunks, events, macro_indicators)
│ └── 002_tasks_memory.sql # tasks + research_memory tables
├── data/
│ └── events.json # Hand-curated event timeline
├── docs/
│ ├── failure_analysis.md
│ └── interview_talking_points.md
├── cloud_server/
│ └── server.py # FastAPI inference server (remote embedding)
└── config.yaml # Single source of truth for all configuration
Why PostgreSQL over Pinecone/Chroma? Financial RAG requires time filtering + exact numerical queries + vector search in the same transaction. pgvector enables all three without data synchronization complexity.
Why PER Loop over ReAct? Financial Q&A is a closed domain. PER Loop's fixed structure (3 LLM calls minimum) is more predictable and cheaper than ReAct's open-ended tool use.
Why Tool Use for structured output instead of regex?
Planner, Critic, and Memory Extractor previously parsed LLM output with re + json.loads, which fails silently when the LLM adds surrounding text or produces malformed JSON. Tool Use with tool_choice forces the LLM to fill a validated schema — format errors are impossible.
Why send full context to Synthesizer instead of pre-filtering?
Financial report chunks are highly uniform — all contain dense numbers and financial terminology. A separate filtering step forces the LLM to judge relevance before seeing the answer, which is harder than finding the answer directly. The Synthesizer's LLM naturally ignores irrelevant chunks while reading and only cites what it uses. Post-hoc filtering of the Sources panel by [n] citations achieves clean presentation at zero extra cost.
Why compute via Tool Use instead of <compute> tags?
Inline tag embedding requires regex extraction and a second parse pass, and produces orphaned result lines when the LLM places the tag between paragraphs. Tool Use integrates computation into the generation stream: the LLM calls the tool mid-sentence, receives the result, and continues writing — no post-processing needed.
Why Code Executor instead of LLM arithmetic? LLM arithmetic on multi-year financial data is a hallucination risk. The Code Executor moves computation into deterministic Python — every derived number in the answer is verifiable by the code shown.
Why custom task queue over LangGraph? The PER Loop is a bounded 4-step pipeline, not a complex graph. A PostgreSQL task table + asyncio worker is simpler, fully observable, and consistent with the "independently testable components" philosophy.
Why no LangChain? Every component is independently testable. The retrieval SQL, Planner prompt, Critic logic, and Code Executor can each be evaluated in isolation.
Why Fixed chunking over Semantic? Ablation results: Fixed achieves higher precision (0.062 vs 0.000) with uniform chunk sizes that produce stable RRF rankings.
18 documented bugs and optimizations in docs/failure_analysis.md, including:
sec-parserreturning 3.8M empty nodes → replaced with BeautifulSoup + regex- Synthesizer hallucination (faithfulness=0) → hardened to mandatory
[n]citation rules - Planner repeating identical sub-queries → added
already_searchedlist to prompt - Chunk ablation scoring 0 due to year mismatch → reversed sort to select most recent filings
- Critic dead loop → anti-repeat fixed Set B +0.029
<compute>output appearing as isolated line → replaced<compute>tag + regex with compute Tool Use agentic loop; orphaned-line cleanup no longer needed- Section detection bug (4 compounding issues:
\xa0, case, TOC vs body, startswith ambiguity) → fixed; MD&A 2→30 chunks, Risk Factors 0→34 chunks; faithfulness +0.123, ragas_score +0.071 - Eval pipeline reimplementation bypassing
already_searched→ replaced with directper_loop.run()call - Eval context truncation (3 000 chars) hiding evidence from judge → expanded to 10 000 chars / 25 chunks
context_precisionholistic estimate not reflecting ranking quality → replaced with Precision@Kcontext_recallholistic score masking partial coverage → replaced with atomic fact decomposition- Gemini 2.5 Pro
resp.textreturningNone→ guarded with(resp.text or "").strip() - Eval duplicate rows when ragas_score is None → fixed
:.3fformat crash that triggered the except branch answer_relevancypenalising correct refusals (C03: 0.0) → prompt updated to score 1.0 for unanswerable questions handled correctly- Synthesizer supplementing missing context with background knowledge (B01/B02 faithfulness) → added hard Rule 5: general knowledge does not exist for this answer
Sango Wu | AI Engineer Portfolio Project | 2026