Repository navigation
perf(query): optimize query execution layer (#76) - #93
Conversation
Addresses the four performance hot spots in the query execution layer called out by issue #76 (subtask of #72): * max_rows is pushed down to SQL as a LIMIT clause instead of being applied via Python rows[:max_rows] slicing in QueryService. The executor injects the effective limit into a template's limit variable when one exists and only appends a fresh LIMIT when the rendered SQL has no trailing one, so no template gets a duplicate LIMIT clause. * QueryExecutor caches compiled Jinja templates by name so long-lived owners (MCP server, SnapshotAnalyzer) skip the Environment.from_string parse step on every render. * A new ContextCache (LRU, mtime-aware, default maxsize=4) shares a persistent Context across execute_query calls so the MCP server and SnapshotAnalyzer reuse a single read-only SQLite connection and skip dictionary schema validation per call. Cache entries are evicted automatically when the db file's mtime changes (e.g. after 'pt-snap import') and can be force-refreshed via invalidate(). * The memory_peak template is rewritten as a CTE that materialises the (optional) start_id/end_id range once, computes the three MAX values in a single scan, then reuses the CTE for the MIN(id) event-id lookups. Output is bit-identical to the original nested-subquery form across every start_id/end_id combination (covered by tests/query/test_memory_peak_cte.py). leak_detection and callstack_analysis gain a 'limit' parameter so the max_rows pushdown applies there too.
跨模型代码评审报告
目标:PR #93 已核实的发现(按严重度排序)1. HIGH —
|
| 评审者 | 模型 | 状态 | 会话 |
|---|---|---|---|
| DeepSeek | coding-plan/deepseek-v4-flash |
完成 | ses_01e1bbbbdffeZcMK87DInDnQv3 |
| GLM | coding-plan/glm-5.2 |
完成 | ses_01e1bbb97ffeu32x0a0x1wogLZ |
| Kimi | unraid-wg/wb/kimi-k3 |
完成 | ses_01e1bbb8affevnV9BitpXeuvhf |
三者独立识别出 total 语义回归;两者独立识别出 max_rows < top_n 违约与编译缓存无效。并行评审会增加 token 用量与成本。
…ion-optimization # Conflicts: # src/pt_snap_cli/query/templates/business/leak_detection.yaml
Closes #76 (subtask of #72).
What changed
Four query-execution-layer optimizations called out by the issue:
max_rowspushdown.QueryExecutor.rendernow injects a positivemax_rowsinto the template'slimitvariable (when one exists) andappends
LIMIT <n>only when the rendered SQL has no trailing LIMITclause.
QueryService.execute_queryno longer slices in Python --total == returnedreflects what the query actually produced.Jinja compilation cache.
QueryExecutor._compiled_template()cachesparsed templates by name so long-lived owners (MCP server,
SnapshotAnalyzer) skipEnvironment.from_stringon every render.Context caching for MCP. A new
ContextCache(LRU, mtime-aware,default
maxsize=4) shares a persistentContextacross calls.Context(persistent=True)keeps the read-only SQLite connectionalive across
connect()calls; default behaviour is unchanged.st_mtimechange, sopt-snap importoverwriting the db file transparently forces a reopen.
invalidate()provides an explicit refresh path.
QueryServiceandSnapshotAnalyzer; the analyzeraccepts an externally-supplied cache so the MCP server can share
one across instances.
memory_peakCTE rewrite. The 6 nested subqueries arereplaced by a
WITH filtered AS ..., peaks AS ...CTE thatmaterialises the (optional)
start_id/end_idrange once,computes the three MAX values in a single scan, and reuses the CTE
for the MIN(id) event-id lookups. Output is byte-identical to the
original nested-subquery form for every start_id/end_id combination
(covered by
tests/query/test_memory_peak_cte.py, includinginverted and out-of-range windows).
leak_detectionandcallstack_analysisgained alimitparameter sothe
max_rowspushdown applies there too.Verification
pytest tests/-- 625 passed, 1 xfailed (pre-existing).ruff check src/ tests/clean.black --check src/ tests/clean.pt-snap query <db> --template-use leak_detection -n 2returns 2 rows; without
-nreturns all 5.memory_peakwith andwithout
start_id/end_idreturns the same numbers as theoriginal nested-subquery SQL.
tests/core/test_context_cache.py-- 11 tests covering LRUeviction, mtime invalidation, schema error propagation, persistent
vs non-persistent Context.
tests/query/test_memory_peak_cte.py-- 11 parametrized casestests/query/test_query_max_rows_pushdown.py-- 7 tests forleak_detectionandcallstack_analysispushdown.tests/query/test_executor.py-- 6 new tests formax_rowspushdown and Jinja cache.
tests/test_snapshot_analyzer_cache.py-- 4 tests forend-to-end analyzer caching.
tests/test_query_cache_perf.py-- 1 perf backstop that 10cached queries finish under 2 s.
Snapshot provenance decision: no-update
Snapshot provenance no-update reason: Issue #76 touches only the query execution layer (templates, executor, QueryService, ContextCache). No files under src/pt_snap_cli/snapshot/ are modified, so the first-party replay/import runtime history is unchanged.