Skip to content

perf(query): optimize query execution layer (#76) - #93

Merged
codeasier merged 3 commits into
mainfrom
issue/76-query-execution-optimization
Aug 8, 2026
Merged

codeasier merged 3 commits into
mainfrom
issue/76-query-execution-optimization

Conversation

@codeasier

Copy link
Copy Markdown
Owner

Closes #76 (subtask of #72).

What changed

Four query-execution-layer optimizations called out by the issue:

  1. max_rows pushdown. QueryExecutor.render now injects a positive
    max_rows into the template's limit variable (when one exists) and
    appends LIMIT <n> only when the rendered SQL has no trailing LIMIT
    clause. QueryService.execute_query no longer slices in Python --
    total == returned reflects what the query actually produced.

  2. Jinja compilation cache. QueryExecutor._compiled_template() caches
    parsed templates by name so long-lived owners (MCP server,
    SnapshotAnalyzer) skip Environment.from_string on every render.

  3. Context caching for MCP. A new ContextCache (LRU, mtime-aware,
    default maxsize=4) shares a persistent Context across calls.

    • Context(persistent=True) keeps the read-only SQLite connection
      alive across connect() calls; default behaviour is unchanged.
    • The cache evicts on st_mtime change, so pt-snap import
      overwriting the db file transparently forces a reopen. invalidate()
      provides an explicit refresh path.
    • Wired through QueryService and SnapshotAnalyzer; the analyzer
      accepts an externally-supplied cache so the MCP server can share
      one across instances.
  4. memory_peak CTE rewrite. The 6 nested subqueries are
    replaced by a WITH filtered AS ..., peaks AS ... CTE that
    materialises the (optional) start_id/end_id range once,
    computes the three MAX values in a single scan, and reuses the CTE
    for the MIN(id) event-id lookups. Output is byte-identical to the
    original nested-subquery form for every start_id/end_id combination
    (covered by tests/query/test_memory_peak_cte.py, including
    inverted and out-of-range windows).

leak_detection and callstack_analysis gained a limit parameter so
the max_rows pushdown applies there too.

Verification

  • pytest tests/ -- 625 passed, 1 xfailed (pre-existing).
  • ruff check src/ tests/ clean.
  • black --check src/ tests/ clean.
  • CLI smoke: pt-snap query <db> --template-use leak_detection -n 2
    returns 2 rows; without -n returns all 5. memory_peak with and
    without start_id/end_id returns the same numbers as the
    original nested-subquery SQL.
  • New focused tests:
    • tests/core/test_context_cache.py -- 11 tests covering LRU
      eviction, mtime invalidation, schema error propagation, persistent
      vs non-persistent Context.
    • tests/query/test_memory_peak_cte.py -- 11 parametrized cases
      • 1 smoke test.
    • tests/query/test_query_max_rows_pushdown.py -- 7 tests for
      leak_detection and callstack_analysis pushdown.
    • tests/query/test_executor.py -- 6 new tests for max_rows
      pushdown and Jinja cache.
    • tests/test_snapshot_analyzer_cache.py -- 4 tests for
      end-to-end analyzer caching.
    • tests/test_query_cache_perf.py -- 1 perf backstop that 10
      cached queries finish under 2 s.

Snapshot provenance decision: no-update
Snapshot provenance no-update reason: Issue #76 touches only the query execution layer (templates, executor, QueryService, ContextCache). No files under src/pt_snap_cli/snapshot/ are modified, so the first-party replay/import runtime history is unchanged.

Addresses the four performance hot spots in the query execution layer
called out by issue #76 (subtask of #72):

* max_rows is pushed down to SQL as a LIMIT clause instead of being
  applied via Python rows[:max_rows] slicing in QueryService. The
  executor injects the effective limit into a template's limit
  variable when one exists and only appends a fresh LIMIT when the
  rendered SQL has no trailing one, so no template gets a duplicate
  LIMIT clause.

* QueryExecutor caches compiled Jinja templates by name so long-lived
  owners (MCP server, SnapshotAnalyzer) skip the Environment.from_string
  parse step on every render.

* A new ContextCache (LRU, mtime-aware, default maxsize=4) shares a
  persistent Context across execute_query calls so the MCP server and
  SnapshotAnalyzer reuse a single read-only SQLite connection and skip
  dictionary schema validation per call. Cache entries are evicted
  automatically when the db file's mtime changes (e.g. after
  'pt-snap import') and can be force-refreshed via invalidate().

* The memory_peak template is rewritten as a CTE that materialises the
  (optional) start_id/end_id range once, computes the three MAX values
  in a single scan, then reuses the CTE for the MIN(id) event-id
  lookups. Output is bit-identical to the original nested-subquery
  form across every start_id/end_id combination (covered by
  tests/query/test_memory_peak_cte.py).

leak_detection and callstack_analysis gain a 'limit' parameter so the
max_rows pushdown applies there too.
@codeasier

Copy link
Copy Markdown
Owner Author

跨模型代码评审报告

由 cross-review 技能驱动:三个独立只读评审会话并行执行(DeepSeek / GLM / Kimi),父会话负责核实、去重与定级。本次评审未修改任何文件。

目标:PR #93 perf(query): 优化查询执行层(commit a90bbed,对比基线 main)


已核实的发现(按严重度排序)

1. HIGH — max_rows 对使用其他限制变量的模板被静默忽略

active_memory_callstack_at_event 模板用 top_n 而非 limit 控制行数。执行器仅在渲染上下文注入 limit,模板仍渲染出 LIMIT 20;随后 _has_trailing_limit 检测到尾部已有 LIMIT,不再追加 max_rows,导致调用方上限失效。QueryService 又移除了原先的 Python rows[:max_rows] 兜底切片,于是 max_rows=5 仍返回 20 行,违反 -n/max_rows 契约。

  • 位置:src/pt_snap_cli/query/executor.py:154、src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml:73
  • 复现:render(q, {"event_id": 42}, device_id=0, max_rows=5) 末行输出 LIMIT 20(已实际复现)
  • 影响范围:任何尾部 LIMIT 不由 limit 变量驱动的模板(含用户经 register_template 注册的模板)
  • 建议修复:在 QueryService 恢复 Python 兜底切片(rows = rows[:max_rows]),或在执行器中对尾部 LIMIT 数值与 effective_limit 取 min。至少补充 max_rows < top_n 的测试

2. MEDIUM — QueryResult.total 语义变更,CLI 截断提示与文档失配

原先 total 为完整匹配数、returned 为截断后行数;现在 total == returned == len(rows)(已截断)。

  • CLI ... and N more (use -n to show more) 分支永远不可达:src/pt_snap_cli/cli.py:439-443
  • 中英文文档仍展示 Found 150 results, showing 2: ... and 148 more:docs/en/querying.md:164-171、docs/zh/querying.md:164-171
  • MCP execute_query 默认 max_rows=100,agent 无法再判断结果是否被截断
  • 位置:src/pt_snap_cli/core/query_service.py:150-158
  • 建议修复:二选一 —(a)执行 LIMIT max_rows+1 或独立 COUNT(*) 以保留真实 total;(b)同步更新 CLI 与两侧文档并补充契约测试

3. MEDIUM — Jinja 编译缓存在生产路径中无效

QueryService.execute_query 每次都新建 QueryExecutor 实例(query_service.py:135),而 _compiled_cache 是实例属性,调用结束即丢弃。MCP/SnapshotAnalyzer 热路径均经此入口,故"跳过 Environment.from_string 解析"这一优化在生产中零收益,仅手工构造长生命周期 executor 的测试受益。

  • 位置:src/pt_snap_cli/core/query_service.py:135、src/pt_snap_cli/query/executor.py:60
  • 附带隐患:缓存仅以 template.name 为键(executor.py:71),若同名模板被 register_template 替换为不同 SQL,将命中旧编译体(当前生产无此路径,风险低)
  • 建议修复:将编译缓存上移至 QueryService 或按 (name, hash(query)) 键控;或每 QueryService 复用一个 executor

4. MEDIUM — ContextCache 仅按 st_mtime 失效,同时间戳替换会返回陈旧数据

os.replace() 在 1 秒粒度文件系统(HFS+/FAT/NFS/exFAT)上可能产生相同 st_mtime,缓存命中后继续使用绑定旧 inode 的只读连接,查询将持久返回旧库内容,直至 LRU 淘汰或显式 invalidate()。测试用 time.sleep(0.05) 规避了该情形(tests/core/test_context_cache.py:97),反向印证风险存在。

  • 位置:src/pt_snap_cli/core/context_cache.py:65-72
  • 建议修复:额外比较 st_ino/st_size,或在 pt-snap import 发布后显式调用 invalidate_context_cache()

5. LOW — 非持久 Context 的嵌套 connect() 会提前关闭外层连接

重构后 re-entrant 分支改为 try/finally: if not self._persistent: self.close()。内层 with ctx.connect(): 退出即 close(),外层仍持已关闭连接。当前生产路径未触发(查询用持久缓存,非持久仅一次性使用),属潜在隐患。

  • 位置:src/pt_snap_cli/context.py:137-143
  • 建议修复:按重入深度计数,仅在深度归零时关闭;或恢复 yield; return 语义

未采纳的候选发现(经核实降级)

  • sqlite3 跨线程安全:当前 FastMCP 在事件循环线程内同步调用工具,SnapshotAnalyzer 实际单线程使用,属潜在而非现实缺陷。建议以文档约束 SnapshotAnalyzer 单线程。
  • ContextCache.get 抛 OSError:exists() 与 stat() 间的窄窗口竞争在实际只读路径上极罕见,且 CLI 上层已捕获通用异常兜底。
  • 持久连接 GC 才关闭:mode=ro 句柄资源开销极低,CLI 每 command 重建 QueryService,影响有限,归入测试缺口。

测试缺口

  1. 缺 max_rows < top_n 用例:test_max_rows_does_not_double_limit 仅覆盖 max_rows=100 > top_n=20,恰好是唯一无害分支
  2. 缺 total/returned 语义契约测试:tests/test_contract_cli_mcp.py 用 max_rows=0,无法暴露截断回归
  3. 缺服务路径编译缓存复用测试:仅验证手工 executor 实例内复用
  4. 缺同时间戳 os.replace 失效测试
  5. 缺多线程 SnapshotAnalyzer/ContextCache 测试
  6. 缺非持久 Context 嵌套 connect() 测试
  7. tests/test_query_cache_perf.py 为 2s 软预算计时测试,CI 负载下可能 flaky

验证记录

  • 实际复现 HIGH 发现:render(..., max_rows=5) 末行输出 LIMIT 20
  • 聚焦测试通过:tests/query/test_executor.py、tests/core/test_query_service.py、tests/core/test_context_cache.py 共 38 passed
  • git diff --check main...HEAD 无异常
  • 评审过程未修改任何源文件

评审者来源

评审者 模型 状态 会话
DeepSeek coding-plan/deepseek-v4-flash 完成 ses_01e1bbbbdffeZcMK87DInDnQv3
GLM coding-plan/glm-5.2 完成 ses_01e1bbb97ffeu32x0a0x1wogLZ
Kimi unraid-wg/wb/kimi-k3 完成 ses_01e1bbb8affevnV9BitpXeuvhf

三者独立识别出 total 语义回归;两者独立识别出 max_rows < top_n 违约与编译缓存无效。并行评审会增加 token 用量与成本。

codeasier added 2 commits August 9, 2026 00:09
…ion-optimization

# Conflicts:
#	src/pt_snap_cli/query/templates/business/leak_detection.yaml
@codeasier
codeasier merged commit 9a80b44 into main Aug 8, 2026
6 checks passed
@codeasier
codeasier deleted the issue/76-query-execution-optimization branch August 8, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] 查询执行层优化:max_rows 下推、memory_peak SQL 重写、模板缓存与连接复用(子任务 #72-4)

1 participant