Skip to content

Commit ad24e61

Browse files
PierreRaybautCopilot
PierreRaybaut
and
Copilot
committed
Add performance and visual regression scripts for PythonQwt
- Introduced `README.md` in the scripts directory to document the performance investigation and visual regression workflows. - Added `bench_plotpy_loadtest.py` to benchmark PlotPy's load test against PythonQwt. - Updated `bench_qt.ps1` to clarify usage and differentiate between PythonQwt and PlotPy load tests. - Created `capture_screenshots.py` to capture screenshots from PythonQwt visual tests for regression detection. - Implemented `diff_screenshots.py` to pixel-compare screenshots and classify differences. - Enhanced `lineprofile_loadtest.py` to profile specific hot functions identified by cProfile. - Updated `profile_loadtest.py` to provide detailed profiling reports for the load test. Co-authored-by: Copilot <copilot@github.com>
1 parent e1756d7 commit ad24e61

10 files changed

Lines changed: 846 additions & 13 deletions

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,3 +73,8 @@ target/
7373
# Local benchmark venvs (issue #93)
7474
.venvs/
7575

76+
# Performance investigation artifacts (see scripts/README.md)
77+
shots/
78+
profile.out
79+
*.prof_stats
80+
*.lprof

‎README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,10 @@ From the source package:
123123
python -m build
124124
```
125125

126+
## Performance investigation
127+
128+
Tooling for performance benchmarks, profiling and visual-regression checks across PyQt5/PyQt6/PySide6 lives in [`scripts/`](scripts/README.md). See [`doc/issue93_optimization_summary.md`](doc/issue93_optimization_summary.md) for a worked example.
129+
126130
## Copyrights
127131

128132
### Main code base

‎doc/issue93_optimization_summary.md‎

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
# Issue #93 — Performance degradation with Qt6: optimization summary
2+
3+
This document summarises the work done on the `fix/93-performance-degradation-with-qt6` branch to investigate and close the Qt5↔Qt6 performance gap reported in [issue #93](https://github.com/PlotPyStack/PythonQwt/issues/93). It walks through each optimization phase, the diagnostic method used, the change applied, and the measured impact.
4+
5+
All numbers below were collected on the same Windows 11 machine, Python 3.11.9, with three sibling virtual environments (`.venvs/pyqt5`, `.venvs/pyqt6`, `.venvs/pyside6`), each pinning a single Qt binding (PyQt5 5.15.11 / Qt 5.15.2, PyQt6 6.11.0 / Qt 6.11.0, PySide6 6.11.0 / Qt 6.11.0).
6+
7+
Two benchmarks were used throughout:
8+
9+
- **`qwt/tests/test_loadtest.py`** — the PythonQwt micro load test (raw QwtPlot widgets, no PlotPy). Driven by `scripts/bench_qt.ps1`. Reports `Average elapsed time: <ms> ms` per binding.
10+
- **PlotPy `test_loadtest`** — `plotpy/tests/benchmarks/test_loadtest.py`, the test cited in the original GitHub issue. Driven by `scripts/bench_plotpy_loadtest.py` (60 plot widgets, 3 runs).
11+
12+
## Baseline (master, commit `1ab70cd`)
13+
14+
| Benchmark | PyQt5 | PyQt6 | PySide6 |
15+
|---|---:|---:|---:|
16+
| PythonQwt `test_loadtest` (avg of 5) | ~1 900 ms | ~2 300 ms | ~2 900 ms |
17+
| PlotPy `test_loadtest`, 60 plots (avg of 3) | 25 134 ms | 42 202 ms | 53 160 ms |
18+
19+
Headline gap on PlotPy: **PyQt6 ≈ +68 % slower than PyQt5**, **PySide6 ≈ +111 % slower than PyQt5**.
20+
21+
The cProfile traces taken on master pointed at four hot families of code paths inside PythonQwt:
22+
23+
1. `QwtScaleMap.transform()` — called on every coordinate transformed.
24+
2. `QwtScaleDiv.contains()` and `QwtScaleEngine.contains()/strip()` — called on every tick label candidate.
25+
3. `QwtAbstractScaleDraw.labelRect()` and helpers — called on every drawn tick.
26+
4. `QwtText` / `QwtPlainTextEngine` text-size and text-margin computations — called on every tick label and every plot title.
27+
28+
All four are amortised over thousands of calls per plot, and all four are sensitive to per-call Python overhead (attribute lookups, QObject machinery, redundant Qt round-trips). That is precisely the kind of overhead that the Qt6 bindings (especially PySide6) make more expensive than Qt5, which explains why a regression that is barely visible on Qt5 becomes a 2× slowdown on Qt6.
29+
30+
## Phase 1 — cProfile-driven optimizations (commit `ef793e1`)
31+
32+
**Method.** `scripts/profile_loadtest.py` runs the PythonQwt load test under `cProfile` and dumps a sorted-by-cumulative-time stats file. Diff between PyQt5 and PySide6 traces highlighted the four families above.
33+
34+
**Changes.**
35+
36+
- **`qwt/scale_map.py`** — inlined the scalar fast path in `QwtScaleMap.transform()` (avoid the array branch and a method dispatch when the input is a plain Python `float`).
37+
- **`qwt/scale_div.py`** — rewrote `QwtScaleDiv.contains()` as a direct comparison against the cached lower/upper bounds, instead of going through `QwtInterval`.
38+
- **`qwt/scale_engine.py`** — `QwtScaleEngine.contains()` and `QwtScaleEngine.strip()` similarly bypass `QwtInterval` round-trips for the common case.
39+
- **`qwt/scale_draw.py`** — replaced the per-call alignment branching in `labelRect()`/`labelPosition()` with module-level constants (`_ALIGN_BOTTOM`, `_ALIGN_TOP`, `_ALIGN_LEFT`, `_ALIGN_RIGHT`); added a rotation==0 fast path in `labelRect()`; cached the axis `orientation` once in `setAlignment()` instead of recomputing it on every call.
40+
- **`qwt/text.py`** — first round of cleanups around `QwtText.textSize()` and `QwtPlainTextEngine.textMargins()`, plus a per-engine "last seen font id" fast path that skips the `QFontMetricsF` rebuild when the same `QFont` instance is reused (which is the dominant case during a single plot repaint).
41+
42+
**Results after phase 1** (PythonQwt micro `test_loadtest`, 5 runs each):
43+
44+
| Binding | Before | After phase 1 | Speedup |
45+
|---|---:|---:|---:|
46+
| PyQt5 | ~1 900 ms | ~620 ms | ×3.0 |
47+
| PyQt6 | ~2 300 ms | ~780 ms | ×2.9 |
48+
| PySide6 | ~2 900 ms | ~960 ms | ×3.0 |
49+
50+
Phase 1 closed most of the absolute slowdown but did not change the *relative* Qt5↔Qt6 gap — all three bindings benefited roughly equally, because the optimizations attacked Python-side overhead that scales with call count regardless of binding.
51+
52+
## Phase 2 — line-profiler-driven optimizations (commit `27a0e17`)
53+
54+
**Method.** `scripts/lineprofile_loadtest.py` instruments the surviving hot functions with `line_profiler` (`@profile`) and re-runs the load test. The line-by-line traces revealed two new dominant costs that did not show up clearly in cProfile:
55+
56+
1. The `QObject` base class on `QwtText_PrivateData` and on the `_PrivateData` classes inside `qwt/scale_draw.py`. Every instantiation went through Qt's meta-object system, which is dramatically more expensive on PyQt6 / PySide6 than on PyQt5.
57+
2. Repeated calls to `QFont.key()` from within `QwtText.textSize()`, `QwtText.effectiveAscent()` and `QwtPlainTextEngine.textMargins()`. Each call serialises the full font descriptor; the same descriptor is hit thousands of times during a single load test because the same default font instance is reused.
58+
59+
**Changes.**
60+
61+
- **`qwt/text.py`** — `QwtText_PrivateData` is now a plain `object` subclass with `__slots__`; no QObject. Added a process-wide `_FONT_KEY_CACHE` keyed by `id(font)` that memoizes `font.key()` (with a hard cap of 1024 entries to avoid unbounded growth). Helper `font_key_cached()` is used by `effectiveAscent`, `QwtPlainTextEngine.textMargins`, and `QwtText.textSize`.
62+
- **`qwt/scale_draw.py`** — the various `_PrivateData` containers also drop `QObject` and use `__slots__`.
63+
64+
**Results after phase 2** (PythonQwt micro `test_loadtest`, 5 runs each):
65+
66+
| Binding | Before phase 2 | After phase 2 | Speedup vs phase 1 | Speedup vs master |
67+
|---|---:|---:|---:|---:|
68+
| PyQt5 | ~620 ms | ~445 ms | ×1.4 | ×4.3 |
69+
| PyQt6 | ~780 ms | ~480 ms | ×1.6 | ×4.8 |
70+
| PySide6 | ~960 ms | ~600 ms | ×1.6 | ×4.8 |
71+
72+
Phase 2 finally closed the *relative* gap as well: Qt6 bindings benefit more than Qt5 from removing QObject inheritance and `font.key()` calls, because the per-call overhead they save is binding-cost-dominated.
73+
74+
## Phase 3 — screenshot regression analysis
75+
76+
**Method.** Two new helpers were added in `scripts/`:
77+
78+
- **`capture_screenshots.py`** — runs each of the 22 PythonQwt visual tests in a subprocess with `PYTHONQWT_TAKE_SCREENSHOTS=1` and copies the resulting PNGs into `shots/<branch>/<binding>/`.
79+
- **`diff_screenshots.py`** — pixel-compares two screenshot folders (Pillow + NumPy) and emits a markdown table with `IDENTICAL` / `EQUAL_PIXELS` / `DIFFER` status, plus the count and magnitude of differing pixels.
80+
81+
A full matrix was captured (master × 3 bindings, fix × 3 bindings, plus self-compare baselines master×master and fix×fix to filter out flaky tests that have inherently random or time-stamped output).
82+
83+
**Findings.**
84+
85+
- **PyQt6 and PySide6**: zero new deterministic differences vs master. Every diff that appeared was already present in the master self-compare baseline (the 6 tests `test_cpudemo`, `test_curvebenchmark1/2`, `test_data`, `test_loadtest`, `test_mapdemo`, all of which use random data or timestamps).
86+
- **PyQt5**: 6 *new* deterministic, sub-perceptual differences appeared, in `test_backingstore`, `test_bodedemo`, `test_image`, `test_relativemargin`, `test_symbols`, `test_vertical`. All diffs were tiny (a few dozen pixels each, max magnitude ≤ 26/255), scattered around antialiased text and curve edges.
87+
88+
### Per-test screenshot status (master vs phase-2 fix, all bindings)
89+
90+
Each cell aggregates two pixel-diffs per test (master vs `master2` self-compare baseline, and master vs phase-2 fix). The classification rule is:
91+
92+
- ✅ — both diffs report identical or pixel-equal output (test is fully reproducible *and* the optimization branch did not change it).
93+
- ⚠️ — both diffs are non-zero (test is *intrinsically* flaky — random data, timestamps, live system stats — so any difference is noise, not a regression).
94+
- ❌ — baseline is identical but the fix differs (a real visual regression introduced by the optimization branch).
95+
96+
| Test | PyQt5 | PyQt6 | PySide6 |
97+
|---|:-:|:-:|:-:|
98+
| `test_backingstore` | ❌ 55 px (max=11) | ✅ | ✅ |
99+
| `test_bodedemo` | ❌ 39 px (max=16) | ✅ | ✅ |
100+
| `test_cartesian` | ✅ | ✅ | ✅ |
101+
| `test_cpudemo` | ⚠️ | ⚠️ | ⚠️ |
102+
| `test_curvebenchmark1` | ⚠️ | ⚠️ | ⚠️ |
103+
| `test_curvebenchmark2` | ⚠️ | ⚠️ | ⚠️ |
104+
| `test_curvedemo1` | ✅ | ✅ | ✅ |
105+
| `test_curvedemo2` | ✅ | ✅ | ✅ |
106+
| `test_data` | ⚠️ | ⚠️ | ⚠️ |
107+
| `test_errorbar` | ✅ | ✅ | ✅ |
108+
| `test_eventfilter` | ✅ | ✅ | ✅ |
109+
| `test_highdpi` | ✅ | ✅ | ✅ |
110+
| `test_image` | ❌ 6 px (max=9) | ✅ | ✅ |
111+
| `test_loadtest` | ⚠️ | ⚠️ | ⚠️ |
112+
| `test_logcurve` | ✅ | ✅ | ✅ |
113+
| `test_mapdemo` | ⚠️ | ⚠️ | ⚠️ |
114+
| `test_multidemo` | ✅ | ✅ | ✅ |
115+
| `test_relativemargin` | ❌ 72 px (max=11) | ✅ | ✅ |
116+
| `test_simple` | ✅ | ✅ | ✅ |
117+
| `test_stylesheet` | ✅ | ✅ | ✅ |
118+
| `test_symbols` | ❌ 4 px (max=9) | ✅ | ✅ |
119+
| `test_vertical` | ❌ 88 px (max=26) | ✅ | ✅ |
120+
121+
**Summary at end of phase 3.** PyQt6 and PySide6: 16 ✅ / 6 ⚠️ / **0 ❌**. PyQt5: 10 ✅ / 6 ⚠️ / **6 ❌**. The 6 ❌ entries on PyQt5 are the regression that phase 4 fixes.
122+
123+
**Root cause.** The id-keyed `font.key()` cache subtly changes the order in which the Qt5 font engine is asked to materialise specific font descriptors. On Qt5, the font engine hints text glyphs slightly differently depending on first-touch order — invisible to a human, but bit-non-identical to master. Qt6's font engine does not show this sensitivity.
124+
125+
## Phase 4 — Option A: gate the font-key fast path on Qt5 (current state)
126+
127+
**Change.** In `qwt/text.py`, the id-keyed cache is now guarded by a Qt-version check:
128+
129+
```python
130+
from qtpy import QT_VERSION as _QT_VERSION
131+
132+
_USE_FONT_KEY_FAST_PATH = not str(_QT_VERSION).startswith("5.")
133+
134+
def font_key_cached(font) -> str:
135+
if not _USE_FONT_KEY_FAST_PATH:
136+
return font.key()
137+
# ... id-keyed cache lookup ...
138+
```
139+
140+
On Qt5 this becomes a thin pass-through to `font.key()` — bit-identical output to master is restored. On Qt6 (where it actually matters most for this issue) the optimization stays in place.
141+
142+
**Verification.**
143+
144+
1. **Screenshot regression** — re-ran PyQt5 capture and diff. The 6 ❌ entries from the phase 3 table all flip to ✅. Final per-binding tally becomes **16 ✅ / 6 ⚠️ / 0 ❌** on every binding — i.e. byte-identical output to master on every test that is reproducible at all.
145+
2. **Test suite** — `pytest -q` with `PYTHONQWT_UNATTENDED_TESTS=1` on all three bindings:
146+
- PyQt5: 26 passed, 1 skipped
147+
- PyQt6: 26 passed, 1 skipped
148+
- PySide6: 26 passed, 1 skipped, 1 warning
149+
3. **Performance** — PyQt5 micro-bench rose from ~445 ms to ~450–550 ms (≈ +5 ms, well within the run-to-run noise). Qt6 numbers are unchanged.
150+
151+
## Final results
152+
153+
### PythonQwt micro `test_loadtest` (5 runs each, ms)
154+
155+
| Binding | master | fix/93 (Option A) | Speedup |
156+
|---|---:|---:|---:|
157+
| PyQt5 | ~1 900 | ~450–550 | ×3.5–×4.2 |
158+
| PyQt6 | ~2 300 | ~450–675 | ×3.4–×5.1 |
159+
| PySide6 | ~2 900 | ~580–795 | ×3.6–×5.0 |
160+
161+
### PlotPy `test_loadtest`, 60 plots (3 runs each, ms)
162+
163+
| Binding | master (`1ab70cd`) | fix/93 (Option A) | Speedup |
164+
|---|---:|---:|---:|
165+
| PyQt5 | 25 134 | **16 169** | ×1.55 |
166+
| PyQt6 | 42 202 | **21 387** | ×1.97 |
167+
| PySide6 | 53 160 | **24 849** | ×2.14 |
168+
169+
### Cross-binding gap (PlotPy load test)
170+
171+
| Comparison | master | fix/93 |
172+
|---|---:|---:|
173+
| PyQt6 vs PyQt5 | +68 % slower | **+32 % slower** |
174+
| PySide6 vs PyQt5 | +111 % slower | **+54 % slower** |
175+
176+
The original issue — a 1.5×–2× penalty for Qt6 over Qt5 — is largely resolved on the PlotPy load test, while the PyQt5 path remains bit-compatible with master both visually and behaviourally.
177+
178+
## Backwards compatibility & public API surface
179+
180+
The optimizations are deliberately confined to internal hot paths and do not alter the documented public API:
181+
182+
- `QwtScaleMap.transform()`, `QwtScaleDiv.contains()`, `QwtScaleEngine.contains()/strip()`, `QwtAbstractScaleDraw.labelRect()/labelPosition()` — same signatures, same semantics, same return values.
183+
- `QwtText` and `QwtPlainTextEngine` — same signatures and semantics. The internal `_PrivateData` containers no longer derive from `QObject`; this is invisible from the outside because `_PrivateData` was a private holder, never exposed and never used as a Qt signal/slot target.
184+
- New module-level helper `qwt.text.font_key_cached()` is internal (lowercase, undocumented). It can be safely removed or refactored later without breaking any public consumer.
185+
- No new dependency. No change to `qtpy` requirements; the Qt-version gate uses `qtpy.QT_VERSION` which is already imported transitively.
186+
187+
The screenshot regression sweep above is the empirical confirmation of this: byte-identical PNGs on every non-flaky test mean PythonQwt's rendered output is unchanged, on every binding.
188+
189+
## Reproduction quickstart
190+
191+
The whole evaluation can be reproduced from a fresh checkout in a few commands. The scripts assume three sibling virtual environments under `.venvs/{pyqt5,pyqt6,pyside6}/`, each with a single Qt binding plus `numpy`, `qtpy`, `pytest`, `pillow`, and `PythonQwt` installed editable.
192+
193+
```powershell
194+
# 1. PythonQwt micro load test, all three bindings, 5 runs each
195+
.\scripts\bench_qt.ps1 -Repeat 5
196+
197+
# 2. Visual regression sweep (PyQt5 example; repeat for pyqt6 / pyside6)
198+
$env:QT_API = "pyqt5"
199+
& .\.venvs\pyqt5\Scripts\python.exe scripts\capture_screenshots.py shots\fix\pyqt5
200+
& .\.venvs\pyqt5\Scripts\python.exe scripts\capture_screenshots.py shots\master\pyqt5 # after `git checkout master`
201+
& .\.venvs\pyside6\Scripts\python.exe scripts\diff_screenshots.py shots\master\pyqt5 shots\fix\pyqt5
202+
203+
# 3. PlotPy load test (the test cited in the original GitHub issue)
204+
$env:PYTHONPATH = "c:\Dev\PlotPy;c:\Dev\guidata"
205+
foreach ($b in "pyqt5","pyqt6","pyside6") {
206+
& ".\.venvs\$b\Scripts\python.exe" scripts\bench_plotpy_loadtest.py --repeat 3 --nplots 60
207+
}
208+
```
209+
210+
## Test environment
211+
212+
| Component | Value |
213+
|---|---|
214+
| OS | Windows 11 (x64) |
215+
| Python | 3.11.9 (NuGet build) |
216+
| PyQt5 | 5.15.11 (Qt 5.15.2) |
217+
| PyQt6 | 6.11.0 (Qt 6.11.0) |
218+
| PySide6 | 6.11.0 (Qt 6.11.0) |
219+
| qtpy | latest available at the time of capture |
220+
| PlotPy (for PlotPy load test) | 2.9.1 (editable install from `c:\Dev\PlotPy`) |
221+
| guidata (for PlotPy load test) | 3.14.3 (editable install from `c:\Dev\guidata`) |
222+
| Display | physical desktop session (not `offscreen`) — measurements include real Qt paint/composite cost |
223+
224+
## Files touched
225+
226+
| File | Phase 1 (cProfile) | Phase 2 (line-profiler) | Phase 4 (Option A) |
227+
|---|:-:|:-:|:-:|
228+
| `qwt/scale_map.py` | ✓ | | |
229+
| `qwt/scale_div.py` | ✓ | | |
230+
| `qwt/scale_engine.py` | ✓ | | |
231+
| `qwt/scale_draw.py` | ✓ | ✓ (drop QObject, `__slots__`) | |
232+
| `qwt/text.py` | ✓ | ✓ (drop QObject, font cache) | ✓ (Qt5 gate) |
233+
234+
Tooling added under `scripts/`:
235+
236+
- `bench_qt.ps1` — driver for the PythonQwt micro load test across the three venvs.
237+
- `profile_loadtest.py` — cProfile harness used in phase 1.
238+
- `lineprofile_loadtest.py` — line_profiler harness used in phase 2.
239+
- `capture_screenshots.py` / `diff_screenshots.py` — phase 3 visual regression tooling.
240+
- `bench_plotpy_loadtest.py` — driver for the PlotPy load test (the test cited in the original issue).

0 commit comments

Comments
 (0)