Skip to content

fix(metrics): bound histogram sample storage to prevent unbounded memory growth#4119

Open
UGVicV wants to merge 16 commits into
orchestration-agent:mainfrom
UGVicV:fix/bounty-982-histogram-bound
Open

fix(metrics): bound histogram sample storage to prevent unbounded memory growth#4119
UGVicV wants to merge 16 commits into
orchestration-agent:mainfrom
UGVicV:fix/bounty-982-histogram-bound

Conversation

@UGVicV
Copy link
Copy Markdown

@UGVicV UGVicV commented May 25, 2026

Description

This PR resolves the issue where MetricsCollector.observe() stored every raw observation value in an unbounded list, causing unbounded memory growth in long-running agents with high-volume metrics.

Fix

  • Replaced raw list storage with a new _HistogramBucket class using __slots__ for memory efficiency.
  • Each bucket tracks: count, total (sum), min_val, max_val, and bounded samples (deque with maxlen).
  • Added MAX_HISTOGRAM_SAMPLES = 1000 constant — only the most recent 1000 samples are retained.
  • Aggregates (count, sum, min, max) are computed incrementally O(1) per observation, not recomputed from raw data.
  • Snapshot now includes min and max fields alongside existing count, sum, avg.
  • Wrapped observe() in try-except with logger.error() for robust error logging.
  • Replaced print() with logger.error() in snapshot().
  • Removed unused List import.
  • Ensured 100% PEP8 compliance (all lines under 79 characters).

Verification (Proof)

Added 4 regression tests to tests/test_metrics.py:

  • test_histogram_min_max: Verifies min/max tracking across observations.
  • test_histogram_bounded_samples: Verifies memory is bounded after 1500 observations while count/sum remain accurate.
  • test_histogram_aggregates_accurate: Verifies count/sum/avg/min/max for a known set.
  • test_histogram_empty: Verifies no crash when querying unobserved metrics.

Test Execution Output

69 passed in 0.07s

Lint & Formatting Output

  • flake8 src/common/metrics.py tests/test_metrics.py -> Passed (0 errors/warnings)
  • git diff --check -> Passed (0 errors)

No secrets, tokens, or hidden context are included in the code.

Closes #982.

Vic added 16 commits May 22, 2026 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ Bounty $5k ] [ Metrics ] Bound histogram sample storage — memory pressure

1 participant