Skip to content

feat(poi): improve benchmark latency measurement and reliability#225

Merged
sainijit merged 2 commits into
intel-retail:mainfrom
sainijit:main
May 22, 2026
Merged

feat(poi): improve benchmark latency measurement and reliability#225
sainijit merged 2 commits into
intel-retail:mainfrom
sainijit:main

Conversation

@sainijit

Copy link
Copy Markdown
Contributor
  • Use VLM metrics files as primary latency source (matching SAD pattern)
  • Prefer mqtt_received_at over frame timestamp in alerts API fallback
  • Add early exit on first alert for single benchmark mode
  • Add benchmark_duration parameter for configurable wait time
  • Filter stale alerts using iteration start timestamp
  • Skip container recreation when services already running
  • Add dual latency method: single run vs stream density
  • Clean metrics files before each iteration

PR Checklist

  • Added label to the Pull Request for easier discoverability and search
  • Commit Message meets guidelines as indicated in the URL https://github.com/intel-retail/performance-tools/blob/main/CONTRIBUTING.md
  • Every commit is a single defect fix and does not mix feature addition or changes
  • Unit Tests have been added for new changes
  • Updated Documentation as relevant to the changes
  • All commented code has been removed
  • If you've added a dependency, you've ensured license is compatible with repository license and clearly outlined the added dependency.
  • PR change contains code related to security
  • PR introduces changes that breaks compatibility with other modules (If YES, please provide details below)

What are you changing?

Issue this PR will close

close: #issue_number

Anything the reviewer should know when reviewing this PR?

Test Instructions if applicable

If the there are associated PRs in other repositories, please link them here (i.e. intel-retail/performance-tools )

- Use VLM metrics files as primary latency source (matching SAD pattern)
- Prefer mqtt_received_at over frame timestamp in alerts API fallback
- Add early exit on first alert for single benchmark mode
- Add benchmark_duration parameter for configurable wait time
- Filter stale alerts using iteration start timestamp
- Skip container recreation when services already running
- Add dual latency method: single run vs stream density
- Clean metrics files before each iteration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the POI stream-density benchmark script to improve latency measurement fidelity and benchmark reliability by prioritizing VLM metrics files, improving alerts-based fallback, and adding a dedicated single-benchmark mode with configurable duration and early-exit behavior.

Changes:

  • Prefer vlm_application_metrics files as the primary latency source, with a stream-density-specific extractor (last N pairs) and alerts API as a fallback.
  • Improve alerts API parsing by preferring mqtt_received_at, filtering stale alerts using an iteration start timestamp, and applying the same filtering when saving thumbnails.
  • Add single-run benchmark enhancements: configurable BENCHMARK_DURATION, early exit on first alert, and skipping container recreation when key services are already running.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread benchmark-scripts/poi_stream_density.py Outdated
Comment thread benchmark-scripts/poi_stream_density.py
Comment thread benchmark-scripts/poi_stream_density.py
Comment thread benchmark-scripts/poi_stream_density.py
- Normalize timestamps to UTC-aware before subtraction to prevent
  TypeError on mixed tz-aware/naive datetime arithmetic
- Track which start field was used (mqtt_received_at vs timestamp)
  and log an accurate label accordingly
- Fix poll overshoot: sleep min(poll_interval, remaining) so
  _wait_for_alert_or_timeout respects the exact duration
- Use actual elapsed time for docker log collection window in
  single-run mode instead of stabilise_duration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sainijit sainijit merged commit 5c68ea1 into intel-retail:main May 22, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants