Skip to content

Update flagging for SD Card data and implement flag-aware resampling#33

Open
emmleroy wants to merge 8 commits into
emmleroy-v1.0.0from
sdcard-reflagging-sc-20242
Open

Update flagging for SD Card data and implement flag-aware resampling#33
emmleroy wants to merge 8 commits into
emmleroy-v1.0.0from
sdcard-reflagging-sc-20242

Conversation

@emmleroy

@emmleroy emmleroy commented Jul 10, 2026

Copy link
Copy Markdown

Summary

This PR updates the flagging module to support data source - specific logic (i.e. SD Card vs Database).

Changes

  • FLAG_CRITERIA are data-source specific as defined in variables.py (DATABASE_CRITERIA, CLOUDAPI_CRITERIA, RAWSD_CRITERIA).
  • FLAG_DEFINITIONS are NOT data-source specific, nor data-model specific, since the MODULAIR and MODULAIR-PM firmware will be updated to be consistent with MOD-X devices
  • The evaluate_criterion() logic in flag.py (the function that masks rows that meet the flag criteria) is only data-model specific for FLAG_STARTUP (i.e. if the criterion instance is ``Gap'') --> we only add FLAG_STARTUP if the device is a modulair or modulair-x since this only affects gas columns.
  • The resample_dataframe() now supports an optional flag-aware resampling method that mirrors the logic from the firmware.
    If flag_aware = True, resampling follows this logic:
    - If there's at least 1 good record in the resampling bin, average all good records (set the flag to 0).
    - If there are no good records, average all bad records (bitwise OR all the flags in the bin).
    The old version of resample.py mistakenly treated the 'flag' column similarly as the other numeric columns.
  • Added new tests for flagging rawSD data and for flag-aware resampling.

Notes/Questions for Review

  • What columns should be nan'd if FLAG_OVERHEAT is set? Currently, none are nan'd.
  • Flag criteria for FLAG_OPC differ for database/cloudAPI vs rawSD --> if FLAG_OPC in the database was original set by the logic in the firmware, this re-flagging module will lead to different flags --> not sure we want that?
  • The same^ applies for FLAG_STARTUP (the flag criteria differ between database/cloudAPI and rawSD, but only in that rawSD flagging requires LIPO SOC < 10% in addition to being offline for 1-hour).

@emmleroy emmleroy requested a review from dhhagan July 10, 2026 17:24
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.54726% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.11%. Comparing base (06e2356) to head (f3cf262).
⚠️ Report is 1 commits behind head on emmleroy-v1.0.0.

Files with missing lines Patch % Lines
quantaq_cli/toolkit/resample.py 66.66% 10 Missing ⚠️
quantaq_cli/toolkit/flag.py 86.66% 8 Missing ⚠️
quantaq_cli/utilities.py 50.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                 Coverage Diff                 @@
##           emmleroy-v1.0.0      #33      +/-   ##
===================================================
+ Coverage            90.97%   91.11%   +0.13%     
===================================================
  Files                   21       21              
  Lines                  931     1069     +138     
===================================================
+ Hits                   847      974     +127     
- Misses                  84       95      +11     
Flag Coverage Δ
unittests 91.11% <90.54%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
quantaq_cli/cli.py 91.33% <100.00%> (+0.06%) ⬆️
quantaq_cli/schema.py 100.00% <ø> (ø)
quantaq_cli/toolkit/expunge.py 77.27% <100.00%> (-13.64%) ⬇️
quantaq_cli/toolkit/munge.py 80.00% <ø> (-5.72%) ⬇️
quantaq_cli/variables.py 90.32% <100.00%> (+16.79%) ⬆️
tests/test_clean.py 100.00% <100.00%> (ø)
tests/test_concat.py 100.00% <100.00%> (ø)
tests/test_expunge.py 92.22% <100.00%> (+0.75%) ⬆️
tests/test_flag.py 100.00% <100.00%> (ø)
tests/test_merge.py 100.00% <100.00%> (ø)
... and 5 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant