Add re-flagging based on multiple filters by emmleroy · Pull Request #30 · quant-aq/cli

emmleroy · 2026-06-26T15:58:12Z

Summary

This PR adds re-flagging of QuantAQ data based on multiple flag criteria (filters)

Changes

Remove arisense flagging functionality
Define FLAG_DEFINITIONS (flag names, flag bitmask values, and columns to nan) as a generic list of namedtuples common across all QuantAQ products
Define FLAG_CRITERIA as a nested dict that permits different flag criteria/logic for different data sources (rawSD, cloudAPI, database) and data models (modulair, modulair-pm, modulair-x, modulair-x-pm)
Infer data source from the dataframe timestamp frequency
Infer data model from the dataframe 'sn' column (added to rawSD dataframes at load)
Modify expunge.py and ports echo_flag_table() and flag_summary() from Isoprene
Add additional tests

Testing

Added additional tests in test_expunge.py
poetry run pytest --cov=./ --cov-report=xml -rP passes

Other notes

Re-flagging of rawSD card can be easily implemented, but the flag criteria/logic are not yet defined in variables.py (this will raise a NotImplementedError if attempting to re-flag rawSD card data for now)
Planning to remove support for feather files in favor of parquet in sc-20240
Planning to overhaul logging and convert to flatter file structure / library-first approach in sc-20240

…tdiff)

… FLAG_CRITERIA

…mns definition

…soprene

codecov · 2026-06-30T19:07:31Z

Codecov Report

❌ Patch coverage is 93.13305% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.49%. Comparing base (8b95643) to head (9e729a7).
⚠️ Report is 1 commits behind head on emmleroy-v1.0.0.

Files with missing lines	Patch %	Lines
quantaq_cli/console/commands/flag.py	90.00%	7 Missing ⚠️
quantaq_cli/utilities.py	84.44%	7 Missing ⚠️
quantaq_cli/variables.py	92.00%	2 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##           emmleroy-v1.0.0      #30      +/-   ##
===================================================
- Coverage            96.57%   95.49%   -1.08%     
===================================================
  Files                   17       18       +1     
  Lines                  846      932      +86     
===================================================
+ Hits                   817      890      +73     
- Misses                  29       42      +13

Flag	Coverage Δ
unittests	`95.49% <93.13%> (-1.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
quantaq_cli/console/__init__.py	`98.55% <100.00%> (-0.14%)`	⬇️
quantaq_cli/console/commands/expunge.py	`95.65% <100.00%> (-0.58%)`	⬇️
quantaq_cli/console/commands/resample.py	`94.93% <100.00%> (-1.27%)`	⬇️
tests/test_expunge.py	`100.00% <100.00%> (ø)`
tests/test_flag.py	`100.00% <ø> (ø)`
tests/test_merge.py	`100.00% <ø> (ø)`
tests/test_resample.py	`100.00% <ø> (ø)`
tests/test_utilities.py	`100.00% <100.00%> (ø)`
quantaq_cli/variables.py	`92.00% <92.00%> (-8.00%)`	⬇️
quantaq_cli/console/commands/flag.py	`90.69% <90.00%> (+2.60%)`	⬆️
... and 1 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ests

dhhagan · 2026-06-30T12:53:30Z

+# i.e. are flag names and values different for database vs rawSD data?
+FLAGS = {}

 FLAGS["v100"] = [


We can remove all references to ARISense

Removed this, as well as all tests for arisense data in 8162df4 (in PR #32)

dhhagan · 2026-07-02T14:07:23Z

+    df["flag"] = df["flag"].astype(int, errors='ignore')
+
+    # Drop NaNs
+    df = df.dropna(how='any', subset=["flag"])


Definitely want to log this, as it should never occur (for future PR)

Got it! Added a logger.warning in 349b388 (in PR #32)

dhhagan · 2026-07-02T14:08:14Z

-            for item in data:
-                click.echo(item)
-
+        echo_flag_table(df)


This will be removed once we have logging implemented

dhhagan · 2026-07-02T14:08:40Z

+from quantaq_cli.exceptions import InvalidFileExtension, InvalidArgument, InvalidDeviceModel
+
+
+def add_flag(df, flag_name, flag_value, criterion):


Let's make sure we add docstrings that describe types and what the purpose of the function is.

Added google-style docstrings with types in 349b388 (in PR #32) but let me know if you'd like type hints as well!

dhhagan · 2026-07-02T14:09:54Z

+        tscol = determine_timestamp_column(df)
+
+        # Force timestamp type
+        df[tscol] = df[tscol].map(pd.to_datetime)


Should check the type first and only do this if it's not already a datetime

Got it, added helper function fix_timestamps() in 349b388 (in PR #32) which only forces the timestamp type if not already a datetime

dhhagan · 2026-07-02T14:16:57Z

+def infer_data_source(df):
+    # Determine the best timestamp column and sort
+    tscol = determine_timestamp_column(df)
+    df[tscol] = pd.to_datetime(df[tscol])


Should only do this if needed

We also have this in several places, so probably a good idea to create a helper function

Added helper function fix_timestamps() in 349b388 (in PR #32) which only forces the timestamp type if not already a datetime

dhhagan · 2026-07-02T14:18:47Z

+    has_1min = 60.0 in tdiffs
+    has_5sec = 5.0 in tdiffs
+
+    if has_1min and has_5sec:


So, it's highly, highly likely that you're always going to have some sort of mixed tdiffs. It is likely rarer for you to encounter both 1min and 5s exactly, but it's definitely within the realm of possibility. We way want to be a bit more lenient and just look at the frequency of tdiffs? Similarly, the UFP data is 10s on the SD card, not 1min.

Changed this to most common tdiff in d838aaa (in PR#32) and added logging

dhhagan · 2026-07-02T14:19:54Z

+    MOD-X-00993    -> modulair-x
+    MOD-X-PM-01685 -> modulair-x-pm
+    """
+    prefix, _, _ = device_sn.rpartition("-")


Will this catch MOD-UFP-XXXXX? We should probably add to the docstring at minimum.

It will! Added to docstring and tests in 2ca1564(PR#32)

dhhagan · 2026-07-02T14:21:37Z

+    Flag("FLAG_BAT", 2048, ["bat_voltage", "soc", "vbat"]),
+]
+
+SUPPORTED_MODELS = ("modulair", "modulair-x", "modulair-pm", "modulair-x-pm")


Should add modulair-ufp

Sounds good, will hold off for now until I add modulair-ufp flagging logic (at least for database/api data)

dhhagan · 2026-07-02T14:22:41Z

+Range = namedtuple("Range", ["column", "lo", "hi"])
+Gap = namedtuple("Gap", ["gap_in_seconds", "post_gap_flag_length_seconds"])
+
+FLAG_CRITERIA = {


I think it might be easiest to just go over the criteria before dumping too much time into this part. The SD and db sides are fairly different and the logic may be difficult to capture in this way, though I could be wrong.

emmleroy marked this pull request as draft June 26, 2026 16:07

emmleroy marked this pull request as ready for review June 26, 2026 16:08

emmleroy marked this pull request as draft June 26, 2026 16:08

emmleroy added 6 commits June 30, 2026 09:36

Use namedtuple for FLAG and FLAG_CRITERIA

6c74da7

Replace single-filter flag command with multi-filter flagging

02606fb

Add util for determining the timestamp col

28197e1

Refactor - rename bit to flag_value, fix click commands

1f8f9ba

Update tests

79e26bf

Fix indentation

003d5ad

emmleroy force-pushed the emmleroy-sc-20242-reflagging-v2 branch from cb24699 to 003d5ad Compare June 30, 2026 13:44

emmleroy added 14 commits June 30, 2026 09:50

Remove relative imports

d4b2c97

Remove arisense flagging functionality

22b7560

Modify safe_loader() to add device serial number to rawSD columns

61f6d34

Add func to infer data model from the sn column

f0b40aa

Add func to infer data source from the dataframe sampling frequency (…

5ad1d1f

…tdiff)

Infer data model and source in flag_dataframe() instead of taking args

045e34b

Define identical FLAG_DEFINITIONS for QuantAQ products and modifiable…

d04686a

… FLAG_CRITERIA

Convert model-specific FLAG_DEFINITIONS into one generic list

e19c028

Remove tests for rawSD flagging - currently not implemented

fe018eb

Upgrade deps to Click8.x

30646f7

Port expunge_dataframe() and echo_flag_table() from Isoprene

89a2ce6

Move flag_summary and echo_flag_table to flag.py and fix all_nan_colu…

49c186a

…mns definition

Remove arisense from test_expunge() and add some expunge tests from I…

cae63d5

…soprene

Remove feather tests and fix --wind unpack error in resample CLI

2e67857

emmleroy marked this pull request as ready for review June 30, 2026 19:08

emmleroy added 3 commits July 1, 2026 10:55

Remove data model and data source as required args to flag_command

eb2c043

Remove data model and data source as required args in cli logic and t…

04d34ed

…ests

Fix cli args again

9e729a7

dhhagan reviewed Jul 2, 2026

View reviewed changes

emmleroy merged commit 9e729a7 into emmleroy-v1.0.0 Jul 8, 2026
6 of 8 checks passed

emmleroy deleted the emmleroy-sc-20242-reflagging-v2 branch July 9, 2026 13:21

		from quantaq_cli.exceptions import InvalidFileExtension, InvalidArgument, InvalidDeviceModel


		def add_flag(df, flag_name, flag_value, criterion):

Uh oh!

Conversation

emmleroy commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Other notes

Uh oh!

codecov Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

emmleroy commented Jun 26, 2026 •

edited

Loading

codecov Bot commented Jun 30, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading