Refactor to library-first architecture with thin CLI wrapper by emmleroy · Pull Request #32 · quant-aq/cli

emmleroy · 2026-07-02T15:10:35Z

Summary

This PR restructures quantaq_py as an importable library with cli.py reduced to a thin wrapper around it.

Changes

renamed quantaq_cli/ to quantaq_py/
core functions moved to quantaq_py/toolkit/, still organized by individual module (e.g. concat.py), but re-exported in quantaq_py/__init__.py --> so they're importable directly as from quantaq_py import concat_files
cli commands on files are moved to cli.py (which first calls safe_load for functions that operate on dataframes)
added logging via loguru, with logging configs specified in log.py
removed feather support in favor of parquet (and added testing for parquet support)
infers data_model and data_source from the dataframe (for flagging)
always add sn column when calling safe_load on rawSD data
added pandera dtype checks when calling safe_load
removed tests for arisense data
added helper for fixing timestamp columns

…tdiff)

… FLAG_CRITERIA

…mns definition

…soprene

…ests

…o cli.py

…cli.py

…mand() to cli.py

…and to cli.py

…SD, convert clean_file to clean_dataframe

codecov · 2026-07-02T15:15:10Z

Codecov Report

❌ Patch coverage is 87.64045% with 77 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.97%. Comparing base (8b95643) to head (06e2356).
⚠️ Report is 1 commits behind head on emmleroy-v1.0.0.

Files with missing lines	Patch %	Lines
quantaq_cli/utilities.py	84.31%	16 Missing ⚠️
quantaq_cli/cli.py	91.26%	11 Missing ⚠️
quantaq_cli/toolkit/flag.py	82.81%	11 Missing ⚠️
quantaq_cli/toolkit/merge.py	66.66%	11 Missing ⚠️
quantaq_cli/log.py	59.09%	9 Missing ⚠️
quantaq_cli/variables.py	73.52%	9 Missing ⚠️
quantaq_cli/toolkit/concat.py	82.35%	3 Missing ⚠️
tests/test_expunge.py	93.33%	3 Missing ⚠️
quantaq_cli/toolkit/expunge.py	90.90%	2 Missing ⚠️
quantaq_cli/toolkit/munge.py	85.71%	2 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##           emmleroy-v1.0.0      #32      +/-   ##
===================================================
- Coverage            96.57%   90.97%   -5.60%     
===================================================
  Files                   17       21       +4     
  Lines                  846      931      +85     
===================================================
+ Hits                   817      847      +30     
- Misses                  29       84      +55

Flag	Coverage Δ
unittests	`90.97% <87.64%> (-5.60%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
quantaq_cli/__init__.py	`100.00% <100.00%> (ø)`
quantaq_cli/schema.py	`100.00% <100.00%> (ø)`
quantaq_cli/toolkit/resample.py	`98.24% <100.00%> (ø)`
tests/test_clean.py	`100.00% <100.00%> (ø)`
tests/test_concat.py	`100.00% <100.00%> (ø)`
tests/test_flag.py	`100.00% <100.00%> (ø)`
tests/test_merge.py	`100.00% <100.00%> (ø)`
tests/test_resample.py	`100.00% <100.00%> (ø)`
tests/test_utilities.py	`100.00% <100.00%> (ø)`
quantaq_cli/toolkit/expunge.py	`90.90% <90.90%> (ø)`
... and 9 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dhhagan · 2026-07-02T18:55:11Z

+    df = fix_timestamps(df, sort_values=True)
+
+    if df.empty:
+        raise Exception("No data")


Should add an error message here before the exception

Done in 1e1d347

dhhagan · 2026-07-02T18:58:54Z

+    df = flag_dataframe(df)
+
+    logger.info("New flag summary:")
+    echo_flag_table(df)


I don't think we want to include a table, at least not by default as loghandlers can't pick it up easily. Should use logs instead where possible and/or include the table as a separate utility that can optionally be triggered?

Got it -- I removed this from the flag_command in 44a89e5. The logging in _add_flag() logs which flags are added and for how many rows (although we will modify _add_flag() later so that it is device-specific).

dhhagan · 2026-07-02T19:00:21Z

+        tmp = fix_timestamps(tmp, set_index=True, sort_values=True, localize_tz=True)
+
+        # merge with the other files
+        df = pd.merge(df, tmp, left_index=True, right_index=True, how='outer')


Is the goal here to merge two files together or n files? We probably want someway to pass the desired suffix for each file in the event of duplicate column names and/or provide the option to only keep 1?

oh hmm I only considered using this to merge two files together, but I'm not sure?
I added an error for attempting to merge >2 files in a49159a.
Also passed desired suffixes for each file + option to only keep one in e739f6d.

dhhagan · 2026-07-02T19:07:11Z

+from quantaq_py.exceptions import InvalidFileExtension
+from quantaq_py.schema import build_dtype_schema, COLUMN_DEFINITIONS
+
+def infer_data_source(df):


See previous comments on how we may want to improve this auto-detection

Made changes in d838aaa

dhhagan · 2026-07-02T19:08:48Z

+        df = df.sort_values(by=tscol)
+
+    # localize the timezone if needed
+    if localize_tz:


We should make sure we only do this if tscol is timestamp or timestamp_iso and NOT timestamp_local

Oops good catch - see 583a903

dhhagan · 2026-07-02T19:09:57Z

@@ -1,7 +1,7 @@
 [tool.poetry]
-name = "quantaq-cli"
+name = "quantaq-py"


So, we can change the name from quantaq-cli, but we probably don't want to use quantaq_py as we already have and maintain py_quantaq which may get a bit confusing...

kept as quantaq-cli in 06e2356

…ulair-ufp

…NOT timestamp_local

…rror msg before exception

…lag() instead

…ep only the overlapping columns from both, left, or right

…frame

…col names still need to be specified

emmleroy added 30 commits June 30, 2026 09:36

Use namedtuple for FLAG and FLAG_CRITERIA

6c74da7

Replace single-filter flag command with multi-filter flagging

02606fb

Add util for determining the timestamp col

28197e1

Refactor - rename bit to flag_value, fix click commands

1f8f9ba

Update tests

79e26bf

Fix indentation

003d5ad

Remove relative imports

d4b2c97

Remove arisense flagging functionality

22b7560

Modify safe_loader() to add device serial number to rawSD columns

61f6d34

Add func to infer data model from the sn column

f0b40aa

Add func to infer data source from the dataframe sampling frequency (…

5ad1d1f

…tdiff)

Infer data model and source in flag_dataframe() instead of taking args

045e34b

Define identical FLAG_DEFINITIONS for QuantAQ products and modifiable…

d04686a

… FLAG_CRITERIA

Convert model-specific FLAG_DEFINITIONS into one generic list

e19c028

Remove tests for rawSD flagging - currently not implemented

fe018eb

Upgrade deps to Click8.x

30646f7

Port expunge_dataframe() and echo_flag_table() from Isoprene

89a2ce6

Move flag_summary and echo_flag_table to flag.py and fix all_nan_colu…

49c186a

…mns definition

Remove arisense from test_expunge() and add some expunge tests from I…

cae63d5

…soprene

Remove feather tests and fix --wind unpack error in resample CLI

2e67857

Remove data model and data source as required args to flag_command

eb2c043

Remove data model and data source as required args in cli logic and t…

04d34ed

…ests

Fix cli args again

9e729a7

Add jupyter for playbook notebooks

cf1fe9a

Rename quantaq_cli to quantaq_py

61ccdea

Add logging configuration file

7d09cc4

Create importable concat_files() function and move concat_command() t…

b11c7c5

…o cli.py

Create importable merge_files() function and move merge_command() to …

554d0fe

…cli.py

Create importable resample_dataframe() function and move resample_com…

6afab69

…mand() to cli.py

Add importable flag_dataframe() and echo_flag_table(), move flag_comm…

736358d

…and to cli.py

emmleroy added 3 commits July 1, 2026 16:53

Move library function into quantaq_py/toolkit

bfe66fb

Add pandera

c50de58

Add Pandera schema checks, always add sn column when safe_loading raw…

a625733

…SD, convert clean_file to clean_dataframe

emmleroy requested a review from dhhagan July 2, 2026 15:10

Define dtypes and schema checks in schema.py

9966b89

emmleroy added 3 commits July 2, 2026 11:22

Remove tests for arisense

8162df4

Update docstrings with types, add helper for fixing timestamps

349b388

Add MOD-UFP to sn_to_model docstrings and tests

2ca1564

emmleroy mentioned this pull request Jul 2, 2026

Add re-flagging based on multiple filters #30

Merged

dhhagan reviewed Jul 2, 2026

View reviewed changes

emmleroy added 17 commits July 2, 2026 16:04

Infer data source based on the frequency of tdiffs, add tests for mod…

d838aaa

…ulair-ufp

Fix: only local timezone if tscol is timestamp or timestamp_iso, and …

583a903

…NOT timestamp_local

Fix: raise error if no files found before fixing timestamps and log e…

84d4fc0

…rror msg before exception

Remove echo_flag_table() from flag_command, rely on logging in _add_f…

44a89e5

…lag() instead

Create a save_file utility function

7cf009c

Passed desired suffixes for overlapping columns, and add option to ke…

e739f6d

…ep only the overlapping columns from both, left, or right

Fix: merge_df bug and add error for attempting to merge >2 files

a49159a

Ensure all errors are logged and that errors are referenced when raised

1e1d347

More informative docstring for clean_dataframe()

c3a2060

Move drop_unnamed() to a helper func used in safe_load and clean_data…

405bc1d

…frame

Update docstring

69b39e2

Specify that u/v columns don't need be present in dataframe, but the …

1311341

…col names still need to be specified

Add intercept handler to route standard logging calls through loguru

537bb1e

Rename quantaq_py to quantaq_tools

e981a49

Set log level for noisy loggers to WARNING

8839d93

Add init file

a294b6a

Undo package rename

06e2356

emmleroy merged commit 3201cd4 into emmleroy-v1.0.0 Jul 8, 2026
6 of 8 checks passed

emmleroy deleted the emmleroy-sc-20240-library-sc-20270-logging-v2 branch July 8, 2026 21:32

Uh oh!

Conversation

emmleroy commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

codecov Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emmleroy Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

emmleroy commented Jul 2, 2026 •

edited

Loading

codecov Bot commented Jul 2, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading

emmleroy Jul 2, 2026 •

edited

Loading