Argo.design

[covesturtevant]

Mostly merging in recent (isolated) Argo development work. Change that impacts current Pachyderm processing is an update to all assignment modules to optionally accept a date range to assign instead of inputting a file with years in it (current behavior). All assignment modules were added to the respective combined modules, as these modules will be run for the data day at the point of processing in the Argo design. The changes are backward compatible, and have been tested in pachy-dev with all downstream datums skipped.

Note that the gcs_data module is only used in Argo, not Pachyderm.

[Copilot]

Pull request overview

This PR merges recent Argo-focused workflow development into the NEON IS data-processing repo, including new Argo workflow templates/config and a new GCS-manifest-driven data loader, plus updates to assignment modules to optionally accept a date range instead of a year-file.

Changes:

• Update SRF/location/group assignment modules to optionally accept DateBgn/DateEnd as an alternative to FileYear.
• Add a new gcs_data module (Docker image + Python scripts) to load L0 parquet from GCS using a JSON manifest and to build datum path selectors from manifest records.
• Add/extend Argo WorkflowTemplate/CronWorkflow examples and a “generic template design” pattern using Kustomize bases/overlays and ConfigMaps.

Reviewed changes

Copilot reviewed 51 out of 80 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pipe/radShortPrimary/radShortPrimary_srf_assignment.yaml	Update Pachyderm SRF assignment image tag to a new SHA build.
pipe/radShortPrimary/radShortPrimary_group_assignment.yaml	Update Pachyderm group assignment image tag to a new SHA build.
pipe/cmp22/cmp22_location_asset_assignment.yaml	Update Pachyderm location/asset assignment image tag to a new SHA build.
pipe/cmp22/cmp22_location_active_dates_assignment.yaml	Update Pachyderm location active-dates assignment image tag to a new SHA build.
modules/gcs_data/tests/init.py	Add placeholder test package for the new gcs_data module.
modules/gcs_data/requirements.txt	Define Python dependencies for the new gcs_data module.
modules/gcs_data/manifest_paths_builder.py	New helper to convert manifest records into normalized output path selectors.
modules/gcs_data/l0_gcs_loader_by_manifest.py	New manifest-driven GCS downloader for L0 parquet selection and download.
modules/gcs_data/Dockerfile	New/updated container build for gcs_data (Ubuntu base, Python, rclone/yq).
modules_combined/location_group_and_restructure/Dockerfile	Extend combined module image to include flow.loc.grp.asgn and install jq.
modules_combined/level1_group_consolidate_srf/Dockerfile	Extend combined module image to include flow.srf.asgn and install jq.
modules_combined/fill_date_gaps_nonregularized/Dockerfile	Extend combined module image to include flow.loc.grp.asgn and install jq.
modules_combined/fill_date_gaps_and_regularize/Dockerfile	Extend combined module image to include flow.loc.grp.asgn and install jq.
modules_combined/calibration_group_and_convert/Dockerfile	Extend combined module image to include flow.cal.asgn and install jq.
flow/flow.srf.asgn/wrap.srf.asgn.R	Roxygen docs update for date-range args.
flow/flow.srf.asgn/flow.srf.asgn.R	Add DateBgn/DateEnd option for SRF assignment time window selection.
flow/flow.loc.grp.asgn/wrap.loc.grp.asgn.R	Roxygen docs update for date-range args and formatting.
flow/flow.loc.grp.asgn/flow.loc.grp.asgn.R	Add DateBgn/DateEnd option for location/group assignment time window selection.
flow/flow.cal.asgn/wrap.cal.asgn.R	Roxygen docs update for date-range args and formatting.
flow/flow.cal.asgn/flow.cal.asgn.R	Add DateBgn/DateEnd option for calibration assignment time window selection.
argo/utilities/testInputs.R	Add simple helper script for inspecting R CLI args in Argo testing.
argo/test_workflows/test_bucket_input_b.yaml	Add Argo test template for GCS artifact mounting (variant B).
argo/test_workflows/test_bucket_input_a.yaml	Add Argo test template for GCS artifact mounting (variant A).
argo/test_workflows/submit_workflow_with_datum_manifest.yaml	Add Argo resource-template example for submitting workflows with datum manifests.
argo/test_workflows/sensor-upstream-workflows-succeeded.yaml	Add Argo Events Sensor example for triggering on upstream workflow success.
argo/test_workflows/sensor-log-everything.yaml	Add Argo Events Sensor example to log events.
argo/test_workflows/my-shared-template.yaml	Add minimal shared WorkflowTemplate example.
argo/test_workflows/location_asset_template.yaml	Add WorkflowTemplate example for location-asset loader usage.
argo/test_workflows/eventsource-transition-workflow-completions.yaml	Add EventSource example watching workflow completion events.
argo/test_workflows/configmap-sitelist.yaml.tmpl	Add gomplate template for generating a sitelist ConfigMap from NetBox.
argo/test_workflows/configmap-sitelist.yaml	Add generated sitelist ConfigMap (full list).
argo/test_workflows/configmap-sitelist-small.yaml	Add smaller sitelist ConfigMap example.
argo/test_workflows/configmap-cmp22-next-workflows.yaml	Add ConfigMap defining next workflows list for chaining.
argo/test_workflows/cmp22_location_group_and_restructure.yaml	Add Argo workflow template example combining loader + combined module + upload.
argo/test_workflows/cmp22_calibration_group_and_convert.yaml	Add Argo workflow template example for calibration group+convert.
argo/test_workflows/cmp22_calibration_group_and_convert_loopEachManifest.yaml	Add loop-per-manifest variant of the calibration group+convert workflow.
argo/test_workflows/call-which-template.yaml	Add example of selecting templates via parameterized conditional steps.
argo/test_workflows/argo_ptb330a_ds_trino_containerset.yaml	Add containerset-based trino + linkmerge example workflow.
argo/test_workflows/argo_ptb330a_data_source_trino.yaml	Add step-based trino load + linkmerge example workflow.
argo/test_workflows/argo_ptb330a_data_source_trino_PVmount.yaml	Add PV-mounted trino load + linkmerge example workflow.
argo/test_workflows/argo_ptb330a_data_source_trino_emptyDir.yaml	Add emptyDir-based trino load + linkmerge example workflow.
argo/test_workflows/argo_ds_trino_workflowTemplate.yaml	Add parameterized WorkflowTemplate for trino + linkmerge containerset.
argo/test_workflows/argo_ds_trino_templateRef.yaml	Add example Workflow referencing the trino WorkflowTemplate.
argo/test_workflows/argo_ds_trino_templateRef_loopSite.yaml	Add example of looping sites and calling a templateRef workflow.
argo/test_workflows/argo_ds_trino_containerset.yaml	Add full workflow example using containerset for trino + linkmerge.
argo/generic_template_design/workflows/README.md	Document the intended base/overlays workflow-template family structure.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/cmp22/kustomization.yaml	Add Kustomize overlay wiring for cmp22 calibration workflow family.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/cmp22/configmap-resource-request.yaml	Add sensor-specific resource ConfigMap for cmp22 overlay.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/cmp22/configmap-env.yaml	Add sensor-specific env ConfigMap for cmp22 overlay.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/aepg600m/kustomization.yaml	Add Kustomize overlay wiring for aepg600m calibration workflow family.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/aepg600m/configmap-resource-request.yaml	Add sensor-specific resource ConfigMap for aepg600m overlay.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/aepg600m/configmap-env.yaml	Add sensor-specific env ConfigMap for aepg600m overlay.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/aepg600m_heated/kustomization.yaml	Add Kustomize overlay wiring for aepg600m_heated calibration workflow family.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/aepg600m_heated/configmap-resource-request.yaml	Add sensor-specific resource ConfigMap for aepg600m_heated overlay.
argo/generic_template_design/workflows/calibration-group-and-convert/overlays/aepg600m_heated/configmap-env.yaml	Add sensor-specific env ConfigMap for aepg600m_heated overlay.
argo/generic_template_design/workflows/calibration-group-and-convert/base/kustomization.yaml	Add family base Kustomize definition for calibration group+convert.
argo/generic_template_design/workflows/calibration-group-and-convert/base/configmap-upload-output-instructions.yaml	Add base upload-output instruction script ConfigMap.
argo/generic_template_design/workflows/calibration-group-and-convert/base/configmap-schemas.yaml	Add base schema repo/revision ConfigMap.
argo/generic_template_design/workflows/calibration-group-and-convert/base/configmap-resource-request.yaml	Add default resource request/limit ConfigMap.
argo/generic_template_design/workflows/calibration-group-and-convert/base/configmap-processing-instructions.yaml	Add base processing-instructions script ConfigMap.
argo/generic_template_design/workflows/calibration-group-and-convert/base/configmap-load-data-instructions.yaml	Add base load-data-instructions script ConfigMap.
argo/generic_template_design/workflows/calibration-group-and-convert/base/calibration-group-and-convert.yaml	Add platform/sensor-agnostic WorkflowTemplate using ConfigMap-injected behavior.
argo/generic_template_design/ARCHITECTURE.md	Add architecture overview for the generic template design approach.
argo/assignment_workflows/location_asset_assignment/location_asset_assignment.yaml	Add CronWorkflow for location-asset assignment using assignment-template.
argo/assignment_workflows/location_asset_assignment/config/configmap-location-asset-assignment-env.yaml	Add env ConfigMap for location-asset assignment CronWorkflow.
argo/assignment_workflows/location_asset_assignment/config/configmap-location-asset-assignment-config.yaml	Add per-source-type R-args ConfigMap for location-asset assignment.
argo/assignment_workflows/location_active_dates_assignment/location_active_dates_assignment.yaml	Add CronWorkflow for location active-dates assignment using assignment-template.
argo/assignment_workflows/location_active_dates_assignment/config/configmap-location-active-dates-assignment-env.yaml	Add env ConfigMap for location active-dates assignment CronWorkflow.
argo/assignment_workflows/location_active_dates_assignment/config/configmap-location-active-dates-assignment-config.yaml	Add per-source-type R-args ConfigMap for location active-dates assignment.
argo/assignment_workflows/group_assignment/group_assignment.yaml	Add CronWorkflow for group assignment using assignment-template.
argo/assignment_workflows/group_assignment/config/configmap-group-assignment-env.yaml	Add env ConfigMap for group assignment CronWorkflow.
argo/assignment_workflows/group_assignment/config/configmap-group-assignment-config.yaml	Add per-source-type R-args ConfigMap for group assignment.
argo/assignment_workflows/calibration_assignment/config/configmap-calibration-assignment-env.yaml	Add env ConfigMap for calibration assignment CronWorkflow.
argo/assignment_workflows/calibration_assignment/config/configmap-calibration-assignment-config.yaml	Add per-source-type R-args ConfigMap for calibration assignment.
argo/assignment_workflows/calibration_assignment/calibration_assignment.yaml	Add CronWorkflow that downloads calibrations and runs calibration assignment per source type.
argo/assignment_workflows/assignment_template.yaml	Add reusable WorkflowTemplate for assignment workflows (load->assign->manifest).

Comments suppressed due to low confidence (2)

modules/gcs_data/l0_gcs_loader_by_manifest.py:186

• This comparison also assumes manifest_source_id is a string; if it’s numeric the .replace(...) call will fail and abort processing mid-run.
  modules/gcs_data/l0_gcs_loader_by_manifest.py:200
• resolved_source_id can be non-string (e.g., numeric source_id from the manifest). Calling .replace(...) without string coercion can raise at runtime and prevent downloads from being written.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.