Use slicks for scanning and discovery#20
Merged
haoruizhou merged 14 commits intomainfrom Feb 18, 2026
Merged
Conversation
Ensure DATE_BIN uses a fixed origin (config.start) instead of the per-query t0 so bucket boundaries remain consistent across chunked/split queries. Adds a bin_origin variable and updates the SQL DATE_BIN(...) third argument to reference that origin, preventing misaligned bins when scanning runs in multiple segments.
Replace custom InfluxDB SQL scans with the slicks library: add slicks to requirements, delegate data-availability scanning to slicks.scan_data_availability and convert its ScanResult into the flat API dict format, and replace manual sensor SQL queries with slicks.discover_sensors (including fallback behavior). Update service connectivity check to use slicks.connect_influxdb3 / slicks.get_influx_client. Add docstrings and small refactors (private helper renames) while removing direct InfluxDBClient3 SQL query logic.
Handle Slack event errors
This reverts commit ab09885.
_build_sensor_fallback_range now finds the run with the longest duration and uses its time range for the fallback.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8569782d98
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
haoruizhou
force-pushed
the
data-downloader-slicks-integration
branch
from
a6d3e1f to
62776dc
Compare
Add a simple HTML status page at / (app.py) that reports InfluxDB connectivity, runs, sensors and scanner status for quick debugging (imports HTMLResponse). Adjust periodic worker retry behavior to wait 60s after failures and keep the regular interval after successful scans (periodic_worker.py). Fix timezone handling in sensor discovery (sql.py) by converting UTC-aware datetimes to naive datetimes to work around slicks 0.1.3 generating invalid SQL for InfluxDB 3; apply same fix to fallback ranges. Add several debugging/reproduction scripts for inspecting slicks and InfluxDB behavior (installer/data-downloader/testing/*) to help reproduce timestamp/connection issues. Update docker-compose to inject INFLUX_URL, INFLUX_TOKEN and INFLUX_DB into the api and scanner services so slicks child processes can self-configure.
Introduce multi-season configuration and handling across the data-downloader service and frontend. Key changes: - Config: add SeasonConfig and _parse_seasons to read SEASONS env var; expose seasons in Settings with sensible defaults. - Backend services: manage per-season Runs/Sensors repositories (file suffixing), return seasons list via /api/seasons, accept optional season parameter for runs/sensors/note/query endpoints, scan all configured seasons and store results per-season, and default to newest season when none provided. - Influx/query: fetch_signal_series now accepts a database override and service query routes pass season-specific DBs. - Scanner/Storage: adjust season start/end boundaries (season starts Aug previous year, ends Jan 1 next year); storage filenames include season suffix. - Periodic worker: support scheduling either by interval or daily time via SCAN_DAILY_TIME env var. - Frontend: fetch seasons, add season selector UI, propagate selected season to all API calls and adapt visuals (accent color), and add season type to types. - Docker compose: add SEASONS and SCAN_DAILY_TIME environment wiring and minor formatting fixes. - .gitignore: ignore installer/data-downloader/data. Fallbacks and defaults are provided when SEASONS is unset, and scanning continues per-season even if individual season scans fail.
haoruizhou
force-pushed
the
main
branch
2 times, most recently
from
fc33293 to
bedc080
Compare
haoruizhou
force-pushed
the
data-downloader-slicks-integration
branch
from
b76ea2e to
93c6d7d
Compare
haoruizhou
force-pushed
the
data-downloader-slicks-integration
branch
from
eb5b063 to
14b610d
Compare
Handle config.table values that may include a schema (e.g. "iox.WFR25"). server_scanner.scan_runs now splits config.table into schema and table_name and passes them to slicks.connect_influxdb3; it also sets the local table variable to None to rely on the global configuration. sql.fetch_unique_sensors now includes schema and table when connecting as well. These changes ensure the DB connector receives explicit schema/table info and that scanning functions behave correctly when config.table is provided as "schema.table".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.