Repository navigation
feat: Doris continuous analytics [DHIS2-22183] - #24440
Merged
Merged
Conversation
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…no-op negative case
…able catalog qualification - DorisSqlBuilder.createTable() now reorders the rendered column list so primary key columns lead, since Doris requires key columns to be an ordered leading prefix of the schema. Aggregate and completeness analytics tables (where id is not already column 0) would otherwise fail CREATE TABLE. - AbstractJdbcTableManager.swapTable()'s Doris insert-into-main branch now references the staging table via quote() instead of qualifyTable(), since the staging table is a native Doris table, not a federated one (same fix already applied to removeUpdatedData() in earlier commits).
…artition requirement Doris's UNIQUE KEY (Merge-on-Write) model requires the partition column to be part of the key. Analytics tables are partitioned by year, so the key must be composite: (id, year) for aggregate/completeness, (event, year) for event. Confirmed working end-to-end against a real Doris 3.0.7 instance. Also fixes DorisSqlBuilder.getOrderedColumns(), which built the leading key-column block from natural column order rather than the primary key's declared order. For a composite key, Doris requires the leading schema columns to match the key clause's declared order exactly -- for the aggregate/completeness tables, year appears before id in natural column order (FIXED_COLS), which would have produced a mismatched schema. removeUpdatedData() DELETE...USING SQL is unchanged in all three managers -- it matches on id/event alone and was confirmed to still correctly find and delete rows regardless of partition.
…st isPartialUpdate() skipMasterTable is computed from params.isPartialUpdate() (lastYears != null || isLatestUpdate()), not isLatestUpdate() alone, so the new Doris insert-into-main branch was also firing for lastYears=1..N (a normal bounded-years rebuild), not just lastYears=0 (the continuous/latest-partition case this plan is scoped to). removeUpdatedData() never runs for lastYears=N>0, so the branch was merging staging data into main without ever purging rows deleted at source -- a real regression versus the pre-PR no-op-drop-only behavior for that path. Gate the insert branch on params.isLatestUpdate() as well, restoring the exact pre-PR (safe, no-op) behavior for lastYears=N>0 on Doris while keeping the insert working for lastYears=0.
jason-p-pickering
added
the
run-api-analytics-tests-doris
jason-p-pickering
marked this pull request as draft
July 14, 2026 05:58
…ey column) Real Doris CI (Sierra Leone e2e run) failed with errCode = 2, "String Type should not be used in key column[id]" while creating the DATA_VALUE staging table -- aborting the whole analytics table generation job, including everything after DATA_VALUE in iteration order. The synthetic id column (concat(...) of several dimension UIDs) was defined as DataType.TEXT, which DorisSqlBuilder maps to the unbounded "string" type. Doris rejects unbounded string/text as a key column on any key model. event/dx don't have this problem -- they're already CHARACTER_11 (bounded char(11)). Give id a bounded VARCHAR_255 (Doris: varchar(1020)) type specifically when sqlBuilder.requiresUniqueKeyAnalyticsTables() is true (Doris only), leaving Postgres/ClickHouse's TEXT unchanged, in both JdbcAnalyticsTableManager and JdbcCompletenessTableManager.
… into feat/doris-continuous-analytics
…inuous-analytics history preservation
tableExists() filtered on t.table_schema = 'public', but native Doris
analytics tables live in whatever database the connection is on (e.g.
'dhis2'), not 'public' ('public' is only correct for the federated
Postgres catalog). This meant tableExists() always returned false for
real Doris tables, so swapTable()'s skipMasterTable was always false,
so every continuous (lastYears=0) run silently replaced the entire
main analytics table with just the latest-window data instead of
inserting into the existing table - destroying history.
Found via a live manual test against a real Doris instance: triggered
lastYears=0, and confirmed the main table's row count and partition
history matched exactly what a full replace (not a merge) would
produce. Fixed by using database() instead of the hardcoded schema
name, which correctly resolves to the connection's actual database.
Re-verified live after the fix: history preserved, create/update/delete
all correctly reflected, "skip master table: true" confirmed in logs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
JUnit 5 test lifecycle methods don't need public visibility; the rest of the class already uses package-private.
1 of 2 tasks
Contributor
|
The Doris analytics CI failures (12 download tests with HTTP 406) are a master regression from #24463, not from this PR. Fix opened: #24479 Stripped-path content negotiation was matching the JSON |
The Doris branch of removeUpdatedData() joined a live federated Postgres datavalue table against local dimension tables inside a single DELETE ... USING statement. Live validation found this takes 40+ minutes (killed, incomplete) versus ~12 minutes for a full analytics rebuild on the same instance, defeating the purpose of continuous analytics. Split into: a plain federated SELECT (the same join shape already proven fast by populateTable) that materializes matching rows' natural dx/co/ao/pe/ou column values into a small native staging table, then a fully local DELETE against that staging table. No change to the main table's schema, key, or the Postgres/ClickHouse branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
luciano-fiandesio
requested changes
Sep 22, 2026
luciano-fiandesio
left a comment
luciano-fiandesio
left a comment
Contributor
There was a problem hiding this comment.
I found a couple of possible issues, it would be good to address them.
Also, we are aware of the issue with lastAnyTableUpdate being global, but this will be addressed later.
…continuous-analytics
…s Doris tables Addresses two review comments from Luciano on #24440: - A main table created before unique-key analytics tables were introduced stays on the duplicate-key model until a full rebuild recreates it, but removeUpdatedData()'s DELETE requires unique-key (Doris only supports DELETE on unique-key/MOW tables). Add AnalyticsTableManager. isReadyForContinuousUpdate(), checked before a continuous run is attempted: it queries SHOW CREATE TABLE and refuses to proceed if the existing main table isn't unique-key, instead of letting the DELETE fail and letting the update continue regardless (removeUpdatedData()'s stage uses FailurePolicy.SKIP_STAGE, so a failure there doesn't abort the job on its own). Also make the abort explicit if removeUpdatedData() fails for any other reason, rather than proceeding to swap. - A bounded lastYears=1..N update against an existing main table on a continuous-analytics engine (Doris) previously discarded the rebuilt staging data silently, since removeUpdatedData() only runs for the latest-partition (lastYears=0) case and merging without it would leave stale/deleted rows behind. Fail loudly instead: only a full rebuild or lastYears=0 are supported once the main table exists on such an engine; bounded updates are tracked separately as follow-up work. Postgres is unaffected (no declarative partitioning); ClickHouse is unaffected (no continuous analytics support). 🤖 AI Assisted
Addresses Luciano's readability comment on #24440 (gist: f93f938afb471511281593203159edd2): swapTable() packed together several independent decisions (full vs. partial vs. continuous update, whether the main table exists, how the database manages partitions, and how to publish: replace/reparent/merge) into nested conditionals. Extract a TableUpdateAction enum (REPLACE_TABLE, REPLACE_PARTITIONS, MERGE_ROWS) and a resolveUpdateAction() method that makes the decision once, up front, then dispatches to three small named methods. This also generalizes the two fail-loudly behaviors from the previous commit from Doris-only to any declarative-partitioning engine without continuous analytics support (currently ClickHouse): a continuous (lastYears=0) or bounded (lastYears=1..N) update against an existing main table on such an engine now throws instead of silently discarding the rebuilt staging data, since there is no purge step available to make either publish safe. 🤖 AI Assisted
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
A bounded lastYears=N update against an existing Doris main table threw an IllegalStateException. Before this PR the same update gave correct results by replacing the whole table, so throwing broke something that worked, including scheduled analytics jobs that use "last years". On declarative-partitioning engines the populate step does not filter by year, so the staging table holds every year and replacing the main table is correct. Return REPLACE_TABLE for this case on engines that support continuous analytics (Doris) and log a warning that the whole table was rebuilt. ClickHouse, where this path has not been verified, still throws. 🤖 AI Assisted
When a Doris analytics table was created before continuous updates were supported, the continuous update for that table type was skipped with an ERROR in the server log only. The job log showed the readiness check at INFO level and the job ended as updated, so administrators had no sign in the job output of why continuous updates were not being applied. Report the readiness check as a failed stage with SKIP_STAGE, so the reason and the required full rebuild appear as an ERROR in the job log while the other table types are still updated. 🤖 AI Assisted
On ClickHouse a continuous (lastYears=0) update threw an IllegalStateException at the swap step, failing the whole analytics job. Before this PR the same update gave correct results by repopulating and replacing the whole table. The latest-update partition change in getTablePartitions() also applied to ClickHouse, narrowing the staging table to changed rows only, so simply replacing the main table would have lost data. Only use the narrowed latest-update partition on engines that support continuous analytics (Doris), so the staging table holds all data on ClickHouse, and replace the whole main table instead of throwing for a continuous update on engines without continuous support and for a bounded lastYears update on any declarative-partitioning engine. Both log a warning that the whole table was rebuilt. 🤖 AI Assisted
PRIMARY_KEY is (event, year), not event alone; year is only part of it because Doris requires the partition column in a unique key. Spell out that the staging table carries just event and the delete therefore matches across all years, so an event moved to another year is still removed. 🤖 AI Assisted
…rted run When a continuous analytics update aborted for a table type (the continuous update readiness check or removeUpdatedData() failed), the type was still stamped as successfully updated, and on Doris the next run's window started after the changes the aborted run never published. AnalyticsTableService.create() now returns whether the update completed; only completed types get their per-type last successful update settings. On engines with a unique key on analytics tables (Doris), the continuous update window now starts at the per-type settings rather than the shared ones, which also advance when a type was skipped or aborted. This is the read path switch deferred from DHIS2-21992 (#24899), for Doris; Postgres still uses the shared settings. create() returns true when there is no source data or nothing to update, and false when the job was cancelled. Co-authored-by: Luciano Fiandesio <luciano@dhis2.org> 🤖 AI Assisted
A program created after the last full rebuild has no event analytics table yet. removeUpdatedData() ran its delete against that missing table, which failed, and the guard against swapping after a failed delete then skipped the swap for every program of the table type. The deletes already committed for the other programs were not followed by their swap, so their updated events disappeared from analytics, and every following continuous update failed the same way until a full rebuild. Affects Postgres and Doris. - Only remove updated data from tables whose main table exists; a new table has no stale rows and becomes the main table when swapped. - Give a program without a main table all of its data rather than the continuous update window, so its new main table is complete. Reported by Luciano Fiandesio. 🤖 AI Assisted
…per table [DHIS2-22183] Review feedback: - Check continuous update readiness right after the tables are known, before any staging work. A table type that is not ready no longer populates its staging tables in full on every scheduled run only to throw them away. - On unique-key databases (Doris), remove updated data per table. A table whose delete fails keeps its staged data unpublished, the other tables are published, and the update is reported as incomplete so the window is not advanced and the next run covers it again (merging a window again is idempotent). Before, one failed table aborted the swap for all tables after rows had already been deleted from the others. - Other databases (Postgres) swap the tables in after the delete whatever its outcome, unchanged from master. - Limit the full-history window for programs without a main table to unique-key databases. On Postgres the latest partition is replaced with the same window on every run, so the epoch window made a new program's older events appear and then vanish; it also no longer queries tableExists per program there. - Tests: drop class-wide lenient strictness for lenient() on the shared stubs that need it, and use assertFalse instead of assertTrue(!...). 🤖 AI Assisted
jason-p-pickering
changed the title
|
luciano-fiandesio
approved these changes
Oct 7, 2026
maikelarabori
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes continuous/incremental analytics table updates (
lastYears=0) for Apache Doris, matching functionality that already exists for PostgreSQL.Implements/fixes https://dhis2.atlassian.net/browse/DHIS2-22183
Additional info on Confluence
🤖 AI Assisted