[ila]

• DuckDB 2.0: port the compiler to its native CTE-based correlated plans and update the extension/CI APIs.
• SUM correctness: fix packed exact-SUM carries, overflow accounting, partial-state merging, and finalization; make approximate cascading lossless; fix negative-only counter worlds and unsigned magnitudes ≥2^63.
• Memory: serialize unfinished stochastic states for DuckDB's disk spilling, and keep small SUM groups buffered during merging/result construction to address TPC-DS Q78 (TPC-DS Q78 SF30: AS aggregation exhausts memory on 32 GB Graviton4 #36).
• Noise: give PAC releases distinct advancing random streams, reset stream allocation per query, and prevent DuckDB from folding random scalar calls into constants.
• Compiler coverage: preserve world-answer types through scalar-subquery wrappers, nested aggregation, CTE definitions/readers, and copied plans.
• UNION: carry masks through UNION ALL, OR masks when deduplicating UNION/DISTINCT, and fix the top-level UNION privacy bypass (UNION and UNION ALL skip the privacy rewrite #37).
• TPC-H: move Q20's nation filter to the outer WHERE clause; optimize Q17/Q21 in extension code while retaining Q17's SQL and Q21's ownership joins.
• Measurement code: calculate actual microbenchmark medians, read DuckDB 2.0 operator profiles, and report profile errors instead of zero timings.
• Applications: add the full SSB query set and published MEPS survey analyses, including official 64/128-replicate designs and joined event records.
• Remaining limitations: explicitly track unfixed negative-only scalar SUM/AVG, clipped MAX, nullable-owner grouping, and benchmark-validation gaps below.

This is a change record relative to the original paper implementation and experiments, including work already merged into main. The SUM fixes, SSB/MEPS additions, initial TPC-DS runner, private UNION support, and #37 fix already landed there. This PR introduces the 2.0 port, state spilling, SUM memory/update-adapter changes, PAC random streams, and correlated-plan optimization. Draft #39 contains the larger handwritten TPC-DS benchmark additions.

Here a membership mask says which worlds contain a tuple. An aggregate state holds the unfinished results for those worlds within one SQL group, including buffers and separately allocated counter arrays.

SUM bugs: what failed and what changed

Exact integer SUM — fixed on main in #28

These were exposed while validating all SSB world/group answers against ordinary SQL. The arithmetic examples below use SET pac_mi=0 and SET pac_sample_diversity_check=false to isolate the aggregate from release noise and diversity checks. The fixes are in b7b88e1, with regression cases in pac_sum.test.

1. The exact-build flag did not select the exact algorithm. PAC_NOAPPROX=ON defined PAC_NOAPPROX, but the source selected the algorithm with PAC_EXACTSUM. CMakeLists.txt now maps the option to the macro actually read by the implementation. This matters when interpreting measurements labelled “exact”.

2. Packed signed additions could change another world's counter. Imagine two 8-bit lanes holding [-1, 0], represented by 0x00ff. Adding [1, 0] as one machine-word addition yields 0x0100: the intended first-lane result is zero, but the carry also increments the second lane. The fix isolates the high bits so carries cannot cross lane boundaries. The relevant implementation is AddToTotalsSWAR:

   uint64_t high_bits = MASK << (BITS - 1);
   uint64_t addend = val_packed & expanded;
   uint64_t sum = (total[i] & ~high_bits) + (addend & ~high_bits);
   total[i] = sum ^ ((total[i] ^ addend) & high_bits);

3. A signed net total was not a valid bound on each world's subtotal. For example, five pairs of +30, -30 have net sum zero, but a world selecting only the positive rows accumulates 150 and can overflow a signed 8-bit counter. Overflow accounting now tracks input magnitudes rather than cancellation-prone net totals. INT64_MIN's magnitude is formed in unsigned arithmetic; the exact signed path routes it to a wider accumulator. See the update code and exact-state overflow/flush logic.

4. Merging partial exact states repeated the packed-word carry problem. Two worker states were combined by adding packed words. The fix combines the individual lane values after making room at the corresponding level. This is distinct from the per-row update fix. See CombineLevel and PacSumCombineInt.

5. Finalization could leave contributions behind. A chain of if / else if flushed only the first allocated width, even when wider levels also contained values; an intermediate level could also be missing. Finalization now visits populated levels bottom-up and allocates an intermediate level when a pending contribution must pass through it. See Flush8/16/32/64 and Flush.

“Exact” describes the integer accumulation path. Returning the aggregate's FLOAT lanes still rounds sufficiently large integers; validation compares at that output precision.

Approximate SUM — loss during cascading/finalization fixed on main

Trigger: SSB Q4.3 had a world with true sum 55,890 but output 53,248, about 4.7% low. A much larger value in a different world caused finalization to push the small world's counter up to a coarse common level. The old cascade also discarded low bits whenever counters moved between levels.

Resolution: 3da3eef retains each counter's low four bits, transfers only its upper bits, sums every level during finalization, and handles a nearly-full source level when merging states. 0bc3082 preserves those fixes but restores per-input truncation after round-to-nearest added about 2–3% to the measured update hot path. The final implementation truncates inputs; it does not round them to nearest.

Simplified pseudocode for one world, from Cascade/GetTotals/CombineFrom:

on input magnitude v:
    k = level_for(v)
    counters[k, world] += v >> (4*k)     # input truncation remains

when level k must cascade:
    counters[k+1, world] += counters[k, world] >> 4
    counters[k, world]   &= 15          # retain the low bits

finish:
    total[world] = sum_k counters[k, world] * 2^(4*k)

The identity x = (x >> 4) * 16 + (x & 15) explains why the transfer is lossless. Finalization no longer forces partially populated levels upward. Separate positive/negative magnitudes are combined afterward in the default signed implementation.

This example now produces 55,888, with only the remaining input truncation, even when the other world contains a large value:

SELECT (as_sum(mask, value))[1] / 2 AS small_world
FROM (VALUES (1::UBIGINT, 55890::BIGINT),
             (2::UBIGINT, 4950674::BIGINT)) t(mask, value);
-- 55888; divide by two to remove the AS API's sampling correction.

The routing code retains approximately 9–12 significant bits and documents a per-input relative truncation bound of 2^-8. This is not the old 0.024% universal bound on a final answer. Cancellation can amplify relative error in a signed sum, and FLOAT result construction contributes output rounding. The paper must describe these distinctions.

Negative-only counters and large magnitudes — fixed on main in #29

Found while cross-checking the PostgreSQL implementation; fixed by 40b15a1, with cases in pac_counters.test.

1. Negative-only worlds disappeared from as_sum's returned counter list. Finalization subtracted the negative totals through a dummy mask, then discarded worlds absent from the positive mask. It now passes the actual output mask so negative membership is included. See PacSumAvgFinalizeCountersInternal.

   SELECT (as_sum(3::UBIGINT, value))[1:2]
   FROM (VALUES (-5), (-7)) t(value);
   -- Before: [0, 0]. Now: [-24, -24], including the normal 2x correction.

2. Unsigned values ≥2^63 were routed as negative signed values. GetLevel(int64_t) chose the wrong level, arithmetic shifts sign-extended unsigned magnitudes, and negating INT64_MIN in signed arithmetic was undefined. The level function now accepts uint64_t, unsigned input uses a logical shift, and magnitude formation uses uint64_t(0) - uint64_t(value). The fix covers large UBIGINT inputs and INT64_MIN's magnitude without adding another per-row state structure.

Important remaining distinction: the counter-list fix does not fix the fused scalar negative-only SUM/AVG problem in #30. The counter example above is correct, while as_noised_sum(3::UBIGINT, value) over the same rows still returns NULL. The open-issues section records that gap explicitly.

TPC-DS findings and the DuckDB 2.0 CTE port

A CTE is a named intermediate query result. In DuckDB's logical plan it has a definition that produces columns and readers that consume them. Each reader has its own column identifiers and cached types. DuckDB 2.0 also uses generated CTEs internally to turn correlated subqueries into shared computation; this is not limited to SQL that explicitly writes WITH.

For example, a correlated average can conceptually become an intermediate table keyed by part. After stochastic rewriting, its average column may contain 64 world-specific values rather than one scalar. Updating only the definition is insufficient: readers and expressions above them must know that the column is a list, where its world membership comes from, and when the values may be finalized.

definition:  part_key -> averages[64]
reader:      part_key -> averages[64]       # same type/lineage, different column ids
predicate:   keep[j] = quantity < 0.2 * averages[j]
next_mask:   original_mask AND pack_bits(keep)
outer SUM:   update the worlds selected by next_mask

The example explains the required propagation; it is not literal generated SQL. The implementation uses the default 2.0 planner rather than enabling the deprecated delim_join_as_cte=false mode. Fixes include:

1. Q09-style scalar subqueries inside CASE retained stale scalar types. DuckDB wraps a scalar subquery in logic that checks whether it returned more than one row. Once its aggregate returns world counters, the wrapper's first(...)/CASE result must carry that list type too. 85346c6 and the 2.0 port update those types without removing the cardinality error. Regression cases cover different CASE branches and a subquery that genuinely returns multiple rows.

2. CTE boundaries lost mask columns or counter lineage. The old traversal could see the reader without following its definition, and a rewritten producer could have list output while its reader still advertised a scalar. The native port 66c9006 traces definitions/readers, propagates needed membership columns, refreshes reader types, and tracks every stochastic input to an expression rather than only the first one. See SyncCTECounterBindings/RewriteBottomUp, counter-source detection, and CTE mask-column propagation. Generated correlation-key deduplication is kept separate from the tuple stream carrying membership; adding a mask to the deduplication key would change the grouping.

3. Q65 exposed AVG being decomposed twice. The AVG rewrite treated an aggregate over a list of world answers like a scalar (mask, value) AVG and attempted another SUM/COUNT decomposition, causing integer/list binding failures. 3e6723d recognizes the one-argument list form and leaves it to the existing list implementation. See pac_avg_rewriter.cpp.

4. Q94 exposed copied readers referencing the wrong CTE. Deep-copying a correlated plan assigned new identifiers to definitions but left a reader referencing the original definition. Later optimization then looked for statistics under the wrong identifier. 3e6723d remaps the reader's definition identifier if that definition belongs to the copied subtree, while preserving references to definitions outside it. See ApplyIndexMapToSubtree.

5. Raw UNION branches needed a shared membership representation. TPC-DS's UNION shapes motivated the mask-carrying/OR-based rewrite below. Branches are not simply counted as supported because they execute; membership propagation and sampled-world semantics are checked separately. Unsupported constructs retain explicit errors.

6. Q78 exceeded memory despite ordinary DuckDB succeeding. Its high-cardinality, three-SUM-per-group states motivated state spilling plus the late-allocation fixes described below. Issue #36 retains the original failure conditions.

The larger handwritten-query/reference expansion is in draft #39. Its per-world validator currently creates a macro referring to nonexistent as_stddev_samp even for queries that do not use STDDEV. Q17 also actually requires a STDDEV rewrite. This remains benchmark work to resolve; it is not claimed as fixed compiler coverage.

UNION, UNION ALL, and DISTINCT

Implemented on main in #38 by 35343a4, with additional join-shape guards in db220e2. See PrivateUnionRewriter and independent sampled-world tests.

• UNION ALL: append both branches, retaining row multiplicity and carrying each row's hidden mask.
• UNION: deduplicate visible values and OR masks. If value x arrives with 001 and 100, emit x,101, not two distinct (value,mask) rows. In a world containing both original rows, x must still appear once.
• Outer SELECT DISTINCT: apply the same mask merging to the selected visible columns, including projected subsets and NULL values. Masks can pass through materialized CTEs and remain hidden from SQL output.
• Scope: one sampling-entity relation and one unambiguous owner per raw branch. DISTINCT ON, unsupported branch-local aggregation/DISTINCT/LIMIT, and joins needing different unmatched rows in different worlds are rejected. This is separate from combining already finalized aggregate answers.

Top-level privacy bypass (#37), fixed directly on main: a different entry-point bug skipped all checks when UNION was the outermost operation. For example, SELECT COUNT(DISTINCT pid) FROM p UNION ALL SELECT 0 returned the exact count, and SELECT protected_column FROM t UNION ALL SELECT 'x' exposed protected rows. 0a833e8 routes top-level UNION through normal checks/rewrite, checks both branches, and follows union columns through outer projection/grouping. The fix is incorporated here, with its 2.0 table-index adaptation in 371825e. Issue #37 is closed.

Regression coverage checks protected output, both branch positions, wrappers, EXPLAIN, CTAS, and required parameters across the privacy modes. Existing restrictions on DP query shapes remain; multiple branches do not silently receive independent full query budgets. Two old tests relied on the bypass: exact full-data UNION expectations were replaced by independently sampled SQL references, and protected-key fixture construction now explicitly disables rewriting only during setup. See privacy_links.test and the fixture change.

PAC noise: independent draws, seeds, and volatile functions

Reused random sequences — fixed in this PR

While checking grouped PAC releases, we found that restarting generators from the same query seed reused sequences across groups/batches/workers. In one Q18 diagnostic, either all 57 groups became NULL or none did. The related row-filter issue #31 reported 0.929–0.930 agreement between decisions at matching row positions in different scan groups, although their independent decisions should agree roughly half the time.

The stream implementation is in e64bdc1; per-query resetting is in 29c8a3c. See PacRandomState/PacRandomStream, the PCG engine, and categorical local-state initialization.

old pattern:
    for each worker or restarted batch:
        rng = Generator(query_seed)        # same sequence starts again
        for cell in batch:
            release(cell, rng.next())

current pattern:
    on QueryBegin:
        query_state.next_stream = 0

    when a worker/batch needs a stream:
        id = atomic_fetch_add(query_state.next_stream, 1)
        rng = PCG(query_seed, id)           # distinct stream
        for cell in batch:
            release(cell, rng.next())       # advance, including equal-input cells

This avoids hashing every cell's 64 counters or initializing a large generator per group. The draw distribution/calibration code remains separate from stream allocation. The change concerns PAC noise and random categorical decisions; it does not replace the DP mechanisms' own noise generator with PCG.

On the current 2.0 build, rerunning #31's four-million-row/eight-thread example gave agreement 0.50007 and 0.50016 at the two reported offsets. That issue remains open on GitHub because this fix is on the unmerged PR branch. Independent noise/NULL-decision and equal-counter regressions are in pac_noise_independence.test.

Seed reproducibility — fixed without requiring stable group identities

Initially, allocating distinct streams without resetting their sequence made subsequent executions differ even with a fixed seed. 29c8a3c resets the stream allocator per query, including prepared executions. Unrelated earlier queries no longer shift later draws.

Bit-identical seeded results require the same build, data, settings, execution order, and initial p-tracking state. Tests use one thread and stable ordering. Parallel scheduling can assign draws to different groups even at an unchanged thread count; invariance to row/thread reordering is not promised. See the documented seed contract.

Constant-folded scalar noise — fixed in registration and planning

If DuckDB treats a random function as deterministic, as_noised(constant_counter_list) can be evaluated once and reused for many output rows. The scalar release functions are registered VOLATILE: each call must be evaluated separately even with equal arguments. The final registration and correlated-plan handling are in 895a563.

incorrect optimizer assumption:
    draw = as_noised(the_same_counters)
    return draw for every row

required behavior:
    for each output row:
        return as_noised(the_same_counters)  # fresh draw

The 4,096-row isolated regression produced distinct scalar outputs for both release functions. P-tracking is disabled in that diagnostic to isolate random-draw assignment from adaptive calibration. These checks detect the implementation defect; they are not a proof of the full privacy mechanism.

TPC-H changes and their motivation

The only final TPC-H SQL edit in this PR is the handwritten AS Q20. AS Q17, AS Q21, the ordinary control SQL, naive SQL, simple-hash SQL, and C++ runner retain their previous text.

1. Q20: an unnecessarily correlated nation filter. n_name='CANADA' referred to the outer nation row but appeared inside the IN subquery. f122d3b moves it to the outer WHERE, matching canonical TPC-H/control SQL. It still selects Canadian suppliers and retains the LINEITEM-to-ORDERS join needed to recover customer membership. The deterministic SF30 output matches all 5,405 reference rows.

2. Q21: too much work finding rows that disqualify an order. The query asks whether other suppliers exist and whether another supplier delivered late. The slow 2.0 plan formed a large candidate set before applying the exclusion. The extension now passes the relevant order-key restriction into the LINEITEM/ORDERS work that finds disqualifying rows, using existing DuckDB execution filters. The restriction is only applied on the side where removing irrelevant candidate rows preserves the NOT EXISTS result. SQL NULL equality, duplicates, output columns, LIMIT barriers, and volatile expressions are checked. All original ORDERS joins remain; an attempted SQL join removal was reverted in 895a563. The SF30 output matches all 100 reference rows. See as_correlation_optimizer.cpp.

3. Q17: retain sharing while evaluating fresh noise per row. Registering scalar noise as VOLATILE correctly prevented draw reuse but also made the new planner repeat deterministic correlated aggregation. The extension separates those operations: compute/reuse counters by the real correlation keys, then evaluate the random release separately for each outer row. Safe brand/container filters still apply before that work. This is implemented in 895a563 rather than modifying Q17 SQL. The noise-disabled reference result is 9,076,953.965714285.

4. Grouped SUM adapter overhead. In the first 2.0 adaptation, updating a group created sliced vector wrappers and incurred their reference-management work. 72f8fc5 reads the original vectors using existing group row selections directly. This changes the engine adapter, not the per-value arithmetic or state layout. See PacSumUpdateValues/PacSumUpdateClusters.

State spilling and the Q78 memory fix

Issue #36 records TPC-DS Q78 exhausting the 32 GB Graviton machine while ordinary DuckDB succeeds. Q78 forms many year/item/customer groups, each with three stochastic sums, then joins the channel aggregates. Two separate memory problems had to be addressed.

1. Make the complete unfinished state spillable

The hash-table entry contains pointers to buffers and counter arrays allocated elsewhere. Writing its pointer bytes does not write those counters, and reading those bytes back does not reconstruct valid ownership. DuckDB 2.0 exposes an aggregate state-export/import interface; the extension implements that interface and lets DuckDB choose and write aggregate partitions.

86050be introduced SUM transport; 3e6723d extends mixed stochastic aggregation. See the shared writer/reader/callbacks, SUM state codec, COUNT codec, and MIN/MAX codec.

• Export: write buffered values and masks, update counts, allocated SUM levels/overflow budgets, positive/negative arrays, floating counters, COUNT's packed bytes/overflow totals, and MIN/MAX/list values and validity. Write contents, never process-local addresses. Preserve SQL types, DECIMAL specialization, and constant correction parameters needed to rebind the aggregate.
• Import: allocate new owned storage, reconstruct the unfinished state, and resume merging. Reconstructing a finalized rounded scalar would discard information needed for later updates and would risk applying sampling correction/noise too early.
• Coverage: membership-mask SUM/COUNT/MIN/MAX and their list-input counterparts expose this interface; scalar AVG uses the SUM/COUNT rewrite. Every aggregate in a shared hash table needs a compatible state layout for DuckDB to choose this path.
• Semantics: exporting does not finalize, select a secret world, or add noise. The format is internal to the implementation/build, not a cross-version interchange format. The ordinary no-spill path does not write temporary state files.

2. Prevent late expansion of millions of small SUM states

Spilling alone still exhausted a 16 GB DuckDB limit. The later merge/result phases expanded groups containing only one or two buffered inputs into full stochastic states, after earlier spilling decisions had already been made. Writing inactive partitions to disk does not bound all such late allocations.

86050be, retained/scoped by 68b575c, keeps small groups buffered when merging and finalizes a temporary copy using storage released after the callback. See CombineSumBuffers/PrepareSumFinalization. The permanent source state no longer grows simply because a result was emitted. These SUM memory changes also apply in memory; COUNT/MIN/MAX retain their existing in-memory update/merge/finalize kernels.

With both changes, the tested automatic Q78 SF30 query completed at four threads with a 6 GB DuckDB limit in 18.2 seconds, about 254 MiB peak temporary storage, and all 100 rows matching the independent world-zero SQL reference. This is a managed-memory limit, not a 6 GB process-RSS claim or a guarantee for every thread/memory configuration. The new Graviton SF30 run also completed five measurements for both handwritten and automatic AS at 16 threads (approximately 2.21 s and 2.23 s medians). Its performance summary labels these executions as not numerically validated; successful execution is separate from the independent output checks in the lower-memory diagnostic.

Additional bugs found while testing state restoration

1. Unaligned state pointers after variable-length aggregates. Fixed by 0b7852d. A string_agg restoration can leave the shared arena's next address unaligned. Buffered stochastic states use low pointer bits as metadata and assume an aligned real pointer. For example:

   next address after a string = 0x1003
   stochastic allocation      = 0x1003
   decode pointer: 0x1003 & ~7 = 0x1000   # wrong address
   
   with AlignNext():
   next allocation            = 0x1008
   decode pointer: 0x1008 & ~7 = 0x1008   # correct
   

   Import now calls AlignNext before rebuilding a state. This is pointer-tagging alignment, not alignment of a SIMD load. The regression in pac_sum.test mixes string_agg(repeat('x',13),'') with as_sum(...), forces external aggregation, and checks all 100,000 groups. It crashed before the fix and passes afterward.

2. NULL imported states lost identity semantics. Fixed by 24b7041. A NULL serialized value must leave the caller's initialized empty state intact: finalizing it remains NULL, while combining it with a real state preserves that real state. Import now does that instead of treating NULL as a normal payload.

3. Windows could select the wrong SUM state layout. Fixed by 1c65642. The linker can merge identical initializer functions, so callback addresses do not uniquely identify integer versus floating layouts. Export/import selection now follows the declared SQL input type in RegisterSumStateExport. Focused MSVC SUM tests pass after this change.

4. Restored small buffers could be overwritten on combine. COUNT/MIN/MAX imports materialize their restored buffers using existing kernels before normal merging; SUM keeps its buffer because its updated combiner handles it. See state-transport documentation and 3e6723d.

State regressions also cover signed/unsigned/floating/DECIMAL types, correction arguments, empty/buffered/high-cascade states, combining restored states, and malformed/truncated payloads. An actual-spill checksum is compared with the in-memory result.

Additional workloads: SSB and MEPS

SSB: all 13 joined queries over customer-sampled worlds

Origin and files. The end-to-end SSB benchmark was added in b7b88e1, merged in #28, and refined in 25409d8. The 2.0 compatibility updates are in 66c9006 and e18010f. Relevant artifacts:

• Methodology README, C++ runner, and build/run wrapper.
• Original SQL, handwritten AS SQL, and ownership schema. Filenames q11.sql through q43.sql denote the standard Q1.1 through Q4.3 names.
• Data preparation, external generator patch, and paper-style plotter.

What the experiment measures. We run all 13 standard SSB queries on the original five tables at SF30. CUSTOMER is the sampling entity, and every LINEORDER row inherits its customer's membership. priv_hash(hash(lo_custkey)) is evaluated inside the timed SQL. Because LINEORDER already has the customer foreign key, obtaining membership does not require an extra CUSTOMER join; queries using customer attributes still retain that join. The original joins, predicates, grouping keys, and ORDER BY clauses remain. Depending on the query, one to four original joins execute.

The fixed configuration is privacy_seed=42, pac_mi=0, and pac_hash_repair=true. Each customer appears in exactly 32 of 64 overlapping worlds, and all its fact rows use the same mask. These are customer-level subsets added to SSB for this experiment; they are not an independent Bernoulli sample for each world or part of the original SSB specification. PART, SUPPLIER, and DATE remain shared. All timed variants read the base tables; neither a prejoined fact table nor a precomputed membership table supplies the AS query.

Baselines and timing. The runner compares ordinary full-data DuckDB, repeated sampled-world SQL, and one AS query returning all 64 worlds. Full-data DuckDB produces a different amount of work and is a single-query cost reference. The repeated-SQL time is the latency of world zero multiplied by 64, following the TPC-H convention; all other worlds execute for validation but their latencies are not added to the reported bar. AS shares the relational work and maintains 64 aggregate answers per group.

Timing includes planning, scans, membership hashing/world filtering, the original joins, aggregation, sorting, and result materialization. Data generation, import, C++ answer conversion, and correctness checks are outside the timer. Each mode has one excluded warmup and five measured runs, with rotating measurement order; figures use per-query medians. The current Graviton run uses 16 threads and a 30 GB DuckDB memory limit. Approximate and exact SUM builds run separately. The exact run measures AS only and reuses ordinary baseline timings, while still executing its independent world references for validation. The wrapper restores the default approximate build afterward.

Correctness and numerical error. Every handwritten AS query is checked against ordinary SQL for every world and group. as_count distinguishes an absent group from a present group whose sum is zero. NULLs, missing/extra groups, and non-finite results are checked. Since the aggregate API scales selected sums by two, validation divides AS sums by two before comparison. Exact SUM must agree after converting the reference to the AS output's FLOAT precision. Approximate SUM records maximum/mean normalized error, maximum absolute error, and output-rounding mismatches; it does not hide numerical discrepancies behind a pass tolerance. Normalized error is abs(actual-reference)/max(1,abs(reference)).

Automatic rewriting/execution of the 13 original queries is audited separately from correctness of the handwritten SQL. AS orders list-valued aggregate keys lexicographically; validation compares per-world groups and values rather than claiming an independently sorted output stream for every world. SSB has no LIMIT. Saved artifacts include raw timings, coverage status, the executed SQL, EXPLAIN plans, build settings, and data-generation provenance/checksums. The SSB validation is also what exposed the exact-SUM bugs and sparse-world approximation error described below.

MEPS: published survey analyses with official replicate designs

Commits and files. The published-workload MEPS benchmark is implemented in 25409d8. Its runner and SQL receive the DuckDB 2.0 API/lambda updates linked above. Relevant artifacts:

• Methodology README, C++ benchmark runner, input converter, build/run wrapper, and plotter.
• Damico 2006 SQL in all four modes and AHRQ 2016 SQL in all four modes.

Why these data and queries. The goal is to evaluate AS on actual repeated-subset analyses, including a person-to-event join, rather than on a synthetic array reduction. The timed implementation is C++/DuckDB throughout. Python downloads/converts input files before the benchmark; no R interpreter or package runtime is included in the performance comparison.

Workload	Published analysis and input	Queries	Replicate design
damico2006	Damico (2009), Table 2, using 2006 HC-105 persons	Mean annual expenditure for adults (age > 17) and nonelderly adults (17 < age < 65)	The historical HC-036BRR 1996–2006 file's 64 indicators; one AS block
ahrq2016	AHRQ's 2016 event-analysis example, using HC-192 persons and HC-188F/G events	Five query files: office events, physician-only office events, outpatient events, physician-only outpatient events, and visits per person	The official 128 indicators; two AS blocks, covering all replicates

The first four AHRQ queries return weighted event counts and mean expenditure by five payment sources. The visits query returns office visits and physician-office visits per person and retains persons with zero visits through a left join. The 2016 person input is checked against its expected 34,655 records, including 6,933 zero-expenditure people and 33,259 positive-weight people. Preparation verifies person uniqueness, person/panel BRR linkage, event ownership, and binary/non-NULL replicate indicators. Recorded download URLs, hashes, and row counts identify the inputs. The exact pandas XPORT zero-decoding artifact 2**-260 is corrected before analysis, preserving real zeros without thresholding other small values.

Statistical computation. These are fixed, published replicate memberships, not newly generated random worlds. Under the official MEPS BRR rule, replicate r uses weight 2 * BRR_r * person_weight. Every dependent event inherits its person's flag and survey weight. A replicate mean is its weighted numerator divided by its own weighted denominator. The AS SUM API's factor of two supplies the BRR scaling for weighted counts and cancels between the numerator and denominator for means. The runner uses all 64 or 128 replicate answers to compute BRR standard errors, including both full-estimate-centered and replicate-mean-centered forms. No privacy noise or new sampling procedure is added to this application workload.

The Damico workload reproduces the published mean/SE examples and checks their one-decimal values. AHRQ's original 2016 example uses Taylor variance estimation; our adaptation retains its estimands and filters but calculates BRR standard errors using the official replicate design. Matching the original Taylor standard errors is therefore not the claim.

Four methods. Each workload has ordinary SQL in queries/, a direct-replicate template in naive_queries/, a shared ordinary-SQL implementation in conditional_queries/, and a handwritten AS version in as_queries/:

1. Full-sample DuckDB: one weighted point estimate per metric. This is a single-query cost reference, not equivalent work to estimating uncertainty from all replicates.
2. Repeated SQL (naive_as): read the actual BRR_r flag and evaluate the query with 2 * weight * BRR_r. Its baseline is not burdened with constructing AS bitmasks. Reported time is the first replicate's query latency multiplied by 64 or 128; the remaining replicates execute to validate outputs and calculate standard errors.
3. Conditional SQL: one ordinary query calculates all replicate numerators/denominators from the BRR columns using conditional aggregate expressions. It already shares scans and joins. Output lists package results rather than expanding every input row into one row per replicate. This comparison tests the benefit of AS aggregate maintenance beyond sharing the relational plan.
4. SIMD-ASA: pack the official indicators into one or two 64-bit masks and use the corresponding as_sum blocks. A non-materialized view joins people to the BRR file and builds masks inside the query. Per-replicate ratios are computed lane-wise in SQL. Event joins and the visits-per-person aggregation also remain inside the query.

Timing and validation. query_ms covers planning, scans, person/replicate linkage, mask construction where applicable, event joins, weighting, aggregate updates, SQL ratios, and result materialization. Download/conversion, import, initial input validation, query-text generation, EXPLAIN, C++ answer checking, and the final standard-error calculation across replicate outputs are outside this timer. Import and input-validation times are reported separately. Thus this measures end-to-end database query execution, not download-to-analysis wall time.

The wrapper runs both SUM builds with one excluded warmup and five measured runs, rotating method order; the Graviton configuration uses 16 threads. It restores the approximate default afterward. Figures report medians for each query/workload/build and use one set of ordinary-control bars. Warmup, raw observations, SQL, plans, build caches, all replicate estimates, and point/SE summaries are retained.

Ordinary repeated SQL supplies the replicate reference. Conditional and AS outputs are compared metric by metric, replicate by replicate, and again after calculating BRR standard errors. Outputs must be finite and have the correct number of metrics/replicates; an undefined ratio is an error rather than a silently dropped replicate. The runner records maximum normalized error and retains the existing guards of 1e-8 for ordinary SQL and 1e-3 for AS. These are validation thresholds, not claimed measured errors or exact-equality guarantees. Damico's published values add an external numerical check; agreement among our three implementations alone is not an independent oracle for the 2016 query translation.

Known bugs and unfinished work

These statuses distinguish an issue being open on GitHub from a defect still present in the current PR build. The relevant examples were rechecked while updating this description.

• #30 — negative-only fused scalar SUM/AVG: still present. The raw as_sum counter-list bug was fixed in Fix as_sum counters losing negative-only worlds and magnitudes >= 2^63 #29, but as_noised_sum still returns NULL if no positive state was allocated. Its early NULL decision also uses positive membership before negative membership is merged. Clipped scalar SUM has the related ordering problem. Thus the counter-list example returning [-24,-24] above does not establish correctness of every scalar SUM/AVG release. See PacSumFinalize and clipped SUM finalization.
• #34 — clipped MAX of negative-only worlds: still present. The negative result is merged with a zero-initialized positive buffer: max(0,-5) returns zero. The current build still returns zero in the issue's negative-only example; ordinary as_max is a different path. See as_clip_min_max.cpp.
• #33 — grouping by a nullable owner key: still present. An outer join can put many different entities into one NULL-key group. Treating that group as one entity is invalid. The issue's nested SUM/COUNT example still returns NULL while the explicit world-zero count is 20. It needs a correct rewrite or explicit rejection, not a relaxed comparison.
• #31 — identical filter RNG sequences across threads: addressed here, pending merge. The distinct-stream fix and fresh eight-thread reproduction are described above. Order/thread-invariant bit-identical outputs remain outside the seed contract.
• #36 — Q78 memory exhaustion: execution recovered; issue remains open. The new 16-thread SF30 run completed both AS modes over five measurements. Keep that execution result separate from per-world numerical validation and from the four-thread/6 GB spill diagnostic. Preserve the original failure record when assessing closure.
• #27 — SQL function documentation/discoverability: open documentation work. This concerns parameter names/examples in catalog metadata; it is not an aggregate arithmetic fix.
• #39 — TPC-DS STDDEV per-world validation: unfinished benchmark work. The unconditional missing-function macro and Q17's real STDDEV requirement must be resolved before claiming complete validation of that path.

CI dependency setup

The format/tidy jobs repeatedly failed during Ubuntu dependency installation, before either check ran. The quality wrapper in d39f0b1 and fb3d274 now uses the hosted image's existing Ninja and clang-tidy 18, installs the pinned clang-format 11/Python check tools through pip, and still runs make format-check and make tidy-check. The official runner manifest lists the preinstalled tools. See CodeQuality.yml. Distribution builds retain the upstream pinned workflow. This is infrastructure maintenance rather than a change to the paper's algorithms.

What needs to be changed in the paper

• Approximate SUM algorithm and error analysis: state that overflow moves upper bits while preserving the low remainder, and finalization reconstructs all levels. Keep per-input truncation explicit; remove a universal 0.024% final-answer bound if still present. Distinguish input truncation, cancellation, and FLOAT output conversion. Use the actual default kernel, not the intermediate round-to-nearest commit.
• Exact SUM description: distinguish exact integer accumulation from FLOAT lane output. If presenting its packed arithmetic, explain lane isolation, magnitude-based overflow accounting, and bottom-up propagation. Confirm that reported “exact” runs used the corrected build flag.
• State allocation/buffering: extend the current description to cover small-group buffers during merge and temporary materialization during finalization, not only buffering at input. Explain how this reduces late memory growth.
• Spilling/design sketch: describe the DuckDB 2.0 state-export/import integration, complete unfinished-state serialization, and ownership reconstruction. Report Q78 with its actual thread count, memory limit, disk usage, and checked output; distinguish the successful diagnostic from a common-configuration result. Do not suggest that spilling removes all memory requirements.
• Compiler implementation: update the DuckDB-specific discussion for native generated CTE definitions/readers, counter-type/mask propagation, and nested list aggregates. Explain that this is an engine-port requirement; the formal sampled-world semantics remain the specification being preserved.
• Query coverage: incorporate supported single-entity UNION ALL/UNION/DISTINCT semantics and their restrictions if the revised coverage claims rely on them. Recheck top-level UNION cases after UNION and UNION ALL skip the privacy rewrite #37; earlier execution success could have been an unrewritten query. Keep coverage, successful execution, and per-world validation separate, including the STDDEV gap in Add handwritten TPC-DS benchmark and sampled-world validation #39.
• PAC release implementation: document advancing independent streams, VOLATILE scalar release functions, and the reproducibility contract if discussing randomness/seed behavior. Explain that deterministic correlated counters can be shared while random releases are evaluated per outer row. Do not imply that fixed seeds reproduce cells after arbitrary parallel reordering.
• TPC-H artifact/methodology: link the revised Q20 filter placement and retain the Q17/Q21 SQL/ownership joins. Explain extension-side correlated-plan optimizations if needed to interpret new performance. Compare matched engine versions/configurations rather than attributing all changes to one kernel.
• SSB: describe CUSTOMER sampling, all 13 original joined queries, query-time masks, both SUM variants, absent-group checks, and measured approximation error. State that repeated-query timing is one world's latency multiplied by 64.
• MEPS: describe the published 2006/2016 workloads and their official 64/128 indicators; use two 64-lane blocks for the latter. Preserve person/event ownership, replicate-specific denominators, and zero-visit people. State the 2016 BRR adaptation rather than claiming reproduction of Taylor SEs, and explain the conditional-SQL baseline and extrapolated repeated-query timing.
• Remaining correctness limitations: do not claim general support/accuracy for SUM/AVG of only negative INTEGER or DECIMAL values return NULL, even with pac_mi = 0 #30, Nested aggregation over a nullable owner key returns NULL instead of sampled counts #33, or as_clip_max / as_noised_clip_max return 0 for worlds that only received negative values #34 until resolved or explicitly excluded. Counter-list correctness is not interchangeable with fused noisy-scalar correctness.
• Microbenchmark summaries: regenerate figures with the actual median in common.sh. The old awk condition printed the first sorted observation rather than the median. The harness now also reads the DuckDB 2.0 profiling tree and treats a missing/unreadable aggregate profile as an error instead of reporting zero operator time; see 2dc137b and the retained implementation in 4f06701.
• Results/artifact provenance: regenerate affected measurements/figures using the final code and record the engine/build configuration. The running Graviton snapshot predates the UNION and UNION ALL skip the privacy rewrite #37 follow-up; its TPC-DS UNION coverage must be reassessed before it is used as evidence for the corrected path.

This section is a checklist for the paper source; this PR does not edit the Overleaf manuscript.