A warehouse-native Bayesian A/B testing pipeline on GCP
A production-style experimentation pipeline that runs frequentist and Bayesian analysis with CUPED variance reduction, heterogeneous treatment effect detection, trust checks, and automated guardrail flagging. It is built end to end on Google Cloud (BigQuery, Cloud Storage, Cloud Run, Looker Studio) on real MovieLens 25M data, with a known treatment effect simulated on the variant arm so every estimate can be checked against the truth.
"We've built a new recommender (Variant B). Should we ship it?"
This is the most common A/B question at any recommender-driven company. The answer is rarely a clean yes or no. The value of an experimentation platform is in producing the right kind of nuance: the probability that B is better, the expected loss if we are wrong, segment-level effects, and guardrail health.
This project demonstrates that full loop at portfolio scale.
| Metric | Raw lift | CUPED-adjusted | P(B>A) | Decision |
|---|---|---|---|---|
| Watch time (primary) | +13.0% | +9.1% | 0.85 | Ship to light users |
| Sessions (guardrail) | +13.1% | +10.0% | 0.70 | Inconclusive (yellow) |
Recommendation produced by the pipeline:
PROCEED WITH CAUTION. Primary lift +9.1% (P(B>A)=0.85). Guardrail metric inconclusive (+10.0%, CI crosses zero). Consider a longer experiment or a segmented rollout. Segment analysis shows light users have stronger evidence than heavy users.
The dataset is deliberately constructed so the raw aggregate lift is misleading. Pre-period covariates show A and B were balanced before assignment, but a chance imbalance in experiment-period activity inflates the raw lift to +13%. The true treatment effect coded into the data is +7% on watch time (with an extra +3% for light users) and -2% on sessions.
The pipeline's job is to recover that signal from noisy data. It does so through:
- Multivariate CUPED with four pre-period covariates, which pulls the lift estimate from +13% down to +9%, closer to the true +7%
- Bayesian inference that produces a calibrated P(B>A) of 0.85, even though the frequentist test fails to reject the null at alpha = 0.05
- Segment-level analysis that surfaces the heterogeneity (low-activity users +23%, mid +10%, high +4%; light users +11%, heavy users +8%)
- Guardrail flagging that catches the unstable sessions metric and returns a yellow status with a segmented recommendation
Real experiments are often underpowered. The skill that matters is making good decisions under uncertainty, not waiting for p < 0.05 to feel certain.
Warehouse-native. All metric computation happens in BigQuery. Raw events are transformed into a star schema and aggregated metric tables inside the warehouse; the analysis job reads only those aggregated tables and writes results back to BigQuery. Metrics are defined once, in SQL, where the data lives.
Trust before results. A lift estimate is only reported alongside the checks that say whether it can be believed.
Decisions, not p-values. The output is a ship / hold / segment recommendation with the evidence behind it, written for a non-technical reader.
Every run reports these before the results are read:
| Check | What it guards against | Result |
|---|---|---|
| Sample ratio mismatch (SRM) | Broken assignment or logging | Pass (chi-square = 0.205) |
| Pre-period balance | Arms that differed before treatment | Pass (balanced on pre-period covariates) |
| Guardrail metrics | Shipping a win that hurts another metric | Sessions flagged yellow |
| Outlier control | A few heavy users dominating arm means | p99 winsorization applied |
MovieLens 25M
│
▼
GCS bucket (raw CSVs)
│
▼
BigQuery raw layer (4 tables)
│
▼
BigQuery staging (cohort, assignment, daily metrics)
│
▼
BigQuery analytics (star schema: 3 dims + 2 facts)
│
▼
Cloud Run job (Python: frequentist + CUPED + Bayesian + segments)
│
▼
analytics.experiment_results
│
▼
Looker Studio dashboard (public)
| Component | Purpose | Tech |
|---|---|---|
| Synthetic experiment build | Real MovieLens behavior as control; injects a known counterfactual treatment effect (lift, novelty, heterogeneity, guardrail regression) into the variant arm | BigQuery SQL |
| Star schema | dim_users, dim_movies, dim_experiments, fact_assignments, fact_daily_metrics |
BigQuery |
| Analysis library | Welch's t-test, univariate and multivariate CUPED, closed-form Bayesian, segment-level HTE | Python (scipy, pandas, numpy) |
| Cloud Run job | Containerized pipeline runs end to end in about 10s and writes structured results to BigQuery | Docker, Artifact Registry, Cloud Run |
| Test suite | 14 tests covering math correctness against scipy, CUPED variance reduction, and Bayesian probability recovery | pytest |
| Dashboard | Single-page executive dashboard with traffic-light decision logic, segment breakdowns, and a methodology table | Looker Studio |
Closed-form Bayesian instead of MCMC. For Normal-Normal A/B tests with weak priors, the posterior over the difference in means is analytically Student-t. The closed-form implementation runs in 5ms and matches PyMC sampling numerically (which took 23 minutes during development). The PyMC version is kept as bayesian_mcmc.py for reference. Closed-form posteriors are the practical choice for normal metrics on large-scale platforms, where thousands of metrics are evaluated per run.
Multivariate CUPED. Univariate CUPED with pre_watch_time_min alone achieved only 6% variance reduction (rho = 0.25), because the cohort filter compresses the covariate range. Multivariate CUPED with four pre-period covariates reaches 14% (multiple R = 0.38). This is more realistic than the textbook 40% figure: it reflects what production CUPED looks like when covariates are weak. Investigating why the reduction is modest, and reporting it honestly, is part of the project's value.
Stratified randomization on activity decile. Hash-based bucketing within NTILE(10) OVER (ORDER BY pre_n_ratings) keeps pre-period activity balanced across arms.
Winsorization at p99 (pooled). Watch time has a heavy tail. Capping at the pooled p99 stops single users from dominating arm averages without biasing one arm relative to the other.
Intent-to-treat analysis. Per-user totals include zero-activity users. Every assigned user contributes one value, including zero, which matches how production experimentation platforms handle exposure.
See DECISIONS.md for a fuller treatment of these and other tradeoffs.
You need a GCP project with billing enabled and the bigquery, storage, run, and artifactregistry APIs enabled.
# 1. Setup
git clone https://github.com/Tussar98/experimentiq.git && cd experimentiq
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1
pip install -e .
pip install -r requirements.txt
# 2. Build the warehouse
python scripts/01_download_movielens.py
python scripts/02_upload_to_gcs.py
python scripts/03_load_to_bigquery.py
python scripts/04_build_experiment.py
python scripts/05_build_star_schema.py
# 3. Run the analysis (writes to analytics.experiment_results)
python scripts/06_run_analysis.py
# 4. Or deploy to Cloud Run
docker build -t $IMAGE .
docker push $IMAGE
gcloud run jobs create experimentiq-analysis --image $IMAGE --region us-central1 \
--service-account $SA_EMAIL \
--set-env-vars "GCP_PROJECT_ID=$PROJECT_ID" \
--set-env-vars "GCS_BUCKET=$BUCKET" \
--set-env-vars "BQ_LOCATION=US"
gcloud run jobs execute experimentiq-analysis --region us-central1 --wait
# 5. Run tests
pytest tests -vexperimentiq/
├── src/
│ └── analysis/
│ ├── data.py # BigQuery loader, ExperimentData container
│ ├── frequentist.py # Welch's t-test + CIs + power
│ ├── cuped.py # univariate + multivariate CUPED
│ ├── bayesian.py # closed-form Bayesian (production)
│ ├── bayesian_mcmc.py # PyMC reference implementation
│ └── segments.py # heterogeneous treatment effects
├── scripts/
│ ├── 01_download_movielens.py
│ ├── 02_upload_to_gcs.py
│ ├── 03_load_to_bigquery.py
│ ├── 04_build_experiment.py
│ ├── 05_build_star_schema.py
│ ├── 06_run_analysis.py
│ ├── main.py # Cloud Run entrypoint
│ └── _run_analysis_entry.py # importable wrapper
├── tests/ # 14 tests, runs in under 2 seconds
├── Dockerfile
├── requirements-prod.txt
├── pyproject.toml
├── DECISIONS.md
└── README.md
This experiment is underpowered. With about 2,400 users and heavy-tailed engagement, the frequentist test fails to reject the null at alpha = 0.05 even though the true effect is real. The Bayesian and segment analyses do the decision-making work. On production traffic of millions of users, the same pipeline would produce a tight interval and a clearly significant result.
CUPED's modest variance reduction (14%) is structural to the cohort. Cohort filtering compresses the range of pre-period covariates and suppresses correlation.
The chance imbalance is the point. A naive frequentist analysis would say "fail to reject, don't ship." A naive aggregate-only Bayesian analysis on raw lift would say "P(B>A) = 0.91, ship." The pipeline does neither: it adjusts for the imbalance, surfaces the heterogeneity, flags the guardrail uncertainty, and produces a segmented recommendation (ship to light users, hold for heavy users).
- Delta method for ratio metrics (e.g., watch time per session), where the analysis unit differs from the randomization unit
- Sequential testing with always-valid inference, so experiments can be monitored without inflating false positives
- Crossover check for users exposed to both variants, added to the trust-check table
- Metric definitions as config (YAML compiled to BigQuery SQL), so new metrics are defined once without code changes
- Better CUPED covariates: shorter pre-period windows (7 to 14 days vs 90), per-segment CUPED (high-activity users show rho > 0.4), or learned residual models
- Holdout analysis to measure the cumulative value of shipped changes
Python 3.12 · GCP (BigQuery, Cloud Storage, Cloud Run, Artifact Registry) · Docker · scipy · pandas · numpy · pymc · pytest · Looker Studio