From de12f954e9b5ddfa41f384865a9cd4d7a476a6a2 Mon Sep 17 00:00:00 2001 From: TomeHirata Date: Wed, 19 Aug 2026 16:17:01 +0900 Subject: [PATCH 1/3] refine summary and statement of need in paper.md for clarity and audience targeting --- paper/paper.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/paper/paper.md b/paper/paper.md index 2259f30..047cfd0 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -31,7 +31,9 @@ bibliography: paper.bib # Summary -`dte_adj` is a Python package for estimating distributional treatment effects (DTEs) in randomized experiments (RCTs, also known as A/B tests). Unlike traditional approaches that focus on average treatment effects, `dte_adj` enables researchers to analyze the full distributional impact of interventions across different outcome levels. The package implements machine learning-enhanced regression adjustment methods for variance reduction, supports multiple experimental designs including simple randomization, covariate-adaptive randomization, and settings with imperfect compliance, and provides a scikit-learn compatible API with comprehensive functionality for computing distribution functions, probability treatment effects, and quantile treatment effects with confidence intervals. +`dte_adj` is a Python package for analyzing how an intervention — such as a marketing campaign, a medical treatment, or a policy change — affects the entire range of an outcome, not just its average. Practitioners running randomized experiments (RCTs, also known as A/B tests) can use it to answer questions such as "did the treatment help the users who were struggling the most?" or "did it move the whole distribution, or only shift the top end?", together with rigorous confidence bands around those answers. + +More formally, `dte_adj` estimates distributional treatment effects (DTEs) in randomized experiments. Unlike traditional approaches that focus on average treatment effects, `dte_adj` enables researchers to analyze the full distributional impact of interventions across different outcome levels. The package implements machine learning-enhanced regression adjustment methods for variance reduction, supports multiple experimental designs including simple randomization, covariate-adaptive randomization, and settings with imperfect compliance, and provides a scikit-learn compatible API with comprehensive functionality for computing distribution functions, probability treatment effects, and quantile treatment effects with confidence intervals. # Statement of Need @@ -39,6 +41,8 @@ Randomized experiments have been fundamental to scientific inquiry since @fisher Despite the growing importance of distributional analysis in economics, medicine, and technology, the Python ecosystem lacks comprehensive tools for DTE estimation with modern variance reduction techniques. Researchers often resort to basic empirical CDFs or manual implementations that lack statistical rigor. `dte_adj` fills this gap by providing a unified framework for distributional treatment effect analysis that integrates state-of-the-art machine learning methods for improved precision, rigorous confidence interval construction, and support for complex experimental designs. +The target audience for `dte_adj` includes applied researchers in economics, biostatistics, and the social sciences, as well as data scientists and experimentation engineers at technology companies who run A/B tests and need to characterize heterogeneous distributional effects. Compared with general causal inference libraries such as `DoWhy` [@dowhy] and `EconML` [@econml], which target average or conditional average treatment effects, and the R package `qte`, which supports quantile treatment effects but without machine learning-based variance reduction, `dte_adj` is the first Python package specifically designed to estimate the full distributional treatment effect together with modern regression adjustment for variance reduction. + # State of the Field Several Python packages address causal inference, but none focus on distributional treatment effects with machine learning-based variance reduction: @@ -50,6 +54,8 @@ Several Python packages address causal inference, but none focus on distribution In the R ecosystem, packages like `qte` provide quantile treatment effect estimation but lack machine learning integration for variance reduction. `dte_adj` uniquely combines: (1) distributional treatment effect estimation across the full outcome distribution, (2) machine learning-enhanced regression adjustment for precision gains, and (3) support for multiple experimental designs including covariate-adaptive randomization and imperfect compliance settings. +**Build vs. contribute.** We considered contributing DTE estimators to an existing library rather than releasing a standalone package, but concluded that a dedicated package was the more appropriate choice. Existing causal inference libraries are organized around scalar estimands: `DoWhy` around identification and estimation of ATEs through causal graphs, and `EconML` around heterogeneous CATE estimation via meta-learners. Distributional estimation requires a distinct set of primitives — distribution functions evaluated over grids of locations, interval probabilities, and quantile inversion — together with confidence bands (pointwise and uniform) and cross-fitted distributional regression for variance reduction. These primitives do not map cleanly onto the point-estimate abstractions used by those libraries, and retrofitting them would either bloat the host libraries' interfaces or force awkward compromises for users. A focused package also lets us track a rapidly evolving methodological literature (multiple recent papers on CAR, imperfect compliance, and multi-task learning for DTEs) without being constrained by the release cadence and API stability requirements of a much larger project. + # Software Design `dte_adj` follows a class-based architecture with a template method pattern, where a base class defines the algorithm structure and subclasses implement design-specific computations: @@ -60,6 +66,10 @@ In the R ecosystem, packages like `qte` provide quantile treatment effect estima All estimators implement a consistent API with three primary methods: `predict_dte()` for distributional treatment effects, `predict_pte()` for probability treatment effects over intervals, and `predict_qte()` for quantile treatment effects. The adjusted estimators use K-fold cross-fitting to prevent overfitting and support both single-task and multi-task learning modes [@hirata2025efficientscalableestimationdistributional] for computational efficiency. Bootstrap methods provide confidence intervals with multiple variance estimation approaches. +**Design trade-offs.** Two design decisions deserve explicit discussion. First, we chose a *template method* pattern over pure composition or a strategy-based configuration. The estimators share a common outer algorithm — evaluate a distribution function on a grid, difference across treatment arms, and construct confidence bands — but differ in the inner step of how the conditional distribution is estimated, which in turn depends on the experimental design (simple vs. covariate-adaptive randomization vs. imperfect compliance) and on whether a plug-in empirical estimator or a cross-fitted machine learning estimator is used. A template method keeps this outer algorithm defined once in the base class while subclasses override the design-specific inner step, which we found easier to read, test, and extend than either a chain of injected strategy objects (which pushes the algorithm's structure into configuration and obscures the invariants each design must satisfy) or deep composition (which would fragment the algorithm across many small collaborators). The main trade-off is a shallow inheritance hierarchy that users must learn, but the tree is intentionally kept flat (two levels) and matches the taxonomy of the underlying methods. + +Second, we chose *separate estimator classes* (`SimpleDistributionEstimator`, `AdjustedStratifiedDistributionEstimator`, and so on) rather than a single configurable estimator with flags such as `adjusted=True, stratified=True, compliance="imperfect"`. The alternative would compress the API surface but at the cost of a large space of flag combinations, many of which are meaningless (e.g., stratification without a strata argument) or correspond to different statistical objects with different identifying assumptions. Distinct classes make the required inputs explicit at construction time, allow the type system and docstrings to describe each estimator precisely, and let us evolve one estimator (for example, adding multi-task support to the adjusted variants) without changing the interface of the others. The cost is some repetition across constructors and documentation, which we accepted in exchange for clarity about which estimator is appropriate for which design. + ![Distributional treatment effects for the Hillstrom email marketing dataset [@hillstrom2008], comparing Women's vs Men's email campaigns. The simple estimator (left, purple) and ML-adjusted estimator (right, green) show that adjustment substantially tightens confidence bands, demonstrating the variance reduction benefit of regression adjustment.](hillstorm_dte.png) ![Local distributional treatment effects for emergency department costs in the Oregon Health Insurance Experiment [@finkelstein2012], estimated using `SimpleLocalDistributionEstimator` (left) and `AdjustedLocalDistributionEstimator` (right). Health insurance coverage shifts the distribution of ED costs, with ML adjustment again yielding narrower confidence intervals.](oregon_ldte_costs_comparison.png) From 189adb4ce521976161a252c94416d4c337c4c7a3 Mon Sep 17 00:00:00 2001 From: TomeHirata Date: Wed, 19 Aug 2026 16:31:40 +0900 Subject: [PATCH 2/3] fix: improve clarity in documentation by adjusting phrasing in summary and AI usage sections Signed-off-by: TomeHirata --- paper/paper.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/paper/paper.md b/paper/paper.md index 047cfd0..59fb467 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -31,7 +31,7 @@ bibliography: paper.bib # Summary -`dte_adj` is a Python package for analyzing how an intervention — such as a marketing campaign, a medical treatment, or a policy change — affects the entire range of an outcome, not just its average. Practitioners running randomized experiments (RCTs, also known as A/B tests) can use it to answer questions such as "did the treatment help the users who were struggling the most?" or "did it move the whole distribution, or only shift the top end?", together with rigorous confidence bands around those answers. +`dte_adj` is a Python package for analyzing how an intervention, such as a marketing campaign, a medical treatment, or a policy change, affects the entire range of an outcome, not just its average. Practitioners running randomized experiments (RCTs, also known as A/B tests) can use it to answer questions such as "did the treatment help the users who were struggling the most?" or "did it move the whole distribution, or only shift the top end?", together with rigorous confidence bands around those answers. More formally, `dte_adj` estimates distributional treatment effects (DTEs) in randomized experiments. Unlike traditional approaches that focus on average treatment effects, `dte_adj` enables researchers to analyze the full distributional impact of interventions across different outcome levels. The package implements machine learning-enhanced regression adjustment methods for variance reduction, supports multiple experimental designs including simple randomization, covariate-adaptive randomization, and settings with imperfect compliance, and provides a scikit-learn compatible API with comprehensive functionality for computing distribution functions, probability treatment effects, and quantile treatment effects with confidence intervals. @@ -54,7 +54,7 @@ Several Python packages address causal inference, but none focus on distribution In the R ecosystem, packages like `qte` provide quantile treatment effect estimation but lack machine learning integration for variance reduction. `dte_adj` uniquely combines: (1) distributional treatment effect estimation across the full outcome distribution, (2) machine learning-enhanced regression adjustment for precision gains, and (3) support for multiple experimental designs including covariate-adaptive randomization and imperfect compliance settings. -**Build vs. contribute.** We considered contributing DTE estimators to an existing library rather than releasing a standalone package, but concluded that a dedicated package was the more appropriate choice. Existing causal inference libraries are organized around scalar estimands: `DoWhy` around identification and estimation of ATEs through causal graphs, and `EconML` around heterogeneous CATE estimation via meta-learners. Distributional estimation requires a distinct set of primitives — distribution functions evaluated over grids of locations, interval probabilities, and quantile inversion — together with confidence bands (pointwise and uniform) and cross-fitted distributional regression for variance reduction. These primitives do not map cleanly onto the point-estimate abstractions used by those libraries, and retrofitting them would either bloat the host libraries' interfaces or force awkward compromises for users. A focused package also lets us track a rapidly evolving methodological literature (multiple recent papers on CAR, imperfect compliance, and multi-task learning for DTEs) without being constrained by the release cadence and API stability requirements of a much larger project. +**Build vs. contribute.** We considered contributing DTE estimators to an existing library rather than releasing a standalone package, but concluded that a dedicated package was the more appropriate choice. Existing causal inference libraries are organized around scalar estimands: `DoWhy` around identification and estimation of ATEs through causal graphs, and `EconML` around heterogeneous CATE estimation via meta-learners. Distributional estimation requires a distinct set of primitives, distribution functions evaluated over grids of locations, interval probabilities, and quantile inversion, together with confidence bands (pointwise and uniform) and cross-fitted distributional regression for variance reduction. These primitives do not map cleanly onto the point-estimate abstractions used by those libraries, and retrofitting them would either bloat the host libraries' interfaces or force awkward compromises for users. A focused package also lets us track a rapidly evolving methodological literature (multiple recent papers on CAR, imperfect compliance, and multi-task learning for DTEs) without being constrained by the release cadence and API stability requirements of a much larger project. # Software Design @@ -66,7 +66,7 @@ In the R ecosystem, packages like `qte` provide quantile treatment effect estima All estimators implement a consistent API with three primary methods: `predict_dte()` for distributional treatment effects, `predict_pte()` for probability treatment effects over intervals, and `predict_qte()` for quantile treatment effects. The adjusted estimators use K-fold cross-fitting to prevent overfitting and support both single-task and multi-task learning modes [@hirata2025efficientscalableestimationdistributional] for computational efficiency. Bootstrap methods provide confidence intervals with multiple variance estimation approaches. -**Design trade-offs.** Two design decisions deserve explicit discussion. First, we chose a *template method* pattern over pure composition or a strategy-based configuration. The estimators share a common outer algorithm — evaluate a distribution function on a grid, difference across treatment arms, and construct confidence bands — but differ in the inner step of how the conditional distribution is estimated, which in turn depends on the experimental design (simple vs. covariate-adaptive randomization vs. imperfect compliance) and on whether a plug-in empirical estimator or a cross-fitted machine learning estimator is used. A template method keeps this outer algorithm defined once in the base class while subclasses override the design-specific inner step, which we found easier to read, test, and extend than either a chain of injected strategy objects (which pushes the algorithm's structure into configuration and obscures the invariants each design must satisfy) or deep composition (which would fragment the algorithm across many small collaborators). The main trade-off is a shallow inheritance hierarchy that users must learn, but the tree is intentionally kept flat (two levels) and matches the taxonomy of the underlying methods. +**Design trade-offs.** Two design decisions deserve explicit discussion. First, we chose a *template method* pattern over pure composition or a strategy-based configuration. The estimators share a common outer algorithm, evaluate a distribution function on a grid, difference across treatment arms, and construct confidence bands, but differ in the inner step of how the conditional distribution is estimated, which in turn depends on the experimental design (simple vs. covariate-adaptive randomization vs. imperfect compliance) and on whether a plug-in empirical estimator or a cross-fitted machine learning estimator is used. A template method keeps this outer algorithm defined once in the base class while subclasses override the design-specific inner step, which we found easier to read, test, and extend than either a chain of injected strategy objects (which pushes the algorithm's structure into configuration and obscures the invariants each design must satisfy) or deep composition (which would fragment the algorithm across many small collaborators). The main trade-off is a shallow inheritance hierarchy that users must learn, but the tree is intentionally kept flat (two levels) and matches the taxonomy of the underlying methods. Second, we chose *separate estimator classes* (`SimpleDistributionEstimator`, `AdjustedStratifiedDistributionEstimator`, and so on) rather than a single configurable estimator with flags such as `adjusted=True, stratified=True, compliance="imperfect"`. The alternative would compress the API surface but at the cost of a large space of flag combinations, many of which are meaningless (e.g., stratification without a strata argument) or correspond to different statistical objects with different identifying assumptions. Distinct classes make the required inputs explicit at construction time, allow the type system and docstrings to describe each estimator precisely, and let us evolve one estimator (for example, adding multi-task support to the adjusted variants) without changing the interface of the others. The cost is some repetition across constructors and documentation, which we accepted in exchange for clarity about which estimator is appropriate for which design. @@ -80,7 +80,7 @@ The methods implemented in `dte_adj` have been published across machine learning # AI Usage Disclosure -Generative AI tools (Claude) were used to assist with documentation writing and code review during development. All AI-generated content was reviewed and validated by the human authors. +Generative AI tools (Claude) were used to assist with documentation writing and code review during development, and with copy-editing this paper. All AI-generated content was reviewed and validated by the human authors. # Acknowledgements From 05ada023edc4a7546dc688954d990e6f0f80a6cd Mon Sep 17 00:00:00 2001 From: TomeHirata Date: Thu, 20 Aug 2026 01:36:24 +0900 Subject: [PATCH 3/3] refine statement of need in paper.md for clarity and audience targeting --- paper/paper.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/paper/paper.md b/paper/paper.md index 59fb467..f17db65 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -39,9 +39,7 @@ More formally, `dte_adj` estimates distributional treatment effects (DTEs) in ra Randomized experiments have been fundamental to scientific inquiry since @fisher1935design, providing the gold standard for causal inference. While most experimental analyses focus on average treatment effects (ATEs), many research questions require understanding how treatments affect the entire distribution of outcomes. Distributional treatment effects (DTEs) capture these richer patterns, revealing heterogeneous impacts across different outcome levels that averages can mask. For example, a policy intervention might have no effect on average income while substantially reducing poverty rates at lower quantiles, or a medical treatment might benefit patients at the tails of the distribution differently than those near the median. -Despite the growing importance of distributional analysis in economics, medicine, and technology, the Python ecosystem lacks comprehensive tools for DTE estimation with modern variance reduction techniques. Researchers often resort to basic empirical CDFs or manual implementations that lack statistical rigor. `dte_adj` fills this gap by providing a unified framework for distributional treatment effect analysis that integrates state-of-the-art machine learning methods for improved precision, rigorous confidence interval construction, and support for complex experimental designs. - -The target audience for `dte_adj` includes applied researchers in economics, biostatistics, and the social sciences, as well as data scientists and experimentation engineers at technology companies who run A/B tests and need to characterize heterogeneous distributional effects. Compared with general causal inference libraries such as `DoWhy` [@dowhy] and `EconML` [@econml], which target average or conditional average treatment effects, and the R package `qte`, which supports quantile treatment effects but without machine learning-based variance reduction, `dte_adj` is the first Python package specifically designed to estimate the full distributional treatment effect together with modern regression adjustment for variance reduction. +Despite the growing importance of distributional analysis in economics, medicine, and technology, the Python ecosystem lacks comprehensive tools for DTE estimation with modern variance reduction techniques. Applied researchers in economics, biostatistics, and the social sciences, along with data scientists and experimentation engineers running A/B tests in industry, are often left resorting to basic empirical CDFs or manual implementations that lack statistical rigor. `dte_adj` fills this gap for these users with a unified framework that integrates state-of-the-art machine learning methods for improved precision, rigorous confidence interval construction, and support for complex experimental designs. It complements general causal inference libraries such as `DoWhy` [@dowhy] and `EconML` [@econml], which target average or conditional average treatment effects, and the R package `qte`, which supports quantile treatment effects but without machine learning-based variance reduction. # State of the Field @@ -54,7 +52,7 @@ Several Python packages address causal inference, but none focus on distribution In the R ecosystem, packages like `qte` provide quantile treatment effect estimation but lack machine learning integration for variance reduction. `dte_adj` uniquely combines: (1) distributional treatment effect estimation across the full outcome distribution, (2) machine learning-enhanced regression adjustment for precision gains, and (3) support for multiple experimental designs including covariate-adaptive randomization and imperfect compliance settings. -**Build vs. contribute.** We considered contributing DTE estimators to an existing library rather than releasing a standalone package, but concluded that a dedicated package was the more appropriate choice. Existing causal inference libraries are organized around scalar estimands: `DoWhy` around identification and estimation of ATEs through causal graphs, and `EconML` around heterogeneous CATE estimation via meta-learners. Distributional estimation requires a distinct set of primitives, distribution functions evaluated over grids of locations, interval probabilities, and quantile inversion, together with confidence bands (pointwise and uniform) and cross-fitted distributional regression for variance reduction. These primitives do not map cleanly onto the point-estimate abstractions used by those libraries, and retrofitting them would either bloat the host libraries' interfaces or force awkward compromises for users. A focused package also lets us track a rapidly evolving methodological literature (multiple recent papers on CAR, imperfect compliance, and multi-task learning for DTEs) without being constrained by the release cadence and API stability requirements of a much larger project. +A standalone package is warranted because existing libraries are organized around scalar estimands (ATEs in `DoWhy`, heterogeneous CATEs in `EconML`), whereas distributional estimation requires a distinct set of primitives: distribution functions evaluated over grids of locations, interval probabilities, quantile inversion, pointwise and uniform confidence bands, and cross-fitted distributional regression for variance reduction. These do not map cleanly onto point-estimate abstractions, and a focused package also allows the implementation to track a rapidly evolving methodological literature on covariate-adaptive randomization, imperfect compliance, and multi-task learning for DTEs. # Software Design @@ -66,9 +64,7 @@ In the R ecosystem, packages like `qte` provide quantile treatment effect estima All estimators implement a consistent API with three primary methods: `predict_dte()` for distributional treatment effects, `predict_pte()` for probability treatment effects over intervals, and `predict_qte()` for quantile treatment effects. The adjusted estimators use K-fold cross-fitting to prevent overfitting and support both single-task and multi-task learning modes [@hirata2025efficientscalableestimationdistributional] for computational efficiency. Bootstrap methods provide confidence intervals with multiple variance estimation approaches. -**Design trade-offs.** Two design decisions deserve explicit discussion. First, we chose a *template method* pattern over pure composition or a strategy-based configuration. The estimators share a common outer algorithm, evaluate a distribution function on a grid, difference across treatment arms, and construct confidence bands, but differ in the inner step of how the conditional distribution is estimated, which in turn depends on the experimental design (simple vs. covariate-adaptive randomization vs. imperfect compliance) and on whether a plug-in empirical estimator or a cross-fitted machine learning estimator is used. A template method keeps this outer algorithm defined once in the base class while subclasses override the design-specific inner step, which we found easier to read, test, and extend than either a chain of injected strategy objects (which pushes the algorithm's structure into configuration and obscures the invariants each design must satisfy) or deep composition (which would fragment the algorithm across many small collaborators). The main trade-off is a shallow inheritance hierarchy that users must learn, but the tree is intentionally kept flat (two levels) and matches the taxonomy of the underlying methods. - -Second, we chose *separate estimator classes* (`SimpleDistributionEstimator`, `AdjustedStratifiedDistributionEstimator`, and so on) rather than a single configurable estimator with flags such as `adjusted=True, stratified=True, compliance="imperfect"`. The alternative would compress the API surface but at the cost of a large space of flag combinations, many of which are meaningless (e.g., stratification without a strata argument) or correspond to different statistical objects with different identifying assumptions. Distinct classes make the required inputs explicit at construction time, allow the type system and docstrings to describe each estimator precisely, and let us evolve one estimator (for example, adding multi-task support to the adjusted variants) without changing the interface of the others. The cost is some repetition across constructors and documentation, which we accepted in exchange for clarity about which estimator is appropriate for which design. +The template method pattern is a natural fit here because every estimator shares the same outer algorithm (evaluate a distribution function on a grid, difference across treatment arms, and construct confidence bands) and differs only in how the conditional distribution is estimated, which depends on the experimental design and on whether a plug-in or a cross-fitted machine learning estimator is used. Defining the outer algorithm once in the base class keeps its statistical invariants in one place, whereas a strategy-based configuration would push that structure into runtime flags and obscure them. For the same reason the estimators are exposed as distinct classes rather than a single configurable one: many flag combinations would be confusing (e.g., stratification without strata) or correspond to different estimands with different identifying assumptions, and separate classes make the required inputs explicit at construction time and let each estimator evolve independently. ![Distributional treatment effects for the Hillstrom email marketing dataset [@hillstrom2008], comparing Women's vs Men's email campaigns. The simple estimator (left, purple) and ML-adjusted estimator (right, green) show that adjustment substantially tightens confidence bands, demonstrating the variance reduction benefit of regression adjustment.](hillstorm_dte.png)