diff --git a/_subfiles/_def-prediction-error.qmd b/_subfiles/_def-prediction-error.qmd index 2e931a026..77458f440 100644 --- a/_subfiles/_def-prediction-error.qmd +++ b/_subfiles/_def-prediction-error.qmd @@ -1,7 +1,7 @@ :::{#def-prediction-error} #### Prediction error -A [prediction function](#def-prediction) $\predfnf{\Predvar}$ has as its [estimand](https://morrison-lab.github.io/sds/estimation.html#def-estimand) +A [prediction function](#def-prediction-function) $\predfn$ has as its [estimand](https://morrison-lab.github.io/sds/estimation.html#def-estimand) the observed [outcome](#def-outcome) $\Outvar$ of a [new observation](#def-new-observation) with [input](#def-input) $\Predvar$, not the [conditional mean](#def-cond-expectation-function) of that outcome. The **prediction error** of $\predfn$ is the prediction minus that outcome: @@ -9,16 +9,3 @@ The **prediction error** of $\predfn$ is the prediction minus that outcome: $$\prederrf{\predfn} \eqdef \predfnf{\Predvar} - \Outvar$$ ::: - -{{< slidebreak >}} - -:::{#exm-prediction-error} -#### Prediction error in the best-predictor exercise - -In @exr-best-predictor, -consider the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$. -Its [prediction error](#def-prediction-error) is $\prederrf{\predfn_1} = 3 - \Outvar$. -This is $3$ when $\Outvar = 0$. -It is $-3$ when $\Outvar = 6$. - -::: diff --git a/_subfiles/_def-prediction-function.qmd b/_subfiles/_def-prediction-function.qmd new file mode 100644 index 000000000..6e53fc525 --- /dev/null +++ b/_subfiles/_def-prediction-function.qmd @@ -0,0 +1,7 @@ +:::{#def-prediction-function} +#### Prediction function + +A **prediction function** $\predfn$ is a [function](https://morrison-lab.github.io/mds/functions.html#def-function) that takes a value $\predvar$ of the [input](#def-input) +and returns a [prediction](#def-prediction) $\predfnf{\predvar}$ of the [outcome](#def-outcome). + +::: diff --git a/_subfiles/_def-reducible-risk.qmd b/_subfiles/_def-reducible-risk.qmd index da24be460..692aab404 100644 --- a/_subfiles/_def-reducible-risk.qmd +++ b/_subfiles/_def-reducible-risk.qmd @@ -1,11 +1,10 @@ :::{#def-reducible-risk} #### Reducible risk -In the squared-error risk decomposition of @thm-risk-decomposition, -the excess risk from choosing the prediction function $\predfn$ instead of the conditional mean, +The **reducible risk** of a [prediction function](#def-prediction-function) $\predfn$ +is the [expectation](#def-expectation) of the squared difference +between $\predfnf{\Predvar}$ and the [conditional mean](#def-cond-expectation-function) $\truemeanf{\Predvar}$ of the [outcome](#def-outcome): -$$\E{\sqf{\E{\Outvar \mid \Predvar} - \predfnf{\Predvar}}} \ge 0$$ - -is the **reducible risk**. +$$\E{\sqf{\predfnf{\Predvar} - \truemeanf{\Predvar}}}$$ ::: diff --git a/_subfiles/_def-squared-error-risk.qmd b/_subfiles/_def-squared-error-risk.qmd index 719e84d22..354202090 100644 --- a/_subfiles/_def-squared-error-risk.qmd +++ b/_subfiles/_def-squared-error-risk.qmd @@ -1,21 +1,10 @@ :::{#def-squared-error-risk} #### Squared-error risk (expected prediction error) -The **squared-error risk** of a prediction function $\predfnf{\Predvar}$, +The **squared-error risk** of a [prediction function](#def-prediction-function) $\predfn$, also called its **expected prediction error** [@hastie2009elements, p. 18], is the expected square of its [prediction error](#def-prediction-error): $$\popriskf{\predfn} \eqdef \E{\sqf{\prederrf{\predfn}}}$$ ::: - -{{< slidebreak >}} - -:::{#exm-squared-error-risk} -#### Squared-error risk in the best-predictor exercise - -In @exr-best-predictor, -consider the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$. -Its [squared-error risk](#def-squared-error-risk) is $\popriskf{\predfn_1} = 5$. - -::: diff --git a/_subfiles/_exm-cross-term-zero.qmd b/_subfiles/_exm-cross-term-zero.qmd new file mode 100644 index 000000000..c34f90fb9 --- /dev/null +++ b/_subfiles/_exm-cross-term-zero.qmd @@ -0,0 +1,11 @@ +:::{#exm-cross-term-zero} +#### The vanishing cross term in the best-predictor exercise + +In @exr-best-predictor, take the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$. +When $\Predvar = 0$, $\truemeanf{0} = 1$, so $\predfn_1\paren{0} - \truemeanf{0} = 2$, +and $\prederrf{\truemean} = 1 - \Outvar$ is $1$ or $-1$ with equal [probability](#def-probability), +so the product has conditional mean $0$. +The same cancellation happens when $\Predvar = 1$, +so the expectation of the product is $0$. + +::: diff --git a/_subfiles/_exm-prediction-error.qmd b/_subfiles/_exm-prediction-error.qmd new file mode 100644 index 000000000..944ea01bf --- /dev/null +++ b/_subfiles/_exm-prediction-error.qmd @@ -0,0 +1,10 @@ +:::{#exm-prediction-error} +#### Prediction error in the best-predictor exercise + +In @exr-best-predictor, +consider the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$. +Its [prediction error](#def-prediction-error) is $\prederrf{\predfn_1} = 3 - \Outvar$. +This is $3$ when $\Outvar = 0$. +It is $-3$ when $\Outvar = 6$. + +::: diff --git a/_subfiles/_exm-prediction-function.qmd b/_subfiles/_exm-prediction-function.qmd new file mode 100644 index 000000000..79f728a7a --- /dev/null +++ b/_subfiles/_exm-prediction-function.qmd @@ -0,0 +1,8 @@ +:::{#exm-prediction-function} +#### A constant prediction function + +In @exr-best-predictor, +the [function](https://morrison-lab.github.io/mds/functions.html#def-function) $\predfn_1\paren{\predvar} = 3$ is a [prediction function](#def-prediction-function): +it returns the [prediction](#def-prediction) $3$ for every value of the [input](#def-input). + +::: diff --git a/_subfiles/_exm-reducible-risk.qmd b/_subfiles/_exm-reducible-risk.qmd index 8facd6765..e12e1035c 100644 --- a/_subfiles/_exm-reducible-risk.qmd +++ b/_subfiles/_exm-reducible-risk.qmd @@ -2,7 +2,7 @@ #### Reducible risk in the best-predictor exercise In @exr-best-predictor, -the constant prediction $\predfn_1\paren{\Predvar} = 3$ has reducible risk $5 - 1 = 4$, -and the prediction $\predfn_2\paren{\Predvar} = 4\Predvar$ has reducible risk $2 - 1 = 1$. +the constant [prediction function](#def-prediction-function) $\predfn_1$, with $\predfn_1\paren{\Predvar} = 3$, has [reducible risk](#def-reducible-risk) $5 - 1 = 4$, +and the prediction function $\predfn_2$, with $\predfn_2\paren{\Predvar} = 4\Predvar$, has reducible risk $2 - 1 = 1$. ::: diff --git a/_subfiles/_exm-risk-decomposition.qmd b/_subfiles/_exm-risk-decomposition.qmd new file mode 100644 index 000000000..e80bc0eee --- /dev/null +++ b/_subfiles/_exm-risk-decomposition.qmd @@ -0,0 +1,9 @@ +:::{#exm-risk-decomposition} +#### The risk decomposition in the best-predictor exercise + +In @exr-best-predictor, +the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$ has [squared-error risk](#def-squared-error-risk) $\popriskf{\predfn_1} = 5$, +which @thm-risk-decomposition splits as $5 = 1 + 4$: +the risk $1$ of the [conditional expectation function](#def-cond-expectation-function) plus the excess $4$ from using $\predfn_1$. + +::: diff --git a/_subfiles/_exm-squared-error-risk.qmd b/_subfiles/_exm-squared-error-risk.qmd new file mode 100644 index 000000000..f264f2076 --- /dev/null +++ b/_subfiles/_exm-squared-error-risk.qmd @@ -0,0 +1,8 @@ +:::{#exm-squared-error-risk} +#### Squared-error risk in the best-predictor exercise + +In @exr-best-predictor, +consider the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$. +Its [squared-error risk](#def-squared-error-risk) is $\popriskf{\predfn_1} = 5$. + +::: diff --git a/_subfiles/_lem-cross-term-zero.qmd b/_subfiles/_lem-cross-term-zero.qmd index fc5ffdda9..49eb84085 100644 --- a/_subfiles/_lem-cross-term-zero.qmd +++ b/_subfiles/_lem-cross-term-zero.qmd @@ -2,7 +2,7 @@ #### The conditional mean's error is uncorrelated with any function of the input Let $\truemeanf{\Predvar} \eqdef \E{\Outvar \mid \Predvar}$ be the [conditional expectation function](#def-cond-expectation-function) of $\Outvar$ given $\Predvar$. -For any [prediction function](#def-prediction) $\predfnf{\Predvar}$ with $\E{\predfnf{\Predvar}^2} < \infty$: +For any [prediction function](#def-prediction-function) $\predfn$ with $\E{\predfnf{\Predvar}^2} < \infty$: $$\E{\prederrf{\truemean} \cd \paren{\predfnf{\Predvar} - \truemeanf{\Predvar}}} = 0$$ @@ -44,17 +44,3 @@ $$ $$ ::: - -{{< slidebreak >}} - -:::{#exm-cross-term-zero} -#### The vanishing cross term in the best-predictor exercise - -In @exr-best-predictor, take the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$. -When $\Predvar = 0$, $\truemeanf{0} = 1$, so $\predfn_1\paren{0} - \truemeanf{0} = 2$, -and $\prederrf{\truemean} = 1 - \Outvar$ is $1$ or $-1$ with equal [probability](#def-probability), -so the product has conditional mean $0$. -The same cancellation happens when $\Predvar = 1$, -so the expectation of the product is $0$. - -::: diff --git a/_subfiles/_rem-reducible-risk-notes.qmd b/_subfiles/_rem-reducible-risk-notes.qmd index 6f1a4e125..9178c15fa 100644 --- a/_subfiles/_rem-reducible-risk-notes.qmd +++ b/_subfiles/_rem-reducible-risk-notes.qmd @@ -8,5 +8,6 @@ $$\E{\sqf{\Outvar - \predfnf{\Predvar}}} = \E{\sqf{\Outvar - \E{\Outvar \mid \Pr shows that the total squared-error risk is the sum of the residual risk (@def-irreducible-risk) and the reducible risk (@def-reducible-risk). +@james2021islr2e [Section 2.1.1, p. 18] calls the squared difference between a [prediction function](#def-prediction-function) and the [conditional mean](#def-cond-expectation-function) the "reducible error". ::: diff --git a/_subfiles/_thm-best-predictor.qmd b/_subfiles/_thm-best-predictor.qmd index 944f0f581..9d11521a4 100644 --- a/_subfiles/_thm-best-predictor.qmd +++ b/_subfiles/_thm-best-predictor.qmd @@ -3,8 +3,8 @@ Let $\Predvar$ and $\Outvar$ be [jointly distributed](random-variables.qmd#def-jointly-distributed) random variables with $\E{\Outvar^2} < \infty$. Let $\truemeanf{\Predvar} \eqdef \E{\Outvar \mid \Predvar}$ be the [conditional expectation function](#def-cond-expectation-function) of $\Outvar$ given $\Predvar$. -Write $\bestpredfn \eqdef \truemean$ for the prediction function that uses it. -Then for any [prediction function](#def-prediction) $\predfnf{\Predvar}$ with $\E{\predfnf{\Predvar}^2} < \infty$: +Write $\bestpredfn \eqdef \truemean$ for the [prediction function](#def-prediction-function) that uses it. +Then for any prediction function $\predfn$ with $\E{\predfnf{\Predvar}^2} < \infty$: $$\popriskf{\predfn} \ge \popriskf{\bestpredfn}$$ diff --git a/_subfiles/_thm-risk-decomposition.qmd b/_subfiles/_thm-risk-decomposition.qmd index 247563295..984bef6bd 100644 --- a/_subfiles/_thm-risk-decomposition.qmd +++ b/_subfiles/_thm-risk-decomposition.qmd @@ -2,7 +2,7 @@ #### Squared-error risk decomposition Let $\truemeanf{\Predvar} \eqdef \E{\Outvar \mid \Predvar}$ be the [conditional expectation function](#def-cond-expectation-function) of $\Outvar$ given $\Predvar$. -For any [prediction function](#def-prediction) $\predfnf{\Predvar}$ with $\E{\predfnf{\Predvar}^2} < \infty$: +For any [prediction function](#def-prediction-function) $\predfn$ with $\E{\predfnf{\Predvar}^2} < \infty$: $$\popriskf{\predfn} = \popriskf{\truemean} + \E{\sqf{\predfnf{\Predvar} - \truemeanf{\Predvar}}}$$ @@ -38,15 +38,3 @@ $$ $$ ::: - -{{< slidebreak >}} - -:::{#exm-risk-decomposition} -#### The risk decomposition in the best-predictor exercise - -In @exr-best-predictor, -the constant [prediction](#def-prediction) $\predfn_1\paren{\Predvar} = 3$ has [squared-error risk](#def-squared-error-risk) $\popriskf{\predfn_1} = 5$, -which @thm-risk-decomposition splits as $5 = 1 + 4$: -the risk $1$ of the [conditional expectation function](#def-cond-expectation-function) plus the excess $4$ from using $\predfn_1$. - -::: diff --git a/_subfiles/cor-best-predictor-argmin/_cor-best-predictor-argmin.qmd b/_subfiles/cor-best-predictor-argmin/_cor-best-predictor-argmin.qmd index 9e0b58ff3..cf2ea1a52 100644 --- a/_subfiles/cor-best-predictor-argmin/_cor-best-predictor-argmin.qmd +++ b/_subfiles/cor-best-predictor-argmin/_cor-best-predictor-argmin.qmd @@ -2,7 +2,7 @@ #### Conditional mean as the minimizer of squared-error risk Let $\Predvar$ and $\Outvar$ be [jointly distributed](random-variables.qmd#def-jointly-distributed) random variables with $\E{\Outvar^2} < \infty$. -Among [prediction functions](#def-prediction) $\predfnf{\Predvar}$ with $\E{\predfnf{\Predvar}^2} < \infty$, +Among [prediction functions](#def-prediction-function) $\predfn$ with $\E{\predfnf{\Predvar}^2} < \infty$, the [conditional expectation function](#def-cond-expectation-function) is a [minimizer](https://morrison-lab.github.io/mds/algebra-basics.html#def-global-minimizer) of the [squared-error risk](#def-squared-error-risk): $$\E{\Outvar \mid \Predvar} \in \argmin_{\predfn} \E{\sqf{\Outvar - \predfnf{\Predvar}}},$$ @@ -13,7 +13,7 @@ where $\argmin$ is the [set of minimizers](https://morrison-lab.github.io/mds/al ::: proof -By @thm-best-predictor, $\bestpredfnf{\Predvar} = \E{\Outvar \mid \Predvar}$ has [risk](#def-risk) no larger than that of any [prediction function](#def-prediction) $\predfn$, +By @thm-best-predictor, $\bestpredfnf{\Predvar} = \E{\Outvar \mid \Predvar}$ has [risk](#def-risk) no larger than that of any [prediction function](#def-prediction-function) $\predfn$, so it is a [minimizer](https://morrison-lab.github.io/mds/algebra-basics.html#def-global-minimizer). ::: diff --git a/_subfiles/cor-best-predictor-argmin/_exm-best-predictor-argmin.qmd b/_subfiles/cor-best-predictor-argmin/_exm-best-predictor-argmin.qmd index 786ae98b1..84e506b5e 100644 --- a/_subfiles/cor-best-predictor-argmin/_exm-best-predictor-argmin.qmd +++ b/_subfiles/cor-best-predictor-argmin/_exm-best-predictor-argmin.qmd @@ -2,7 +2,7 @@ #### The minimizer in the best-predictor exercise In @exr-best-predictor, the [conditional expectation function](#def-cond-expectation-function) $\bestpredfnf{\Predvar}$ has [risk](#def-risk) $1$. -The constant prediction $\predfn_1\paren{\Predvar} = 3$ has risk $5$ and $\predfn_2\paren{\Predvar} = 4\Predvar$ has risk $2$, both larger than $1$. +The constant [prediction function](#def-prediction-function) $\predfn_1$, with $\predfn_1\paren{\Predvar} = 3$, has risk $5$ and $\predfn_2\paren{\Predvar} = 4\Predvar$ has risk $2$, both larger than $1$. So $\bestpredfn$ is the [minimizer](https://morrison-lab.github.io/mds/algebra-basics.html#def-global-minimizer) among the three. ::: diff --git a/_subfiles/cor-best-predictor-unique/_cor-best-predictor-unique.qmd b/_subfiles/cor-best-predictor-unique/_cor-best-predictor-unique.qmd index 1db240f65..c4ab82294 100644 --- a/_subfiles/cor-best-predictor-unique/_cor-best-predictor-unique.qmd +++ b/_subfiles/cor-best-predictor-unique/_cor-best-predictor-unique.qmd @@ -2,7 +2,7 @@ #### The minimizer of squared-error risk is unique up to probability-1 equality Let $\Predvar$ and $\Outvar$ be [jointly distributed](random-variables.qmd#def-jointly-distributed) random variables with $\E{\Outvar^2} < \infty$. -Every [prediction function](#def-prediction) $\predfnf{\Predvar}$ with $\E{\predfnf{\Predvar}^2} < \infty$ +Every [prediction function](#def-prediction-function) $\predfn$ with $\E{\predfnf{\Predvar}^2} < \infty$ that is a [minimizer](https://morrison-lab.github.io/mds/algebra-basics.html#def-global-minimizer) of the [squared-error risk](#def-squared-error-risk) (see @cor-best-predictor-argmin) satisfies: diff --git a/expectation.qmd b/expectation.qmd index 461cffa2d..a7bdc1935 100644 --- a/expectation.qmd +++ b/expectation.qmd @@ -103,6 +103,14 @@ format: {{< slidebreak >}} +{{< include pds/_subfiles/_def-prediction-function.qmd >}} + +{{< slidebreak >}} + +{{< include pds/_subfiles/_exm-prediction-function.qmd >}} + +{{< slidebreak >}} + {{< include pds/_subfiles/_def-new-observation.qmd >}} {{< slidebreak >}} @@ -111,10 +119,18 @@ format: {{< slidebreak >}} +{{< include pds/_subfiles/_exm-prediction-error.qmd >}} + +{{< slidebreak >}} + {{< include pds/_subfiles/_def-squared-error-risk.qmd >}} {{< slidebreak >}} +{{< include pds/_subfiles/_exm-squared-error-risk.qmd >}} + +{{< slidebreak >}} + {{< include pds/_subfiles/_rem-squared-error-risk-is-risk.qmd >}} {{< slidebreak >}} @@ -127,10 +143,18 @@ format: {{< slidebreak >}} +{{< include pds/_subfiles/_exm-cross-term-zero.qmd >}} + +{{< slidebreak >}} + {{< include pds/_subfiles/_thm-risk-decomposition.qmd >}} {{< slidebreak >}} +{{< include pds/_subfiles/_exm-risk-decomposition.qmd >}} + +{{< slidebreak >}} + {{< include pds/_subfiles/_thm-best-predictor.qmd >}} {{< slidebreak >}}