Define cross-entropy as the expectation of the surprisal measured by q

_subfiles/_def-cross-entropy.qmd

@@ -1,17 +1,20 @@
 :::{#def-cross-entropy}
 #### Cross-entropy
 
-Let $p$ and $q$ be [probability mass functions](random-variables.qmd#def-pmf)
-on the same finite set of outcomes $\set{1, \ldots, K}$,
-with $p_k$ and $q_k$ the probabilities of outcome $k$.
+Let $X$ be a [random variable](random-variables.qmd#def-random-variable) with
+[probability mass function](random-variables.qmd#def-pmf) or [density](random-variables.qmd#def-pdf) $p$,
+and let $q$ be a [mass function](random-variables.qmd#def-pmf) or density on the same values.
 The **cross-entropy** of $q$ relative to $p$
 [@wikipedia_cross_entropy]
-is
+is the [expectation](#def-expectation), when $X$ has [distribution](random-variables.qmd#def-cdf) $p$,
+of the [surprisal](#def-surprisal) (or its [continuous](#def-surprisal-continuous) form) $\surprisal_q$ measured by $q$,
+where $\surprisal_q\paren{x} \eqdef -\log q\paren{x}$:
 
-$$\crossentropyf{p}{q} \eqdef -\sumto{k}{K} p_k \log q_k.$$
+$$\crossentropyf{p}{q} \eqdef \E{\surprisal_q\paren{X}}.$$
 
-We take $-\log 0 = \infty$, so when $q_k = 0$ for an outcome with $p_k > 0$, the cross-entropy is infinite.
-A term with $p_k = 0$ is taken to be $0$ whatever $q_k$ is, as for [entropy](#def-entropy).
+We take $-\log 0 = \infty$,
+so when $\P(q(X) = 0) > 0$,
+the cross-entropy is infinite.
 
 :::
 
@@ -49,11 +52,11 @@ A term with $p_k = 0$ is taken to be $0$ whatever $q_k$ is, as for [entropy](#de
 
 {{< slidebreak >}}
 
-{{< include pds/_subfiles/def-cross-entropy/_exr-cross-entropy-general-expectation.qmd >}}
+{{< include pds/_subfiles/def-cross-entropy/_exr-cross-entropy-general-sum.qmd >}}
 
 {{< slidebreak >}}
 
-{{< include pds/_subfiles/def-cross-entropy/_sol-cross-entropy-general-expectation.qmd >}}
+{{< include pds/_subfiles/def-cross-entropy/_sol-cross-entropy-general-sum.qmd >}}
 
 {{< slidebreak >}}
 
@@ -73,7 +76,15 @@ A term with $p_k = 0$ is taken to be $0$ whatever $q_k$ is, as for [entropy](#de
 
 {{< slidebreak >}}
 
-{{< include pds/_subfiles/def-cross-entropy/_thm-cross-entropy-expectation.qmd >}}
+{{< include pds/_subfiles/def-cross-entropy/_thm-cross-entropy-discrete.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include pds/_subfiles/def-cross-entropy/_thm-cross-entropy-continuous.qmd >}}
+
+{{< slidebreak >}}
+
+{{< include pds/_subfiles/def-cross-entropy/_exm-cross-entropy-continuous.qmd >}}
 
 {{< slidebreak >}}
 
@@ -117,60 +128,16 @@ A term with $p_k = 0$ is taken to be $0$ whatever $q_k$ is, as for [entropy](#de
 
 {{< slidebreak >}}
 
-:::{#def-kl-divergence}
-#### Kullback--Leibler divergence (KL divergence, relative entropy)
-
-For $p$ and $q$ as in @def-cross-entropy,
-the **Kullback--Leibler divergence** (**KL divergence**, or **relative entropy**)
-of $q$ from $p$ is
-
-$$\kldivf{p}{q} \eqdef \sumto{k}{K} p_k \log \frac{p_k}{q_k} = \crossentropyf{p}{q} - \crossentropyf{p}{p}.$$
-
-:::
-
-::: {#rem-kl-asymmetric}
-#### The divergence is not symmetric
-
-In general, $\kldivf{p}{q} \ne \kldivf{q}{p}$,
-so KL divergence is not a distance between $p$ and $q$.
-
-:::
+{{< include pds/_subfiles/def-cross-entropy/_def-kl-divergence.qmd >}}
 
 {{< slidebreak >}}
 
-:::{#exm-cross-entropy}
-#### Cross-entropy and divergence for two coins
-
-Let $p = (\tfrac{1}{2}, \tfrac{1}{2})$ be a fair coin
-and $q = (\tfrac{1}{4}, \tfrac{3}{4})$ a coin biased toward the second outcome.
-Using logarithms to base 2, $\log_2 \tfrac{1}{4} = -2$
-and $\log_2 \tfrac{3}{4} = \log_2 3 - 2$, so
-
-$$
-\ba
-\crossentropyf{p}{q} &= -\tfrac{1}{2} \log_2 \tfrac{1}{4} - \tfrac{1}{2} \log_2 \tfrac{3}{4} \\
-&= 1 + \tfrac{1}{2} \paren{2 - \log_2 3} \\
-&= 2 - \tfrac{1}{2} \log_2 3 \\
-&\approx 1.208, \\
-\crossentropyf{p}{p} &= -\tfrac{1}{2} \log_2 \tfrac{1}{2} - \tfrac{1}{2} \log_2 \tfrac{1}{2} \\
-&= 1, \\
-\kldivf{p}{q} &= \crossentropyf{p}{q} - \crossentropyf{p}{p} = 1 - \tfrac{1}{2} \log_2 3 \approx 0.208.
-\ea
-$$
+{{< include pds/_subfiles/def-cross-entropy/_rem-kl-asymmetric.qmd >}}
 
-Here the cross-entropy is larger than the entropy of $p$,
-and the divergence is the difference.
-
-:::
+{{< slidebreak >}}
 
-::: {#rem-cross-entropy-source}
-#### Where the definitions come from
+{{< include pds/_subfiles/def-cross-entropy/_exm-cross-entropy.qmd >}}
 
-@murphy2012mlpp [section 2.8.2, pp. 57--58] defines the KL divergence
-and the cross-entropy as above,
-and gives the coding interpretation:
-the cross-entropy is the average number of bits needed
-to encode data from a source with distribution $p$
-using a code designed for $q$.
+{{< slidebreak >}}
 
-:::
+{{< include pds/_subfiles/def-cross-entropy/_rem-cross-entropy-source.qmd >}}

_subfiles/def-cross-entropy/_cor-cross-entropy-entropy.qmd

@@ -1,7 +1,7 @@
 :::{#cor-cross-entropy-entropy}
 #### Entropy is the cross-entropy of a distribution with itself
 
-Let $X$ be a [discrete random variable](random-variables.qmd#def-discrete-rv) with [probability mass function](random-variables.qmd#def-pmf) $p$ on $\set{1, \ldots, K}$.
+Let $X$ be a [random variable](random-variables.qmd#def-random-variable) with [probability mass function](random-variables.qmd#def-pmf) or [density](random-variables.qmd#def-pdf) $p$.
 Then
 
 $$\crossentropyf{p}{p} = \entropyf{X}.$$
@@ -10,10 +10,8 @@ $$\crossentropyf{p}{p} = \entropyf{X}.$$
 
 ::: proof
 
-By @thm-cross-entropy-expectation with $q = p$, $\crossentropyf{p}{p} = \E{-\log p_X}$.
-Since $\P(X = k) = p_k$, the [random variable](random-variables.qmd#def-random-variable) $-\log p_X$ is the [surprisal](#def-surprisal) $\surprisalf{X}$.
-The surprisal $\surprisalf{X}$ is finite with [probability](probability-basics.qmd#def-probability) 1, because $X$ takes only values with $p_k > 0$,
-and the values with $p_k = 0$ contribute $0$ to both $\crossentropyf{p}{p}$ and $\entropyf{X}$ by the convention in their definitions.
+By the definition of [cross-entropy](#def-cross-entropy) with $q = p$, $\crossentropyf{p}{p} = \E{-\log p(X)}$.
+The [random variable](random-variables.qmd#def-random-variable) $-\log p(X)$ is the [surprisal](#def-surprisal) (or its [continuous](#def-surprisal-continuous) form) $\surprisalf{X}$.
 So $\crossentropyf{p}{p} = \E{\surprisalf{X}} = \entropyf{X}$ by the definition of [entropy](#def-entropy).
 
 :::

_subfiles/def-cross-entropy/_def-kl-divergence.qmd

@@ -0,0 +1,10 @@
+:::{#def-kl-divergence}
+#### Kullback--Leibler divergence (KL divergence, relative entropy)
+
+For $p$ and $q$ as in @def-cross-entropy,
+the **Kullback--Leibler divergence** (**KL divergence**, or **relative entropy**)
+of $q$ from $p$ is
+
+$$\kldivf{p}{q} \eqdef \sumto{k}{K} p_k \log \frac{p_k}{q_k} = \crossentropyf{p}{q} - \crossentropyf{p}{p}.$$
+
+:::

_subfiles/def-cross-entropy/_exm-cross-entropy-continuous.qmd

@@ -0,0 +1,19 @@
+:::{#exm-cross-entropy-continuous}
+#### Cross-entropy of two densities on the unit interval
+
+Let $p(x) = 1$ and $q(x) = 2x$ for $0 < x < 1$ be [densities](random-variables.qmd#def-pdf).
+By @thm-cross-entropy-continuous, using the natural logarithm,
+
+$$
+\ba
+\crossentropyf{p}{q}
+&= -\int_0^1 \log (2x) \, dx \\
+&= -\log 2 - \int_0^1 \log x \, dx \\
+&= -\log 2 + 1 \\
+&\approx 0.307 \text{ nats}.
+\ea
+$$
+
+<!-- no-figure: a single closed-form value -->
+
+:::

_subfiles/def-cross-entropy/_exm-cross-entropy.qmd

@@ -0,0 +1,24 @@
+:::{#exm-cross-entropy}
+#### Cross-entropy and divergence for two coins
+
+Let $p = (\tfrac{1}{2}, \tfrac{1}{2})$ be a fair coin
+and $q = (\tfrac{1}{4}, \tfrac{3}{4})$ a coin biased toward the second outcome.
+Using logarithms to base 2, $\log_2 \tfrac{1}{4} = -2$
+and $\log_2 \tfrac{3}{4} = \log_2 3 - 2$, so
+
+$$
+\ba
+\crossentropyf{p}{q} &= -\tfrac{1}{2} \log_2 \tfrac{1}{4} - \tfrac{1}{2} \log_2 \tfrac{3}{4} \\
+&= 1 + \tfrac{1}{2} \paren{2 - \log_2 3} \\
+&= 2 - \tfrac{1}{2} \log_2 3 \\
+&\approx 1.208, \\
+\crossentropyf{p}{p} &= -\tfrac{1}{2} \log_2 \tfrac{1}{2} - \tfrac{1}{2} \log_2 \tfrac{1}{2} \\
+&= 1, \\
+\kldivf{p}{q} &= \crossentropyf{p}{q} - \crossentropyf{p}{p} = 1 - \tfrac{1}{2} \log_2 3 \approx 0.208.
+\ea
+$$
+
+Here the cross-entropy is larger than the entropy of $p$,
+and the divergence is the difference.
+
+:::

_subfiles/def-cross-entropy/_exr-cross-entropy-general-expectation.qmd → _subfiles/def-cross-entropy/_exr-cross-entropy-general-sum.qmd

@@ -1,9 +1,9 @@
-:::{#exr-cross-entropy-general-expectation}
+:::{#exr-cross-entropy-general-sum}
 
-#### Cross-entropy as an expectation when $q$ is positive where $p$ is
+#### Cross-entropy as a sum when $q$ is positive where $p$ is
 
 Let $p$ and $q$ be [probability mass functions](random-variables.qmd#def-pmf) on $\set{1, \ldots, K}$ with $q_k > 0$ whenever $p_k > 0$.
 Let $X$ have [mass function](random-variables.qmd#def-pmf) $p$, and write $q_X$ for the [random variable](random-variables.qmd#def-random-variable) that equals $q_k$ when $X = k$.
-Show that the [cross-entropy](#def-cross-entropy) $\crossentropyf{p}{q}$ equals the [expectation](#def-expectation) $\E{-\log q_X}$.
+Show that the [cross-entropy](#def-cross-entropy), the [expectation](#def-expectation) $\E{-\log q_X}$, equals $-\sumto{k}{K} p_k \log q_k$.
 
 :::

_subfiles/def-cross-entropy/_rem-cross-entropy-expected-surprisal.qmd

@@ -1,8 +1,8 @@
 :::{#rem-cross-entropy-expected-surprisal}
 #### Cross-entropy is an expected surprisal
 
-By @thm-cross-entropy-expectation, the [cross-entropy](#def-cross-entropy) $\crossentropyf{p}{q}$ is the average [surprisal](#def-surprisal) of the values drawn from $p$,
-with each surprisal measured by $q$ as $-\log q_k$.
+By definition, the [cross-entropy](#def-cross-entropy) $\crossentropyf{p}{q}$ is the average [surprisal](#def-surprisal) of the values drawn from $p$,
+with each surprisal measured by $q$ as $-\log q(x)$.
 By @cor-cross-entropy-entropy, measuring with $p$ itself gives the [entropy](#def-entropy).
 
 :::

_subfiles/def-cross-entropy/_rem-cross-entropy-source.qmd

@@ -0,0 +1,11 @@
+::: {#rem-cross-entropy-source}
+#### Where the definitions come from
+
+@murphy2012mlpp [section 2.8.2, pp. 57--58] defines the KL divergence
+and the cross-entropy as above,
+and gives the coding interpretation:
+the cross-entropy is the average number of bits needed
+to encode data from a source with distribution $p$
+using a code designed for $q$.
+
+:::

_subfiles/def-cross-entropy/_rem-kl-asymmetric.qmd

@@ -0,0 +1,7 @@
+::: {#rem-kl-asymmetric}
+#### The divergence is not symmetric
+
+In general, $\kldivf{p}{q} \ne \kldivf{q}{p}$,
+so KL divergence is not a distance between $p$ and $q$.
+
+:::

_subfiles/def-cross-entropy/_sol-cross-entropy-general-self.qmd

@@ -1,10 +1,7 @@
 :::{#sol-cross-entropy-general-self}
 
-By @thm-cross-entropy-expectation with $q = p$,
-$\crossentropyf{p}{p} = \E{-\log p_X}$.
-Since $\P(X = k) = p_k$, we have $-\log p_X = \surprisalf{X}$, the [surprisal](#def-surprisal) of $X$.
-The surprisal $\surprisalf{X}$ is finite with [probability](probability-basics.qmd#def-probability) 1, because $X$ takes only values with $p_k > 0$,
-and the values with $p_k = 0$ contribute $0$ to both $\crossentropyf{p}{p}$ and $\entropyf{X}$ by the convention in their definitions.
+By definition, $\crossentropyf{p}{p} = \E{-\log p(X)}$.
+Since $\P(X = k) = p_k$, we have $-\log p(X) = \surprisalf{X}$, the [surprisal](#def-surprisal) of $X$.
 So $\crossentropyf{p}{p} = \E{\surprisalf{X}} = \entropyf{X}$ by the definition of [entropy](#def-entropy).
 
 :::

_subfiles/def-cross-entropy/_sol-cross-entropy-general-expectation.qmd → _subfiles/def-cross-entropy/_sol-cross-entropy-general-sum.qmd

@@ -1,21 +1,23 @@
-:::{#sol-cross-entropy-general-expectation}
+:::{#sol-cross-entropy-general-sum}
 
-Let $g(k) \eqdef -\log q_k$, so that $-\log q_X = g(X)$.
+Let $g(k) \eqdef -\log q_k$,
+so that $-\log q_X = g(X)$.
 Since $q_k > 0$ whenever $p_k > 0$,
 $g(k)$ is finite for every value $k$ that $X$ takes with positive [probability](probability-basics.qmd#def-probability).
 By [LOTUS](#thm-lotus),
 
 $$
 \ba
-\E{-\log q_X}
+\crossentropyf{p}{q}
+&= \E{-\log q_X} \\
 &= \E{g(X)} \\
 &= \sum_{k : p_k > 0} g(k) \cd \P(X = k) \\
 &= -\sum_{k : p_k > 0} p_k \log q_k \\
-&= -\sumto{k}{K} p_k \log q_k \\
-&= \crossentropyf{p}{q}.
+&= -\sumto{k}{K} p_k \log q_k.
 \ea
 $$
 
-The fourth line adds the terms with $p_k = 0$, which are $0$ by the convention in the definition.
+The last line adds the terms with $p_k = 0$,
+which are $0$ by the convention in @thm-cross-entropy-discrete.
 
 :::

_subfiles/def-cross-entropy/_thm-cross-entropy-continuous.qmd

@@ -0,0 +1,42 @@
+:::{#thm-cross-entropy-continuous}
+#### Cross-entropy of two densities
+
+Let $X$ be a [continuous random variable](random-variables.qmd#def-continuous-rv) with [density](random-variables.qmd#def-pdf) $p$ on its [range](random-variables.qmd#def-range) $\rangef{X}$,
+and let $q$ be a density on $\rangef{X}$,
+with $q(x) > 0$ wherever $p(x) > 0$.
+Then the [cross-entropy](#def-cross-entropy) is
+
+$$\crossentropyf{p}{q} = -\int_{x \in \rangef{X}} p(x) \log q(x) \, dx.$$
+
+The integrand is taken to be $0$ where $p(x) = 0$.
+Otherwise, if $\P(q(X) = 0) > 0$,
+both sides are $\infty$.
+
+:::
+
+::: proof
+
+Let $g(x) \eqdef -\log q(x)$,
+so that $-\log q(X) = g(X)$.
+Then [LOTUS](#thm-lotus) gives
+
+$$
+\ba
+\crossentropyf{p}{q}
+&= \E{g(X)} && \text{(definition)} \\
+&= \int_{x \in \rangef{X}} g(x) \cd p(x) \, dx && \text{(LOTUS)} \\
+&= -\int_{x \in \rangef{X}} p(x) \log q(x) \, dx. && \text{(definition of } g \text{)}
+\ea
+$$
+
+Now suppose $\P(q(X) = 0) > 0$.
+Then $-\log q(X) \ge 0$,
+and $-\log q(X) = \infty$ with positive [probability](probability-basics.qmd#def-probability),
+so $\crossentropyf{p}{q} = \infty$.
+The integrand $-p(x) \log q(x)$ is $\infty$
+wherever $q(x) = 0$ and $p(x) > 0$,
+which is a region of positive probability under $p$,
+and it is never $-\infty$,
+so the right side is also $\infty$.
+
+:::

_subfiles/def-cross-entropy/_thm-cross-entropy-discrete.qmd

@@ -0,0 +1,41 @@
+:::{#thm-cross-entropy-discrete}
+#### Cross-entropy of two mass functions
+
+Let $p$ and $q$ be [probability mass functions](random-variables.qmd#def-pmf) on $\set{1, \ldots, K}$,
+with $p_k$ and $q_k$ the probabilities of outcome $k$.
+Then the [cross-entropy](#def-cross-entropy) is
+
+$$\crossentropyf{p}{q} = -\sumto{k}{K} p_k \log q_k.$$
+
+A term with $p_k = 0$ is taken to be $0$ whatever $q_k$ is, as for [entropy](#thm-entropy-discrete).
+
+:::
+
+::: proof
+
+Let $X$ have [mass function](random-variables.qmd#def-pmf) $p$,
+and let $g(k) \eqdef -\log q_k$,
+so that $-\log q(X) = g(X)$.
+First suppose $q_k > 0$ whenever $p_k > 0$.
+Then $g(k)$ is finite for every value $k$ that $X$ takes with positive [probability](probability-basics.qmd#def-probability),
+and [LOTUS](#thm-lotus) gives
+
+$$
+\ba
+\crossentropyf{p}{q}
+&= \E{g(X)} && \text{(definition)} \\
+&= \sum_{k : p_k > 0} g(k) \cd \P(X = k) && \text{(LOTUS)} \\
+&= -\sum_{k : p_k > 0} p_k \log q_k && \text{(definition of } g \text{ and } \P(X = k) = p_k \text{)} \\
+&= -\sumto{k}{K} p_k \log q_k. && \text{(terms with } p_k = 0 \text{ are } 0 \text{)}
+\ea
+$$
+
+Now suppose $q_k = 0$ for some $k$ with $p_k > 0$.
+Then $-\log q(X) \ge 0$,
+and $-\log q(X) = \infty$ with probability at least $p_k > 0$,
+so $\crossentropyf{p}{q} = \infty$.
+The right side is also $\infty$,
+because its term for $k$ is $-p_k \log 0 = \infty$
+and its other terms are finite or $\infty$ but never $-\infty$.
+
+:::