Conditional Probability

The banner image suggests the decomposition of a sample space $\mathrm{\Omega }$ by values of a random variable $X$ into different level sets $\{X=x\}$.

$$$$

Index | Other Stuff | Background

0.1 Nutshell Summary

Let $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ be a probability space and $\mathcal{G}\subset \mathcal{F}$ a sub-sigma algebra. The conditional probability of a set $A\in \mathcal{F}$ given information in $\mathcal{G}$ is a $\mathcal{G}$-measurable function $\mathsf{P}(A\mid \mathcal{G})$ on $\mathrm{\Omega }$ such that $$\mathsf{P}(B\cap A)={\int }_{B}P(A\mid \mathcal{G})d\mathsf{P}$$ for all $B\in \mathcal{G}$. Any two such functions are equal a.e. $[\mathsf{P}]$.

If $X$ is a real-valued random variable on $\mathrm{\Omega }$ then, the conditional expectation of $X$ given $\mathcal{G}$ is a $\mathcal{G}$-measurable function $\mathsf{E}[X\mid \mathcal{G}]$ such that $${\int }_B\mathsf{E}[X\mid \mathcal{G}](\omega )\mathsf{P}(d\omega )={\int }_{B}X\mathsf{P}(d\omega )$$ for all $B\in \mathcal{G}$. Again, any two such functions are equal a.e. $[\mathsf{P}]$.

If $T$ is another real-valued random variable on $\mathrm{\Omega }$, then $\mathsf{E}[X\mid T]$ means $\mathsf{E}[X\mid \sigma (T)]$. Since it is $\sigma (T)$-measurable, it factors through $T$ giving a function $\mathbb{R}\to \mathbb{R}$ so that $\mathsf{E}[X\mid T]=\varphi (T)$. Then $\mathsf{E}[X\mid T=t]$ equals $\varphi (t)$. Here, $\mathsf{E}[X\mid T=t]$ is defined as a function with the “right” integrals against real Borel sets: $\mathrm{\forall }B\in \mathcal{B}(\mathbb{R}),{\int }_B\mathsf{E}[X\mid T=t]{\mathsf{P}}_T(dt)={\int }_{T^{-1}}(B)Xd\mathsf{P}$.

Conditional probabilities are simply conditional expectations of indicator functions $1_A$. Using Pollard’s all-in-one notation, conditional probabilities and expectations satisfy $$\mathsf{P}(BA)=\mathsf{P}(B\mathsf{P}(A\mid G)),\mathsf{P}[BX]=\mathsf{P}(B\mathsf{E}[X\mid G]).$$ We try to write the $\mathcal{G}$-measurable function first.

It is usually possible to combine the functions $\mathsf{P}(A\mid \mathcal{G})$ for different $A$ into a function $p:\mathrm{\Omega }\times \mathcal{F}\to [0,1]$ called a regular conditional probability satisfying:

1. MEAS: $\mathrm{\forall }\omega \in \mathrm{\Omega }$, $B\mapsto p(\omega ,B)$ is a probability measure.
2. VER: $\mathrm{\forall }A\in \mathcal{F}$, the function $\omega \mapsto p(\omega ,A)$ is a version of $\mathsf{P}(A\mid \mathcal{G})$.

If the condition in MEAS only holds a.e., then the missing values of $\omega$ can be replaced with an arbitrary probability measure so that it holds for all $\omega$ and condition VER is still true. Thus for all or a.e. can be used interchangeably in MEAS. Since $\mathsf{P}(A\mid \mathcal{G})$ is $\mathcal{G}$-measurable, so is the function in VER.

The conditional and regular conditional probabilities are connected via $$\begin{array}{c}\mathsf{P}(A\cap B)={\int }_B{1}_A({\omega }^{\prime })p(\omega ,d{\omega }^{\prime })=p_{\omega }BA\\ \mathsf{E}[X\mid \mathcal{G}](\omega )={\int }_{\mathrm{\Omega }}X({\omega }^{\prime })p(\omega ,d{\omega }^{\prime })=p_{\omega }X.\end{array}$$

There is a chain of increasing generality, from working on $(\mathrm{\Omega },\mathcal{F})$ with $\mathsf{P}$ to working on $S$ and $\sigma (T)$ with $P_X$.

It was one of Kolmogorov’s many great ideas to use the Radon-Nikodým theorem (RNT) to define conditional probabilities and conditional expectations (Kolmogorov and Bharucha-Reid 2013). The idea can be applied in several flavors, which we now explore.

0.2 Notation

• $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ is a probability space and $(L,\mathcal{A})$ and $(M,\mathcal{B})$ is are measure spaces
• $\mathcal{G}\subset \mathcal{F}$ is a sub-sigma algebra
• $A\in \mathcal{F}$ and $B\in \mathcal{G}$ are measurable subsets
• $X:(\mathrm{\Omega },\mathcal{F})\to (\mathbb{R},\mathcal{B}(\mathbb{R}))$ is an $\mathcal{F}$-measurable random variable
• $H:(\mathrm{\Omega },\mathcal{G})\to (\mathbb{R},\mathcal{B}(\mathbb{R}))$ is a $\mathcal{G}$-measurable random variable
• $S:(\mathrm{\Omega },\mathcal{F})\to (L,\mathcal{A})$ is a $\mathcal{F}$ measurable random element, taking values in $(L,\mathcal{A})$
• $T:(\mathrm{\Omega },\mathcal{F})\to (M,\mathcal{B})$ is a $\mathcal{F}$ measurable random element, taking values in $(M,\mathcal{B})$
• ${\mathsf{P}}_T$ is a measure of $\mathcal{B}$ defined by ${\mathsf{P}}_T(B)=\mathsf{P}(T^{-1}(B))=\mathsf{P}(T\in B)$

The (Lebesgue) integral used in probability is written in many different ways: $${\int }_{B}Xd\mathsf{P}={\int }_{B}X(\omega )\mathsf{P}(d\omega )=\int 1_{B}Xd\mathsf{P}=\mathsf{E}[1_{B}X]=\mathsf{P}{1}_{B}X.$$ The last notation suggested by Pollard (2002), emphasizes the role of expectation as a linear operator and makes the connection to $\mathsf{P}$ is especially clear.

0.3 Conditional Probabilities and Conditional Expectations

We can define a measure $\nu$ on $\mathcal{G}$ by $$\nu (B)={\int }_{B}Xd\mathsf{P}.$$ The measure $\nu$ is absolutely continuous with respect to $\mathsf{P}$, $\nu \ll \mathsf{P}$, meaning $$\mathsf{P}(B)=0⟹\nu (B)=0.$$ Therefore, RNT gives a $\mathcal{G}$-measurable function $\xi :\mathrm{\Omega }\to \mathbb{R}$ so that $$\nu (B)={\int }_B\xi d\mathsf{P}=\mathsf{P}{1}_B\xi$$ i.e., $\xi$ and $X$ have the same integrals over $\mathcal{G}$-measurable subsets of $\mathrm{\Omega }$. Since we can build any $\mathcal{G}$-measurable function $H$ from sums of indicator functions, we can extend the identity $\nu (B)=\mathsf{P}{1}_B\xi$ to any $H$ to obtain the fundamental identity of conditional probability $$\mathsf{P}H\xi =\mathsf{P}HX.$$ The function $\xi$ is given special names and notations in different situations, the most common of which are conditional probability and conditional expectation.

Define $\mathsf{E}[X\mid T=t]$ to be any $\mathcal{B}$-measurable function so that $$\mathrm{\forall }B\in \mathcal{B},{\displaystyle {\int }_B\mathsf{E}[X\mid T=t]{\mathsf{P}}_T(dt)={\displaystyle {\int }_{T^{-1}(B)}Xd\mathsf{P}.}}$$ Generalizing to $H$ we could use Pollard’s notation to write¹ $${\mathsf{P}}_T^{t}H(t)\mathsf{E}[X\mid T=t]=\mathsf{P}H(T)X.$$ The conditional expectation of $X$ given $T=t$ exists by the standard RNT argument (applied to measure on $\mathcal{B}$ defined by integral on the right) combined with the transformation theorem REF.

The next table summarizes particular instances of these definitions, illustrating their connection to the fundamental equality $\mathsf{P}H\xi =\mathsf{P}HX$.

General definition of conditional probabilities and three common specifications.

$\mathcal{G}$	$H$	$X$	$\xi$ notation	$\mathsf{P}H\xi$	$\mathsf{P}HX$
General case				fundamental equality: $\mathsf{P}H\xi =\mathsf{P}HX$
$\mathcal{G}$	$H$	$X$	$\xi$	${\displaystyle \int H\xi d\mathsf{P}}$	${\displaystyle \int HXd\mathsf{P}}$
Specific examples
$\mathcal{G}$	$1_B$	$1_A$	$\mathsf{P}(A\mid \mathcal{G})$	${\displaystyle {\int }_B\mathsf{P}(A\mid \mathcal{G})d\mathsf{P}}$	$\mathsf{P}(A\cap B)$
$\mathcal{G}$	$1_B$	$X$	$\mathsf{E}[X\mid \mathcal{G}]$	${\displaystyle {\int }_B\mathsf{E}[X\mid \mathcal{G}]d\mathsf{P}}$	${\displaystyle {\int }_{B}Xd\mathsf{P}}$
$\sigma (T)$	$1_{T^{-1}(B)}$	$X$	$\mathsf{E}[X\mid T]=\phi (T)$ factorization theorem   $\mathsf{E}[X\mid T=t]=\phi (t)$ uniqueness, right	${\displaystyle {\int }_{T^{-1}(B)}\mathsf{E}[X\mid T]d\mathsf{P}}$ $={\displaystyle {\int }_{T^{-1}(B)}^{\phantom{X}}\phi (T)d\mathsf{P}}$ factorization $={\displaystyle {\int }_B^{\phantom{X}}\phi d{\mathsf{P}}_T}$ transformation $={\displaystyle {\int }_B^{\phantom{X}}\mathsf{E}[X\mid T=t]{\mathsf{P}}_T(dt)}$ definition	${\displaystyle {\int }_{T^{-1}(B)}Xd\mathsf{P}}$

0.4 Regular Conditional Probability

The function $\xi$ given by RNT depends on $\mathcal{G}$ and $X$ and is a function of $\omega \in \mathrm{\Omega }$. In applications, $\mathcal{G}$ is usually fixed while different $X$ are considered. To reflect these dependences, write $\xi (\omega )=\xi (X\mid \mathcal{G})(\omega )$. In the case $X=1_A$, we can write $\xi (A\mid \mathcal{G})(\omega )$ using Pollard’s identification of a set with its indicator function. It would be very handy if the different $\xi$ (one for each $X$) could be stitched together consistently to create a function $P:\mathcal{F}\times \mathrm{\Omega }\to [0,1]$ with

1. $\mathrm{\forall }A\in \mathcal{F},\omega \to P(A,\omega )$ a $\mathcal{G}$-measurable function
2. $\mathrm{\forall }\omega \in \mathrm{\Omega },A\to P(A,\omega )$ is a probability measure
3. The function in item 1 is a version of the function $\xi (A\mid \mathcal{G})$

Notice that item 3 implies the function in item 1 is $\mathcal{G}$-measurable because $\xi$ is $\mathcal{G}$-measurable. Such a $P$ is called a regular conditional probability (rcp) function, because we are stitching together all the functions in item 1 consistently to form a regular² probability measure in item 2. See Section 0.8 for other names given by different textbooks.

When $\mathcal{G}=\sigma (T)$ then $P:\mathcal{F}\times M\to [0,1]$ and some authors (Pollard 2002, p. Dellacherie2011bk) write $P(A,t)=P_t(A)$. Hoffman-Jørgensen (2017a) writes $P^T(A\mid t)$. In these cases we have a decomposition of $\mathsf{P}$ $$\mathsf{P}H(T)X={\mathsf{P}}_T^t{P}_t^{\omega }H(T(\omega ))X(\omega ).$$

Conditional expectations can be computed using an rcp $P$ $$\mathsf{E}[X\mid \mathcal{G}](\cdot )=\int X(\omega )P(d\omega ,\cdot ).$$ To see this, start with a simple function $X=1_A$. Then, by rcp definition item 3, $$\mathsf{E}[1_A\mid \mathcal{G}](\cdot )=\mathsf{P}(A\mid \mathcal{G})(\cdot )=P(A,\cdot )=\int 1_A(x)P(dx,\cdot ).$$ The result follows for general $X$ by approximating using simple functions and using monotone convergence. See (Billingsley 2017 Thm 34.5)

A function satisfying conditions 1 and 2 above is called a probability kernel (or Markov kernel) from $(M,\mathcal{B})$ to $(\mathrm{\Omega },\mathcal{F})$. If the probability measure in item 2 is just a measure, then $P$ is called a kernel.

More generally, a second random element $S$ is used to create different distributions ${\mathsf{P}}_S$ on $(L,\mathcal{A})$. A regular conditional distribution of $S$ given $T$ is a Markov kernel $P_S^T$ from $(M,\mathcal{B})$ to $(L,\mathcal{A})$ so that $P_S^T(A\mid t)$ is a conditional distribution of $S$ given $T$. The fundamental equality for $P_S^T$ is³ $$\mathrm{\forall }A\in \mathcal{A},\mathrm{\forall }B\in \mathcal{B},\mathsf{P}(S\in A,T\in B)={\int }_B{P}_S^T(A\mid t){\mathsf{P}}_T(dt).$$ This decomposition can be understood as a generalization $$\mathsf{P}={\mathsf{P}}_{T}P\leftrightarrow \mathsf{P}={\mathsf{P}}_T{P}_S^T.$$ (Apologies for the hodge-podge notation!) A special case is $S=X$ is a (real-valued) random variable.

0.5 Disintegrations

In addition, if $P_t\{T\ne t\}=0$ then the decomposition of $\mathsf{P}$ further simplifies $$\mathsf{P}H(T)X={\mathsf{P}}_T^t{P}_t^{\omega }H(T(\omega ))X(\omega )={\mathsf{P}}_T^{t}H(t)P_t^{\omega }X(\omega )={\mathsf{P}}_T^{t}H(t)\phi (t)$$ where, again, $\phi (t)=P_t^{\omega }X(\omega )=\mathsf{E}[X\mid T=t]$. In this case, Pollard⁴ calls $\{P_t{\}}_t$ a disintegration of $\mathsf{P}$ with respect to ${\mathsf{P}}_T$. Following Dellacherie and Meyer (1978), he proves existence using a general disintegration theorem that starts with a measure on a product space and decomposes it into a kernel between the products and the marginal distribution of one product. The general theorem is applied to the image of $\mathsf{P}$ under the map $\omega \mapsto (\omega ,T\omega )\in \mathrm{\Omega }\times M$ assuming the graph of this embedding is $\mathcal{A}\otimes \mathcal{B}$-measurable. Graphs are measurable when $\mathcal{B}$ is separably (countably) generated, which is the assumption in (Rogers and Williams 1994, p. II.89, Theorem).

0.6 Existence Results

In the general case, an rcd for $S$ given $T$ exists if $(L,\mathcal{A})$ is Borel, Kallenberg (2001a) Theorem 6.3. Hoffman-Jørgensen (2017a) 10.29 describes a more complicated set of assumptions which are satisfied for Borel $(L,\mathcal{A})$ and $P_S$ a Radon measure.

Rogers and Williams (1994) II.89.1 works with probabilities rather than distributions (i.e., no $S$) on $\mathrm{\Omega }$ a Lusin space and relative to $\mathcal{G}\subset \mathcal{F}$. The proof starts assuming $\mathrm{\Omega }$ is compact metrizable and works with a countable dense subset of the continuous functions on $\mathrm{\Omega }$. It makes a version of the conditional expectation of each element using Riesz Representation. From there, bootstrap to general functions by using uniform convergence (for all continuous functions), to bounded measurable functions using a monotone class argument, and finally to integrable functions by truncation. To get a proper (correct support, “live where they should”) conditional expectation, they assume that $\mathcal{G}$ is a countably generated sub-sigma algebra. The latter assumption shows that the set with correct support is $\mathcal{G}$ measurable with measure 1—see also discussion of Chang and Pollard (1997) below. Their results highlight there are two assumptions driving different aspects of the conclusion:

• Existence of an rcd depends on topological properties of $\mathrm{\Omega }$: they essentially limit measurable functions; and
• Correct support of the rcd depends on separability of $\mathcal{G}$, which makes the graph measurable and allows application of the disintegration theorem.

Dellacherie and Meyer (1978) also starts with a compact metric space and Borel sigma-field. He extends to any separable metrizable space with Borel sigma-field and assumes that $\mathsf{P}$ is tight (is approximated by a compact set). Then a rcp relative to any sub-sigma algebra.

Pollard (2002) proves a disintegration theorem on a metric space with its Borel sigma-algebra and a function $T$ into $(M,\mathcal{B})$. He assumes that the graph of $T$ is $\mathcal{F}\otimes \mathcal{B}$-measurable. The metric space assumption limits the topological properties of $\mathrm{\Omega }$ and the measurable graph assumption results in a proper rcd. Chang and Pollard (1997) Theorem 1 (existence) is the same: works on a metric space with a sigma-finite Radon measure⁵ and where $T$ takes values in a space with a countably generated sigma algebra that contains all singleton sets.

Chang and Pollard (1997) Theorem 2 states that if $\lambda$ on a metric space $\mathrm{\Omega }$ with Borel sigma algebra with has a $(T,\mu )$-disintegration ${_t}, with $\lambda$ and $\mu$ each sigma finite then

1. $T\lambda \ll \mu$ with density ${\lambda }_t\mathrm{\Omega }$
2. $\{{\lambda }_t\}$ are finite for $\mu$-a.e. $t$ iff $T\lambda$ is sigma-finite
3. $\{{\lambda }_t\}$ are probabilities for $\mu$-a.e. $t$ iff $\mu =T\lambda$
4. If $T\lambda$ is sigma-finite then $(T\lambda )\{{\lambda }_t\mathrm{\Omega }=0\}=0$ and $(T\lambda )\{{\lambda }_t\mathrm{\Omega }=\mathrm{\infty }\}=0$. For $T\lambda$-a.e. $t$ the measures $${\overline{\lambda }}_t(\cdot )=\frac{{\lambda }_t(\cdot )}{{\lambda }_t\mathrm{\Omega }}\{0<{\lambda }_t\mathrm{\Omega }<\mathrm{\infty }\}$$ are probabilities that give a $T$-disintegration of $\lambda$.

Proof. Let $l(t)={\lambda }_t\mathrm{\Omega }$. For non-negative measurable $g$ $$\begin{array}{}\text{(1)}& (T\lambda )g={\lambda }^{x}g(Tx)={\mu }^t{\lambda }_t^{x}g(Tx)={\mu }^{t}g(t)l(t)\end{array}$$ because $g(Tx)=g(t)$ mod ${\lambda }_t$.

1. If $g\ge 0$ and $\mu g=0$ then $g(t)=0$ mod $\mu$ and so $g(t)l(t)=0$ mod $\mu$. Equation 1 exactly expresses that $l(t)$ is the density(!).
2. Sigma-finite iff there exists a strictly positive real-valued function with finite integral, so there exists $h>0$ with $\mu h<\mathrm{\infty }$, so by assumption $h$ exist. If $l(t)<\mathrm{\infty }$ mod $\mu$, the function $g(t)=h(t)/(1+l(t))>0$ mod $\mu$ and $(T\lambda )g={\mu }^{t}g(t)l(t)\le \mu h<\mathrm{\infty }$ so $T\lambda$ is sigma-finite. Conversely, if $(T\lambda )k=\mu kl<\mathrm{\infty }$ for $k>0$ then $k(t)l(t)<\mathrm{\infty }$ mod $\mu$ (by 1? why needed), which gives finiteness of $l(t)$ mod $\mu$.
3. If $l(t)=1$ mod $\mu$ then Equation 1 shows that $(T\lambda )g=\mu g$. Conversely, let $h>0$ have $\mu h<\mathrm{\infty }$ (by sigma finite assumption). Choose $g(t)=h(t)\{l(t)<1\}$ in Equation 1 and using the assumption that $T\lambda =\mu$ gives $$\mathrm{\infty }>{\mu }^{t}h(t)\{l(t)<1\}=??$$ which implies $\mu \{l<1\}=0$. Similarly $\mu \{l>1\}=0$.
4. …SORT OUT.

Chang and Pollard (1997) appendix proves the existence of the disintegration theorem, mirroring Rogers and Williams (1994). EXPAND.

Breiman (1992a) uses a random element to move the problem to a “nice” space and describes this as “like passing over to representation space” to get rid of a difficulty.

0.7 Blackwell and Proper rcps

Blackwell (1956) - Lusin spaces and three weird examples (Doob’s example).

Blackwell and Ryll-Nardzewski (1963) - everywhere proper cds

Blackwell and Dubins (1975) (w Dubins): for countably generated $\mathcal{F}$, if $\mathcal{G}\subset \mathcal{F}$ is not countably generated then no rcp given $\mathcal{G}$ is proper (lives in the right place). Not countably generated is implied by the existence of an extreme (only taking values 0 or 1) probability measure on $\mathcal{G}$ and which is supported by no $\mathcal{G}$-atom in $\mathcal{G}$. READ THIS ONE!

0.8 Common Notation

Table 1 describes the notation used by different authors for (r)cps, (r)cds, and disintegrations (an rcp with extra assumptions). Here we use the abbreviations regular, conditional, probability, and distribution in various combinations. Rcps apply to $\mathsf{P}$, rcds apply to ${\mathsf{P}}_X$, so $X$ allows you to vary the probability. Per Breiman (1992a), $X$ can be used to move to a “nice” space, ensuring that an rcd exists. Existence is determined by properties of the range measure space, see

• rc probabiliity $\leftrightarrow$ $\mathsf{P}$ on $\mathrm{\Omega }$
• rc distributions $\leftrightarrow$ conditional distribution of $X$ on $\mathrm{\Omega }$

Table 1: Common nomenclature in various textbooks.

Reference	Nomenclature	Disintegration
Billingsley (2017)	cd, Thm 33.3	n/a
Breiman (1992a)	rcp, 4.3 Def 4.27 rcd for rv	n/a
Dellacherie and Meyer (1978)	c “laws”	Ch III.70 w support
Dudley (2018)	rcp, 10.2; rcd p.342	n/a
Hoffman-Jørgensen (2017a)	rcd, II.10.1	II.10.30
Kallenberg (2001a)	rcp Ch.6; rcd $S\mid \mathcal{B}$	Thm 6.4
Parthasarathy (1967)	rcpd given $T$, V.8 p.146
Pollard (2002)	rcd, remark w/o	cpd Ch 5.2<4> w support
Rogers and Williams (1994)	rcp, II.42.3	see II.89.1 iii, iv
Williams (1991a)	rcp 9.9	n/a

https://stats.stackexchange.com/questions/531705/regular-conditional-distribution-vs-conditional-distribution

If $X$ is $\mathcal{F}$-measurable and $Y$ is $\mathcal{G}$-measurable and $f(X,Y)$ is jointly measurable and integrable then $$\mathsf{E}[f(X,Y)\mid \mathcal{G}](\omega )=\int f(x,Y(\omega )){\mathsf{P}}_{\omega }(dx)$$ ??

Kolmogorov and regular conditional expectations and disintegration of measure.

Concept	Notation	Conditions
conditional probability	$\mathsf{P}(A\mid \mathcal{F})(\cdot )$	$\nu (B)=\mathsf{P}(B\cap A)$ for $B\in \mathcal{F}$, $\nu \ll \mathsf{P}$ $\nu (B)={\int }_{B}P(A\mid \mathcal{G})d\mathsf{P}$ for $B\in \mathcal{G}$
regular conditional probability defined by $\mathcal{G}$	$p(\cdot ,A)$	MEAS: $\mathrm{\forall }\omega \in \mathrm{\Omega }$, $B\mapsto p(\omega ,B)$ is a probability measure on $\mathcal{F}$. VER$(\mathcal{F})$: $\mathrm{\forall }A\in \mathcal{F}$, $\omega \mapsto p(\omega ,A)$ is a version of $\mathsf{P}(A\mid \mathcal{G})$.
regular conditional probability defined by $\mathcal{G}=\sigma (T)$	$p_t(A)$   $P^T(A\mid t)$	$T:(\mathrm{\Omega },\mathcal{F})\to (E,\mathcal{E})$ random element MEAS: $\mathrm{\forall }t\in E$, $B\mapsto p(t,B)$ is a probability measure on $\mathcal{F}$. VER$(\mathsf{P},\sigma (T))$: $\mathrm{\forall }A\in \mathcal{F}$, $t\mapsto p(t,A)$ is a version of $\mathsf{P}(A\mid \sigma (T))$.
(regular) conditional distribution for rv $X$ and ${\mathsf{P}}_X$	$\mathrm{rcdf}(\mathsf{P},\mathcal{G})$	$X:(\mathrm{\Omega },\mathcal{F})\to (\mathbb{R},\mathcal{B}(\mathbb{R}))$ is a rv MEAS: as rcp VER$({\mathsf{P}}_X,\sigma (T))$: as rcp
(regular) conditional distribution for random element $S:(\mathrm{\Omega },\mathcal{F})\to (L,\mathcal{A})$ defining ${\mathsf{P}}_S$	$\mathrm{rcd}({\mathsf{P}}_S,\sigma (T))$	$T:(\mathrm{\Omega },\mathcal{F})\to (M,\mathcal{B})$ random variable MEAS: $\mathrm{\forall }t\in M$, $A\mapsto p(t,A)$ is a probability measure on $\mathcal{A}$. VER$({\mathsf{P}}_S,\sigma (T))$: $\mathrm{\forall }A\in \mathcal{A}$, $t\mapsto p(t,A)$ is a version of ${\mathsf{P}}_S(A\mid T)$.
disintegration as above, $\mu =T\mathsf{P}$	$\mathsf{P}=\mu P_t$	$P_t\{T\ne t\}=0$ ${\mathsf{P}}^{x}f(x,Tx)={\mu }^t{P}_t^{x}f(x,t)$

1 Notation

Pith. These notes follow Hoffman-Jørgensen (2017b). Thus, the general set up is a measure space $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ with random elements (variables) $S:(L,\mathcal{A})\to ??$ and $T:(M,\mathcal{B})\to ??$. The variable $S$ is used to create a distribution $S\mathsf{P}={\mathsf{P}}_S$ and $T$ is observed information. We are interested in predicting $S$ given $T$. The most common special case is $S=T=$ the identity, $(L,\mathcal{A})=(\mathrm{\Omega },\mathcal{F})$ and $M=\mathrm{\Omega }$ but $\mathcal{B}\subset \mathcal{F}$.

Periphery. Probabilitists, analysts, set theorists, and statisticians all write probability and they tend to use slightly different notation. These differences can make it hard to learn probability.

Compounding these differences, probability needs notation for spaces, sigma-algebras, measures, subsets, and random variables. The one thing everyone agrees that $\mathcal{B}$ denotes Borel sets, the sigma algebra generated by the open sets when the sample space has a topology. Table 3 provides a summary of notation across some common textbooks.

We mix and match notation for expectations and rcps. Hoffman-Jørgensen’s notation $P_S^T(\cdot \mid \cdot )$ makes all the components of the rcp clear. However, Pollard’s de Finetti inspired notation that dispenses with distinctions between sets and indicator functions, and between probabilities and expectations is very concise and helps emphasize the parallel’s between probabilities and expectations (Pollard 2002). Pollard uses $P_t$ for $P^T(\cdot \mid t)$. Conditional probabilities/expectations become $P_{t}f$ or $P_{t}B$ and decomposition ${\mu }^t{P}_t^{x}f(x,t)$.

Always we use $\mu f=\int fd\mu =\int f(x)\mu (dx)$ and allow $\mu A=\mu 1_A$.

2 Motivation

The general theory of conditional probability builds on the elementary discrete theory, which conditions on sets of positive measure.

S1.⁶ Let $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ be a probability space, and let $A\in \mathcal{F}$ be an event with $\mathsf{P}(A)>0$. The conditional probability with respect to $A$ means

$$\mathsf{P}(B\mid A)=\frac{\mathsf{P}(AB)}{\mathsf{P}(A)}.$$

If $X$ is an integrable random variable then

$$\mathsf{E}[X\mid A]=\frac{\mathsf{E}[X{1}_A]}{\mathsf{P}(A)}.$$

S2. Let $\mathcal{A}=\{A_1,A_2,\dots \}$ be a countable decomposition of $\mathrm{\Omega }$ into sets with $\mathsf{P}(A_i)>0$. The conditional probability with respect to $\mathcal{A}$ is the random variable $\mathsf{P}(B\mid \mathcal{A})(\cdot )$ defined by

$$\mathsf{P}(B\mid \mathcal{A})(\omega )=\sum _i\mathsf{P}(B\mid A_i)1_{A_i}(\omega ).$$

Thus

$$\mathsf{P}(B\mid \mathcal{A})(\omega )=\frac{\mathsf{P}(B{A}_{\omega })}{\mathsf{P}(A_{\omega })}$$

where $A_{\omega }$ is the member of the decomposition containing $\omega$. This definition makes it clear that if $\mathcal{A}=\{\mathrm{\Omega }\}$ then $\mathsf{P}(B\mid \mathcal{A})=\mathsf{P}(B)$. Clearly

$$\mathsf{E}\mathsf{P}(B\mid \mathcal{A})=\mathsf{P}(B).$$

The random variable $\mathsf{P}(B\mid \mathcal{A})$ is measurable wrt $\mathcal{G}=\sigma (\mathcal{A})$ and is also denoted $\mathsf{P}(B\mid \mathcal{G})$.

S3. Countable decompositions are naturally generated by random variables taking a finite or countably infinite number of different values. If $X$ takes the value $x_i$, define $A_i=\{X=x_i\}$. It is natural to define the conditional expectation wrt $X$ using $\mathsf{P}(\cdot \mid \mathcal{A})$. It is denoted $\mathsf{P}(\cdot \mid X)$.

S4. What about conditional probabilities wrt sets of measure zero? These occur naturally when conditioning on the outcome of a continuous random variable.

H1. Think about varying the conditioning set $A$ systematically. Let $Y=\{y_1,\dots ,y_n\}$ be a discrete space with the subset sigma algebra. Let $T:\mathrm{\Omega }\to Y$ map $A_i$ to $y_i$. The values of $Y$ split up $\mathrm{\Omega }$, providing the sets over which we wish to condition. Now let $B\in \mathcal{F}$ and $A\subset Y$, and define a measure on $Y$ by ${\nu }_B(A)=\mathsf{P}(B\cap T^{-1}(A))$. Clearly ${\nu }_B\ll T\mathsf{P}$ as measures on $Y$, using Pollard notation $T\mathsf{P}(A)=\mathsf{P}(T^{-1}(A))$. Therefore RNT gives an integrable function $p_B$ on $Y$ so that $${\nu }_B(A)=\mathsf{P}(B\cap T^{-1}(A))={\int }_A{p}_B(y)d(T\mathsf{P})(y).$$ Call $p_B(y)=p(B,y)$ the conditional probability of $B$ given $y$ or given that $T=y$.

If $A$ has $T\mathsf{P}(A)=\mathsf{P}(T^{-1}(A))>0$ we can divide to get $$\mathsf{P}(B\mid T^{-1}(A))=\frac{\mathsf{P}(B\cap T^{-1}(A))}{\mathsf{P}(T^{-1}(A))}=\frac{1}{\mathsf{P}(T^{-1}(A))}{\int }_{A}p(B,y)d(T\mathsf{P})(y)$$ Halmos then says

Since the left term is the conditional probability of $B$ given $T^{-1}(A)$, it is formally possible that as “$A$ shrinks to $y$” the left term should tend to the conditional probability of $B$ given $y$ and the right term should tend to the integrand $p(B,y)$. The RNT is a rigorous substitute for this rather shaky “difference quotient” approach.

He then shows that the collection of functions $p(\cdot ,y)$ act like a measure for each $y$, e.g., they lie between 0 and 1 and are countably additive for $T\mathsf{P}$ a.e. $y$. For example if $B_n$ is a disjoint sequence in $\mathrm{\Omega }$ then $$\begin{array}{rl}{\int }_{A}p(\bigcup _n{B}_n,y)d(T\mu )(y)& =\mathsf{P}((\bigcup _n{B}_n)\cap T^{-1}(A))\\ & =\sum _n\mathsf{P}(B_n\cap T^{-1}(A))\\ & =\sum _n{\int }_{A}p(B_n,y)d(T\mu )(y)\\ & ={\int }_A\sum _{n}p(B_n,y)d(T\mu )(y)\end{array}$$ and apply uniqueness from RNT.

H2. Thus $p(\cdot ,y)$ behaves like a measure. Similar proofs show it is monotone, increases from zero to 1 etc. In all these proofs the excesptional sets depends on the particular $B_n$ under consideration, and “it is in general incorrect to conclude that $p(\cdot ,y)$ is a measure a.e. $y$.”

H3. Extending $$\begin{array}{rl}{\int }_{B}p(A,y)d(T\mathsf{P})(y)& =\mathsf{P}(A\cap T^{-1}(B))\\ & ={\int }_{T^{-1}(B)}{1}_A(x)d\mathsf{P}(x)\end{array}$$ from $1_A$ to any integrable $f$ on $\mathrm{\Omega }$, we can consider its integral $$\nu (f)={\int }_{T^{-1}(B)}f(x)d\mathsf{P}(x)$$ for measurable $B\subset Y$ as a measure on subsets of $Y$. Then $\nu \ll T\mathsf{P}$ (because you would be integrating over a set of measure zero), we can apply RNT to get an integrable function $e(f,y)$ on $Y$ so that $${\int }_{T^{-1}(B)}f(x)d\mathsf{P}(x)={\int }_{B}e(f,y)d(T\mathsf{P})(y)$$ for measurable $A$. Call $e(f,y)$ the conditional expectation of $f$ given $y$. We’d like $$e(f,y)=\int f(x)p(dx,y)$$ but $p(dx,y)$ isn’t a measure a.e. $y$. We can say that if $f$ is integrable on $Y$, then $fT$ is integrable on $X$ and $e(fT,y)=f(y)$ $T\mathsf{P}$ a.e. $y$. That is, the conditional expectation lives where it is supposed to.

Notice that Halmos starts with conditional probability by considering $f=1_A$ and then considers general $f$ to get conditional expectations. In contrast, other approaches start with general $f$ to get conditional expectations immediately and then specialize to $f=1_A$ for conditional probabilities.

3 Kolmogorov conditional probability

Kolmogorov’s axiomatic treatment of probability defines probability based three axioms: non-negativity, normalization, and countable additivity. Kolmogorov’s axioms apply to a wide range of probabilistic scenarios, including both discrete and continuous cases, allowing for generalization and application in various fields.

3.1 Conditional Expectation wrt Sets of Measure Zero

S5. Let $\mathcal{G}\subset \mathcal{F}$ be a sub-sigma algebra. As usual splitting $X=X^{+}-X^{-}$ lets us reduce to the case of non-negative integrable variables. The conditional expectation of $X$ wrt $\mathcal{G}$ is a non-negative (extended) random variable $\mathsf{E}[X\mid \mathcal{G}]$ such that

1. $\mathsf{E}[X\mid \mathcal{G}]$ is $\mathcal{G}$-measureable;
2. for every $A\in \mathcal{G}$

$${\int }_{A}X(\omega )\mathsf{P}(d\omega )={\int }_A\mathsf{E}[X\mid \mathcal{G}](\omega )\mathsf{P}(d\omega ).$$

We generally write the integral on the left simply as ${\int }_{A}Xd\mathsf{P}$.

5.R-W. Rogers and Williams p. 138 say

An experiment has been performed. The only information available to you regarding which sample point $\omega$ has been chosen is the set of values $Z(\omega )$ for every $\mathcal{G}$-measurable random variable $Z$, or, equivalently, the values $1_G(\omega )$ for every $G\in \mathcal{G}$. Then $\mathsf{E}[X\mid \mathcal{G}](\omega )$ is regarded as (a.s. equal to) the “expected value of $X(\omega )$ given this information.”

6. Existence. The set function on $\mathcal{G}$ $$\mathsf{Q}(A)={\int }_{A}Xd\mathsf{P}$$ is a measure on $(\mathrm{\Omega },\mathcal{G})$ and is absolutely continuous wrt $\mathsf{P}$ (if $\mathsf{P}(A)=0$ then $\mathsf{Q}(A)=0$). Therefore the Radon-Nikodým theorem gives a non-negative random variable $\mathsf{E}[X\mid \mathcal{G}]$ such that $$\mathsf{Q}(A)={\int }_A\mathsf{E}[X\mid \mathcal{G}]d\mathsf{P}.$$ It is unique up to sets of $\mathsf{P}$-measure zero. Thus there are many “variants” of the conditional expectation. Conditional expectations are Radon-Nikodým derivatives $$\mathsf{E}[X\mid \mathcal{G}]=\frac{d\mathsf{Q}}{d\mathsf{P}}.$$

3.2 Conditional Probability

S7. The conditional probability of an event $A\in \mathcal{F}$ with respect to $\mathcal{G}$ is $\mathsf{P}(A\mid \mathcal{G})=\mathsf{E}[1_A\mid \mathcal{G}]$. It inherits two properties: it is $\mathcal{G}$-measurable and $$\mathsf{P}(A\cap B)={\int }_B\mathsf{P}(A\mid \mathcal{G})d\mathsf{P}$$ for all $B\in \mathcal{G}$.

S8. If $\mathcal{G}=\sigma (Y)$ is generated by a random variable $Y$ then $\mathsf{E}[\cdot \mid \mathcal{G}]$ (resp. $\mathsf{P}(\cdot \mid \mathcal{G})$) is written $\mathsf{E}[\cdot \mid Y]$ ( $\mathsf{P}[\cdot \mid Y]$) and is called the conditional expectation (probability) wrt $Y$.

S9. These definitions extend those in the finite case.

S10. Conditional probabilities have all the “usual properties”: monotone, Jensen, linear, tower. In addition

1. If ${\mathcal{G}}_1\subset {\mathcal{G}}_2$ then $\mathsf{E}[\mathsf{E}[X\mid {\mathcal{G}}_2]\mid {\mathcal{G}}_1]=\mathsf{E}[X\mid {\mathcal{G}}_1]$ and $\mathsf{E}[\mathsf{E}[X\mid {\mathcal{G}}_1]\mid {\mathcal{G}}_2]=\mathsf{E}[X\mid {\mathcal{G}}_1]$.
2. If $X$ is independent of $\mathcal{G}$ (i.e., independent of $1_B$ for $B\in \mathcal{G}$) then $\mathsf{E}[X\mid \mathcal{G}]=\mathsf{E}[X]$ (a.s.).
3. If $X$ is $\mathcal{G}$-measurable then $\mathsf{E}[XY\mid \mathcal{G}]=X\mathsf{E}[Y\mid \mathcal{G}]$ (a.s.).
4. Bounded convergence, monotone convergence, Fatou, countably additive.

10.1. If $A\in \mathcal{G}$ then $\mathsf{P}(A\mid \mathcal{G})=1_A$: to know the outcome of the “experiment” $\mathcal{G}$ tells you which element of $\mathcal{G}$ occurs. Note that $1_A$ is $\mathcal{G}$-measurable and satisfies $\mathsf{P}(A\cap B)={\int }_B{1}_{A}p\mathsf{P}$ for all $B\in \mathcal{G}$ showing it satisfies the requirements to be a conditional probability.

10.2. If $\mathcal{G}=\{\mathrm{\varnothing },\mathrm{\Omega }\}$ then $\mathsf{P}(A\mid \mathcal{G})$ is a constant and so must equal $\mathsf{P}(A)$ ($\mathsf{P}(A)=\mathsf{P}(A\cap \mathrm{\Omega })={\int }_{\mathrm{\Omega }}{1}_{B}d\mathsf{P}=\mathsf{P}(B)$. Nothing is learnt from this $\mathcal{G}$-experiment.

10.3. If the set $A$ is independent of $\mathcal{G}$ then $\mathsf{P}(A\cap G)=\mathsf{P}(A)\mathsf{P}(G)$ for each $G\in \mathcal{G}$. Since then $\mathsf{P}(A\cap G)={\int }_G\mathsf{P}(A)d\mathsf{P}$ it follows that $\mathsf{P}(A\mid \mathcal{G})=P(A)$ with probability 1.

3.3 Conditional probability and conditional expectation comparison

The next table compares the Kolmogorov RNT approach to conditional probabilities and expectations showing the transition via $A\leftrightarrow 1_A$.

Step	Probability	Transition	Expectation
Given ingredients a sub-sigma algebra and set or rv	$\mathcal{G}\subset \mathcal{F}$ $A\in \mathcal{F}$	$\mathcal{G}\subset \mathcal{F}$ $1_A:\mathrm{\Omega }\to \{0,1\}$	$\mathcal{G}\subset \mathcal{F}$ $X:\mathrm{\Omega }\to \mathbb{R}$.
Consider the measure $\nu$ on $B\in \mathcal{G}$, $\nu \ll P$	$\nu (B)=\mathsf{P}(B\cap A)$	$\nu (B)={\int }_B{1}_{A}d\mathsf{P}$	$\nu (B)={\int }_{B}Xd\mathsf{P}=\mathsf{E}[X{1}_B]$.
RNT gives function $\xi$, where $\mathrm{\forall }B\in \mathcal{G}$ ${\int }_B\xi d\mathsf{P}$ equals	$\mathsf{P}(B\cap A)$	both	${\int }_{B}Xd\mathsf{P}$
and $\xi$ is called	$\mathsf{P}(A\mid \mathcal{G})$	$\mathsf{E}[1_A\mid \mathcal{G}]$	$\mathsf{E}(X\mid \mathcal{G})$.
Using $\xi$, $nu(B)=$	${\int }_B\mathsf{P}(A\mid \mathcal{G})(\omega )\mathsf{P}(d\omega )$	same	${\int }_B\mathsf{E}(X\mid \mathcal{G})(\omega )\mathsf{P}(d\omega )$.
Countable additivity is proved by converting back to $\mathsf{P}$ and appealing to RNT uniqueness. Other properites handled similarly.	For disjoint $A_i$, $A=\bigcup _i{A}_i$, ${\int }_B\mathsf{P}(\bigcup _i{A}_i\mid \mathcal{G})d\mathsf{P}$ $\phantom{X}=\mathsf{P}(B\cap \bigcup A_i)$ $\phantom{X}=\mathsf{P}(\bigcup _i(B\cap A_i)$ $\phantom{X}=\sum _i\mathsf{P}(B\cap A_i)$ $\phantom{X}=\mathsf{P}(B\cap A)$ $\phantom{X}={\int }_B\mathsf{P}(A\mid \mathcal{G})d\mathsf{P}$		For $X_i\ge 0$, $X=\sum _i{X}_i$, and $X$ integrable ${\int }_B\mathsf{E}(\sum X_i\mid \mathcal{G})d\mathsf{P}$ $\phantom{X}={\int }_B\sum _i{X}_{i}d\mathsf{P}$ $\phantom{X}=\sum _i{\int }_B{X}_{i}d\mathsf{P}$ $\phantom{X}=\sum _i{\int }_B\mathsf{E}[X_i\mid \mathcal{G}]d\mathsf{P}$ $\phantom{X}={\int }_B\sum _i\mathsf{E}[X_i\mid \mathcal{G}]d\mathsf{P}$

3.4 Expectation Conditional on Another Random Variable

S11. Since the function $\mathsf{E}[X\mid Y]$ is ${\mathcal{G}}_Y=\sigma (Y)$ measurable, it factors through $Y$ (Thm II.4.3) giving a Borel function $m$ so that

$$\mathsf{E}[X\mid Y](\omega )=m(Y(\omega ))$$

for all $\omega$. Denote $m(y)$ by $\mathsf{E}[X\mid Y=y]$ and call it the conditional expectation of $X$ wrt the event $\{Y=y\}$. Thus for $A\in {\mathcal{G}}_Y$

$${\int }_{A}Xd\mathsf{P}={\int }_A\mathsf{E}[X\mid Y]d\mathsf{P}={\int }_{A}m(Y)d\mathsf{P}$$

Note shorthand notation: ${\int }_{A}m(Y)d\mathsf{P}={\int }_{A}m(Y(\omega ))\mathsf{P}(d\omega )$.

S12. Let $P_Y(y)=\mathsf{P}(Y\le y)$ be the distribution function of $Y$. Then by the usual change of variable formula

$${\int }_{\{Y\in B\}}Xd\mathsf{P}={\int }_{B}m(y)P_Y(y)$$

for $B$ Borel in $\mathbb{R}$.

S13. The conditional probability of $A\in \mathcal{F}$ given $Y=y$ is $\mathsf{P}(A\mid Y=y)=\mathsf{E}[1_A\mid Y=y]$. This notation extends the discrete case and agrees with calculations performed with a bivariate density.

3.5 The problem of uncountably many measure zero sets

S14. If $B_1,B_2,\dots$ is a sequence of pairwise disjoint sets then

$$\mathsf{P}(\sum B_i\mid \mathcal{G})=\sum P(B_i\mid \mathcal{G})\text{(a.s.)}.$$

However, we cannot assume that $\mathsf{P}(\cdot \mid \mathcal{G})(\omega )$ is a measure on $\mathcal{G}$ because of the equation only holds a.s. and there are uncountably many sequences of $B_i$. (For each sequence countable additivity holds a.e., but we would like it to apply a.e. over all sequences. That is, we want to interpret $\mathsf{P}(B\mid \mathcal{G})(\omega )$ as a function $P(\omega ,B)$ on $\mathrm{\Omega }\times \mathcal{F}$ giving a measure for each $\omega$ and a function for each $B$. If that were so, then we could write

$$\mathsf{E}[X\mid \mathcal{G}](\cdot )={\int }_{\mathrm{\Omega }}X(\omega )P(\cdot ,d\omega ).$$

4 Kernels

$(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ is a probability space. $\mathcal{G}$ is a sub-sigma algebra of $\mathcal{F}$. A kernel combines two random elements

• $T:(\mathrm{\Omega },\mathcal{F})\to (E,\mathcal{E})$, and
• $X:(\mathrm{\Omega },\mathcal{F})\to (S,\mathcal{S})$.

$T$ defines what you observe and it gives a partition of $\mathrm{\Omega }$ into measurable subsets $\{T=t\}$. $X$ is what you care about or are trying to estimate.

A kernel or random measure from $E$ to $S$ is a function $p:E\times \mathcal{S}\to [0,1]$ satisfying

1. MF: For fixed $A\in \mathcal{S}$ the function $t\mapsto p(t,A)$ is $\mathcal{E}$ measurable.
2. MEAS: For each $A\in \mathcal{S}$, $p(\cdot ,A)$ is a (probability) measure. In applications it will be a version of conditional probability.

If the measure in 1. is a probability then the kernel is a probability kernel or Markov kernel.

A kernel determines an operator on suitable functions on $S$ via $$f\mapsto pf(t)=\int f(s)p(t,ds)=E[f\mid T=t].$$

5 Regular conditional probabilities

5.1 Regular Conditional Probability (rcp)

S15. A function $P(\omega ,B)$ on $\mathrm{\Omega }\times \mathcal{F}$ is a regular conditional probability wrt $\mathcal{G}$ if it is a probability kernel from ?? to ?? where for each $B\in \mathcal{F}$ the function $\omega \mapsto P(\omega ,B)$ is a variant of the conditional probability $\mathsf{P}(B\mid \mathcal{G})$, i.e., $P(\omega ,B)=\mathsf{P}(B\mid \mathcal{G})(\omega )$ a.s.

The next theorem shows why rcps are useful.

16. Theorem. Let $P$ be a regular conditional probability wrt $\mathcal{G}$, and let $X$ be an integrable random variable. Then

$$\mathsf{E}[X\mid \mathcal{G}](\omega )={\int }_{\mathrm{\Omega }}X(\nu )P(\omega ,d\nu )\text{(a.s.)}.$$

Proof. If $X=1_B$ for $B\in \mathcal{F}$, unraveling the definitions shows the claim becomes simply

$$\begin{array}{rl}\mathsf{E}[1_B\mid \mathcal{G}](\omega )& =\mathsf{P}(B\mid \mathcal{G})(\omega )\\ & ={\int }_{\mathrm{\Omega }}{1}_B(\nu )\mathsf{P}(d\nu \mid \mathcal{G})(\omega )\\ & ={\int }_{\mathrm{\Omega }}{1}_B(\nu )P(\omega ,d\nu )\end{array}$$

by part 2 of the rcp definition. The general result holds by approximating using simple functions and using the limit properties of conditional expectations.

5.2 Regular Conditional Distributions and Regular Distribution Functions

S17. A regular conditional probability is determined by a probability and a sub-sigma algebra. Often we want to adjust the probability to the distribution of a random variable (retaining the same sub-sigma algebra). The next two definitions handle these cases.

1. Let $(E,\mathcal{E})$ be a measurable space, $X$ a random variable on $\mathrm{\Omega }$ with values in $E$, and $\mathcal{G}\subset \mathcal{F}$ a sub-sigma algebra. A function $Q:\mathrm{\Omega }\times \mathcal{E}\to \mathbb{R}$ is a regular conditional distribution (rcd) of $X$ wrt $\mathcal{G}$ is a probability kernel from $E$ to $\mathcal{F}$ where for each $B\in \mathcal{E}$ the function $\omega \mapsto Q(\omega ,B)$ is a variant of the conditional probability $\omega \mapsto \mathsf{P}(X\in B\mid \mathcal{G})(\omega )$, i.e.,

$$Q(\omega ,B)=\mathsf{P}(X\in B\mid \mathcal{G})(\omega )\text{(a.s.)}.$$

2. Let $X$ be a random variable^[1]. A function $F:\mathrm{\Omega }\times \mathbb{R}\to \mathbb{R}$ is a regular distribution function (rcf) for $X$ with respect to $\mathcal{G}$ if
   1. $F(\omega ,x)$ is a distribution function on $\mathbb{R}$ for each $\omega$;
   2. $F(\omega ,x)=\mathsf{P}(X\le x\mid \mathcal{G})(\omega )$ a.s. for each $x\in \mathbb{R}$.

The distribution function only measures events $\{X\le x\}$ whereas the distribution measures any $B\in \mathcal{E}$.

S18. Theorem. Let $X$ be a random variable. Then $X$ has a regular distribution function and a regular conditional distribution function with respect to $\mathcal{G}$.

Proof. It is important that $X$ takes values in $\mathbb{R}$ which has a countable dense subset, the rationals. It is also important that a distribution function is increasing and continuous from the right. This means that it is specified by its values at rationals. The proof starts by constructing a regular distribution function. For each rational, it uses any variant of the conditional probability to define distribution functions depending on $\omega$. These functions fail to be monotone, have right limits, start at 0, or end at 1 on sets of measure zero. However, there are only countably many of these (they are index by rationals). Therefore, their union is also of measure zero and on its complement we have a distribution function. Off it, we just select any distribution function (to satisfy 2.i). Condition 2.ii follows by continuity and the fact that conditional probabilities respect limits. To extend to a regular conditional distribution integrate the rdf just constructed over $B$. Since all Borel sets can be constructed from $\{X\le x\}$, where the rdf gives the right answer, the rcd will too using a monotone class argument.

Here are a few more details to give the flavor. For each rational $r\in \mathbb{R}$, define

$$F_r(\omega )=\mathsf{P}(X\le r\mid \mathcal{G})(\omega )$$

where $\mathsf{P}(X\le r\mid \mathcal{G})=E[1_{X\le r}\mid \mathcal{G}]$ is any variant of the conditional probability wrt $\mathcal{G}$ of the event $\{X\le x\}$. Let $\{r_i\}$ be an enumeration of the rationals. If $r_i<r_j$ then

$$\mathsf{P}(X\le r_i\mid \mathcal{G})\le \mathsf{P}(X\le r_i\mid \mathcal{G})\text{(a.e.)}$$

and therefore if $A_{ij}=\{\omega \mid F_{r_j}(\omega )<F_{r_i}(\omega )\}$, $A=\bigcup A_{ij}$, we have $\mathsf{P}(A)=0$. Off this set we have a monotonic function. Play a similar game with

$$B_i=\{\omega \mid \underset{n\to \mathrm{\infty }}{lim}{F}_{r_i+1/n}(\omega )\ne F_{r_i}(\omega )\},B=\bigcup _i{B}_i$$

to ensure right limits and

$$C=\{\omega \mid \underset{n\to \mathrm{\infty }}{lim}{F}_n(\omega )\ne 0\}\bigcup \{\omega \mid \underset{n\to -\mathrm{\infty }}{lim}{F}_n(\omega )>0\}$$

to ensure appropriate left/right tail behavior. Then define

$$F(\omega ,x)=\{\begin{array}{ll}\underset{r\downarrow x}{lim}{F}_r(x)& \omega \notin A\cup B\cup C\\ G(x)& \omega \in A\cup B\cup C\end{array}$$

where $G$ is any distribution function. $F(\omega ,\cdot )$ is the distribution function required by 2.i. It takes the correct values, 2.ii by construction on the rationals and then because both $F$ and the underlying cp.

Using $F$, define a contender rcd as a Lebesgue-Stieltjes integral

$$Q(\omega ,B)={\int }_{B}F(\omega ,dx).$$

Standard nonsense shows this satisfies 1.i and ii.

Billingsley (2017) (p.460) calls this a conditional distribution of $X$ given $\mathcal{G}$. The proof is the same.

5.3 Extension to Borel Spaces

S19. A measure space $(E,\mathcal{E})$ is a Borel space if there is a bi-measurable injection $\varphi :(E,\mathcal{E})\to (\mathbb{R},\mathcal{B}(\mathbb{R}))$. (It is isomorphic to a Borel subset of a complete separable metric space=Polish space). Complete separable metric spaces, ${\mathbb{R}}^n$ and ${\mathbb{R}}^{\mathrm{\infty }}$ are all Borel spaces.

20. Theorem. Let $X$ be a random element with values in a Borel space $(E,\mathcal{E})$. Then there is a rcd of $X$ wrt $\mathcal{G}$.

Proof. Apply the previous result to the real-valued variable $\varphi \circ X$ and pull back.

Br.1. Breiman (1992b) explains that by moving the problem to $(E,\mathcal{E})$ you get rid of the piling up measure zero sets, and can guarantee the existence of a rcd. (It is called a distribution because $X\mathsf{P}=\mathsf{P}(X^{-1}(\cdot )))$ is the distribution of $X$. You need the random variable in addition to the original measure $\mathsf{P}$. See discussion in Kallenberg p. 47 where $X$ is called a random element (or variable if real-valued).)

5.4 Hoffmann-Jørgensen’s “Useful Rules”

Hoffman-Jørgensen (2017a), p.121 shows conditional expectations generally behave as you expect, provided CONDITION ON $M$.

Let $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ be a probability space and let

• $S,T$ be random elements on $\mathrm{\Omega }$ with values in $(L,\mathcal{A})$, and $(M,\mathcal{B})$ respectively.
• $\psi :(L\times M,\mathcal{A}\otimes \mathcal{B})\to \mathbb{R}$ so that $\psi (S,T)$ is $\mathsf{P}$-integrable.
• $\varphi :(L,\mathcal{A})\to \mathbb{R}$ is $\mathsf{P}$-measurable.
• $X$ be a random variable on $\mathrm{\Omega }$.

Let $P_S^T(\cdot \mid \cdot ):\mathcal{A}\times M\to [0,1]$ be an rcp for $S$ given $T$. Thus $P_S^T(A\mid t)$ is

• a probability measure in $A$ for all $t\in M$,
• is $\mathcal{B}$-measurable as a function of $t$ for all $A\in \mathcal{A}$ (probability kernel), and
• satisfies the rcp condition $$\begin{array}{}\text{(2)}& \mathsf{P}(S\in A,T\in B)={\int }_B{P}_S^T(A\mid t){\mathsf{P}}_T(dt).\end{array}$$ for $P_T$ a.e. $t\in M$.

TEST

a.${\mathsf{P}}_S={\mathsf{P}}_T{P}_S^T$ b.$\mathsf{E}[\psi (S,T)]={\displaystyle {\int }_M{\mathsf{P}}_T(dt){\int }_L\psi (s,t)P_S^T(ds\mid t)}$ c.$\mathsf{E}[X\mid T=t]={\displaystyle {\int }_L\mathsf{E}[X\mid S=s,T=t]P_S^T(ds}$

Then, the useful rules say that $$\begin{array}{}\text{(3)}& \begin{array}{rl}a.& {\mathsf{P}}_S={\mathsf{P}}_T{P}_S^T\\ b.& \mathsf{E}[\psi (S,T)]={\int }_M{\mathsf{P}}_T(dt){\int }_L\psi (s,t)P_S^T(ds\mid t)\\ c.& \mathsf{E}[X\mid T=t]={\int }_L\mathsf{E}[X\mid S=s,T=t]P_S^T(ds\mid t)\\ d.& \mathsf{E}[\psi (S,T)\mid T=t]=\mathsf{E}[\psi (S,t)\mid T=t]\\ e.& \mathsf{E}[\varphi (S)\mid T=t]=(P_S^T(\varphi ))(t)={\int }_L\varphi (s)P_S^T(ds\mid t)\\ f.& \mathsf{E}[\psi (S,T)\mid T=t]=\mathsf{E}[\psi (S,t)\mid T=t]\end{array}\end{array}$$

provided that we use the rcp to compute conditional expectations in the last formula. Let $\mathrm{\Gamma }(S,T)=\{(S(\omega ),T(\omega )\mid \omega \in \mathrm{\Omega }\}$ be the graph of $T$ against $S$. If $\mathrm{\Gamma }(S,T)\in \mathcal{A}\otimes \mathcal{B}$ then $$\begin{array}{rl}g.& P_S^T(S(T^{-1}(t))\mid T=t)=1P_T\text{ a.e. }t\in M.\end{array}$$

In particular, if $P^T(F\mid t)$ is a rcp of $\mathsf{P}$ given $T$, and if the graph $\mathrm{\Gamma }(T)=\{(\omega ,T\omega )\mid \omega \in \mathrm{\Omega }\}$ is $\mathcal{F}\otimes \mathcal{B}$ measurable then $$\begin{array}{rl}h.& P^T(T=t\mid t)=1\end{array}$$ for $P_T$ a.e. $t\in M$.

The measurable requirement is discussed further in XX and it cannot be dropped, see Section 7.2.2.

Proof.

1. This is just Equation 2 applied to $B=M$.
2. By Equation 2, this holds for $\psi =1_{A\times B}$ for $A\in \mathcal{A}$, $B\in \mathcal{B}$. Then use standard integration arguments.
3. Given $B\in \mathcal{B}$, set $\psi (s,t)=1_B(t)\mathsf{E}[X\mid S=s,T=t]$ in b. Then $$\begin{array}{rl}{\int }_{T^{-1}(B)}Xd\mathsf{P}& =\mathsf{E}[1_B(T)X]\\ & =\mathsf{E}[\mathsf{E}[1_B(T)X\mid S,T]]\\ & =\mathsf{E}[1_B(T)\mathsf{E}[X\mid S,T]]\\ & =\mathsf{E}[\psi (S,T)]\\ & ={\int }_B{\mathsf{P}}_T(dt){\int }_L\mathsf{E}[X\mid S=s,T=t]P_S^T(ds)\end{array}$$ so the result follows from the defining property of a conditional expectation.
4. By definition of rcp, $\psi (s,t)=\mathsf{E}[\psi (S,T)\mid S=s,T=t]$ so d follows from c.
5. Same from d.
6. Same.
7. Let $L(t)=S(T^{-1}(t))=\{S(\omega )\mid T(\omega )=t\}$ and $\psi =1_{\mathrm{\Gamma }}(S,T)$. Then $\psi (S(\omega ),T(\omega ))=1$ and $\psi (S(\omega ),t)=1_{L(t)}(S(\omega ))$, $\mathrm{\forall }\omega ,t\in M$. Hence g follows from e. Notice that you need the diagonal to be measurable in order that $\psi$ is $\mathcal{A}\otimes \mathcal{B}$-measurable.
8. Follows from g setting $S(\omega )=\omega$.

6 Disintegrations

Given measures $\lambda$ on $\mathrm{\Omega }$ and $\mu$ on $M$ we can form the product measure $\mathrm{\Lambda }$ on xx. We have $$\mathrm{\Lambda }f={\mu }^t{\lambda }_t^{x}f(x,t).$$ Disintegrations are the opposite: start with an measure on the product, decompose it into the marginal on one projection and a kernel on the other. The connection to rcps comes from embedding $\mathcal{X}$ into the product $\mathcal{X}\times M$ by $\iota :x\mapsto (x,Tx)$. The image $\iota \lambda$ of the measure $\lambda$ on $\mathcal{X}$ becomes a measure on the product space which we can then decompose. Again, under certain regularity conditions, everything is hunky dory.

6.1 Pachl Disintegrations

Pachl (1978) defines a disintegration as follows.

Setup.

Measure spaces $(X,\mathcal{A})$, $(Y,\mathcal{B})$. $\mathsf{R}$ a probability on $\sigma (\mathcal{A}\otimes \mathcal{B})$ and put $\mathsf{Q}={\pi }_Y(\mathsf{R})$ so $\mathsf{Q}(F)=\mathsf{R}(X\times F),\mathrm{\forall }F\in \mathcal{B}$.

Result.

Suppose $\mathrm{\forall }y\in Y$ there is a sigma-algebra ${\mathcal{A}}_y$ on $X$ and a probability $P_y$ on ${\mathcal{A}}_y$ such that

1. $\mathrm{\forall }E\in \mathcal{A},\mathrm{\exists }N\in \mathcal{B}:QN=0$, $E\in {\mathcal{A}}_y\mathrm{\forall }y\in Y\setminus N$ and $y\mapsto P_y(E)$ is $(\mathcal{B},Y\setminus N)$-measurable
2. $\mathrm{\forall }E\in \mathcal{A},\mathrm{\forall }F\in \mathcal{B}$ $${\int }_F{P}_y(E)Q(dy)=R(E\times F).$$

The family $\{({\mathcal{A}}_y,P_y{)}_{y\in Y}\}$ is called a $Q$-disintegration of $R$.

The definition does not assume ${\mathcal{A}}_y\supset \mathcal{A}$ so that complete Lebesgue probability is disintegrable, see Section 7.3.

6.2 Hoffmann-Jørgensen Disintegrations

6.2.1 RCP

Setup.

1. $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$
2. $(L,\mathcal{A})$
3. $(M,\mathcal{B})$
4. $S:(\mathrm{\Omega },\mathcal{F})\to (L,\mathcal{A})$
5. $T:(\mathrm{\Omega },\mathcal{F})\to (M,\mathcal{B})$

Result.

RCP $P_S^T(\cdot \mid \cdot ):\mathcal{A}\times M\to \mathbb{R}$ (kernel) s.t.

1. $P_S^T(A\mid \cdot )$ is $\mathcal{B}$-measurable $\mathrm{\forall }A\in \mathcal{A}$
2. $P_S^T(\cdot \mid t)$ is a measure on $\mathcal{A}$ for ?ae $t\in M$
3. $P_S^T$ is a version of conditional probability $$\mathsf{P}(S\in A,T\in B)={\int }_B{P}_S^T(A\mid t)P_T(dt).$$

6.2.2 Disintegration

Setup.

1. $(L,\mathcal{A},\mu )$
2. $(M,\mathcal{B},\nu )$
3. $\varphi :(L,\mathcal{A})\to (M,\mathcal{B})$
4. $\varphi \mu \ll \nu$

Result.

Disintegration $\xi :\mathcal{A}\times M\to \mathbb{R}$:

1. $\xi (A\mid \cdot )$ is $\mathcal{B}$-measurable $\mathrm{\forall }A\in \mathcal{A}$
2. $\xi (\cdot \mid t)$ is a measure on $\mathcal{A}$ for ?ae $t\in M$
3. $\xi$ is a version of conditional probability $$\mu (A\cap {\varphi }^{-1}(B))={\int }_B\xi (A\mid t)\nu (dt).$$

If the graph of $\varphi$ is $\mathcal{A}\otimes \mathcal{B}$-measurable then $\xi ({\varphi }^{-1}(t)\mid t)=1,\mathrm{\forall }t\in M$.

RCP	Disintegration
$S\mathsf{P}$	$\mu$
$T\mathsf{P}$	$\nu$
$\mathrm{\Gamma }$ graph of $S$ against $T$	Graph of function $\varphi$

$S\mathsf{P}(A)=\mathsf{P}(S\in A)$

6.2.3 Pollard Disintegrations

Pollard (2002) 5.2<4> defines an RCP for $S$ the identity, i.e., for $\mathsf{P}$ rather than $S\mathsf{P}$ and works with $T\mathsf{P}$ on $(M,\mathcal{B})$. He also requires $P^T(T\ne t\mid t)=0$ a.e. $t$. In a remark, he compares this definition to an rcp which has a requirement that $\mathsf{E}[h(T)X]=\mathsf{E}[h(t)\mathsf{E}[X\mid T]]$ and points out the differences is just the diagonal measurability requirement.

Given a kernel $P$ and distribution $\nu$ on $(M,\mathcal{B})$ (Pollard’s $\mathbb{Q}$) you get a measure $\nu \otimes P$ on the product $(\mathrm{\Omega }\times M,\mathcal{F}\otimes \mathcal{B})$ by (Pollard and standard notation) $$\begin{array}{rl}(\nu \otimes P)(f)& ={\nu }^t{P}_t^{x}g(\omega ,t)\\ & ={\nu }^t{P}_t^{\omega }g(\omega ,T\omega )\\ & =P^{\omega }g(\omega ,T\omega )\end{array}$$

The disintegration goes the other way. If you start with a measure $\mathrm{\Gamma }$ on $\mathcal{F}\otimes \mathcal{B}$ and a measure $\mu$ on $\mathcal{B}$ can you decompose $\mathrm{\Gamma }$ into a kernel $\mathrm{\Lambda }$ so that $\mathrm{\Gamma }=\mu \otimes \mathrm{\Lambda }$?

7 Examples

?Others from Chang and Pollard?

7.1 Duplication Map

Let $\mathrm{\Omega }=[0,1]$ and $T(x)=\{2x\}$ where $\{x\}=x-\lfloor x\rfloor$ is the fractional part of $x$.

7.2 Coco Examples / Diagonal not measurable

7.2.1 Billingsley

Following Billingsley (2017), Section 33.

B.1. There are pathological examples showing that the interpretation of conditional probability in terms of an observer with partial information breaks down in certain cases.

Let $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ be the unit interval with Lebesgue measure on the $\sigma$-field of Borel subsets of $\mathrm{\Omega }$. Take $\mathcal{G}$ to be the $\sigma$-field of sets that are either countable or cocountable. Then the function identically equal to $\mathsf{P}(A)$ is a version of $\mathsf{P}(A\mid \mathcal{G})$. The function is $\mathcal{G}$-measurable and integrable, and has the “right” integral over $G\in \mathcal{G}$ because $\mathsf{P}(G)$ is either 0 or 1 for every G in $\mathcal{G}$ . Therefore, $$\begin{array}{}\text{(4)}& \mathsf{P}(A\mid \mathcal{G}{)}_{\omega }=\mathsf{P}(A)\end{array}$$ with probability 1. But since $\mathcal{G}$ contains all one-point sets, to know which elements of $\mathcal{G}$ contain $\omega$ is to know $\omega$ itself. Thus $\mathcal{G}$ viewed as an experiment should be completely informative—the observer given the information in $\mathcal{G}$ should know $\omega$ exactly—and so one might expect that $$\begin{array}{}\text{(5)}& \mathsf{P}(A\mid \mathcal{G}{)}_{\omega }=\{\begin{array}{ll}1& \text{if }\omega \in A,\\ 0& \text{if }\omega \notin A.\end{array}\end{array}$$

The mathematical definition gives Equation 4; the heuristic considerations lead to Equation 5. Of course, Equation 4 is right and Equation 5 is wrong. The heuristic view breaks down in certain cases but is nonetheless illuminating and cannot, since it does not intervene in proofs, lead to any difficulties.

B.2. Switch perspective from fixing $A$, and getting a function $\omega \mapsto \mathsf{P}(A\mid G)(\omega )$, to fixing $\omega$ and looking for a measure. Suppose that $A_0,\dots ,A_r\in \mathcal{F}$ partition $\mathrm{\Omega }$ and set $\mathcal{G}=\sigma (A_1,\dots ,A_r)$. If $\mathsf{P}(A_0)=0$ but other $\mathsf{P}(A_i)>0$, then one version of $\mathsf{P}(B\mid \mathcal{G})is$$ (B = $$\{\begin{array}{ll}17& \omega \in B_0\\ \mathsf{P}(B\cap A_i)& \omega \in B_i,i=1,\dots ,r\end{array}$$

. $$

Now, for each $B$, $B\mapsto \mathsf{P}(B\mid \mathcal{G})(\omega )$ is a measure on $\mathcal{F}$ for $\omega \in B_1\cup \cdots \cup B_r$ but not if $\omega \in B_0$. This is the “wrong version”. If $17$ is replaced by $\mathsf{P}(B)$ then we get a probability measure in $B$.

B.3. If ${\omega }_0$ is an atom of positive probability (any set containing ${\omega }_0$ has positive measure). Fix any versions of $\mathsf{P}(B\mid \mathcal{G})$. For each $B$ the set $\{\omega \mid \mathsf{P}(B\mid \mathcal{G})(\omega )<0\}\in \mathcal{G}$, since the conditional probability is $\mathcal{G}$-measurable. Since, for a fixed $B$, we get a conditional probability, this set must have measure 0. Therefore it cannot contain ${\omega }_0$ and thus $\mathsf{P}(A\mid \mathcal{G}{)}_{{\omega }_0}\ge 0$. Similarly $\mathsf{P}(\mathrm{\Omega }\mid \mathcal{G}{)}_{{\omega }_0}=1$ and $\mathsf{P}(\bigcup _i{B}_i\mid \mathcal{G}{)}_{{\omega }_0}=\sum \mathsf{P}(B_i\mid \mathcal{G}{)}_{{\omega }_0}$ over disjoint unions. Therefore $\mathsf{P}(A\mid \mathcal{G}{)}_{{\omega }_0}$ is a probability as $A$ ranges over $\mathcal{F}$.

Thus conditional probabilities behave like probabilities at points of positive probability. That they may not do so at points of probability $0$ causes no problem because individual such points have not effect on the probabilities of sets.

7.2.2 Hoffmann-Jørgensen

Let $\mathrm{\Omega }$ be diffuse (uncountable, all points measurable with measure zero). $S$ is the identity, but $T$ is the identity to $(\mathrm{\Omega },\mathcal{B})$ where $\mathcal{B}$ is the sigma algebra generated by null sets (countable sets?!), $\mathcal{B}\subset \mathcal{F}$.

Claim. $P^T(F\mid t)=\mathsf{P}(F)$ is a rcp for $\mathsf{P}$ given $T$. (Pf. It is clearly a measure in $F$ for all $t$, and is a measurable function in $t$ for all $F$ because it is constant. Finally, consider $\mathsf{P}(A\cap B)$. If $\mathsf{P}(B)=0$ then $\mathsf{P}(A\cap B)=0$ because $A\cap B\subset B$. If $\mathsf{P}(B)=1$ then $\mathsf{P}(A\cap B)=\mathsf{P}(A)$ by complements. In both cases $\mathsf{P}(A\cap B)=\mathsf{P}(A)\mathsf{P}(B)={\int }_{B}P(A){\mathsf{P}}_T(dt)$.)

Since $\{T=t\}=\{t\}$, $P^T(T=t\mid t)=\mathsf{P}(\{t\})=0$ for all $t\in \mathrm{\Omega }$. If $\mathrm{\Gamma }=\{(\omega ,\omega )\}$ is the diagonal in $\mathrm{\Omega }\times \mathrm{\Omega }$ and $\psi =1_{\mathrm{\Gamma }}$, then $\psi (S,T)=1$ is the identity and $\psi (S,t)=1_{\{t\}}$. Hence $$\mathsf{E}[\psi (S,T)\mid T=t]=1,\mathsf{E}[\psi (S,t)\mid T=t]=0\mathrm{\forall }t.$$ In this case $\mathrm{\Gamma }\notin \mathcal{F}\otimes \mathcal{B}$ and $\psi$ is not $\mathcal{F}\otimes \mathcal{B}$-measurable. This is the same as Billingsley’s coco example, #sec-billcoco.

7.3 Lebesgue measure example

Example (Pachl (1978)). Assume the continuum hypothesis. Put $X=[0,1{]}^2$, $Y=[0,1]$. $\mathsf{P}$, $\mathsf{Q}$ are ordinary Lebesgue measures on the sigma-algebras $\mathcal{A}$, $\mathcal{B}$ of Lebesgue measurable subsets of $X$ and $Y$. Define the probability $\mathsf{R}$ on $\sigma (\mathcal{A}\otimes \mathcal{B})$ (NOT complete!) by $$\mathsf{R}(G)=\mathsf{P}\{(x_1,x_2)\in X\mid ((x_1,x_2),x_2)\in G\}$$ for $G\in \sigma (\mathcal{A}\otimes \mathcal{B})$. There is no $Q$-distintegration with ${\mathcal{A}}_y\supset \mathcal{A}$ for all $y\in Y$. If $\{({\mathcal{A}}_y,P_y{)}_{y\in Y}\}$ is a disintegration (so $P_y$ is a measure on $X=[0,1{]}^2$) then $P_y$ extends one-dimensional Lebesgue measure in $[0,1]\times \{y\}\subset X$ to ${\mathcal{A}}_y$. But $\mathcal{A}$ contains all subsets of $[0,1]\times \{y\}$ (it is a null set). Thus if ${\mathcal{A}}_y\supset \mathcal{A}$ we obtain an extension of Lebesgue probability in $[0,1]$ to the sigma-algebra of all subsets of $[0,1]$, which is inconsistent with the CH per Ulam’s theorem. (This is the “Strange Example” from Hoffmann-Jørgensen (1994) vol II p.150-51.)

Note. This is an example of why Simon says the LM sets are evil.

7.4 Doob’s example

7.4.1 Stoyanov

Following Stoyanov (2013).

Let $(\mathrm{\Omega },\mathcal{F},\mathsf{P})$ be a probability space and ${\mathcal{F}}_1$ a σ-field such that ${\mathcal{F}}_1\subset \mathcal{F}$. Recall that the conditional probability $\mathsf{P}(A\mid {\mathcal{F}}_1)$ is defined $\mathsf{P}$-a.s. as an ${\mathcal{F}}_1$-measurable function of $\omega$ such that $$\mathsf{P}(AB)={\int }_B\mathsf{P}(A\mid {\mathcal{F}}_1)d\mathsf{P}(\omega )\mathrm{\forall }B\in {\mathcal{F}}_1.$$ The conditional probability $\mathsf{P}(A\mid {\mathcal{F}}_1)$, $A\in \mathcal{F}$ is said to be regular if there exists a function $P(A,\omega )$, $A\in \mathcal{F},\omega \in \mathrm{\Omega }$ (notation $P$ not $\mathsf{P}$), which satisfies the following two properties:

1. $P(A,\cdot )=\mathsf{P}(A\mid {\mathcal{F}}_1)(\cdot )$ $\mathsf{P}$-a.s. for an arbitrary $A\in \mathcal{F}$;
2. for each $\omega$, $P(·,\omega )$ is a probability measure on $\mathcal{F}$.

If condition (ii) is satisfied and condition (i) holds for all $\omega$ (not only for $\mathsf{P}$-almost all $\omega$), then $\mathsf{P}(A\mid {\mathcal{F}}_1)$ is called a proper regular conditional probability. (In terms of distributions we speak about regular and proper regular conditional distributions.) Regular conditional probabilities exist in many cases, but proper regular conditional probabilities do not always exist, as can be seen below.

Let $(\mathrm{\Omega },\mathcal{F},\lambda )$ be a probability space with $\omega =[0,1]$, $\mathcal{F}$ the $\sigma$-field of the Lebesgue measurable sets in $[0,1]$ and $\lambda$ the Lebesgue measure. It is well known that in the interval $[0,1]$ there is a non-measurable (in Lebesgue sense) set, say $N$, such that its outer measure is ${\lambda }^{\ast }(N)=1$ and its inner measure is ${\lambda }_{\ast }(N)=0$. For details see Halmos (1974) and my note on the Halmos set $N$ is “half of $\mathrm{\Omega }$” in the sense that $\mathrm{\Omega }=N\cup (N+1)$, so it “should have” measure $1/2$.

Define a new $\sigma$-field $\tilde{\mathcal{F}}$ which is generated by $\mathcal{F}$ and the set $N$. Thus $\tilde{\mathcal{F}}$ consists of sets of the form $N{B}_1\cup N^c{B}_2$ where $B_1,B_2\in \mathcal{F}$. Define also the measure $\mathsf{P}$ on the measurable space $([0,1],\tilde{\mathcal{F}},\mathsf{P})$ by $$\mathsf{P}(N{B}_1\cup N^c{B}_2)=\frac{1}{2}(\lambda (B_1)+\lambda (B_2)).$$

It is easy to check that $\mathsf{P}$ is well defined and defines a probability on $\tilde{\mathcal{F}}$, so the triplet $([0,1],\tilde{\mathcal{F}},\mathsf{P})$ is a probability space. For every $B\in \mathcal{F}$ we have $$\mathsf{P}(NB\cup N^{c}B)=\mathsf{P}(B)=\lambda (B)$$ and hence $\mathsf{P}$ coincides with $\lambda$ on $\mathcal{F}$, that is $\mathsf{P}{|}_{\mathcal{F}}=\lambda$. Moreover, $$\mathsf{P}(N)=\frac{1}{2}.$$ Now we shall prove the following statement: on the probability space $([0,1],\tilde{\mathcal{F}},\mathsf{P})$ there is no regular conditional probability $P(A,\mathcal{F})$, $A\in \tilde{\mathcal{F}}$ with respect to the σ-field $\mathcal{F}$.

Suppose such a probability exists: that is, there is a function, say $P(A,\omega )$, which satisfies the above conditions (i) and (ii). If so, then for any Borel (and Lebesgue) set $A$, $P(A,\omega )=1_A(\omega )$. Therefore if $A$ is a one-point set, $A=\{\omega \}$, then $P(\{\omega \},\omega )=1$. Now take the set $N$. From the definition of a conditional probability and the equality $\mathsf{P}(N)=1/2$ we get $$\begin{array}{}\text{(6)}& \frac{1}{2}=\mathsf{P}(N)={\int }_{\mathrm{\Omega }}P(N,\omega )\lambda (d\omega ).\end{array}$$ On the other hand, if $P(·,\omega )$ is a measure for each $\omega$, then $$P(N,\omega )\ge P(\{\omega \},\omega )\mathrm{\forall }\omega \in N⟹P(N,\omega )=1\mathrm{\forall }\omega \in N.$$

Consider the set $C=\{\omega :P(\{\omega \},\omega )=1\}$. Since $P(·,\omega )$ is a Borel function in $\omega$, then the set $C$ is Borel measurable with $\mathsf{P}(C)=1$. Let $D=\{\omega :P(N,\omega )=1\}$. It is clear that $D$ is Borel-measurable and $D\supset CN$, which implies that $D\cup C^c\supset N$. However, the set $D\cup C^c$ is Borel and covers the (non-measurable!) set $N$ which has ${\lambda }^{\ast }(N)=1$. Therefore $\mathsf{P}(D\cup C^c)=1$ and $\mathsf{P}(D)=1$. In other words, for almost all $\omega$ we get $P(N,\omega )=1$, which implies the following equality $${\int }_{\mathrm{\Omega }}P(N,\omega )\lambda (d\omega )=1.$$ However, this contradicts Equation 6.

Therefore a regular conditional probability need not always exist.

Let us note that in this counterexample the role of the non-measurable set $N$ is essential. Recall that the construction of $N$ relies on the axiom of choice. Using a weakened form of the axiom of choice, Solovay (1970) derived several interesting results concerning the measurability of sets, measures and their properties.

General results on the existence of regular conditional probabilities can be found in the works of J. Pfanzagl (1969), Blackwell and Dubins (1975) and Faden (1985).

7.4.2 Rogers and Williams

Here is the version from Rogers and Williams (1994), page 141. Notation shift: $N$ becomes $Z$ and $\mathcal{F}$ becomes $\mathcal{G}$ the Borel sigma-algebra. $\tilde{\mathcal{F}}$ becomes $\mathcal{F}$. $\lambda$ becomes $\mu$, Lebesgue measure.

The setup is the same modulo pointing out that outer measure 1 is used to confirm the extended measure $\mathsf{P}$ as defined is a measure.

Again, suppose a rcp $P$ exists for $\mathcal{G}\subset \mathcal{F}$ as a function $\mathcal{F}\times \mathrm{\Omega }\to [0,1]$. First think about what this means. Knowing the outcome in $\mathcal{G}$ actually tells you which point occurred because $\{\omega \}\in \mathcal{G}\mathrm{\forall }\omega \in \mathrm{\Omega }$. The extra information from $\mathcal{F}$ is whether or not $Z$ occurred. By definition $Z$ is “half of $\mathrm{\Omega }$”, so the probability $P(\mathrm{\Gamma }\cap Z,\omega )$ should be $\frac{1}{2}{1}_{\mathrm{\Gamma }}(\omega )$ for $\mathrm{\Gamma }\in \mathcal{G}$ and the set $J=\{\omega \mid P(\mathrm{\Gamma }\cap Z,\omega )=\frac{1}{2}{1}_{\mathrm{\Gamma }}(\omega )\mathrm{\forall }\mathrm{\Gamma }\in \mathcal{G}\}$ should have $\mathsf{P}$ and $\mu$ measure 1. But then if $\omega \in J$ $$P(Z\cap J,\omega )=\frac{1}{2}{1}_J(\omega )=1\ne \frac{1}{2}{1}_{J\setminus \{\omega \}}(\omega )=P(Z\cap (J\setminus \{\omega \}),\omega )$$ so that $Z\cap J\ne Z\cap (J\setminus \{\omega \})$, i.e., $\omega \in Z$. But then $J$, a set of inner measure zero, contains a $\mathcal{G}$-measurable set of measure 1, which is a contradiction.

The detailed argument uses the fact that $\mathcal{G}$ is generated by a countable $\pi$-system $\mathcal{I}$ to reduce $J$ to $\{\omega \mid P(\mathrm{\Gamma }\cap Z,\omega )=\frac{1}{2}{1}_{\mathrm{\Gamma }}(\omega )\mathrm{\forall }\mathrm{\Gamma }\in \mathcal{I}\})$ which shows $J$ is in $\mathcal{G}$ and therefore that its $\mathsf{P}$ and $\mu$ measures are the same.

Notice that $Z$ is independent of $\mathcal{G}$ since $P(Z\cap \mathrm{\Gamma })=\frac{1}{2}\mu (\mathrm{\Gamma })=P(Z)P(\mathrm{\Gamma })$ for all $\mathrm{\Gamma }\in \mathcal{G}$.

This example is consistent with the general statement on the existence of rcps which assumes $\mathcal{G}$ is a sub-sigma algebras of the Borel sigma algebra on $[0,1]$.

7.5 Borel’s paradox

Borel’s Paradox is a famous example in the field of probability theory that illustrates the counterintuitive nature of conditional probabilities in certain contexts. The paradox arises when dealing with conditional probabilities on continuous probability spaces.

Consider a uniform distribution on the surface of a sphere. If we condition on a point on the sphere having a particular latitude, the resulting conditional distribution of longitudes is uniform. However, if we instead condition on the point having a specific longitude, the conditional distribution of latitudes is not uniform. This leads to a paradoxical situation where the method of conditioning affects the outcome, despite the symmetry of the problem.

The paradox highlights the importance of carefully defining conditional probabilities in continuous settings and demonstrates that intuitive ideas from discrete probability do not always directly translate to the continuous case.

It is sometimes described in terms of a meteorite strike. Given that is hits on the equator the distribution is uniform. But given that it hits along a line of longitude through London it is not.

7.6 Other awkward sub sigma algebras

• $\mathcal{F}$ in its completion
• Tail sigma algebras (JS)

8 Background theory

8.1 A very short introduction to measure theory

8.1.1 Basic definitions

Generally need a sigma-finite assumption!

A ring $R$ of sets is a nonempty subset of the power set of a set $X$ (or alternatively a class of sets) that is closed under union and difference: $\mathrm{\forall }E,F\in R$: $E\cup F\in R$ and $E\setminus F\in R$. The empty set $E\setminus E\in R$ as are $E\mathrm{△}F=(E\setminus F)\cup (F\setminus E)$ and $E\cap F=(E\cup F)\setminus (E\mathrm{△}F)$.

An algebra is a ring that contains $X$. Hence it is closed under complements $E^{\prime }=X\setminus E$.

A sigma ring/algebra is closed under countable unions.

Given any class of sets $\mathcal{C}$ there is a unique smallest ring (algebra, sigma-ring, sigma-algebra) containing $\mathcal{C}$. Each element in the sigma-ring generated by $\mathcal{C}$ is in a sigma-ring generated by countably many elements of $\mathcal{C}$. (Because $\{E\subset \mathrm{\Omega }\mid E\in \sigma ({\mathcal{C}}^{\prime }),{\mathcal{C}}^{\prime }\subset \sigma (\mathcal{C}),\text{ countable}\}$ is a sigma algebra contained in $\sigma (\mathcal{C})$!)

A measure is a non-negative, countably additive set function taking value $0$ on the empty set. Measures are monotone. The length of an interval is a set function on the class of (semiclosed) intervals.

A class $\mathcal{H}$ is hereditary if $F\subset E\in \mathcal{H}⟹F\in \mathcal{H}$. An outer measure is a countably subadditive set function.

Given a measure $\mu$ on a ring $R$ $${\mu }^{\ast }(E)=inf\{\sum _{n\ge 1}\mu (E_n)\mid E_n\in R,E\subset \bigcup _n{E}_n\}$$ defines an outer measure on the hereditary set defined by $R$.

A set $E$ in a hereditary sigma-ring $\mathcal{H}$ is ${\mu }^{\ast }$-measurable if $$\mathrm{\forall }A\in \mathcal{H},{\mu }^{\ast }(A)={\mu }^{\ast }(A\cap E)+{\mu }^{\ast }(A\cap E^{\prime }).$$

The class $S$ of ${\mu }^{\ast }$-measurable sets is a sigma-ring and ${\mu }^{\ast }$ restricted to $S$ is a (complete) measure.

$\mu$ is a measure on a ring $R$ with induced measure ${\mu }^{\ast }$ on $\mathcal{H}(R)$ and $\overline{\mu }$ on $\overline{S}$ of measurable sets.

1. Every set in $S(R)$ is ${\mu }^{\ast }$-measurable
2. ${\mu }^{\ast }(E)=inf\{\sum _{n\ge 1}\overline{\mu }(E_n)\mid E_n\in \overline{S},E\subset \bigcup _n{E}_n\}$
3. ${\mu }^{\ast }(E)=inf\{\sum _{n\ge 1}\overline{\mu }(E_n)\mid E_n\in S(R),E\subset \bigcup _n{E}_n\}$
4. The outer measure is identical to the completion of the extension of $\mu$ to $S(R)$ on $\overline{S}$

Starting with $S$ the sigma-ring generated by (semiclosed) intervals Borel sets and $\mu$ length, then $\overline{S}$ are Lebesgue measurable sets and $\overline{\mu }$ is Lebesgue measure.

8.1.2 Atoms

Hoffman-Jørgensen (2017b) Ch 6.2-3.

Let $\mathcal{G}$ be a sigma-algebra on $\mathrm{\Omega }$. Define the equivalence relation $${\omega }^{\prime }\equiv \omega \mathrm{mod}\mathcal{G}⟺1_G(\omega )=1_G({\omega }^{\prime })\mathrm{\forall }G\in \mathcal{G}.$$ Equivalence classes are called atoms and the atom containing ${\omega }_0$ is $$\mathcal{G}({\omega }_0)=\{\omega \mid \omega \equiv {\omega }_0\mathrm{mod}\mathcal{G}\}.$$ The atoms are a disjoint partition of $\mathcal{G}$. If $\mathcal{G}(\omega )=\{\omega \}$ then $\mathcal{G}$ separates points. Thus, $\mathcal{G}$ gives complete information about outcomes.

There follows a long list of things you expect to be true that are true.

8.1.3 Product Measures

The product sigma algebra is generated by products of measurable sets. Sections of a measurable set are measurable.

The class of finite, disjoint unions of rectangles from two rings is itself a ring.

The product sigma-algebra is the smallest sigma-algebra that makes projections measurable.

A rectangle is the intersection of two inverse images of projections. The area of the rectangle ${\pi }_1(E_1)\times {\pi }_2(E_2)={\mathsf{P}}_1(E_1){\mathsf{P}}_2(E_2)$.

Let $X=Y$ be an uncountable set and $\mathcal{F}=\mathcal{G}$ the class of countable subsets (=sigma-algebra generated by the one-point sets). If $D=\{(x,y)\mid x=y\}$ is the diagonal in $X\times Y$ then every section of $D$ is measurable but $D$ is not. (Because: )

Given measure spaces $(X,\mathcal{F},\mathsf{P})$ and $(Y,\mathcal{G},\mathsf{Q})$ the product measure $\lambda =\mathsf{P}\times \mathsf{Q}$ is defined by $$\lambda (E)=\int \mathsf{Q}(E_x)\mathsf{P}(dx)=\int \mathsf{P}(E_y)\mathsf{Q}(dy).$$

The product of two complete measure spaces need not be complete. (Take $A\subset [0,1]$ not Lebesgue measurable. $x\times A\subset [0,1{]}^2$ is not in the product measure space: it doesn’t have measurable sections.) This leads Simon (2015a) p. 210 to say that

passing from Borel to Lebesgue measurable functions is the work of the devil. Don’t even consider it!

8.1.4 The Diagonal

From Hoffman-Jørgensen (2017b), exercises to Chapter 1 and …

Let $(\mathrm{\Omega },\mathcal{F})$ be a measure space and $\mathrm{\Delta }=\{(\omega ,\omega )\mid \omega \in \mathrm{\Omega }\}$ be the diagonal in $\mathrm{\Omega }\times \mathrm{\Omega }$.

$\mathcal{F}$ is countably separating if there exists a countable paving $\mathcal{G}\subseteq \mathcal{F}$ such that $\mathrm{\forall }\omega \ne {\omega }^{\prime },\mathrm{\exists }G\in \mathcal{G}:1_G(\omega )\ne 1_{G^{\prime }}({\omega }^{\prime })$

Here are the important facts about the diagonal and countably separable spaces.

1. Let $f:\mathrm{\Lambda }\to \mathrm{\Omega }$ be an injective function. If $\mathcal{F}$ is countably separable then so is $f^{-1}(\mathcal{F})$.
   • Obvious
2. Let $(S,d)$ be a separable metric space. Then $\mathcal{B}(S)$ is countably separable.
   • Let $\{a_n\}$ be a countable dense subset. Then $\mathcal{G}=\{b(a_i,2^{-j}){\}}_{i,j}$ countably separates. (For $\mathbb{R}$ you can use half intervals at the rationals.)
3. (Marczewski function) The $\{A_n\}$ be a sequence of subsets of $\mathrm{\Omega }$. Define a function $f:\mathrm{\Omega }\to \mathbb{R}$ by $$f(\omega )=\sum _n{10}^{-n}{1}_{A_n}(\omega ).$$ Then $f^{-1}(\mathcal{B}(\mathbb{R}))=\sigma (A_n)$ and $f$ is injective iff $\{A_n\}$ separates points.
4. $\mathcal{F}$ is countably separable iff there exists an injective measurable function $\mathrm{\Omega }\to \mathbb{R}$.
   • (only if, $⟹$): use 3, Marczewski function.
   • (if $⟸$): follows from 1 and 2 (applied to $\mathbb{R}$).
5. $\mathcal{F}$ is countably separating iff $\mathrm{\Delta }\in \mathcal{F}\otimes \mathcal{F}$.
   • (only if, $⟹$): if $\mathcal{F}$ is countably separable then there is an injective measurable function to $\mathbb{R}$. Set $g:\mathrm{\Omega }\times \mathrm{\Omega }\to {\mathbb{R}}^2$, $g({\omega }_1,{\omega }_2)=(f({\omega }_1),f({\omega }_2))$. Then $\mathrm{\Delta }=g^{-1}({\mathrm{\Delta }}_{\mathbb{R}})$, and the diagonal in $\mathbb{R}$ is measurable by 2.
   • (if $⟸$): if $\mathrm{\Delta }\in \mathcal{F}\otimes \mathcal{F}$ then $\mathrm{\Delta }$ is in a sigma-algebra generated by countable set $\mathcal{G}\subset \mathcal{F}$ (see potted summary). Suppose the elements of $\mathcal{G}$ do not separate points, so there are $\omega \ne {\omega }^{\prime }$ where both or neither of which is in each set in $\mathcal{G}$. Since the same holds for unions, complements and intersections of sets in $\mathcal{G}$, it holds for all sets in $\sigma (G)$. The section ${\mathrm{\Delta }}_{\omega }=\{w\mid (\omega ,w)\in \mathrm{\Delta }\}$ is $\mathcal{G}$-measurable and contains $\omega$, and therefore ${\omega }^{\prime }\in {\mathrm{\Delta }}_{\omega }$ meaning $(\omega ,{\omega }^{\prime })\in \mathrm{\Delta }$, which is a contradiction. Therefore $\mathcal{G}$ separates points.
6. Suppose $\mathrm{\Omega }$ has cardinality strictly greater than $\mathbb{R}$. Then the diagonal is not in the product sigma algebra.
   • There is no injective function to $\mathbb{R}$.
7. Let $(L,\mathcal{A})$ and $(M,\mathcal{B})$ be measure spaces, $\nu$ a measure on $(M,\mathcal{B})$ and $f:L\to M$ be a function so that $\mathcal{A}=f^{-1}(\mathcal{B})$ and $f(L)\in \mathcal{B}$. Then
   1. $\mathrm{\forall }A\in \mathcal{A},f(A)\in \mathcal{B}$
   • Let $A=f^{-1}(B)$, then $f(A)=B\cap f(L)$.
   2. $\mathrm{\exists }\mu$ on $(L,\mathcal{A})$ so that $\nu =f\mu$ iff $\nu (M\setminus f(L))=0$.
   • $\mu (A)=\nu (f(A))$ has the required properties.
8. Let $f:(L,\mathcal{A})\to (M,\mathcal{B})$ is a measurable function and suppose $(M,\mathcal{B})$ is countably separable (iff has a measurable diagonal). Then, the graph of $f$ $$\mathrm{\Gamma }=\{(x,f(x))\mid x\in L\}\in \mathcal{A}\otimes \mathcal{B}.$$
   • By assumption, there exists a measurable injection $\varphi :M\to \mathbb{R}$. Define $\mathrm{\Phi }(a,b)=(\varphi (f(a)),\varphi (b))$. Then $\mathrm{\Phi }$ is measurable (projections are measurable) and $\mathrm{\Gamma }={\mathrm{\Phi }}^{-1}(D)$, $D$ the diagonal in ${\mathbb{R}}^2$.
   • Thus measurable graph or diagonal only depends on the range space being countably separable.
9. Let $S$ be a separable metric space, and let $(L,\mathcal{A})$ be a measure space. Suppose that $f:S\times T\to \mathbb{R}$ is a function with $f(\cdot ,t)$ continuous for all $t$ and $f(s,\cdot )$ $\mathcal{A}$-measurable for all $s$. Then $f$ is $\mathcal{B}(S)\otimes \mathcal{A}$-measurable.
   • Let $\{a_n\}$ be a countable dense sequence in $S$ and set $$\begin{array}{c}{D}_1^n=b(a_1,2^{-n}),D_j^n=b(a_j,2^{-n})\setminus \bigcup _{1\le i\le j-1}b(a_i,2^{-n})\\ f_n(s,t)=\sum _{j}f(a_j,t)1_{D_j^n}(s).\end{array}$$ The $D_i^n$ are disjoint (by construction) for fixed $n$ and $S=\bigcup _j{D}_j^n$ ($n$ fixed critical) and $f(s,t)=\underset{n}{lim}{f}_n(s,t)$. The $f_n$ are $\mathcal{B}(S)\otimes \mathcal{A}$-measurable and the pointwise limit of mesurable functions is measurable.

The sigma-algebra of countable and co-countable sets on an uncountable $\mathrm{\Omega }$ is not countably generated. If it were, then it is generated by a countable collection of countable sets, i.e., by a countable number of points in total. But it is uncountable.

Therefore, if $\mathrm{\Omega }$ is uncountable and $\mathcal{F}$ is the countable/co-countable sigma algebra, then the diagonal is not measurable in the product. (There can be no countably separable generators.) However, it has measurable sections.

Interestingly, a sigma-algebra cannot be countably infinite. Suppose $\mathcal{F}$ is countably infinite. Since $\mathcal{F}\subset \mathcal{P}(\mathrm{\Omega })$, (power set) $\mathrm{\Omega }$ must be infinite. Define $A_x=\bigcap \{A\in \mathcal{F}\mid x\in A\}\in \mathcal{F}$ (because it can only be a countable intersection). It is the smallest set in $\mathcal{F}$ containing $\mathcal{F}$. The sets $A_x$ are disjoint: if $A_x\cap A_y\ne \mathrm{\varnothing }$ then $A_x\setminus A_y$ is a smaller set containing $x$. But then there must be an infinite number of distinct $A_x$ and hence $\mathcal{F}$ is uncountable.

8.2 Radon-Nikodým theorem

From Halmos measure theory, Theorem 31.N.

Theorem 1 (Radon-Nikódym Theorem) If $(\mathrm{\Omega },\mathcal{F},\mu )$ is a totally $\sigma$-finite measure space and if a $\sigma$-finite measure $\nu$ on $\mathcal{F}$ is absolutely continuous wrt $\mu$, then there iexists a finite valued mesaureable function $f$ on $\mathrm{\Omega }$ such that $$\nu (A)={\int }_{A}fd\mu$$ for every measurable $A$. The function $f$ is unique a.e. $(\mu )$.

Proof. Reduce to finite $\mu ,\nu$. Uniqueness follows from $${\int }_A(f-g)d\mu =0\mathrm{\forall }A⟹\mu \{f>g\}=0,\mu \{f<g\}=0.$$ Let $\mathcal{K}$ be the class of all non-negative $\mu$-integrable function $f$ such that $${\int }_{A}fd\mu \le \nu (A)$$ and write $$\alpha =sup\{{\int }_{\mathrm{\Omega }}fd\mu \le \nu (A)\mid f\in \mathcal{K}\}.$$ Let $f_n$ be a sequence in $\mathcal{K}$ such that $$\underset{n}{lim}{\int }_{\mathrm{\Omega }}fd\mu =\alpha$$ and let $g_n=max\{f_1,\dots ,f_n\}$. Any $A$ can be written as a disjoint union $A=A_1\cup \cdots \cup A_n$ so that $g_n(x)=f_j(x)$ on $A_j$. Consequently $${\int }_A{g}_{n}d\mu =\sum _{j=1}^n{\int }_{A_j}{f}_{j}d\mu \le \sum _{j=1}^n\nu (A_j)=\nu (A).$$ If we write $f_0(x)=\underset{n}{sup}{f}_n(x)$, then $f_0(x)=\underset{n}{lim}{g}_n(x)$ and (monotone convergence) $f_0\in \mathcal{K}$ and ${\int }_{\mathrm{\Omega }}{f}_{0}d\mu =\alpha$. Since $f_0$ is integrable we can find a finite valued function $f=f_0$ a.e. $(\mu )$. Then $f$ is our guy! We show that if $${\nu }_0(A)=\nu (A)-{\int }_{A}fd\mu$$ then ${\nu }_0$ is identically zero. It is positive (definition of $\mathcal{K}$) and so if not zero we can find a set $B$ of positive $\mu$-measure so that $ϵ\mu (A\cap B)\le {\nu }_0(A\cap B)$. But then $g=f+ϵ1_B>f$ is in $\mathcal{K}$, contradicting the maximality of $\int fd\mu$.

8.3 Reisz Representation Theorem

The Riesz Representation Theorem for measure spaces provides a powerful result that connects linear functionals on spaces of continuous functions to measures. There are two main forms of this theorem, one for compact Hausdorff spaces and one for locally compact Hausdorff spaces. Here, we will focus on the form relevant to compact Hausdorff spaces, which is often the more straightforward case.

Theorem 2 (Riesz Representation Theorem for Compact Hausdorff Spaces) Let $X$ be a compact Hausdorff space, and let $C(X)$ denote the space of continuous real-valued functions on $X$. For every continuous linear functional $L$ on $C(X)$, there exists a unique regular Borel measure $\mu$ on $X$ such that for all $f\in C(X)$: $$L(f)={\int }_{X}fd\mu$$

The measure $\mu$ is a regular Borel measure, which means it is defined on the Borel $\sigma$-algebra of $X$, the measure of any Borel set can be approximated by the measure of compact subsets from within, and open supersets.

Sketch proof. (See also Rudin (1987), theorem 2.14, over seven pages!)

For each non-negative continuous function $f$, define a set function ${\mu }_f$ by: $${\mu }_f(A)=inf\{L(g):f\le g\text{ and }g\in C(X)\text{ is non-negative, }g\ge 1_{A}f\}$$ This set function ${\mu }_f$ should satisfy properties of a measure.

Show that ${\mu }_f$ is countably additive. For disjoint sets $A_i$, $${\mu }_f\left(\bigcup _i{A}_i\right)=\sum _i{\mu }_f(A_i)$$

Regularity of the Measure: Define an outer measure ${\mu }^{\ast }$ using ${\mu }_f$. $${\mu }^{\ast }(A)=sup\{\sum _i{\mu }_f(A_i):A\subseteq \bigcup _i{A}_i\}$$ Use Carathéodory’s Extension Theorem to extend ${\mu }^{\ast }$ to a measure $\mu$ on the Borel $\sigma$-algebra of $X$. Prove that $\mu$ is a regular Borel measure. This involves showing that $\mu$ can be approximated from above by open sets and from below by compact sets.

Uniqueness of the Measure: Suppose there are two measures ${\mu }_1$ and ${\mu }_2$ that represent $L$. For any continuous function $f$, $${\int }_{X}fd{\mu }_1=L(f)={\int }_{X}fd{\mu }_2$$ By the properties of measures, ${\mu }_1={\mu }_2$ must hold.

8.4 Borel and Polish Spaces

From Rogers and Williams (1994).

A topological space $S$ is a Polish space if the topology on $S$ arises from a metric wrt which $S$ is complete. $S$ is a Lusin space if $S$ homeomorphic to a Borel subset of a compact metric space $J$ (which can be taken to be $[0,1{]}^{\mathrm{\infty }}$)

All probabilities on a Lusin space are inner regular (equals $sup$ of measure of contained compact subsets).

$S$ is Polish iff it is homeomorphic to a $G_{\delta }$ subset of a compact metric space. Every Polish space is a Lusin space.

8.5 Sufficient Statistics

A sufficient statistic is a function of the data that provides as much information about a parameter of interest as the entire data set does. More formally, for a given parameter $\theta$ and data set $X$, a statistic $T(X)$ is said to be sufficient for $\theta$ if the conditional distribution of the data $X$ given $T(X)$ does not depend on $\theta$.

Given a random sample $X=(X_1,X_2,\dots ,X_n)$ from a probability distribution with parameter $\theta$, a statistic $T(X)$ is sufficient for $\theta$ if the conditional distribution of the sample given the statistic is independent of $\theta$. Formally, $T(X)$ is sufficient for $\theta$ if: $$P(X\mid T(X),\theta )=P(X\mid T(X))$$

This can also be understood via the Factorization Theorem. A statistic $T(X)$ is sufficient for the parameter $\theta$ if and only if the joint probability (or probability density) function of $X$ can be factored as: $$f(X\mid \theta )=g(T(X)\mid \theta )h(X)$$ where $g$ is a function that depends on the data $X$ only through the statistic $T(X)$, and $h$ is a function that does not depend on $\theta$.

Example: Binomial Distribution

Consider a random sample $X_1,X_2,\dots ,X_n$ from a binomial distribution with parameters $n$ (number of trials) and $p$ (probability of success). The probability mass function is: $$f(x_i\mid p)=(\genfrac{}{}{0ex}{}{n}{x_i})p^{x_i}(1-p{)}^{n-x_i}$$

For a sample of size $k$, the joint probability function is: \[ f(x_1, x_2, \ldots, x_k \mid p) = \prod_{i=1}^{k} \binom{n}{x_i} p^{x_i} (1 - p)^{n - \]i} ]

This can be rewritten as: \[ f(x_1, x_2, \ldots, x_k \mid p) = \left[ p^{\sum_{i=1}^{k} x_i} (1 - p)^{kn - \s \]{i=1}^{k} x_i} ] _{i=1}^{k} ]

Here, $\sum _{i=1}^k{x}_i$ is a sufficient statistic for $p$.

Example: Poisson Distribution

A Poisson distribution with parameter $\lambda$ (rate parameter) has the probability mass function: $$P(X=x\mid \lambda )=\frac{{\lambda }^x{e}^{-\lambda }}{x!}$$ for $ x = 0, 1, 2, $.

Consider a random sample $X_1,X_2,\dots ,X_n$ from a Poisson distribution with parameter $\lambda$. The joint probability mass function of the sample is: $$P(X_1=x_1,X_2=x_2,\dots ,X_n=x_n\mid \lambda )=\prod _{i=1}^n\frac{{\lambda }^{x_i}{e}^{-\lambda }}{x_i!}$$

We can rewrite the joint probability mass function as $$P(X_1=x_1,X_2=x_2,\dots ,X_n=x_n\mid \lambda )=\left(\prod _{i=1}^n\frac{1}{x_i!}\right){\lambda }^{\sum _{i=1}^n{x}_i}{e}^{-n\lambda }$$

In this factorized form, we have

• $g(T(X)\mid \lambda )={\lambda }^{\sum _{i=1}^n{x}_i}{e}^{-n\lambda }$
• $h(X)=\prod _{i=1}^n\frac{1}{x_i!}$

Here, $T(X)=\sum _{i=1}^n{X}_i$ is the statistic that appears in $g(T(X)\mid \lambda )$.

According to the factorization theorem, $T(X)=\sum _{i=1}^n{X}_i$ is a sufficient statistic for the parameter $\lambda$. This is because the joint probability mass function can be written in the form: $$f(X|\lambda )=g(T(X)|\lambda )h(X)$$ where: $$g(T(X)|\lambda )={\lambda }^{\sum _{i=1}^n{x}_i}{e}^{-n\lambda }$$ and $$h(X)=\prod _{i=1}^n\frac{1}{x_i!}.$$

9 Book treatments

Table 2: Probability and measure theory text books and some relevant papers.

Domain	Treatments
Probability
Ash (2014)
Billingsley (2017)
Bobrowski2005bk
Borovkov (2003), Borovkov (2003)
Breiman (1992b)
Chung (2001)
Durrett (2013)
Dellacherie and Meyer (1978)
Doob (1953), Doob (1984)
Dudley (2018)
Fristedt and Gray (2013)
Gray (2009)
Grimmett and Stirzaker (2020a), Grimmett and Stirzaker (2020b)
Gut (2006)
Hoffman-Jørgensen (2017b), Hoffman-Jørgensen (2017a)
Itô (1984)
Kallenberg (2001b)
Klenke (2014)
Loève
Malliavin (2012)
Moran (1968)
Meyer
Parthasarathy (1967)
Pollard (2002)
Rogers and Williams (1994)
Shiryaev (1996)
Stoyanov (2013)
Stromberg (1994)
Stroock (2010)
Tucker (2013)
Williams (1991b)
Wise and Hall (1993)
Analysis
Folland (1999)
König (2009)
Doob (2012)
Tao (2011)
Stein and Shakarchi (2009)
Simon (2015b)
Kelley and Srinivasan (2012)
Halmos (1974)
Papers
Hoffmann-Jørgensen (1971)
P. Pfanzagl (1979), J. Pfanzagl (1969)
Pachl (1978)
Faden (1985)
Chang and Pollard (1997)

Table 3: Summary of notation across some common textbooks.

Domain, author	Spaces	Sigma Alg	Subsets	RVs	Probabilities
Probability
Hoffmann-J	$x\in \mathrm{\Omega },L,t\in M$	$\mathcal{F},\mathcal{A},\mathcal{B}$	$A,B$	$S,T$	$\mathsf{P},\mu ,P_S^T(\cdot \mid \cdot )$
Kallenberg	$\mathrm{\Omega }$		$\psi$
Billingsley	$\mathrm{\Omega }$		$\psi$
Shiryaev	$\mathrm{\Omega }$		$\psi$
Pollard	$\mathrm{\Omega }$ probability $\mathcal{X}$ metric(izable)				$\lambda ,\mu$
Analysis
Halmos	$X,Y$		$E,F$	$f$	n/a?
Statistics
Generic	$\mathrm{\Omega }$	?	$X$	$X,Y,T$

Literature comparison. This was an intermediate step in my understanding. May be duplicative or not necessary in final version.

Item	Conditional Probability	Regular Conditional Probability	(Regular) Conditional Probability Distritbution	Distingegration
Abbreviation	cp	rcp	cpd	dis
Ingredients	$A\in \mathcal{F}$, $\mathcal{G}\subset \mathcal{F}$	Same as cp	Same as cp, plus $T:(\mathrm{\Omega },\mathcal{F})\to (E,\mathcal{E})$	$T$, $Q=T\mathsf{P}$
Results per Shiryaev (Sh)	A $\mathcal{G}$-measurable function $\mathsf{P}(A\mid \mathcal{G}):\mathrm{\Omega }\to [0,1]$ a.e. so that $\mathrm{\forall }B\in \mathcal{G}$, $\mathsf{P}(B\cap A)={\int }_B\mathsf{P}(A\mid \mathcal{G})d\mathsf{P}$	$p:\mathrm{\Omega }\times \mathcal{F}\to [0,1]$ s.t. (i) $\mathrm{\forall }A\in \mathcal{F}$, $p(\cdot ,A)$ is version of $\mathsf{P}(A\mid \mathcal{G})(\cdot )$, and (ii) $\mathrm{\forall }\omega$, $p(\omega ,\cdot )$ is a probability measure on $\mathcal{F}$	$p:\mathrm{\Omega }\times \mathcal{E}\to [0,1]$ s.t. (i) $\mathrm{\forall }A\in \mathcal{E}$, $p(\cdot ,A)$ is version of $\mathsf{P}(T\in A\mid \mathcal{G})(\cdot )$, and (ii) $\mathrm{\forall }\omega$, $p(\omega ,\cdot )$ is a probability measure on $\mathcal{E}$
Existence	Always—Radon-Nikódym theorem
Books
Billingsley, Sec 33		⁓	As Sh for $X$ a rv
Brieman 4.2,4.3		As Sh.; shows $\mathsf{E}[X\mid \mathcal{G}](\omega )=\int X(\nu )p(\omega ,d\nu )$, i.e., conditional expectation is taken wrt rcp (Sh note 16 above)	As Sh, notation $\mathcal{D}\subset \mathcal{F}$; $(E,\mathcal{E})$ a Borel space; same as Sh works via rdf
Dudley	Sec 10.1	In (ii) can replace every with a.e. $\omega$: if it holds a.e. then pick any prob. meas. on the set of measure zero where it fails; this does not affect (i).	A cpd with $T$ the identify on $E=\mathrm{\Omega }$ is a rcp
Halmos	Sec 48	⁓
Hoffmann-Jørgensen
Kalleberg Ch 6. Thm 6.3, 6.4		CHECK!	In (i) requires measurable in def of kernel. Adds random element $X$ with values in $(S,\mathcal{S})$ rcd is a random measure, aka kernel from $E$ to $S$, $p(t,B)=\mathsf{P}(S\in B\mid T):E\times \mathcal{S}\to [0,1]$; $S$ Borel for existence
Meyer-D v. 1
Rogers & Williams (II.42,89)	$\mathsf{P}$ measurable, integrable	As Sh	In Sec 42 (ii) only require $p(\omega ,\cdot )$ to be a measure a.e. $\omega$. Existence: $\mathrm{\Omega }$ is a Lusin space and proves $p$ is a measure $\mathrm{\forall }\omega$; in addition, if $\mathcal{G}$ is countably generatged then (iii) $\{\omega \mid p(\omega ,A)=1_A(\omega ),\mathrm{\forall }A\in \mathcal{G}\}$ is a $\mathcal{G}$ set of $\mathsf{P}$ measure 1; (iv) if $A(\omega )$ denotes the atom of $\mathcal{G}$ containing $\omega$ then $\{\omega \mid p(\omega ,A(\omega ))=1\}$ is in $\mathcal{G}$ and has $\mathsf{P}$ measure 1
Shiryaev (II.7)		$\mathrm{\Omega }$ is a Borel space (isomorphic to a Borel subset of a complete, separable metric space with the Borel sigma-algebra CHECK)	$(E,\mathcal{E})$ a Borel space; If $T$ is a random variable, $F:\mathrm{\Omega }\times \mathbb{R}\to [0,1]$ is an rdf if (i) $\mathrm{\forall }x$, $F(\cdot ,F)=\mathsf{P}(T\le x\mid \mathcal{G})(\omega )$ a.s. and (ii) $\mathrm{\forall }\omega$, $F(\omega ,\cdot )$ is a distribution function on $\mathbb{R}$ Prepends regular

10 Abbreviations

Common abbreviations.

Abbreviation	Meaning	Characteristics
ce	conditional expectations	integral of $X$ and $\mathsf{E}[X\mid \mathcal{G}]$ over $\mathcal{G}$-sets agree, unique a.e.; produces a different function for each $X$ from RNT
cp	conditional probability	any version of $\mathsf{E}[1_B\mid \mathcal{G}]$, $B\in \mathcal{F}$; produces different function for each $B\in \mathcal{F}$ from RNT
rcp	regular conditional probability	a single function on $\mathrm{\Omega }\times \mathcal{F}$ that is a measure $\mathrm{\forall }\omega$ and a.e. equal to the cp for each $B$
(r)cd	regular conditional distribution	synonyms and specializations
rdf	regular distribution function
decomposition

11 References and Footnotes

The banner image suggests the decomposition of a sample space $\mathrm{\Omega }$ by values of a random variable $X$ into different level sets $\{X=x\}$.

References

Ash, R. B. (2014). Real analysis and probability: Probability and mathematical statistics: A series of monographs and textbooks. Academic press.

Billingsley, P. (2017). Probability and measure. John Wiley & Sons.

Blackwell, D. (1956). On a class of probability distributions. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 3, 1–7. https://doi.org/10.1002/mma.1670090116

Blackwell, D., & Dubins, L. E. (1975). On Existence and Non-Existence of Proper, Regular, Conditional Distributions. The Annals of Probability, 3(5), 741–752. https://doi.org/10.1214/aop/1176996261

Blackwell, D., & Ryll-Nardzewski, C. (1963). Non-existence of everywhere proper conditional distributions. Annals of Mathematical Statistics, 34(1), 223–225.

Borovkov, A. A. (2003). Probability Theory. (Fourth.). Springer. https://doi.org/10.2307/2316129

Breiman, L. (1992b). Probability. SIAM.

Breiman, L. (1992a). Probability, volume 7 of classics in applied mathematics.

Chang, J. T., & Pollard, D. (1997). Conditioning as disintegration. Statistica Neerlandica, 51(3), 287–317. https://doi.org/10.1111/1467-9574.00056

Chung, K. L. (2001). A course in probability theory. Academic press.

Dellacherie, C., & Meyer, P.-A. (1978). Probabilities and potential. Hermann.

Doob, J. L. (1953). Stochastic processes. John Wiley; Sons, Inc.

Doob, J. L. (1984). Classical potential theory and its probabilistic counterpart (Vol. 262). Springer.

Doob, J. L. (2012). Measure theory (Vol. 143). Springer Science & Business Media.

Dudley, R. M. (2018). Real analysis and probability. Chapman; Hall/CRC.

Durrett, R. (2013). Probability: Theory and Examples. Cambridge University Press.

Faden, A. M. (1985). The Existence of Regular Conditional Probabilities: Necessary and Sufficient Conditions. The Annals of Probability, 13(1), 288–298. http://projecteuclid.org/euclid.aop/1176996548

Folland, G. B. (1999). Real analysis: Modern techniques and their applications (Vol. 40). John Wiley & Sons.

Fristedt, B. E., & Gray, L. F. (2013). A modern approach to probability theory. Springer Science & Business Media.

Gray, R. M. (2009). Probability, random processes, and ergodic properties. Springer Science & Business Media.

Grimmett, G., & Stirzaker, D. (2020a). Probability and random processes. Oxford university press.

Grimmett, G., & Stirzaker, D. (2020b). One thousand exercises in probability. Oxford University Press, USA.

Gut, A. (2006). Probability: A graduate course (Vol. 200). Springer.

Halmos, P. R. (1974). Measure theory (Vol. 18). Springer.

Hoffman-Jørgensen, J. (2017b). Probability with a view towards statistics, volume i. Routledge.

Hoffman-Jørgensen, J. (2017a). Probability with a view towards statistics, volume II. Routledge.

Hoffmann-Jørgensen, J. (1971). Existence of Conditional Probabilities. https://doi.org/10.7146/math.scand.a-11023

Hoffmann-Jørgensen, J. (1994). Probability with a view towards statistics (Vol. 1). Chapman; Hall. https://doi.org/10.1201/9780203742013

Itô, K. (1984). An introduction to probability theory. Cambridge University Press.

Kallenberg, O. (2001b). Foundations of Modern Probability (Second.). Springer Verlag.

Kallenberg, O. (2001a). Foundations of Modern Probability (Second.). Springer.

Kelley, J. L., & Srinivasan, T. (2012). Measure and integral: Volume 1 (Vol. 116). Springer Science & Business Media.

Klenke, A. (2014). Probability Theory (p. 638). https://doi.org/10.1007/978-1-4471-5361-0

Kolmogorov, A. N., & Bharucha-Reid, A. T. (2013). Foundations of the theory of probability: Second english edition. Marino Publishing.

König, H. (2009). Measure and integration: An advanced course in basic procedures and applications. Springer Science & Business Media.

Malliavin, P. (2012). Integration and probability (Vol. 157). Springer Graduate Texts in Mathematics.

Moran, P. A. P. (1968). An Introduction to Probability Theory. Oxford: Clarendon Press.

Pachl, J. K. (1978). Disintegration and compact measures. https://doi.org/10.7146/math.scand.a-11771

Parthasarathy, K. R. (1967). Probability Measures on Metric Spaces. Academic Press.

Pfanzagl, J. (1969). On the existence of regular conditional probabilities. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 11(3), 244–256. https://doi.org/10.1007/BF00536383

Pfanzagl, P. (1979). Conditional Distributions as Derivatives. The Annals of Probability, 7(6), 347–370. https://doi.org/10.1214/aop/1176994897

Pollard, D. (2002). A user’s guide to measure theoretic probability. Cambridge University Press.

Rogers, L. C. G., & Williams, D. (1994). Diffusions, Markov Processes, and Martingales, Volume 1: Foundations (Second.). John Wiley; Sons.

Rudin, W. (1987). Real and complex analysis.

Shiryaev, A. N. (1996). Probability (Second.). New York: Springer-Verlag.

Simon, B. (2015a). Real Analysis: A Comprehensive Course in Analysis, Part 1. American Mathematical Society. https://doi.org/10.2307/3618959

Simon, B. (2015b). Real analysis. American Mathematical Soc.

Solovay, R. M. (1970). A Model of Set-Theory in Which Every Set of Reals is Lebesgue Measurable. The Annals of Mathematics, 92(1), 1. https://doi.org/10.2307/1970696

Stein, E. M., & Shakarchi, R. (2009). Real analysis: Measure theory, integration, and hilbert spaces. Princeton University Press.

Stoyanov, J. M. (2013). Counterexamples in Probability (Third.). Dover.

Stromberg, K. R. (1994). Probability for Analysts. New York: Chapman; Hall. https://doi.org/10.2307/3618362

Stroock, D. W. (2010). Probability theory: An analytic view. Cambridge university press.

Tao, T. (2011). An introduction to measure theory (Vol. 126). American Mathematical Soc.

Tucker, H. G. (2013). A graduate course in probability. Courier Corporation.

Williams, D. (1991b). Probability with Martingales. Cambridge University Press. https://doi.org/10.1017/CBO9780511813658

Williams, D. (1991a). Probability with martingales. Cambridge university press.

Wise, G. L., & Hall, E. B. (1993). Counterexamples in probability and real analysis. Oxford University Press.

Footnotes

1. Pollard (2002), Remark in Ch 5.2.

2. Normal, usual, not in the technical sense of regular measure

3. Hoffman-Jørgensen (2017a) Ch 10.1.7

4. Pollard (2002) Appendix F.2<5>.

5. A Radon measure is a measure on the Borel sets where every compact set has finite measure.

6. Paragraphs numbered Sn follow Shiryaev (1996), II.7 and I.8; Hn paragraphs follow Halmos (1974), and Bn Billingsley2017bk.