1.3 Families of Distributions

Above, we introduced the basic concepts of random variables and probability distributions. Now, let’s explore some common distributions.

1.3.1 Discrete probability distributions

Bernoulli distribution

Definition 1.1 A Bernoulli trial is a random experiment with exactly two possible outcomes: “success” and “failure.” These outcomes are typically denoted as 1 for success and 0 for failure. The probability of success is often denoted by \(p\), and the probability of failure is then given by \(q = 1 - p\).

The Bernoulli distribution is a discrete probability distribution that describes the probability of getting exactly successes in a Bernoulli trials with the probability of success \(p\in(0,1)\).

Denote \(X \sim \text{Ber}(p)\) if a random variable \(X\) follows Bernoulli distribution. The probability mass function of the Bernoulli distribution is given by: \[ \begin{split} p_X(x) &= \begin{cases} p, & x = 1 \\ 1-p, & x = 0 \\ 0, & \mbox{otherwise} \end{cases} \\ &= p^x(1-p)^{1-x}, \quad x=0,1 \end{split} \] and the range \(R_X=\{0,1\}\).

Proposition 1.1 \(\mathbb{E}(X)=p, \quad \mathrm{Var}(X)=p(1-p)\)

Binomial distribution

The Binomial distribution describes the probability of getting exactly \(k\) successes in \(n\) Bernoulli trials with the probability of success \(p\in(0,1)\).

Denote \(X \sim \text{Bin}(n,p)\) if a random variable \(X\) follows Binomial distribution. The probability mass function is given by: \[ p_X(x) = {n \choose x} p^x(1-p)^{n-x}, \quad x=0,1,\cdots,n. \] and the range \(R_X=\{0,1,\cdots,n\}\).

Proposition 1.2

1. \(\mathbb{E}(X)=np, \quad \mathrm{Var}(X)=np(1-p)\)
2. \(\text{Ber}(p) \sim \text{Bin}(1,p)\)
3. (additive property) Let \(X \sim \text{Bin}(n_1,p), Y \sim Bin(n_2,p)\) and \(X \perp\!\!\!\perp Y\). Then \(X+Y \sim \text{Bin}(n_1+n_2,p)\)

Poisson distribution

Before we introduce the Poisson distribution, we need to know what is Poisson process.

Definition:

A Poisson process is a stochastic process that models a sequence of events occurring randomly over time or space. It is characterized by the following properties:

1. The probability that exactly 1 event occurs in a given interval of length \(h\) is equal to \(\lambda h + o(h)\).
2. The probability that 2 or more events occur in an interval of length \(h\) is equal to \(o(h)\).
3. For any integers \(n\), \(j_1, j_2, \cdots, j_n\) and any set of \(n\) non-overlapping intervals, if we define \(E_i\) to be the event that exactly \(j_i\) of the events under consideration occur in the \(i\)-th of these intervals, then events \(E_1,E_2,\cdots,E_n\) are independent.

Little \(o\) notation: \(o(h)\) stands for any function \(f(h)\) for which \(\displaystyle \lim_{h \to 0} \frac{f(h)}{h} = 0\).

\[ p_X(k) = \frac{\lambda^{k} e^{-\lambda}}{k!}, \quad k \in \mathbb{N}, \lambda \in (0,\infty) \]

Zero-inflated Poisson distribution, Zero-truncated Poisson distribution, Conway–Maxwell–Poisson distribution, Skellam distribution

Geometric distribution

\(p_X(k) = (1-p)^{k-1}p, \quad 0 < p \leq 1\)

Negative binomial distribution

\(f(k;r,p)\equiv p_X(k) = {\binom{k+r-1}{k}} (1-p)^k p^r\)

Hypergeometric distribution

\(p_{X}(k) = \frac{{\binom{K}{k}}{\binom{N-K}{n-k}}}{\binom{N}{n}}\)

Zeta distribution

\(p_{X}(k) = \frac{1}{\zeta(s)}k^{-s}\)

where \(\zeta(s)\) is the Riemann zeta function and \(s>1\).

1.3.2 Continuous probability distributions

Uniform distribution (discrete and continuous)

\[ f(k) = \begin{cases} \frac{1}{b-a+1}, & \text{for } a \leq k \leq b, \quad k \in \mathbb{N} \\ 0, & \text{otherwise} \end{cases} \] where \(a\) and \(b\) are integers.

\[ f(x) = \begin{cases} \frac{1}{b-a}, & \text{for } a \leq x \leq b \\ 0, & \text{otherwise} \end{cases} \] where \(a\) and \(b\) are real numbers.

(log) Normal distribution

\[ f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) \]

where \(\mu\) is the mean and \(\sigma^2\) is the variance.

\[ f(x) = \frac{1}{x\sigma\sqrt{2\pi}} \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right) \]

(shifted/double) Exponential distribution

\[ f(x) = \begin{cases} \lambda e^{-\lambda x} & \text{for } x \geq 0 \\ 0 & \text{otherwise} \end{cases} \]

where \(\lambda\) is the rate parameter.

\[ f(x) = \begin{cases} \lambda e^{-\lambda (x-\mu)} & \text{for } x \geq \mu \\ 0 & \text{otherwise} \end{cases} \]

where \(\lambda\) is the rate parameter and \(\mu\) is the shift parameter.

Double Exponential Distribution (Laplace Distribution)

\[ f(x) = \frac{\lambda}{2} e^{(-\lambda |x-\mu|)} \]

where \(\lambda\) is the rate parameter and \(\mu\) is the location parameter.

Gamma distribution (Erlang distribution)

\[ f(x) = \frac{\lambda^k}{\Gamma(k)} x^{k-1} e^{-\lambda x} \] where \(k\) is the shape parameter and \(\lambda\) is the rate parameter.

Erlang Distribution

\[ f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} e^{-\lambda x} \]

where \(k\) is an integer (number of events) and \(\lambda\) is the rate parameter.

Beta distribution

\[ f(x) = \frac{1}{B(\alpha, \beta)} x^{\alpha-1} (1-x)^{\beta-1} \]

where \(\alpha\) and \(\beta\) are shape parameters and \(B(\alpha,\beta)\) is the beta function.

Weibull distribution (Rayleigh distribution)

\[ f(x) = \frac{k}{\lambda} \left(\frac{x}{\lambda}\right)^{k-1} \exp\left(-\left(\frac{x}{\lambda}\right)^k\right) \]

where \(k\) is the shape parameter and \(\lambda\) is the scale parameter.

Rayleigh Distribution

\[ f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right) \]

where \(\sigma\) is the scale parameter.

Pareto distribution

\[ f(x) = \begin{cases} \frac{\alpha x_m^\alpha}{x^{\alpha+1}} & \text{, for } x \geq x_m \\ 0 & \text{, otherwise} \end{cases} \]

where \(\alpha\) is the shape parameter and \(x_m\) is the scale parameter.

Logistic distribution

\[ f(x) = \frac{e^{-(x-\mu)/s}}{s\left(1+e^{-(x-\mu)/s}\right)^2} \]

where \(\mu\) is the location parameter and \(s\) is the scale parameter.

Cauchy distribution

\[ f(x) = \frac{1}{\pi \gamma} \left[ \frac{1}{1 + \left( \frac{x - x_0}{\gamma} \right)^2} \right] = \frac{1}{\pi} \left[ \frac{\gamma}{(x - x_0)^2 + \gamma^2} \right] \] where \(x_0\) is the location parameter (also the median and mode), \(\gamma\) is the scale parameter.

1.3.3 Distributions derive from Normal

Students’ \(t\)-distribution

\[ f(t) = \frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\sqrt{\nu\pi} \, \Gamma\left(\frac{\nu}{2}\right)} \left(1 + \frac{t^2}{\nu}\right)^{-\frac{\nu+1}{2}} \]

where \(t\) is the test statistic, \(\nu\) is the degrees of freedom (df.).

(Doubly) non-central \(t\) distribution ? The PDF of the non-central t-distribution is more complex and generally expressed in terms of hypergeometric functions.

\(\chi^2\) distribution

\[ f(x) = \frac{1}{2^{\nu/2} \Gamma(\nu/2)} x^{\nu/2 - 1} e^{-x/2} \]

where \(x\) is the chi-squared variable, \(\nu\) is the degrees of freedom.

non-central \(\chi^2\)

\[ f(x) = \frac{1}{2} \left( \frac{x}{\lambda} \right)^{\frac{\nu}{4} - \frac{1}{2}} e^{-\frac{1}{2}(x+\lambda)} I_{\frac{\nu}{2}-1}(\sqrt{\lambda x}) \]

where \(x\) is the non-central chi-squared variable, \(\nu\) is the degrees of freedom, \(\lambda\) is the non-centrality parameter, \(I_{\nu}(x)\) is the first kind modified Bessel function.

\(F\) distribution

\[ f(x) = \frac{\Gamma\left(\frac{\nu_1 + \nu_2}{2}\right)}{\Gamma\left(\frac{\nu_1}{2}\right) \Gamma\left(\frac{\nu_2}{2}\right)} \left(\frac{\nu_1}{\nu_2}\right)^{\frac{\nu_1}{2}} x^{\frac{\nu_1}{2} - 1} \left(1 + \frac{\nu_1}{\nu_2} x\right)^{-\frac{\nu_1 + \nu_2}{2}} \]

where \(x\) is the F statistic, \(\nu_1\) is the degrees of freedom for the numerator, \(\nu_2\) is the degrees of freedom for the denominator.

(Doubly) non-central \(F\) distribution ?

Non-central F-distribution

The PDF of the non-central F-distribution is also quite complex and involves hypergeometric functions.

Doubly Non-central F-distribution

The doubly non-central F-distribution has two non-centrality parameters and its PDF is even more intricate.