机器学习-白板推导 1 概率基础

1. MLE

一元高斯分布：

$P(x) = {\frac {1}{ {\sqrt {2\pi }\sigma}}}\operatorname{ exp } \left(-{\frac {\left(x-\mu \right)^{2}}{2\sigma ^{2}}}\right) \tag{1}$

多元高斯分布：

$P(X) = \frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma|^{\frac{1}{2}}}\operatorname{ exp }\left\{ -\frac{1}{2}(X-\mu)^T\Sigma^{-1}(X-\mu) \right\} \tag{2}$

用极大似然估计参数 $\theta$ 有：

$\begin{align} L(\theta) = \operatorname{ \log } P(x | \theta) = &\operatorname{ \log } \prod_{i=1}^N P(x_i|\theta)=\sum_{i=1}^N \operatorname{ \log } P(x_i|\theta) \\ = & \sum_{i=1}^N \operatorname{ \log } \frac{1}{\sqrt{2\pi}} + \operatorname{ \log } \frac{1}{\sigma} - \frac{(x_i-\mu)^2}{2\sigma^2} \end{align} \tag{3}$

求解 $\mu_{\text{MLE}}$ ，就是对式3求 $\mu$ 的偏导，

$\frac{\partial L(\theta)}{ \partial \mu} = \sum_{i=1}^N(x_i-\mu)=0 \tag{4}$

即得到：

$\sum_{i=1}^{N} x_i - N\mu=0 \Longrightarrow \mu_{MLE}=\frac{1}{N}\sum_{i=1}^{N} x_i$

求解 $\sigma^2_{MLE}$ 时，计算目标为 $\frac{\partial\log p(x|\theta)}{\partial \sigma}$ ，推导公式如下：

$\begin{gather} \frac{\partial\log p(x|\theta)}{\partial \sigma} = \sum_{i=1}^N - \frac{1}{\sigma} - \frac{1}{2}(x_i-\mu)^2(-2)\sigma^{-3} = 0 \\ \sum_{i=1}^N \sigma^2 = \sum_{i=1}^N (x_i-\mu)^2 \\ \sigma^2_{MLE} = \frac{1}{N} \sum_{i=1}^N (x_i-\mu)^2 \end{gather}$

实际上这里的$\mu$是$\mu_{MLE}$，所以，

$\begin{equation} \sigma^2_{MLE} = \frac{1}{N} \sum_{i=1}^N (x_i-\mu_{MLE})^2 \end{equation}$

2. 无偏有偏

首先需要明确什么是无偏估计，所谓无偏估计也就是， $\mathbb{E}(\hat{x})=x$ 。那么利用这个性质我们就可以很方便的判断一个估计是否为无偏估计。验证 $\sigma^2_{MLE}$ 有偏无偏

$\begin{equation} \begin{split} \mathbb{E}[\sigma^2_{MLE}] = & \mathbb{E}[ \frac{1}{N}\sum_{i=1}^N (x_i-\mu_{MLE})^2] \\ = & \mathbb{E}[ \frac{1}{N}\sum_{i=1}^N (x_i^2-2\mu_{MLE} x_i + \mu_{MLE}^2)] \\ = & \mathbb{E}[ \frac{1}{N}\sum_{i=1}^N (x_i^2- \mu_{MLE}^2)]\\ = & \mathbb{E}[ \frac{1}{N}\sum_{i=1}^N (x_i^2-\mu^2)-(\mu_{MLE}^2-\mu^2)] \\ = & \mathbb{E}[ \frac{1}{N}\sum_{i=1}^N (x_i^2-\mu^2)]-\mathbb{E}[(\mu_{MLE}^2-\mu^2)]\\ = & \mathbb{E}[ \frac{1}{N}\sum_{i=1}^N (x_i^2-(\frac{1}{N}\sum_{i=1}^Nx_i)^2)]-\mathbb{E}[(\mu_{MLE}^2-\mu^2)]\\ = & \frac{1}{N}\sum_{i=1}^{N}(\mathbb{E}[x_i^2]-\mathbb{E}[x]^2)-\mathbb{E}[(\mu_{MLE}^2-\mu^2)] \\ = & \sigma^2 - (\mathbb{E}[\mu_{MLE}^2] - \mathbb{E}[\mu^2]) \\ = & \sigma^2 - (\mathbb{E}[\mu_{MLE}^2] - \mathbb{E}[\mathbb{E}[\mu_{MLE}]^2]) \\ = & \sigma^2 - (\mathbb{E}[\mu_{MLE}^2] - \mathbb{E}[\mu_{MLE}]^2] \\ = & \sigma^2 - Var[\mu_{MLE}] \\ = & \sigma^2 - Var[\frac{1}{N}\sum_{i=1}^Nx_i] \\ = & \sigma^2 - \frac{1}{N^2}Var[\sum_{i=1}^Nx_i] \\ = & \sigma^2 - \frac{1}{N^2}\sum_{i=1}^NVar[x_i] \\ = & \sigma^2 - \frac{1}{N^2} N \sigma^2 \\ = & \frac{N-1}{N}\sigma^2 \end{split} \end{equation}$