Chapter05:Vector calculus

Chapter 05: Vector calculus

1. 单变量函数的微分

1. 微分

• derivative:

$$\frac{\mathrm{d} f}{\mathrm{d} x}:=\lim _{h \rightarrow 0} \frac{f(x+h)-f(x)}{h} \tag{1}$$

2. Taylor 级数

$$T_{\infty}(x)=\sum_{k=0}^{\infty} \frac{f^{(k)}\left(x_{0}\right)}{k !}\left(x-x_{0}\right)^{k} \tag{2}$$

3. 微分法则

• Differentiation Rules:

$$\begin{align} \text{Product rule: } &\quad(f(x) g(x))^{\prime}=f^{\prime}(x) g(x)+f(x) g^{\prime}(x) \tag{3} \\ \text{Quotient rule: } &\left(\frac{f(x)}{g(x)}\right)^{\prime}=\frac{f^{\prime}(x) g(x)-f(x) g^{\prime}(x)}{(g(x))^{2}} \tag{4} \\ \text{Sum rule: } &(f(x)+g(x))^{\prime}=f^{\prime}(x)+g^{\prime}(x) \tag{5} \\ \text{Chain rule: } &(g(f(x)))^{\prime}=(g \circ f)^{\prime}(x)=g^{\prime}(f(x)) f^{\prime}(x) \tag{6} \end{align}$$

注：$g \circ f$ 记为复合函数$x \mapsto f(x) \mapsto g(f(x))$

$$\begin{align} \\ \text{例子：} \\ &h(x) = (2x+1)^{4} = g(f(x)) \\ \text{即}&f(x) = 2x + 1 \text{,}\qquad g(f) = f^{4} \\ &f^{\prime}(x) = 2 \text{,}\qquad g^{\prime}(f)=4f^{3} \\ \text{所以}&h^{\prime}(x) = g^{\prime}(f)f^{\prime}(x)=4f^{3}\cdot2=8(2x+1)^{3} \end{align}$$

2.部分微分和梯度

• Partial Differentiation andGradients

1.部分微分

• Partial Derivative:

$$\nabla_{\boldsymbol{x}} f=\operatorname{grad} f=\frac{\mathrm{d} f}{\mathrm{d} \boldsymbol{x}}=\left[\begin{array}{llll} \frac{\partial f(\boldsymbol{x})}{\partial x_{1}} & \frac{\partial f(\boldsymbol{x})}{\partial x_{2}} & \cdots & \frac{\partial f(\boldsymbol{x})}{\partial x_{n}} \end{array}\right] \in \mathbb{R}^{1 \times n} \tag{7}$$

即把对$x$的一阶导的集合写在一起形成一行,也可以转置后形成一列。

2. 梯度

• Gradient as a Row Vector

将Gradient写作行向量的原因：

1. 我们始终可以把梯度推广到向量值函数$f:\mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$;这能让梯度变成矩阵。

2. 我们可以不考虑梯度维数来应用多变量链式法则。

   $$\begin{align} \\ &\text{函数} f(x_{1}, x_{2}) = x_1^{2}x_2 + x_1x_2{3}\quad\text{其梯度为：} \\&\frac{\mathrm{d} f}{\mathrm{d} \boldsymbol{x}} = \begin{bmatrix} \frac{\partial f(x_1, x_2)}{\partial x_1} & \frac{\partial f(x_1, x_2)}{\partial x_2} \end{bmatrix} = \begin{bmatrix} 2x_1x_2^{3} & x_1^{2}+3x_1x_2^{2} \end{bmatrix} \in \mathbb{R}^{\color{red} {1 \times 2}} \end{align}$$

3. Chain Rule

$$\frac{\mathrm{d} f}{\mathrm{d} t}=\left[\begin{array}{ll} \frac{\partial f}{\partial x_{1}} & \frac{\partial f}{\partial x_{2}} \end{array}\right]\left[\begin{array}{l} \frac{\partial x_{1}(t)}{\partial t} \\ \frac{\partial x_{2}(t)}{\partial t} \end{array}\right]=\frac{\partial f}{\partial x_{1}} \frac{\partial x_{1}}{\partial t}+\frac{\partial f}{\partial x_{2}} \frac{\partial x_{2}}{\partial t} \tag{8}$$

用矩阵乘法获得梯度:

$$\frac{\mathrm{d} f}{\mathrm{d}(s, t)}=\frac{\partial f}{\partial \boldsymbol{x}} \frac{\partial \boldsymbol{x}}{\partial(s, t)}= \underbrace{\left[\frac{\partial f}{\partial x_{1}}, \frac{\partial f}{\partial x_{2}}\right]}_{=\frac{\partial f}{\partial \boldsymbol{x}}} \underbrace{\left[\begin{array}{ll} \frac{\partial x_{1}}{\partial s} & \frac{\partial x_{1}}{\partial t} \\ \frac{\partial x_{2}}{\partial s} &\frac{\partial x_{2}}{\partial t} \end{array}\right]}_{=\frac{\partial \boldsymbol{x}}{\partial(s, t)}}$$

4. 向量值函数的梯度(从实数域推广到vector field)

$$\boldsymbol{f}(\boldsymbol{x})=\left[\begin{array}{c} f_{1}(\boldsymbol{x}) \\ \vdots \\ f_{m}(\boldsymbol{x}) \end{array}\right] \in \mathbb{R}^{m}$$

5. Jacobian

所有向量值函数$\boldsymbol{f}: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$的一阶导数的集合称作$\mathit{Jacobian}$。见如下公式：

$$\begin{align} J=\nabla_{x} f=\frac{\mathrm{d} f(x)}{\mathrm{d} x}&=\left[\frac{\partial f(x)}{\partial x_{1}} \quad \cdots \quad \frac{\partial f(x)}{\partial x_{n}}\right] \tag{9} \\&=\left[\begin{array}{ccc} \frac{\partial f_{1}(\boldsymbol{x})}{\partial x_{1}} & \cdots & \frac{\partial f_{1}(\boldsymbol{x})}{\partial x_{n}} \\ \vdots & & \vdots \\ \frac{\partial f_{m}(\boldsymbol{x})}{\partial x_{1}} & \cdots & \frac{\partial f_{m}(\boldsymbol{x})}{\partial x_{n}} \end{array}\right] \tag{10} \\ \boldsymbol{x}=\left[\begin{array}{c} x_{1} \\ \vdots \\ x_{n} \end{array}\right], \quad J(i, j) &=\frac{\partial f_{i}}{\partial x_{j}} \tag{11} \end{align}$$

确定$\boldsymbol{f}(x)$偏导数维度：

偏导数维度：引用自https://mml-book.com

​ 偏导数的维度：column由$\boldsymbol{f}(x)$决定；row由$\boldsymbol{x}$决定

详解：

1. $f:\mathbb{R} \rightarrow \mathbb{R}$ 梯度是个标量
2. $f:\mathbb{R}^{N} \rightarrow \mathbb{R}$ 梯度是$1\times $N行向量(row)
3. $f:\mathbb{R} \rightarrow \mathbb{R}^{M}$ 梯度是$M \times 1$列向量(column)
4. $f:\mathbb{R}^{N} \rightarrow \mathbb{R}^{M}$ 梯度是$M \times N$矩阵(matrix)

可视化矩阵关于Vector的梯度计算（两种等价方法）

• 方法(a):先计算$\frac {\partial \boldsymbol{A}} {\partial {x_1}}, \frac {\partial \boldsymbol{A}} {\partial {x_2}}, \frac {\partial \boldsymbol{A}} {\partial {x_3}}$（都是$4 \times 2$的tensor;再整理成$4\times2\times3$的tensor。

方法(a) 引用自https://mml-book.com

• 方法(b):先把$\boldsymbol {A} \in \mathbb{R}^{4 \times 2}$展平成向量$\tilde{A} \in \mathbb{R}^{8}$，然后计算其梯度$\frac{\mathrm{d} \tilde{\boldsymbol{A}}}{\mathrm{d} \boldsymbol{x}} \in \mathbb{R}^{8 \times 3}$,我们把得到的梯度tensor再reshape成图例形状。

方法(b) 引用自https://mml-book.com