京都大学 情報学研究科 知能情報学専攻 2022年8月実施 専門科目 S-3

标签：

• Kyoto-University
• Feedforward-Neural-Network
• Multi-Label-Classification
• Binary-Cross-Entropy
• Backpropagation
• Vanishing-Gradient-and-Mitigation
• Feature-Standardization
• Mahalanobis-Distance
• Covariance-Matrix

Author

祭音Myyura

Description

設問1

Let us consider a fully-connected feed-forward neural network that has an input of $d$ dimensions, an output of $c$ classes, and $m$ intermediate layers, each having $n$ nodes. A sigmoid function is used in all nodes including output nodes. A weight between node $i$ and node $j$ is denoted as $w_{ij}$, and there are no bias terms.

(1) Draw this network and specify $d$, $c$, $m$, and $n$. Answer the total number of the network weights.

(2) Show the output $g_k$ of an output node (indexed with $k$) using the output $g_j$ of the nodes of the preceding layer (indexed with $j$).

(3) Consider the problem of detecting the source(s) from music recording composed of the sounds of one or more from violin, flute, piano, and singing voices. Describe how the training label $t_k$ will be given for the output nodes (indexed with $k$). Explain why it is not appropriate to use a softmax function in the output nodes for this problem.

(4) Show the binary cross-entropy of the output $g_k$ and the training label $t_k$ of an output node (indexed with $k$).

(5) Show the formula to update the weight $w_{jk}$ of an output node (indexed with $k$) and a node of the preceding layer (indexed with $j$) based on the gradient descent method with the objective function of the sum of the binary cross-entropy defined above over all classes. Show how you derive the formula.

(6) Show the formula to update the weight $w_{ij}$ of the nodes (indexed with $j$ and $i$), both of which are not in the output layer, based on the error back-propagation method. You do not have to show how you derive it.

(7) Explain why it is difficult to update the weights effectively as the number of network layers becomes large. Describe the methods to mitigate this problem.

設問2

Let us consider $n$ training samples of a $d$-dimensional vector $X = (x_1, \dots, x_d)^T$, with their mean vector and covariance matrix denoted as $M = (m_1, \dots, m_d)^T$ and $\Sigma$, respectively, where $T$ denotes the transpose.

(1) Show the formula to compute the component $\sigma_{ij}$ of the covariance matrix $\Sigma$.

(2) Show the formula of the Mahalanobis distance between a sample $X$ and this training sample distribution.

(3) In neural network training, we often conduct normalization of inputs so that the distribution for each dimension has a mean of $0$ and a variance of $1$. Let $X$ and $Z$ be an original input and its normalized one. Discuss the relationship between the square root of the sum of the squared values in each dimension of $Z$, which is regarded as the Euclidean norm $\|Z\|_2$, and the above Mahalanobis distance.

题目描述

1. 考虑全连接前馈神经网络：输入维数 $d$，输出类别数 $c$，有 $m$ 个隐藏层且每层 $n$ 个节点；所有节点包括输出层均用 sigmoid，无偏置，节点 $i,j$ 间权重记为 $w_{ij}$。
   1. 画网络并标注 $d,c,m,n$，求权重总数。
   2. 用前一层节点输出 $g_j$ 表示输出节点 $k$ 的输出 $g_k$。
   3. 要从可能同时含小提琴、长笛、钢琴、歌声中一种或多种的录音识别声源，说明各输出节点训练标签 $t_k$ 如何给定，并解释输出层为何不宜用 softmax。
   4. 写出单个输出 $g_k$ 与标签 $t_k$ 的二元交叉熵。
   5. 以各类二元交叉熵之和为目标，推导梯度下降更新输出层权重 $w_{jk}$ 的公式。
   6. 写出反向传播更新非输出层权重 $w_{ij}$ 的公式，无需推导。
   7. 解释网络加深后为何难以有效更新权重，并说明缓解方法。
2. 有 $n$ 个 $d$ 维训练样本 $X=(x_1,\ldots,x_d)^\top$，均值 $M=(m_1,\ldots,m_d)^\top$、协方差矩阵 $\Sigma$。
   1. 写出 $\Sigma$ 的元素 $\sigma_{ij}$ 的计算式。
   2. 写出样本 $X$ 到训练分布的 Mahalanobis 距离。
   3. 神经网络常把每维输入标准化为均值 0、方差 1。设原输入为 $X$、标准化后为 $Z$，讨论 $\|Z\|_2$ 与上述 Mahalanobis 距离的关系。

考点

• 前馈网络与多标签分类：计算参数量，写 sigmoid 前向传播，并用独立多热标签和二元交叉熵处理多声源共存。
• 反向传播与梯度消失：推导输出层梯度、递推隐藏层误差信号，并说明激活饱和、初始化、归一化、残差连接等缓解手段。
• 协方差与 Mahalanobis 距离：从样本计算协方差，辨析逐维标准化欧氏范数只有在忽略相关性/协方差对角时才等同 Mahalanobis 距离。

Kai

設問1

(1)

the total number of the network weights is

$$dn+(m-1)n^{2}+nc$$

(2)

the output $g_k$ is

$$g_{k} = \sigma\left(\sum_{j=1}^{n}w_{jk}g_{j}\right)$$

where $\sigma$ is the sigmoid function.

(3)

The classification classes consist of four categories: violin, flute, piano, and vocals, thus $c = 4$. Let $k = 1, \dots, c$, and construct the training label $t_k$ such that it equals $1$ if a particular source $k$ is present in the music recording, and $0$ otherwise. Using the softmax function in the output layer is inappropriate in this context because it would force the neural network to solve a four-class classification problem, making it unable to represent cases containing multiple sources.

(4)

$$L_k = -t_{k}\log g_{k}-(1-t_{k})\log (1-g_{k})$$

(5)

Let $h_{j} = \sum_{i=1}^{n}w_{ij}g_{i}$. The loss function $L$ is defined as follows:

$$L = \sum_{k=1}^{c}L_{k} = \sum_{k=1}^{c}\left(-t_{k}\log g_{k}-(1-t_{k})\log (1-g_{k})\right)$$

the weight $w_{jk}$ is updated as follows:

$$w_{jk} \leftarrow w_{jk} - \eta\frac{\partial L}{\partial w_{jk}}$$

where $\eta$ is learning rate. Since

$$\begin{aligned} \frac{\partial L}{\partial w_{jk}} &= \frac{\partial L}{\partial g_{k}}\cdot\frac{\partial g_{k}}{\partial h_{k}}\cdot \frac{\partial h_{k}}{\partial w_{jk}}\\ &= \frac{\partial L}{\partial g_{k}}\cdot \sigma^{\prime}(h_{k})\cdot g_{j}\\ &= \left(-\frac{t_{k}}{g_{k}}+\frac{1-t_{k}}{1-g_{k}}\right)\cdot\sigma^{\prime}(h_{k})\cdot g_{j}\\ &= \frac{g_{k}-t_{k}}{g_{k}(1-g_{k})}\cdot\sigma^{\prime}(h_{k})\cdot g_{j}\\ &= \frac{\sigma(h_{k})-t_{k}}{\sigma(h_{k})(1-\sigma(h_{k}))}\cdot\sigma^{\prime}(h_{k})\cdot g_{j} \end{aligned}$$

$$\begin{aligned} \sigma^{\prime}(x) &= \frac{e^{-x}}{(1+e^{-x})^{2}} = \frac{e^{-x}}{1+e^{-x}}\cdot \frac{1}{1+e^{-x}}\\ &= \left(1-\frac{1}{1+e^{-x}}\right)\frac{1}{1+e^{-x}} \\ &= \sigma(x)(1-\sigma(x)) \end{aligned}$$

we have

$$\frac{\partial L}{\partial w_{jk}} = (\sigma(h_{k})-t_{k})g_{j} = \left(\sigma\left(\sum_{j=1}^{n}w_{jk}g_{j}\right)-t_{k}\right)g_{j}$$

$$w_{jk} \leftarrow w_{jk} - \eta \left(\sigma\left(\sum_{j=1}^{n}w_{jk}g_{j}\right)-t_{k}\right)g_{j}$$

(6)

$$\begin{align} w_{ij} &\leftarrow w_{ij} - \eta \left(\sum_{k=1}^{n}\frac{\partial L}{\partial h_k}w_{jk}\right)\sigma^{\prime}(h_{j})g_{i} \end{align}$$

(7)

Vanishing and exploding gradients.

Methods to mitigate the above issue:

• Weight Initialization
  • Xavier Initialization
  • He Initialization
• Activation Functions
  • ReLU (Rectified Linear Unit)
• Batch Normalization
• Gradient Clipping

設問2

(1)

$$\begin{aligned} \sigma_{ij} &= \text{Cov}[X_{i},X_{j}]\\ &= E[X_{i}X_{j}] - E[X_{i}]E[X_{j}]\\ &= E[X_{i}X_{j}] - m_{i}m_{j} \end{aligned}$$

where $X_i, X_j$ denote the $i$-th and $j$-th training sample.

(2)

$$\sqrt{(X-M)^{T}\Sigma^{-1}(X-M)}$$

(3)

When input $X$ is normalized, if the covariance matrix $\Sigma$ is a diagonal matrix, the Mahalanobis distance and the Euclidean distance are equivalent.

Suppose that

$$Z = A^{-1}(X-M)$$

since $Z$ is obtained by normalizing $X$, when $\Sigma$ is diagonal, the covariance matrix of $Z$ is identity matrix. Hence

$$\begin{aligned} \Sigma &= E[(X-M)(X-M)^{T}]\\ &= E\left[(AZ)(AZ)^{T}\right]\\ &= E\left[AZZ^{T}A^{T}\right]\\ &= AE[ZZ^{T}]A^{T}\\ &= AA^{T} \end{aligned}$$

and the Mahalanobis distance is

$$\begin{aligned} \sqrt{(X-M)^{T}\Sigma^{-1}(X-M)} &= \sqrt{Z^{T}A^{T}(AA^{T})^{-1}AZ}\\ &= \sqrt{Z^{T}A^{T}(A^{T})^{-1}A^{-1}AZ}\\ &= \sqrt{Z^{T}Z} = \|Z\| \end{aligned}$$