跳到主要内容

東京大学 情報理工学系研究科 コンピュータ科学専攻 2019年8月実施 専門科目II 問題6

Author​

zephyr

Description​

The probability density function of the normal distribution with mean and variance is given by

Let and be random variables that independently follow and , respectively, and define for some constant . For an integer , let be two-dimensional random variables that independently follow the same distribution as , for which we write and .

Answer the following questions.

(1) Express the expectation and variance of using and .

(2) Show that the conditional distribution of given is a normal distribution, and express its expectation and variance using , , and .

(3) Let denote a realization of . Express the joint probability density function of using and .

(4) Consider maximum-likelihood estimation of by the EM algorithm for the case where the observation of is missing from , that is, the case where is observed. Then the update rule of estimators of by the EM algorithm for some initial value is given by

where and are the values obtained by the substitution in the expressions of and obtained in question (2), respectively, and denotes the expectation when follows and is fixed.

  • (i) Express using and .
  • (ii) Express using and .

Kai​

(1)​

The random variable is defined as , where and . Since and are independent, we can calculate the expectation and variance of as follows:

  1. Expectation of :
  1. Variance of :

(2)​

To find the conditional distribution of given , note that , where and . The joint distribution of is bivariate normal, which implies that the conditional distribution is also normal.

  1. Expectation of :
  1. Variance of :

This can be derived using the properties of conditional distributions for bivariate normal distributions.

(3)​

The joint probability density function for the random variables and can be expressed as the product of the marginal distributions of and the conditional distributions of given :

Expanding this, we get:

(4)​

(i)​

The expectation is given by:

Simplifying further using the properties of the expectation for a normal distribution:

(ii)​

The update rule for in the EM algorithm is obtained by maximizing the expression found in part (i):

Solving this for and , we find:

This update rule depends on the observed data and the estimates obtained from the conditional expectation.

Knowledge​

正态分布 条件分布 数值期望 EM算法 最大似然估计

难点思路​

推导条件分布涉及到二元正态分布的性质，尤其是推导条件期望和方差时，需要对协方差矩阵有深刻理解。EM 算法的难点在于构建对数似然函数的期望，并通过优化找到参数的更新规则。

解题技巧和信息​

  1. 条件分布：对于二元正态分布，条件分布仍然是正态分布，且其参数可以通过边际分布的参数计算得到。
  2. EM 算法：EM 算法通过最大化对数似然函数的期望来迭代更新参数，对于缺失数据的问题尤为有效。
  3. 最大似然估计：通常情况下，EM 算法能够保证参数的渐进一致性，即经过多次迭代，参数估计会收敛到真值。

重点词汇​

  • Expectation-Maximization (EM) Algorithm: 期望最大化算法
  • Conditional distribution: 条件分布
  • Maximum likelihood estimation: 最大似然估计
  • Normal distribution: 正态分布

参考资料​

  1. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Chapter 9: Mixture Models and EM.
  2. Casella, G., & Berger, R. L. (2001). Statistical Inference (2nd ed.). Duxbury. Chapter 7: Estimation.