Maximum Likelihood Estimation VS Maximum A Posteriori Estimation

Introduction

In non-probabilistic machine learning, maximum likelihood estimation (MLE) is one of the most common methods for optimizing a model. In probabilistic machine learning, we often see maximum a posteriori estimation (MAP) rather than maximum likelihood estimation for optimizing a model.

In this blog post, I would like to discuss the connections between the MLE and MAP methods.

Bayes’ Theorem

Bayes’ theorem is stated mathematically as the following equation.

$$\underset{\text{posterior}}{\underbrace{P(A|B)}}=\frac{\underset{\text{likelihood}}{\underbrace{P(B|A)}}\underset{\text{prior}}{\underbrace{P(A)}}}{\underset{\text{marginal}}{\underbrace{P(B)}}}$$

where $P(A|B)$ is the posterior probability, $P(B|A)$ is the likelihood probability, $P(A)$ and $P(B)$ are prior probabilities, and $P(B)$ is also often referred as the marginal probability as it is an marginalization of the joint probability $P(A,B)$ over variable $A$.

If $A$ is a continuous variable,

$$P(B)={\int }_A^{}P(A,B)dA={\int }_A^{}P(B|A)P(A)dA$$

If $A$ is a discrete variable,

$$P(B)=\sum _A^{}P(A,B)=\sum _A^{}P(B|A)P(A)$$

Notice that $P(B)$ is a constant with respect to the variable $A$, so we could safely say $P(A|B)$ is proportional to $P(B|A)P(A)$ with respect to the variable $A$. Mathematically,

$$P(A|B)\propto P(B|A)P(A)$$

Maximum Likelihood Estimation (MLE)

Maximum likelihood estimation, as is stated in its name, maximizes the likelihood probability $P(B|A)$ in Bayes’ theorem with respect to the variable $A$ given the variable $B$ is observed.

Mathematically, maximum likelihood estimation could be expressed as

$$a_{\text{MLE}}^{\ast }=\underset{A}{\mathrm{argmax}}P(B=b|A)$$

It is equivalent to optimizing in the log domain since $P(B=b|A)\ge 0$ and assuming $P(B=b|A)\ne 0$.

$$a_{\text{MLE}}^{\ast }=\underset{A}{\mathrm{argmax}}\log P(B=b|A)$$

Maximum A Posteriori Estimation (MAP)

Maximum a posteriori estimation, as is stated in its name, maximizes the posterior probability $P(A|B)$ in Bayes’ theorem with respect to the variable $A$ given the variable $B$ is observed.

Mathematically, maximum a posteriori estimation could be expressed as

$$a_{\text{MAP}}^{\ast }=\underset{A}{\mathrm{argmax}}P(A|B=b)$$

It is equivalent to optimizing in the log domain since $P(A|B=b)\ge 0$ and assuming $P(A|B=b)\ne 0$.

$$a_{\text{MAP}}^{\ast }=\underset{A}{\mathrm{argmax}}\log P(A|B=b)$$

MLE and MAP Relationship

By applying Bayes’ theorem, we have

$$\begin{array}{rl}P(A|B=b)& =\frac{P(B=b|A)P(A)}{P(B=b)}\\ & \propto P(B=b|A)P(A)\end{array}$$

Therefore, maximum a posteriori estimation could be expanded as

$$\begin{array}{rl}{a}_{\text{MAP}}^{\ast }& =\underset{A}{\mathrm{argmax}}P(A|B=b)\\ & =\underset{A}{\mathrm{argmax}}\log P(A|B=b)\\ & =\underset{A}{\mathrm{argmax}}\log \frac{P(B=b|A)P(A)}{P(B=b)}\\ & =\underset{A}{\mathrm{argmax}}(\log P(B=b|A)+\log P(A)-\log P(B=b))\\ & =\underset{A}{\mathrm{argmax}}(\log P(B=b|A)+\log P(A))\end{array}$$

If the prior probability $P(A)$ is uniform distribution, i.e., $P(A)$ is a constant, we further have

$$\begin{array}{rl}{a}_{\text{MAP}}^{\ast }& =\underset{A}{\mathrm{argmax}}P(A|B=b)\\ & =\underset{A}{\mathrm{argmax}}(\log P(B=b|A)+\log P(A))\\ & =\underset{A}{\mathrm{argmax}}\log P(B=b|A)\\ & =a_{\text{MLE}}^{\ast }\end{array}$$

Therefore, we could conclude that maximum likelihood estimation is a special case of maximum a posteriori estimation when the prior probability is uniform distribution.

Which One to Use

In optimization, maximum likelihood estimation and maximum a posteriori estimation, which one to use, really depends on the use cases. If we know the probability distribution for both the likelihood probability $P(B|A)$ and the prior probability $P(A)$, we can use maximum a posteriori estimation.

However, in many practical optimization problems, we actually don’t know the distribution for the prior probability $P(A)$. Therefore, applying maximum a posteriori estimation is not possible, and we can only apply maximum likelihood estimation.

For example, suppose we are going to find the optimal parameters for a model. In the model, we have parameter variables $\theta$ and data variables $X$. Given some training data $\{x_1,x_2,\cdots ,x_N\}$, we want to find the most likely parameter ${\theta }^{\ast }$ of the model given the training data. Mathematically, it is essentially maximum a posteriori estimation and it is expressed as

$$\begin{array}{rl}{\theta }^{\ast }& =\underset{\theta }{\mathrm{argmax}}\prod _{i=1}^{N}P(\theta |X=x_i)\\ & =\underset{\theta }{\mathrm{argmax}}\log \prod _{i=1}^{N}P(\theta |X=x_i)\\ & =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N\log P(\theta |X=x_i)\\ & =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N\log (P(X=x_i|\theta )P(\theta ))\\ & =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N(\log P(X=x_i|\theta )+\log P(\theta ))\\ & =\underset{\theta }{\mathrm{argmax}}((\sum _{i=1}^N\log P(X=x_i|\theta ))+N\log P(\theta ))\end{array}$$

As been discussed previously, because in many models, especially the conventional machine learning and deep learning models, we usually don’t know the distribution of $P(\theta )$, we cannot do maximum a posteriori estimation exactly. However, we can still do maximum likelihood estimation by assuming $P(\theta )$ is uniform distribution. Then

$$\begin{array}{rl}{\theta }^{\ast }& =\underset{\theta }{\mathrm{argmax}}\sum _{i=1}^N\log P(X=x_i|\theta )\\ & =\underset{\theta }{\mathrm{argmax}}\prod _{i=1}^{N}P(X=x_i|\theta )\end{array}$$

This is why we often see maximum likelihood estimation, rather than maximum a posteriori estimation, in conventional non-probabilistic machine learning and deep learning models.