Negative log likelihood (NLL) as a cost function for classification, explained!
Let’s start by defining the function under consideration. In probability, the Likelihood function describes the joint probability of observing a specific set of data (\(y\)), as a function of the model's parameters (\(\theta\)). It can be mathematically described as:
$$L(\theta | y) = P(y | \theta)$$
Here, the "\(|\)" symbol denotes conditional probability (read as "given that"). Therefore, \(P(y | \theta)\) means "the probability of observing the event \(y\), given the specific parameter settings \(\theta\)." In the context of the likelihood function \(L(\theta|y)\), the data \(y\) is already observed and fixed, while the parameters \(\theta\) are the variables we adjust to maximize the function.
Practical Example
Let’s take an example of a common classification problem in supervised learning to understand the above description. The goal is to identify the presence of cat in an image.
One way to create this model is by iteratively increasing the likelihood that our model predicts the correct output i.e we need to optimize
$$L(\theta | y) = P(\text{labels } y \text{ | model parameters } \theta)$$
Imagine we have initialized the parameters of the model and we feed the model an image from the training set. The model guesses an arbitrary probability (\(Y^{'}\)) guessing the presence of cat in the image.
Assume \(Y' = 0.3\) and that the example image was indeed an image of a cat (meaning the true label is \(y = 1\)). To calculate the likelihood of our model predicting the correct output for a single image, we use the Bernoulli trial formula:
$$L(\theta | y) = (Y')^y \times (1 - Y')^{1-y}$$
Since it is a cat (\(y = 1\)), the second half of the equation becomes \((1 - Y')^0 = 1\), which leaves us with:
$$L(\theta | y) = (0.3)^1 \times 1 = 0.3$$
Similarly, if the image in question was not that of a cat (\(y = 0\)), the first half of the equation becomes \((Y')^0 = 1\), meaning our likelihood of predicting the correct class is:
$$L(\theta | y) = 1 \times (1 - 0.3)^1 = 0.7$$
From the above equations, we can see that the likelihood function perfectly indicates the correctness of our model and so can be used to optimize our network. Extending this to the entire set of images in the training network. Since the likelihood of independent events multiply, the final equation for the likelihood of the entire training set is \(\prod_{k=1}^{n} L_{k}\), where \(L_{k}\) is the likelihood for image \(k\).
Hence, the final equation is as follows:
$$L(\theta | y) = \prod_{k=1}^{n} L_{k} = \prod_{k=1}^{n} (Y'_k)^{y_k} \times (1 - Y'_k)^{1-y_k}$$
In our example, \(L_{k}\) dynamically evaluates to the correct class probability of the model (\(Y'_k\) for a cat, \(1 - Y'_k\) for a non-cat), providing a generalized equation ready for optimization.
Reason for NLL
For ease of computation, logarithm is usually applied, which translates the above equation to sum of log likelihood. This offers two significant advantages
$$log\ L(\theta|y)\ = \sum_{k=1}^{n}logL_{k}$$
As log is negative in the interval (0,1), we multiple the above function with -1, so that we can deal with positive values. The negative log likelihood will provide a measure of the error of the model since likelihood measures the correctness of the model. Minimization of error being the preferred method of optimization, this fits in perfectly.
$$-log\ L(\theta|y)\ = -\sum_{k=1}^{n}logL_{k}$$
Finally we find the average of above function for m samples so that we get the error contributed by a single sample which can be later optimized by updating the model weights using methods such as gradient descent. This function is called as Cost function of the model.
$$Cost\ Fn=\frac{-1}{n}\sum_{k=1}^{n}logL_{k}$$
The above equation which is the negative of average log likelihood is also called as Cross-entropy in machine learning and it has wide range of applications in Information theory.
Graphical interpretation
From the graph below of \(y=-log(x)\) we can see that \(y->\infty\) when \(x→0\) and \(y=0\) when \(x=1\) which is exactly what an error measuring function should do, i.e unlimited error when model is 0% correct and zero error when model is 100% correct.
Thus we have discussed the likelihood function in Probability and why negative log likelihood is used as cost function for classification tasks in machine learning with an example.
Footnotes
Likelihood is a key part of the Bayes theorem and it was later taken up independently in various areas for statistical modelling. The theorem provides a way to find conditional probability of events by updating probability using new evidence/data.