Author(s): Wuiii

Originally published on Towards AI.

Why Binary Cross Entropy?

If you have made your hands dirty in deep learning or perceptron models and similar such things, you would have encountered new loss functions. You are taught that they are used to represent how much we are at loss while wandering for a solution. If we are losing less as we are traveling, then we believe things are going alright and “maybe” we will reach a point where we are losing nothing. We come up with various strategies to walk or run or hike or sometimes jump in our error surface to finally settle at a deepest valley point where the error is least. These strategies are what we would in technical terms call optimization algorithms or simply optimizers.

A loss function is something that defines how the error surface would, at least under some assumptions, be fitting and easy enough for traveling. These error surfaces have faster training time. However, if error surfaces are chaotically designed, then it would be tough for an algorithm to reach the minima with certainty. A good approach for solving a learning problem would be to design good error surfaces and have good strategies to reach the lowest point in our error valley.

In the series of articles which I am writing recently, I am trying to make sense of how different loss functions design an error surface that would be solved. I use a very adventurous approach while going forward with this. In my last article, I had covered why MSE reduces error over time and why the square is there in the term. In this article, I would tell you about a famous loss function called Binary Cross Entropy. The name looks complicated to many, and I think it is worth an explanation rather than just mere reading for an exam.

When we are dealing with classification problems, we come up with a loss function which discriminates one class over the other class. As the name suggests in our case, it reads “Binary”, i.e., we have two classes and our model tries to associate the input to the right class. In the early period, it would be very bad at doing that (i.e., we will penalize it with heavy loss), and then eventually it learns to classify things well with less error (resulting in less penalty).

If we want to design a similar loss function, we have to come up with solving two cases: while the actual class is 0 (let us believe 0 and 1 are two classes), it should be trained enough to say it is 0. And while the actual class is 1, it should be similarly trained enough to say it is 1.

But you had asked me: we only have 0 or 1. These two are very discrete numbers, and while we are learning, we expect something like a real number so that we could tune in our weights and check the effect on a real number. That is a valid concern. Keeping this concern in context, we come up with something which is sort of a trainable score, and we redefine the problem based on it. Instead of telling that the prediction is 0 or 1, we had rather say that it is 95% chance 1 and rest of the chances 0. That is a very clever way to say it. Here, we are using probabilities to classify something.

And now, we could actually work on training something like: if we adjust some trainable parameters by tuning them, then the probability changes. And we could measure the change and make via this some sort of measure, a.k.a. loss function, that would say how much illiterate our model is and then reduce its illiteracy. So, there is room for improvement via this intuition.

Let us say we start with a scratchpad for designing a loss function for this. Let us say we go on with our first idea, that is to see the difference between what is the actual chance of an output being 1 from what is the chance claimed by the model. So, the loss function would be:

Now you had thought, from the intuition of square from MSE, I would love to put a square on it. That is actually a pretty good start. But you run into a problem when using this loss function. When you train long neural networks, the weight updates get deeper and deeper by the chain rule. During this process, the deepest weight updates are updated with tiny values than the immediate ones. So, when the values during this phase are small, then the weights updated get more tiny as epochs increase, resulting in slower training times.

Even though we are at a great start, we need to work with bigger values. Maybe we need a function which increases the value of probability, and we use it to further develop into a loss function. Our ideal sort of amplifying function would ideally have some properties as:

• Should uniformly apply for both positive and negative real numbers
• How much less the probability input is there, that much lesser the value it should provide
• How much more the probability input is there, that much higher the value it should provide

We have a function which does quite the opposite of this. That is a log function. A few evaluations of it are:

Also, an interesting quality which it has is, when the input becomes as close to zero, then it approaches negative infinity.

Let us consider y = 1. Then if we keep our loss function as:

Where:

• y is the actual label
• y_pred tells chances for it to turn out same as y

Then, we would be seeing the loss function behave as:

• loss increases when model predicts actual label with high chances
• loss decreases when model predicts actual label with low chances

To illustrate the above with an example, for y = 1:

• If y_pred says 1 with 90% chances (i.e., y_pred = 0.9), the loss would be -0.105
• If y_pred says 1 with 20% chances (i.e., y_pred = 0.2), the loss would be -1.609
• Notice here that loss decreased (from -0.105 to -1.609) as y_pred went more wrong. y_pred should have been 100% ideally, but it was 90%, and then it further decreased to 20%, but the loss function shows we are losing less or in other words, we are improving while we clearly are not improving.

Within the same example, if you were to try to fix this, a simple negative sign would help. Then the same example would become:

• If y_pred says 1 with 90% chances (i.e., y_pred = 0.9), the loss would be 0.105
• If y_pred says 1 with 20% chances (i.e., y_pred = 0.2), the loss would be 1.609
• Notice here that loss increased (from 0.105 to 1.609) as y_pred went more wrong. An increase in loss indicates we are moving in the bad direction, which is exactly the case here.

So, our loss function for case y = 1 is simple as this:

As we look at it much closer, it has got a problem. If y_pred says 90% chances that the class is 1 while y (actual class) is actually 0, then taking y_pred directly into the loss function would be bad. Because, as there are 10% chances that it is 0 (actual label), but now if we would force the loss to come down, it would happen by y_pred going up as we designed above (while actual label y was 1). Thus, decreasing loss naively would backfire on us. This also shows that our loss function is incomplete and is only able to handle the case of y being 1.

While we decrease loss, instead of y_pred going up for y = 0, it should go down, indicating the class is 0 actually. That would only happen if we use the probability of occurrence of the opposite event happening (complementary), which is simply given as:

ssIn this case, we use 1 — y_pred. Thus, we have an equation which works perfectly for y = 0, but not actually y = 1:

As we now have two loss functions, each working for a case, we would want to wrap them up into a single loss function. An interesting technique when we have binary cases and, based on values 0 or 1, if we were to choose one is as below:

In this thing, an interesting thing happens:

• When you keep case = 0, you get the answer as y.
• If you keep case = 1, you get the answer as x.

So, just by choosing between cases, you can get an output based on it. Similarly, we use this here, and now it is:

Thus, this is how we get binary cross entropy. Making this article for you was interesting, but costed a lot of thought and learning. If something is wrong, please comment it down, or if you want to add input, please comment it down. I think supporting via subscribing or claps or comments really helps a learner and a writer like me.

Thank you 🙂

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI