Gatsby believed in the green light, the orgastic future that year by year recedes before us. It eluded us then, but that’s no matter–tomorrow we will run faster, stretch out our arms farther….
And one fine morning. So we beat on, boats against the current, borne back ceaselessly into the past.
KL Divergence (Kullback-Leibler Divergence, KLD) Suppose we have two coins: A fair coin with the following probability distribution: $$P=\begin{cases}&0.5,&\text{Heads} \\ &0.5, &\text{Tails}\end{cases}$$ A biased coin with the following probability distribution: $$Q=\begin{cases}&0.9,&\text{Heads} \\ &0.1, &\text{Tails}\end{cases}$$ How can we describe the difference between these two distributions? More importantly, how can we quantify this difference? KL divergence allows us to quantify the difference (or similarity) between two distributions. It measures how one probability distribution $Q(x)$ diverges from a reference distribution $P(x)$, and is defined as: ...