Cross entropy
In information theory, the cross-entropy between two probability distributions and over the same underlying set of events
measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized
for an estimated probability distribution , rather than the true distribution .
Definition
Motivation
Estimation
Relation to log-likelihood
Cross-entropy minimization
Cross-entropy loss function and logistic regression
See also
References
External links
The cross-entropy of the distribution relative to a distribution over a given set is defined as follows:
,
where is the expected value operator with respect to the distribution .
The definition may be formulated using the Kullback–Leibler divergence , divergence of from (also known as the
relative entropy of with respect to ).
,
where is the entropy of .
For discrete probability distributions and with the same support this means
(Eq.1)
The situation for continuous distributions is analogous. We have to assume that and are absolutely continuous with respect to
some reference measure (usually is a Lebesgue measure on a Borel σ-algebra). Let and be probability density functions of
and with respect to . Then
and therefore
(Eq.2)
NB: The notation is also used for a different concept, the joint entropy of and .
In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to
identify one value out of a set of possibilities can be seen as representing an implicit probability distribution
over , where is the length of the code for in bits. Therefore, cross-entropy can be interpreted as
Contents
Definition
Motivation
资源评论
