3
many aspects. We will not cover all of them in this work, but just to
taste the flavor of their scope, we will mention just a few.
The maximum entropy (ME) principle. This is perhaps the oldest
concept that ties the two fields and it has attracted a great deal of
attention, not only of information theorists, but also that of researchers
in related fields like signal processing and image processing. The ME
principle evolves around a philosophy, or a belief, which, in a nutshell,
is the following: if in a certain problem, the observed data comes from
an unknown probability distribution, but we do have some knowledge
(that stems, e.g., from measurements) of certain moments of the under-
lying quantity/signal/random-variable, then assume that the unknown
underlying probability distribution is the one with maximum entropy
subject to (s.t.) moment constraints corresponding to this knowledge.
For example, if we know the first and the second moments, then the ME
distribution is Gaussian with matching first and second order moments.
Indeed, the Gaussian model is perhaps the most common model for
physical processes in information theory as well as in signal- and image
processing. But why maximum entropy? The answer to this philosoph-
ical question is rooted in the second law of thermodynamics, which
asserts that in an isolated system, the entropy cannot decrease, and
hence, when the system reaches thermal equilibrium, its entropy reaches
its maximum. Of course, when it comes to problems in information the-
ory and other related fields, this principle becomes quite heuristic, and
so, one may question its justification, but nevertheless, this approach
has had an enormous impact on research trends throughout the last
50 years, after being proposed by Jaynes in the late fifties of the pre-
vious century [45, 46], and further advocated by Shore and Johnson
afterward [106]. In the book by Cover and Thomas [13, Section 12],
there is a good exposition on this topic. We will not put much emphasis
on the ME principle in this work.
Landauer’s erasure principle. Another aspect of these relations
has to do with a theory whose underlying guiding principle is that
information is a physical entity. Specifically, Landauer’s erasure
principle [63] (see also [6]), which is based on this physical theory
of information, asserts that every bit that one erases, increases the
entropy of the universe by k ln2, where k is Boltzmann’s constant.
