We show how to use “complementary priors” to eliminate the explainingaway
effects thatmake inference difficult in densely connected belief nets
that have many hidden layers. Using complementary priors, we derive a
fast, greedy algorithm that can learn deep, directed belief networks one
layer at a time, provided the top two layers form an undirected associative
memory. The fast, greedy algorithm is used to initialize a slower
learning procedure that fine-tunes the weights using a contrastive version
of thewake-sleep algorithm. After fine-tuning, a networkwith three
hidden layers forms a very good generative model of the joint distribution
of handwritten digit images and their labels. This generative model
gives better digit classification than the best discriminative learning algorithms.
The low-dimensional manifolds on which the digits lie are
modeled by long ravines in the free-energy landscape of the top-level
associative memory, and it is easy to explore these ravines by using the
directed connections to displaywhat the associativememory has in mind.
