(21th-December-2020)
• we saw that many kinds of autoencoders learn the data distribution. 14 There are close connections between score matching, denoisingautoencoders, and contractive autoencoders. These connections demonstrate that some kinds of autoencoders learn the data distribution in some way. We have not yet seen how to draw samples from such models. Some kinds of autoencoders, such as the variationalautoencoder, explicitly represent a probability distribution and admit straightforward ancestral sampling. Most other kinds of autoencoders require MCMC sampling. Contractive autoencoders are designed to recover an estimate of the tangent plane of the data manifold. This means that repeated encoding and decoding with injected noise will induce a random walk along the surface of the manifold (Rifai et al. et al. , ; 2012 Mesnil , ). This manifold diffusion technique is a kind of 2012 Markov chain. There is also a more general Markov chain that can sample from any denoisingautoencoder.
• Markov Chain Associated with any DenoisingAutoencoder
• The above discussion left open the question of what noise to inject and where, in order to obtain a Markov chain that would generate from the distribution estimated by the autoencoder. ( ) showed how to construct Bengio et al. 2013c such a Markov chain for generalized denoisingautoencoders. Generalized denoisingautoencoders are specified by a denoising distribution for sampling an estimate of the clean input given the corrupted input. Each step of the Markov chain that generates from the estimated distribution consists of the following sub-steps, illustrated in figure :
Comments