(1st-Dec-2020)
• Because the RBM admits efficient evaluation and differentiation of ˜ P(v) and efficient MCMC sampling in the form of block Gibbs sampling, it can readily be trained with any of the techniques described in chapter for training models 18 that have intractable partition functions. This includes CD, SML (PCD), ratio matching and so on. Compared to other undirected models used in deep learning, the RBM is relatively straightforward to train because we can compute P(h | v) exactly in closed form. Some other deep models, such as the deep Boltzmann machine, combine both the difficulty of an intractable partition function and the difficulty of intractable inference.
Deep belief networks (DBNs) were one of the first non-convolutional models to successfully admit training of deep architectures (Hinton 2006 Hinton et al., ; , 2007b). The introduction of deep belief networks in 2006 began the current deep learning renaissance. Prior to the introduction of deep belief networks, deep models were considered too difficult to optimize. Kernel machines with convex objective functions dominated the research landscape. Deep belief networks demonstrated that deep architectures can be successful, by outperforming kernelized support vector machines on the MNIST dataset ( , ). Today, deep belief Hinton et al. 2006 networks have mostly fallen out of favor and are rarely used, even compared to other unsupervised or generative learning algorithms, but they are still deservedly recognized for their important role in deep learning history.
Comments