(8th-Dec-2020)
• While the Gaussian RBM has been the canonical energy model for real-valued data, ( ) argue that the Gaussian RBM inductive bias is not Ranzato et al. 2010a well suited to the statistical variations present in some types of real-valued data, especially natural images. The problem is that much of the information content present in natural images is embedded in the covariance between pixels rather than in the raw pixel values. In other words, it is the relationships between pixels and not their absolute values where most of the useful information in images resides. Since the Gaussian RBM only models the conditional mean of the input given the hidden units, it cannot capture conditional covariance information. In response to these criticisms, alternative models have been proposed that attempt to better account for the covariance of real-valued data. These models include the mean and covariance RBM (mcRBM1), the mean-product of t-distribution (mPoT) model and the spike and slab RBM (ssRBM).
• Convolutional Boltzmann Machines
Aextremely high dimensional inputs such as images place 9 great strain on the computation, memory and statistical requirements of machine learning models. Replacing matrix multiplication by discrete convolution with a small kernel is the standard way of solving these problems for inputs that have translation invariant spatial or temporal structure. ( ) Desjardins and Bengio 2008 showed that this approach works well when applied to RBMs. Deep convolutional networks usually require a pooling operation so that the spatial size of each successive layer decreases. Feedforward convolutional networks often use a pooling function such as the maximum of the elements to be pooled. It is unclear how to generalize this to the setting of energy-based models.
Comments