Two different RBM settings

In the thesis "Learning Deep Generative Models" Chapter 2, the RBM is used for two kinds of tasks:

1) Discrimination

2) Dimension Reduction

For Discrimination: the training of RBMs is stopped when the maximum number of epochs is reached. When used for classification, the one more layer is added to the last RBM's hidden layer for discrimination, i.e. the RBMs are not "unrolled". All the RBMs except the first one which deals with inputs directly are binary for both visible and hidden layer. Depending on the inputs, besides the binary visible units, Gaussian units etc. are also adopted.

For Dimension Reduction: RBMs are used to pre-train the autoencoder. Usually, in the final layer of the autoencoder, which corresponding to the codes extracted from the data (similar to the principle components of PCA), the number of codes should be much lower than the input dimension. To allow those low dimensional codes to make good use of continuous  variables, Gaussian units are commonly adopted. After pre-training, the RBMs are "unrolled" to form a encoder and a decoder, then the whole autoendoer is fine tuned using cross entroy errors.

Posted via email from Troy's posterous