[QuickNet] Normalization file

In the QuickNet package, there is a tool called qnnorm, which could be used to go through all the data and compute a normalization file.

IN the normalization file, two vectors are stored:

1) Mean vector - mean values for each dimension;

2) Diagonal Covariance vector - diagonal values of the covariance matrix;

Thus for each dimension of the feature vector f, the normalized value is:

where fi is the original feature value, mi is the corresponding mean and vi is the corresponding diagonal covariance value.

Posted via email from Troy's posterous

Two different RBM settings

In the thesis "Learning Deep Generative Models" Chapter 2, the RBM is used for two kinds of tasks:

1) Discrimination

2) Dimension Reduction

For Discrimination: the training of RBMs is stopped when the maximum number of epochs is reached. When used for classification, the one more layer is added to the last RBM's hidden layer for discrimination, i.e. the RBMs are not "unrolled". All the RBMs except the first one which deals with inputs directly are binary for both visible and hidden layer. Depending on the inputs, besides the binary visible units, Gaussian units etc. are also adopted.

For Dimension Reduction: RBMs are used to pre-train the autoencoder. Usually, in the final layer of the autoencoder, which corresponding to the codes extracted from the data (similar to the principle components of PCA), the number of codes should be much lower than the input dimension. To allow those low dimensional codes to make good use of continuous  variables, Gaussian units are commonly adopted. After pre-training, the RBMs are "unrolled" to form a encoder and a decoder, then the whole autoendoer is fine tuned using cross entroy errors.

Posted via email from Troy's posterous

Higher Order linear chain CRF

Download now or preview on posterous

CRFOverview.pdf (693 KB)

Posted via email from Troy's posterous

Deep Learning

Download now or preview on posterous

thesis.pdf (5055 KB)

Visual Object Recognition Using Generative Models of Images

http://learning.cs.toronto.edu/index.shtml?section=theses

http://learning.cs.toronto.edu/index.shtml?section=courses

Posted via email from Troy's posterous

Check errors: lib file not found

Sometimes, when running the program, it gives the error "*.so" lib file not found. To solve this problem, first try:

ldd <path_to_filename/filename>

to obtain info on shared libraries needed. Copy them to /usr/lib or other suitable folder, and then as root execute

ldconfig -v

Posted via email from Troy's posterous

A useful supporting material for RBM

Click here to download:

1 (439 KB)

A very useful material for RBM trainings.

Posted via email from Troy's posterous

Fine Tuning for RBMs

Download now or preview on posterous

science (1).pdf (360 KB)

Actually, fine tuning of RBM based DBN is not only using the labels to retrain the model by error back-propagation. 

For fine tuning, we re-organize the model to encoder and decoder and concatenate them together. 

For unsupervised training, we use the data as both the input and output to fine tune the model.

Similarly, for supervised training, maybe we should also use both the label and observations as input and output to retrain the model.

Details could be found in the attached paper.

Posted via email from Troy's posterous

Using Neural Networks to Classify Music

Download now or preview on posterous

IMECS2010_pp546-550.pdf (659 KB)

Using Neural Networks to Classify Music
Technology Review (06/03/10) Mims, Christopher 

A neural network built for image recognition is now able to classify music. University of Hong Kong students trained a conventional "kernel machine" neural network to recognize characteristics such as tempo and harmony from a database of songs from 10 genres, but discovered that the optimal number of layers of nodes needed to identify the musical genre was three. The adapted convolutional network was able to correctly and quickly identify a song with greater than 87 percent accuracy. Although the convoluted neural network was not able to identify songs outside of its training library, the team believes its ability to recognize 240 songs within two hours suggests that it is scalable. Cats, which have unique visual cortexes, served as the inspiration for the project. The Hong Kong project is the latest convoluted neural network based on a mammal to show a high level of flexibility. The results raise the question as to why such neural networks have not been used to address other problems involving perception in artificial intelligence.

View Full Article | Return to Headlines

Posted via email from Troy's posterous

Data visualization in Python using Scipy

http://www.scipy.org/doc/api_docs/SciPy.misc.pilutil.html#toimage

If you have numpy and scipy available (and if you are manipulating large arrays in Python, I would recommend them), then the scipy.misc.pilutil.toimage function is very handy. A simple example:

import numpy as np

import scipy.misc.pilutil as smp

# Create a 1024x1024x3 array of 8 bit unsigned integers

data = np.zeros( (1024,1024,3), dtype=np.uint8 )

data[512,512] = [254,0,0]       # Makes the middle pixel red

data[512,513] = [0,0,255]       # Makes the next pixel blue

img = smp.toimage( data )       # Create a PIL image

img.show()                      # View in default viewer

The nice thing is toimage copes with diferent data types very well, so a 2d array of floating point numbers gets sensibly converted to greyscale etc.

to save the image to file, the function smp.imsave(filename, data) could be directly used.

Also there are other kind of methods: http://stackoverflow.com/questions/434583/what-is-the-fastest-way-to-draw-an-image-from-discrete-pixel-values-in-python

An example image:

Posted via email from Troy's posterous

RBM is also called Harmonium

Download now or preview on posterous

output.pdf (1919 KB)

Using two layer Boltzmann Machine to model any distribution.

Posted via email from Troy's posterous

Non-binary units RBM

In the second paper Modeling Human Motion Using Binary Latent Variables, the author stats reasons for when to use sampled values and when to use the expected values for both visible units and hidden units in the part "2.2 Approximations", as shown below:

In the third paper, Exponential Family Harmoniums with an Application to Information Retrieval, the authors extends the binary visible and hidden RBMs to exponential family.

Download now or preview on posterous

gwtaylor_nips.pdf (334 KB)

Download now or preview on posterous

GenHarm3.pdf (526 KB)

Posted via email from Troy's posterous