Tuesday, November 16, 2010
Friday, November 12, 2010
Tuesday, November 9, 2010
[DBN] Machine learning for sequential data: a review
Download now or preview on posterous
Dietterich_2009_Machine Learning for Sequential Data A Review.pdf (145 KB) For sequential problems, the sequences exhibit significant sequential correlation. That is, nearby x and y values are likely to be related to each other.
To model the sequential correlations, we usually adopt the joint probability for the whole sequence as the objective function for learning.
Methods that analyze the entire sequence of x_t values before predicting the y_t labels typically can give better performance on the sequential supervised learning problem.
Non-uniform loss functions usually represented by a cost matrix C(i,j), which gives the cost of assigning label i to an example whose true label is j. In such cases, the goal is to find a classifier with minimum expected cost.
How can these kind of loss function be incorporated into sequential supervised learning? One approach is to view the learning problem as the task of predicting the (conditional) joint probability of all the labels in the output sequence: P(y|x). If this joint distribution can be accurately predicted, then all of the various loss functions can be evaluated.
We would like to find methods to extract features handling long distance interactions.
Approaches reviewed in this paper:
1) The sliding window method;
2) Recurrent sliding window
3) Hidden Markov Models
4) Maximum Entropy Markov Models
5) Input-Output Markov Models
6) Conditional Random Fields
7) Graph Transformer Networks
Sunday, November 7, 2010
[DBN] Advance machine learning lecture notes - III
Regarding the layer size for a deep belief network:
The layers should not get smaller and they should be initialized correctly.
For visible and hidden units, whether to use binary values or probabilities:
Only the first hidden units values are binary values, others like v0, v1, h1 adopt real valued probabilities.
How to measure the quality of the models as the probability is hard to compute due to the partition function:
[DBN] Energy based model for sparse overcomplete representation
Download now or preview on posterous
Teh et al._2003_Energy-based models for sparse overcomplete representations.pdf (691 KB) In this paper, the authors proposed the energy based models for overcomplete sparse representation problems.
They compared it with the Independent Component Analysis and showed some similarity between those training approaches.
1) The causal generative approach for ICA
2) The information maximization approach for ICA
3) The energy based approach
[DBN] Advance machine learning lecture notes - II
Comparison between two model combination approaches:
PoE learning rule:
The derivation from the second equation to the third one is detailed in the attached file.
Subscribe to:
Posts (Atom)