bad data or over pruning

HERest -C src/ConfigHVite -I lists/train.phonemlf -t 250.0 150.0 1000.0 -S train.mfcc.list -H hmm0/macros -H hmm0/hmmdefs -M hmm1 lists/monophones1 ERROR [-7324] StepBack: File ... bad data or over pruning

Possible problems include corrupt mfcc, non-matching or non-existent labels. In this case, I had to re-calculate the mean & variance for the prototype hmm using only 1/2 the data, and the problem went away. If every file is considered bad data, you may have derived the features wrong. Go back to HCopy and check the parameters (config file).

From: http://www.ling.ohio-state.edu/~bromberg/htk_problems.html

WSJCAMP0

Some information about the WSJCAMP0 corpus:
1. Totally 140 speakers and 110 utterances per speaker;

2. 92 training speakers, 20 development test speaker and two sets of 14 evaluation test speakers. Each speaker provides approximately 90 utterances and an additional 18 adaptation utterances.

3. The same set of 18 adaptation sentences was recorded by each speaker, consisting of one recording of background noise, 2 phonetically balanced sentences and the first 15 adaptation sentences from the initial WSJ experiment.

4. Each training speaker read out some 90 training sentences, selected randomly in paragraph units. This is the empirically determined maximum number of sentences that could be squeezed into one hour of speaker time.

5. Each of 48 test speakers read 80 sentences. The final development test group consists of 20 speakers.

The CD-ROM publication consists of six discs, with contents organized as follows:

• discs 1 and 2 - training data from head-mounted microphone
• disc 3 - development test data from head-mounted microphone, plus first set of evaluation test data
• discs 4 and 5 - training data from desk-mounted microphone
• disc 6 - development test data from desk-mounted microphone, plus second set of evaluation test data

There are 90 utterances from each of 92 speakers that are designated as training material for speech recognition algorithms. An additional 48 speakers each read 40 sentences containing only words from a fixed 5,000 word vocabulary, and another 40 sentences using a 64,000 word vocabulary, to be used as testing material. Each of the total of 140 speakers also recorded a common set of 18 adaptation sentences. Recordings were made from two microphones: a far-field desk microphone and a head-mounted close-talking microphone.

http://ccl.pku.edu.cn/doubtfire/CorpusLinguistics/LDC_Corpus/available_corpus_from_ldc.html#wsjcam0

Deep Belief Networks

A introduction YouTube video: http://www.youtube.com/watch?v=AyzOUbkUf3M

On TWiki:
http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/DeepBeliefNetworks

An implementation:
PLearn:
http://plearn.berlios.de/


A tutorial on using PLearn for DBN:
http://plearn.berlios.de/users_guide/node3.html#SECTION00340000000000000000

A 3-hour video lecture:
http://videolectures.net/jul09_hinton_deeplearn/

Doing what the brain does - how computers learn to listen

Max Planck scientists develop model to improve computer language recognition

We see, hear and feel, and make sense of countless diverse, quickly changing stimuli in our environment seemingly without effort. However, doing what our brains do with ease is often an impossible task for computers. Researchers at the Leipzig Max Planck Institute for Human Cognitive and Brain Sciences and the Wellcome Trust Centre for Neuroimaging in London have now developed a mathematical model which could significantly improve the automatic recognition and processing of spoken language. In the future, this kind of algorithms which imitate brain mechanisms could help machines to perceive the world around them. (PLoS Computational Biology, August 12th, 2009) Many people will have personal experience of how difficult it is for computers to deal with spoken language. For example, people who 'communicate' with automated telephone systems now commonly used by many organisations need a great deal of patience. If you speak just a little too quickly or slowly, if your pronunciation isn’t clear, or if there is background noise, the system often fails to work properly. The reason for this is that until now the computer programs that have been used rely on processes that are particularly sensitive to perturbations. When computers process language, they primarily attempt to recognise characteristic features in the frequencies of the voice in order to recognise words. 'It is likely that the brain uses a different process', says Stefan Kiebel from the Leipzig Max Planck Institute for Human Cognitive and Brain Sciences. The researcher presumes that the analysis of temporal sequences plays an important role in this. 'Many perceptual stimuli in our environment could be described as temporal sequences.' Music and spoken language, for example, are comprised of sequences of different length which are hierarchically ordered. According to the scientist’s hypothesis, the brain classifies the various signals from the smallest, fast-changing components (e.g., single sound units like 'e' or 'u') up to big, slow-changing elements (e.g., the topic). The significance of the information at various temporal levels is probably much greater than previously thought for the processing of perceptual stimuli. 'The brain permanently searches for temporal structure in the environment in order to deduce what will happen next', the scientist explains. In this way, the brain can, for example, often predict the next sound units based on the slow-changing information. Thus, if the topic of conversation is the hot summer, 'su…' will more likely be the beginning of the word 'sun' than the word 'supper'. To test this hypothesis, the researchers constructed a mathematical model which was designed to imitate, in a highly simplified manner, the neuronal processes which occur during the comprehension of speech. Neuronal processes were described by algorithms which processed speech at several temporal levels. The model succeeded in processing speech; it recognised individual speech sounds and syllables. In contrast to other artificial speech recognition devices, it was able to process sped-up speech sequences. Furthermore it had the brain’s ability to 'predict' the next speech sound. If a prediction turned out to be wrong because the researchers made an unfamiliar syllable out of the familiar sounds, the model was able to detect the error. The 'language' with which the model was tested was simplified - it consisted of the four vowels a, e, i and o, which were combined to make 'syllables' consisting of four sounds. 'In the first instance we wanted to check whether our general assumption was right', Kiebel explains. With more time and effort, consonants, which are more difficult to differentiate from each other, could be included, and further hierarchical levels for words and sentences could be incorporated alongside individual sounds and syllables. Thus, the model could, in principle, be applied to natural language. 'The crucial point, from a neuroscientific perspective, is that the reactions of the model were similar to what would be observed in the human brain', Stefan Kiebel says. This indicates that the researchers’ model could represent the processes in the brain. At the same time, the model provides new approaches for practical applications in the field of artificial speech recognition. Original work: Stefan J. Kiebel, Katharina von Kriegstein, Jean Daunizeau, Karl J. Friston Recognizing sequences of sequences PLoS Computational Biology, August 12th, 2009.

Max Planck Society for the Advancement of Science Press and Public Relations Department Hofgartenstrasse 8 D-80539 Munich Germany PO Box 10 10 62 D-80084 Munich Phone: +49-89-2108-1276 Fax: +49-89-2108-1207 E-mail: presse@gv.mpg.de Internet: www.mpg.de/english/ Head of scientific communications: Dr. Christina Beck (-1275) Press Officer / Head of corporate communications: Dr. Felicitas von Aretin (-1227) Executive Editor: Barbara Abrell (-1416) ISSN 0170-4656

PDF (121 KB)

Contact:

Dr Christina Schröder
Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig
Tel.: +49 (0)341 9940-132
E-mail: cschroeder@cbs.mpg.de


Dr Stefan Kiebel
Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig
Tel.: +49 (0)341 9940-2435
E-mail: kiebel@cbs.mpg.de

Quicknet Activation Function in Hidden Layer

In the file, “QN_MLP_BunchFlVar.cc”, the function QN_MLP_BunchFlVar::forward_bunch(size_t n_frames, const float* in, float* out)  has the following part of codes:

// Check if we are doing things differently for the final layer.
if (cur_layer!=n_layers - 1)
{
    // This is the intermediate layer non-linearity.
   qn_sigmoid_vf_vf(cur_layer_size, cur_layer_x,
             cur_layer_y);
}
else
{
    // This is the output layer non-linearity.
    switch(out_layer_type)
    {
    case QN_OUTPUT_SIGMOID:
    case QN_OUTPUT_SIGMOID_XENTROPY:
    qn_sigmoid_vf_vf(cur_layer_size, cur_layer_x, out);
    break;
    case QN_OUTPUT_SOFTMAX:
    {
    size_t i;
    float* layer_x_p = cur_layer_x;
    float* layer_y_p = out;

    for (i=0; i<n_frames; i++)
    {
        qn_softmax_vf_vf(cur_layer_units, layer_x_p, layer_y_p);
        layer_x_p += cur_layer_units;
        layer_y_p += cur_layer_units;
    }
    break;
    }
    case QN_OUTPUT_LINEAR:
    qn_copy_vf_vf(cur_layer_size, cur_layer_x, out);
    break;
    case QN_OUTPUT_TANH:
    qn_tanh_vf_vf(cur_layer_size, cur_layer_x, out);
    break;
    default:
    assert(0);
    }
}

The activation function of MLP in quicknet tools, the activation function of hidden layers are all set to sigmoid by default.

Only the activation function can be set by users.

Logical Markov Models

Resources:

http://www.cis.hut.fi/praiko/papers/step02/node1.html

What feature normalization should be used?

From: http://tech.groups.yahoo.com/group/icsi-speech-tools/message/144

It's generally recommended to always use some kind of feature mean and
variance normalization with Quicknet.

The minimum amount of normalization is normalization using a single
set of mean and variance estimates calculated over the entire training
set. In this case, normalization simply amounts to translating the
origin of the feature space (the mean normalization) and re-scaling
the axes of the feature space (the variance normalization), since all
data is normalized using the same mean and variance estimates. This is
recommended as a minimum amount of normalization since very big or
very small numbers can cause sigmoids to saturate, and since the
optimal learning rate during MLP training may depend on the scale of
the data (and thus normalization makes it less likely you'll need to
re-tune the learning rate).

It's very common to go further and do normalization at the utterance
or speaker level. This can be useful for reducing the amount of
variability in the features due to speaker differences and channel
differences.

Speech Recognition Tools

1. quicknet
2. dpwelib
3. feacat
4. pfile_utils
5. libx11-dev
6 HTK

QuickNet II

1. Install dpwelib. Get it from http://www.icsi.berkeley.edu/~dpwe/projects/sprach/sprachcore.html (http://www.icsi.berkeley.edu/%7Edpwe/projects/sprach/dpwelib-2009-02-24.tar.gz).

Just type "./configure"
then "make"
last "sudo make install"
It's done.
You will find those tools in "/usr/local/bin" and "/usr/local/lib".

2. Install feacat. Get it from http://www.icsi.berkeley.edu/~dpwe/projects/sprach/sprachcore.html (ftp://ftp.icsi.berkeley.edu/pub/real/davidj/feacat.tar.gz).

Make sure dpwelib and quicknet are installed.
Then type "./configure --with-dpwelib=/usr/local/lib --with-quicknet=/usr/local/lib"
Next "make"
Last "make install".
It's done.
You will find the tool feacat in "/usr/local/bin".

Using QuickNet with tools not SPRACHcore

The SPRACHcore package has not been updated since 2004. We have
switched to releasing new versions of individual components of
SPRACHcore. We recommend that you use the new versions. The most recent
version of the core neural net code (QuickNet) has many speed
improvements
and other enhancements and it is available here.
The most recent version (compatible with gcc4) of the dpwelib sound utilities
is

dpwelib-2009-02-24.tar.gz.
The most recent versions of feacat, feacalc, noway, and pfile_utils (a set of 18 feature file tools including pfile_klt)
are here.

Not build the whole SPRACHcore again!!!!!!

QuickNet - 1

Platform
--------

This testbed has only been tested on an x86 Red Hat Linux platform.

NOTE: Quicknet includes versions of the qnstrn MLP training tool that
are optimized for specific architectures such as the Pentium 4 or the
AMD Opteron.  Because these different versions of the tool use
different matrix routines, there may be different patterns of rounding
errors.  Thus if you want maximum scientific comparability of
different results, don't mix different versions of the qnstrn tool in
your experiments.  (This is not expected to be a significant issue for
the forward pass tool, qnsfwd, because only training involves a
feedback process which can magnify the significance of different
rounding error patterns.)  For maximum comparability with the results
quoted in this README file, the tools/train script specifically
invokes the Pentium 4 qnstrn binary from Quicknet release 3.11, which
is named qnstrn-v3_11-P4SSE2 at ICSI.  (This binary cannot be used with older
processors which predate the Pentium 4, but it will run on an AMD
Opteron.)  To change this, edit the variable $qnstrnBinary in
tools/train.

tools/train invokes single-threaded MLP training.  On a multi-core
machine, training can be sped up by making it multi-threaded using the mlp3_threads option of qnstrn.  The most convenient way to measure
training speed is by the MCUPS number reported in the qnstrn log
file. If you use more than one thread, you will probably get more
MCUPS if you increase the value of the mlp3_bunch_size option.
However, if it is increased too much, this can reduce the quality of
the trained MLP.  The maximum bunch size before this problem occurs
depends on the corpus.

Feature calculation
-------------------

The neural net software uses the pfile feature file format, which
stores the features for many utterances together in a single file.
The SPRACHcore feacalc tool can calculate a pfile of PLP features
directly.  Pfiles can be created from other feature file formats using
the SPRACHcore feacat tool.

QUICKNET installation on Linux

1. Download the QuickNet on http://www.icsi.berkeley.edu/Speech/qn.html . ( ftp://ftp.icsi.berkeley.edu/pub/real/davidj/quicknet.tar.gz )

2. Download the ATLAS BLAS libraries from http://math-atlas.sourceforge.net/ . ( http://sourceforge.net/project/showfiles.php?group_id=23725 )

3. Download the rtst library for testing from http://www.icsi.berkeley.edu/Speech/qn.html . ( ftp://ftp.icsi.berkeley.edu/pub/real/davidj/rtst.tar.gz )

4. Download the example data files from http://www.icsi.berkeley.edu/Speech/qn.html . ( ftp://ftp.icsi.berkeley.edu/pub/real/davidj/quicknet_testdata.tar.gz )

5. Install ATLAS BLAS libraries.
5.1 Turn off CPU throttling.
Use the command: sudo /usr/bin/cpufreq-selector -g performance
Before this command, we can use the command: cat /proc/cpuinfo to see the current CUP information before changing it. Note the one "cpu MHz". Also if more than one processor is listed, do the cpufreq-selecrtor command for each processor, using the command " sudo /usr/bin/cpufreq-selector -c 1 -g performance" to turn off throttling for the second processor, as the first processor is denoted as processor 0 and is the default one when "-c " is not specified.
After issue the command turning off the CUP throttling, we can use the command "cat /proc/cupinfo" to list out the information of the CPU, then you will see that the "cpu MHz" has been increase.
5.2 After all the processors are set to the performance mode, Extract the source from the downloaded tarball file. And create a build folder under the ATLAS folder, for example named with "build".
Then first "cd" to the ATLAS folder.
cd build
../configure -b 64 -D c -DPentiumCPS=2400
"-b 64" set the target to be 64 bit, which is recommanded if no other reasons.
"-D c -DPentiumCPS=2400" tells the configuration the CPU is Cour2Core 2.4GHz.
After I type the above command, it prompts errors saying that no Fortran compiler is found. Then,
sudo apt-get install gfortran
sudo apt-get install fort77
Next redo the command "../configure -b 64 -D c -DPentiumCPS=2400".
At the last line, "DONE configure" is printed out, yes!
5.3 Type command : make build
After several steps during building, i got following errors:
ATL_dset_xp1yp0aXbX.c: Assembler messages:
ATL_dset_xp1yp0aXbX.c:96: Error: bad register name `%rsp)'
ATL_dset_xp1yp0aXbX.c:97: Error: bad register name `%rsp)'
ATL_dset_xp1yp0aXbX.c:101: Error: bad register name `%rsi)'
ATL_dset_xp1yp0aXbX.c:102: Error: bad register name `%rsi'
make[6]: *** [ATL_dset_xp1yp0aXbX.o] Error 1
make[6]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build/src/blas/level1'
make[5]: *** [dgen] Error 2
make[5]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build/src/blas/level1'
make[4]: *** [dlib] Error 2
make[4]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build/src/blas/level1'
make[3]: *** [lib.grd] Error 2
make[3]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build/src/auxil'
make[2]: *** [IStage1] Error 2
make[2]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build/bin'
ERROR 437 DURING CACHESIZE SEARCH!!.  CHECK INSTALL_LOG/Stage1.log FOR DETAILS.
make[2]: Entering directory `/home/troy/Software/quicknet/ATLAS/build/bin'
cd /home/troy/Software/quicknet/ATLAS/build ; make error_report
make[3]: Entering directory `/home/troy/Software/quicknet/ATLAS/build'
make -f Make.top error_report
make[4]: Entering directory `/home/troy/Software/quicknet/ATLAS/build'
uname -a 2>&1 >> bin/INSTALL_LOG/ERROR.LOG
gcc -v 2>&1  >> bin/INSTALL_LOG/ERROR.LOG
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.3.3-5ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --with-tune=generic --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4)
gcc -V 2>&1  >> bin/INSTALL_LOG/ERROR.LOG
gcc.real: '-V' option must have argument
make[4]: [error_report] Error 1 (ignored)
gcc --version 2>&1  >> bin/INSTALL_LOG/ERROR.LOG
tar cf error_Core264SSE3.tar Make.inc bin/INSTALL_LOG/*
gzip --best error_Core264SSE3.tar
mv error_Core264SSE3.tar.gz error_Core264SSE3.tgz
make[4]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build'
make[3]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build'
make[2]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build/bin'
Error report error_.tgz has been created in your top-level ATLAS
directory.  Be sure to include this file in any help request.
cat: ../../CONFIG/error.txt: No such file or directory
cat: ../../CONFIG/error.txt: No such file or directory
make[1]: *** [build] Error 255
make[1]: Leaving directory `/home/troy/Software/quicknet/ATLAS/build'
make: *** [build] Error 2
Having no idea to go on...
At last I found that somebody install the static library of ATLAS.
Using Synaptic Package Manager to search atlas, and select "libatlas-sse2-dev" to install. At the same time two dependencies "libatlas-headers" and "libatlas3gf-see2" are installed together.
Maybe in the command line, using sudo apt-get install libatlas-sse2-dev will also work.
From: http://seehuhn.de/pages/linear

However, after using that package, I could not find the include and lib file of ATLAS for QuickNet's installation. Thus I removed those package and retry the installation from the source.

Searching the web, I found that the error was caused by the settings of 32 bits or 64 bits. To have a try, I set the configuration to not use the default system archeticuture.
../configure -b 64 -D c -DPentiumCPS=2400 -v 2 -Si archdef 0
make build
Using these two commands, it works. What's more, after the make build command, it automatically make check, make tune and install. Even finally make clean.

In my opinion, maybe for the configuration, -b 64 is changed to be 32 will work either. Or only use the last parameter is enough "-Si archdef".

Finally, it was correctly installed after nearly 3 hours!

However, one thing must be noted is that when type the make install command, do be sure you have the root privillages. Or the operation will be denied. So for this command use "sudo make install".


6. Install QuickNet.
Extract QuickNet source files and the test data files.
"cd" to into the source folder.
mk build
cd build
../configure --with-blas=atlas --with-testdata=/home/troy/Software/quicknet/quicknet_testdata CPPFLAGS="-I/usr/local/atlas/include" LDFLAGS="-L/usr/local/atlas/lib"
make
sudo make install
In this step, errors occurs at the make step.
In file included from ../QN_utils.cc:53:
../QN_fir.h:116: error: extra qualification ‘QN_InFtrStream_FIR::’ on member ‘FillDeltaFilt’
../QN_fir.h:118: error: extra qualification ‘QN_InFtrStream_FIR::’ on member ‘FillDoubleDeltaFilt’
../QN_utils.cc: In function ‘void QN_output_sysinfo(const char*)’:
../QN_utils.cc:87: error: ‘ATL_CCVERS’ was not declared in this scope
../QN_utils.cc:88: error: ‘ATL_CCFLAGS’ was not declared in this scope
What I did is just open the "QN_fir.h" file, remove 'QN_InFtrStream_FIR::'. As in the declearation of the class member function, no need to add the class field.
For the second error, I just commented those two lines as they are just printing out some information.

Also if the option "--with-testdata" is specified, after the install, we can use the scritps built from the source with name "testdata_*.sh" to test the QuickNet.
For example, run command:
cd build
./testdata_qnstrn.sh

Finally, it is installed on my machine. Next thing is to understand how it works, so that I can use it for my experiments. Good luck to myself!

Wu Jun's Beauty of Mathematics

Beauty of Mathematics

http://jun.wu.googlepages.com/beautyofmathematics

数学之美

(Written in Chinese)

I am writing a serial of essays introducing the applications of math in natural language processing, speech recognition and web search etc for non-technical readers . Here are the links

0. Page Rank ( 网页排名算法 )

1. Language Models (统计语言模型)

2. Chinese word segmentation  (谈谈中文分词)

3. Hidden Markov Model and its application in natural language processing (隐含马尔可夫模型)

4. Entropy - the measurement of information (怎样度量信息?)

5. Boolean algebra and search engine index (简单之美：布尔代数和搜索引擎的索引)

6. Graph theory and web crawler (图论和网络爬虫 Web Crawlers)

7. Information theory and its applications in NLP  (信息论在信息处理中的应用)

8. Fred Jelinek and modern speech and language processing (贾里尼克的故事和现代语言处理)

9. how to measure the similarity between queries and web pages.  (如何确定网页和查询的相关性)

10. Finite state machine and local search (有限状态机和地址识别)

11. Amit Singhal: AK-47 Maker in Google (Google 阿卡 47 的制造者阿米特.辛格博士)

12. The Law of Cosines and news classification (余弦定理和新闻的分类)

13.  Fingerprint of information and its applications (信息指纹及其应用)

14. The importance of precise mathematical modeling (谈谈数学模型的重要性)

15. The perfectionism and simplism 繁与简 自然语言处理的几位精英

16. Don't put all of your eggs in one basket - Maximum Entropy Principles 不要把所有的鸡蛋放在一个篮子里 -- 谈谈最大熵模型(A)

17. Don't put all of your eggs in one basket - Maximum Entropy Principles不要把所有的鸡蛋放在一个篮子里 -- 谈谈最大熵模型(B)

18.  闪光的不一定是金子 谈谈搜索引擎作弊问题(Search Engine Anti-SPAM)

19.  Matrix operation and Text classification 矩阵运算和文本处理中的分类问题

20. The Godfather of NLP - MItch Marcus 自然语言处理的教父 马库斯

21. The extension of HMM, Bayesian Networks 马尔可夫链的扩展 贝叶斯网络

22. The principle of cryptography 由电视剧《暗算》所想到的 — 谈谈密码学的数学原理

23. How many keys need we type to input a Chinese character 输入一个汉字需要敲多少个键 — 谈谈香农第一定律

吴军主页的中文首页

吴军 （Jun Wu) 的英文首页

Installing HTK on Linux

 Following the instructions on the page: http://htk.eng.cam.ac.uk/docs/inst-nix.shtml

Download the necessary source files.

Besides the prerequisites, we should make sure the X11 libraries are installed. If not, type following command:
apt-get install lib11-dev

Then following the installation guides:
change to the htk folder,
type:
./configure
make all
make install

Setting up the SPHINX System

Following the instructions listed : http://www.speech.cs.cmu.edu/sphinx/tutorial.html#sphinx3tarball

For java, "sudo apt-get install java6-jdk"
For g++, "sudo apt-get install g++" and other related packages are installed by installing eclipse-cdt.