Moodle - Simple workflow

Finally, it's time to let others see the changes I have made although it just fixes a very simple bug.

The brief workflow should be as follows:

1) At the beginning of the day, check the upstream changes and push to my own repository at github.

2) Do the modification to the code.

3) git diff -U1 // check the changes

4) git status // check the correct status

5) git add [file/directory] // add the changed files or directories for commit

6) git commit -m "[track number] [module changed]: [changes]"

7) git push origin [working branch]

Then the changes are visible on the github.

Moodle - Initial try for bug fixing of MDL-36020

As a prerequisite, fixing existing Moodle bugs is crucial to the GSoC application for Moodle projects. Among the list of bugs suggested by Tim, it is hard to justify their difficulty for a beginner.

Anyway, browsing through the list and finally decide to try this one: https://tracker.moodle.org/browse/MDL-36020

From the description, it is some bug related to the re-grading function. The first step would thus to locate where these actions are processed. 

Directly diving into the codes makes me even confusing and cannot find even a start. Have to do the other way, i.e. trying the system. 

Creating a dummy course, registering a bunch of dummy users and design a set of questions to form a quiz. Using different accounts to attempt the quiz and finally the regrading button shows up.

The regrading lies in the Results section of Quiz module. In the source code it doesn't have a separate page but a set of functions. 

The codes are under the path moodle/mod/quiz/

Especially in the file moodle/mod/quiz/overview/report.php

Following are the functions responsible for the regrading:

process_actions($quiz, $cm, $currentgroup, $groupstudents, $allowed, $redirecturl)

Based on the action received, calling corresponding functions. Actions include "regrade", "regradeall", "regradealldry", "regradealldrydo".

regrade_attempt($attempt, $dryrun = false, $slots = null)

Regrade a particular quiz attempt. This is where the actual regrading happens. It is called by other methods.

regrade_attempts($quiz, $dryrun = false,

            $groupstudents = array(), $attemptids = array())

Regrade a set of attempts for this quiz. The set of attempts are specified by the input parameters. For each attempt, the regrading is done by the above regrade_attempt() function.

regrade_attempts_needing_it($quiz, $groupstudents)

Regrade all the attempts of the specified quiz that are marked as needing regrading.

count_question_attempts_needing_regrade($quiz, $groupstudents)

Count the number of attempts that needs regrading for the specified quiz.

has_regraded_questions($from, $where, $params)

Check whether there are any pending regrades to show.

clear_regrade_table($quiz, $groupstudents)

Remove all information about pending/complete regrades from database.

update_overall_grades($quiz)

Update the final grades for all attempts in the quiz.

These are probably the place to resolving the bug and a general problem for all the regrading action, we may need to fix it in the process_actions() function. 

-------------------------------------------------------------------------------

Next, how to fix the problem. From the description, the problem is due to the long processing time of the regrading which leads to the user session time out. 

Personally, I don't like the questions such as how good are you at xxx? how familiar are you with xxx? I have no idea what standards could be used to evaluated such as goodness or familiarity. No matter what standard you took, the answer will always be subjective and most of time  helpless.

Regarding Tim's question "what the session is". As I have never studies PHP in a systematical way, I only knows it keeps track of a user's information during his/her visiting of the website. And the only thing I have touched session in PHP is during last summer's GSoC. 

a) After a user logs in, the session_start() is called;

b) $_SESSION['variable']=value is used when I need to pass over some variables between different pages;

c) session_destroy() is called when the user logs out.

That's all I know about sessions in PHP.

Regarding the second question "why it might normally lock?", I have no idea. But with the help of Google, I learned following aspects:

a) From http://konrness.com/php5/how-to-prevent-blocking-php-requests/, the session locks due to multiple requests to access the same session file and could be solved by closing the session with session_write_close() before starting the long running process. This function only close the write session capability, the session variables are still readable. 

It seems a solution but how can we guarantee that there will be no session write requests after this?

Moodle - First attempt

After reading the Moodle community documents for several days, I finally decide to try it out and hope to fix several bugs before they give the final decision on my GSoC 2013 application.

Initially, I followed the steps provided in http://docs.moodle.org/dev/Git_for_developers on my current machine. First apply for those accounts and then fork a brunch on Github (https://github.com/troylee/moodle) and checking out a local copy with SHH-key setup. Then I'm totally lost where to start. That's a pretty big project with lots of source files. Need to find an IDE.

During the process of figuring out which is the popular IDE for Moodle development, I decided to use a completely clean system to do all the development. A Ubuntu 12.04 system is thus created in VirtualBox. 

1. Setup the LAMP system following instructions on https://help.ubuntu.com/community/ApacheMySQLPHP . It simply includes two steps:

First install tasksel...

    $ sudo apt-get install tasksel

... and then the LAMP stack:

    $ sudo tasksel install lamp-server

2. Then I followed http://wazai.net/2539/%E5%9C%A8eclipse%E4%B8%8A%E9%96%8B%E7%99%BCphp%E7%A8%8B%E5%BC%8F%E4%BB%A5moodle%E3%80%81egit%E3%80%81pdt%E7%82%BA%E4%BE%8B guide to setup the Eclipse environment for Moodle. 

The only change to the steps is the one that inputs "Source Git repository". Use the previously forked Github repository and my own gihub account and password. Instead of using SSH, the HTTPS is used as there is always error when I tried SSH.

3. Till now, it is almost done. Put a symbolic link under the path /var/www to the git repository folder for Moodle (for mine, it is ~/git/moodle). And make sure the folder has been granted correct permission. I simply used chmod 777 to it as this is for development not for public testing. 

Then, open the browser and type in the http://localhost/moodle . A web-based installation will automatically start for the first time. It requires the creation of a proper permission data folder moodledata, which is better to be under the /var/www path with permission right 777. 

During the installation, it will check the system environments. For my case, I the php5-curl, php5-gd, php5-xmlrpc packages are missing. Using the Ubuntu's apt-get install to get them ready and restart the apache server by /etc/init.d/apache restart. Then click the button at the bottom of the page to redo the checking until all the items are OK. 

Next, the database configuration is required. For Ubuntu, MySql is the default one for LAMP setup, the Improved MySql (native/mysqli) is thus chosen. It at first gave me errors of PHP not configured for mysqli. This was solved by uncomment the line "mysqli.allow_local_infile = On" in the /etc/php5/aphace2/php.ini as all the other lines with mysqli configurations are by default enabled.

Meanwhile, create a MySql database and user for Moodle:

Create a mysql database

    $ mysql> CREATE DATABASE moodel;

Create a mysql user

For creating a new user with all privileges on the moodle database at mysql prompt type:

    $ mysql> GRANT ALL PRIVILEGES ON moodel.* TO 'yourusername'@'localhost' IDENTIFIED BY 'yourpassword' WITH GRANT OPTION;

Next fill out the corresponding information to Moodle.

After updating the admin account information for Moodle site, the final admin workspace will be presented to you.

4. Taking Tim's suggestions on http://dev.moodle.org/mod/forum/discuss.php?d=1803 . Following configuration is done at the admin workspace:

After installing a copy of Moodle for development, the first thing you should do is:

1. Go to Site administration -> Development -> Debugging
2. Set Debug messages to DEVELOPER, and Turn on Display debug messages. (Consider turning on some of the other options too.)
3. In the administration block, search for "Cache" then
1. Turn off Cache all language strings.
2. Set Text cache lifetime to No
3. Turn on Theme designer mode
4. Turn off Cache Javascript

5. Next as the remote repository we used to setup the Eclipse is forked from Moodle and to keep it updated with changes in Moodle, the instructions in the section "Keeping your public repository update to date" on http://docs.moodle.org/dev/Git_for_developers was followed. 

Before addressing the problem, make sure the suggested git configuration is done:

Immediately after the installation, set your name and contact e-mail. The name and e-mail will become part of your commits and they can't be changed later once your commits are accepted into the Moodle code. Therefore we ask contributors to use their real names written in capital letters, eg "John Smith" and not "john smith" or even "john5677".

   git config --global user.name "Your Name"     git config --global user.email yourmail@domain.tld  

Unless you are the repository maintainer, it is wise to set your Git to not push changes in file permissions:

   git config --global core.filemode false

Then register the upstream remote:

   cd moodle     git remote add upstream git://git.moodle.org/moodle.git  

Then use following commands to keep the standard Moodle branches at your Github repository synced with the upstream repository. You may wish to store them in a script so that you can run it every week after the upstream repository is updated.

   #!/bin/sh     git fetch upstream     for BRANCH in MOODLE_19_STABLE MOODLE_20_STABLE MOODLE_21_STABLE MOODLE_22_STABLE MOODLE_23_STABLE MOODLE_24_STABLE master; do         git push origin refs/remotes/upstream/$BRANCH:$BRANCH     done

However, as in the Eclipse configuration, the git repository is using HTTPS instead of SSH, running the update shell script will require the input of both Github username and password for each push command, which is really annoying. To avoid the trouble of tying, first the SHH-Key configuration should be done (c.f. https://help.github.com/articles/generating-ssh-keys) and added to GitHub account's SHH Keys. 

Then change the repository url by:

git remote set-url origin git@github.com:[username]/moodle.git

With this successfully done, running the above update scripts will bypass the log in phase.

6. From my understanding the changes fetched from upstreams are directly pushed to the Gibhub origin. The local working copy is not affected. It may thus require to fetch/pull changes from my own Github repository and merge with my local copy. Not sure yet. To be confirmed later. 

It seems the setup is almost done to me. It's time to start fixing some bugs!

[Paper] Increased Robustness of Noisy Speech Features Using Neural Networks

This is an early paper (1999) from Nuance working on noise robust ASR.

The assumption in this paper is that the noises are additive and can be modeled simply by a bias vector either per frame or per utterance. 

This bias vector is estimated from neural networks using different knowledge sources. They found that estimating the bias vectors for log filterbanks is easier than cepstral vectors.

The paper could be found at

Increased Robustness of Noisy Speech Features Using Neural Networks

Noisy speech

As currently working on recognizing speech under noisy conditions, I'm always wondering what's the best way of solving and evaluating this problem and how human perceive speech under noisy environments.

For research purpose, we usually collect clean data and pure noise data and then generate noisy speech by combining them. Sometimes, we use some filters to create the channel distortion effects. Speech at different SNRs are created and used for evaluation. Like in the Aurora 2 dataset, 6 SNRs from 20dB to -5dB are used for evaluation.

To understand the difficulty in recognizing the noisy speech, the spectrograms of them at different SNRs are plotted. From these relatively high resolution spectrograms, at SNR0 the patterns are already quite confusing. At SNR-5 it is hard to extract speech patterns from the noise.

While in speech recognition, the FBank features used are rather low resolution to the spectrograms. Due to the value ranges, the patterns are relatively hard. That's also why usually we use CMVN to preprocess the features before sending to NNs.

   

The CMVN normalized FBank features are shown as follows. The dynamic parameters actually helps a lot to locate the patterns.

 

 Although from my experience, the dynamic coefficients are really helpful. I always have the question of whether is that because the high dimension. If we use higher dimensional static features, we will have much more detailed information, will that outperform the dynamic features? However, when I try to extract the same number of FBanks, there are several dimensions always giving 0 values. This may be saying the current feature extraction methods are in some aspects limited.
First the per utterance normalized static parts (40D) of the above features are displayed below:

 

Following are illustrations of 80 FBanks of static features.

 Although more FBanks are visually more favorable (at least to me), it is hard to say how the ASR can benefit form them. Maybe some automatically learnt features directly from the waveform signals would be helpful. 

GSoC 2012: Pronunciation Evaluation #Troy – Project Conclusions

This article briefly summarized the Pronunciation Evaluation Web Portal Design and Implementation for the GSoC 2012 Pronunciation Evaluation Project.

The pronunciation evaluation system mainly consists following components:

1) Database management module: Store, retrieve and update all the necessary information including both user information and various data information such as phrases, words, correct pronunciations, assessment scores and etc.

2) User management module: New user registration, information update, change/reset password and so on.

3) Audio recording and playback module: Recording the user's pronunciation for further processing.

4) Exemplar verification module: Justify whether a given recording is an exemplar or not.

5) Pronunciation assessment module: Provide numerical evaluation at the phoneme level (which could be aggregated to form higher level evaluation scores) in both acoustic and duration aspects.

6) Phrase library module: Allow users to create new phrases into the database for evaluation.

7) Human evaluation module: Support human experts to evaluate the users' pronunciations which could be compared with the automatically generated evaluations.

The website could be tested at http://talknicer.net/~li-bo/datacollection/login.php. Do let me know (troy.lee2008@gmail.com) once you encounter any problem as the site needs quite a lot testing before it works robustly. The complete setup of the website could be found athttp://cmusphinx.svn.sourceforge.net/viewvc/cmusphinx/branches/speecheval/troy/. More detailed functionality and implementations could be found in a more manual like report:

Although it is the end of this GSoC, it is just the start of our project that leveraging on open source tools to improve people's lives around the world using speech technologies. We are currently preparing using Amazon Mechanical Turk to collect more exemplar data through our web portal to build a rich database for improved pronunciation evaluation performance and further making the learning much more fun through gamification.

GSoC 2012: Pronunciation Evaluation Week 8 Status

In last week, thanks for Ronanki's help, I finally managed to integrate his exemplar outlier detection and pronunciation scoring scripts into our website, which makes it a more complete system. Following are the details of the two main parts:

1) Exemplar outlier detection

In the exemplar data collection part of the website, after the user recorded each phrase, the system will automatically decide whether the current recording is an exemplar or not. This process involves following steps:

a) Using ffmpeg to convert the FLV recording generated by the rtmplite to WAV file;

b) MFCC feature extraction using sphinx_fe;

c) Do alignment to get the phonetic transcription of the current phrase using sphinx_align;

d) Decode the current recording with edit distance grammar using sphinx_decode;

e) Compute phone error rate for the recognition results compared to the alignment;

f) Based on the phone error rate, the system will decide whether the current recording is an exemplar or not. Currently we require the phone error rate to be less than 50% for an exemplar, which could be further tuned for different cases.

g) The system then either navigate to next phrase for recording or ask the user to re-try the recording for the current phrase according to the above decision.

In the other part of the website, i.e. the student section, two new functionalities are added.

2) Exemplar recordings for student to study

When the student select a phrase for practice, maximum 5 exemplar recordings for this phrase from the database are selected and listed for the student to mimic and learn. Currently, the selection is randomly done. After we collected more exemplar recordings, it could be improved by selecting the top scored exemplars.

3) Student pronunciation scoring

Another important feature for student page is to score the recording he/she recorded. This is done with following steps:

a) Convert the FLV files from rtmplite to WAV using ffmpeg;

b) MFCC feature extraction using sphinx_fe;

c) Generate the phonetic alignment using sphnix_align;

d) Comparing the aligned phone dependent statistics with the average statistics from exemplars to evaluate the student's performance. 

e) Present both the acoustic and duration scores and navigate to next phrase.

GSoC 2012: Pronunciation Evaluation Week 7 Status

Hi All,

Last week, I was still working on the data collection website.

Thank Robert (butler1970@gmail.com) so much for trying out the website and listed the issues he encountered on this page: https://www.evernote.com/pub/butler1970/cmusphinx#b=11634bf8-7be9-479f-a20e-6fa1e54b322b&n=398dc728-b3f0-4ceb-8ccf-89295b98a6d7

Issue #1: The under construction of Student Page

The first stage of the website to collect exemplar recordings, thus the student page is not implemented at that time. 

Issue #2: The inconvenient birthdate control

The birthdate control is now replaced with the standard HTML5 <input type="datetime"> control. Due to the datetime input control is a new element in HTML5, currently only Chrome, Safari and Opera support the popup date selection. On other browsers, which have on support yet, the control will simply be displayed as an input box. The user can just type in the date and the background script will check whether the format is correct or not.

Issue #3: The incorrect error message "Invalid date format" on the additional information update page

After digging into the source code to find the problem for several hours, the bug lies in the order of invoking mysql related functions. The processing steps in the additional information update page is as follows:

a) client side post the user input information to the server;

b) server side first using mysql_escape_string function to preprocess the user information to ensure the security of later mysql queries;

c) check the format of each field including the date time format, whether the user inputs a valid date;

d) update the mysql database with the new information.

As only in step d) the mysql sever action is needed, I thus put the database connection code behind step c), without knowing the mysql_escape_string function also requires mysql database connection. In the previous implementation, the mysql_escape_string returns empty string thus leads to invalid date format. 

Secondly, the exemplar recording page is update with following features:

1) Automatically move to the next utterance after the user record and playback the current recording;

2) Adding extra navigation control for recording phrase selection;

3) When the user opens the exemplar recording page, the first un-recorded utterance will be set to the first one shown the user.

4) Connection the enable and disable of recording and playback buttons of the player with the database information, i.e. if the user has recorded the phrase before, both the recording and playback buttons are enabled, otherwise only recording is allowed.

The third major part done in last week is the student page which is previously left empty.

For the student page, users now can also practice their pronunciation by recording the phrases in the database and also listening to the exemplar recordings in the system. The features are:

1) Full recording and playback functionalities as exemplar recording;

2) When navigating to each phrase, randomly maximum 5 exemplar recordings from the system are retrieved from the database and listed on the page to help the students. 

3) Additionally, to put some exemplar recordings in the system, I have to manually transcribe several sentences and put the recordings into the system for use. After there are many people contributing to the exemplar recordings, I don't need to do manually transcription any more.

For this week, two major tasks to be done: integration with Ronanki's evaluation scripts and mid-term report. 

Regards,

Troy

Troy: GSoC 2012 Pronunciation Evaluation Week 5

Sorry for the late update. Following are the stuff done in week 5, mainly problem solving.

1) Solving the Flash based recorder doesn't allow users to enable their microphone access problem. 

At the very beginning (before the Flash player 11.2 and 11.3 update), the audio recorder I created using Flex works fine. Users can simply right click the recorder and select the "Settings" to allow microphone access. However, with the new updates, that option is disabled without giving any information. 

To solve it, many people suggest adding the websites into the online global privacy list. However, after trying many times still not working for my case. 

Furthermore, checking the http://englishcentral.com/ which also is Flash based recording suddenly gives me a clue. In their website, after clicking recording button (which is a microphone image), a popup window shows up with the Flash Microphone privacy setting dialogue. Yes! Instead of finding the problem disabled the "Setting" option for Flash object, why not checking the accessibility of microphone in code and prompt the setting dialogue when necessary. Here comes the solution:

First, checking whether the microphone is available, if not show the microphone list dialogue of Flash object ask the user to plugin a microphone:

var mic:Microphone = Microphone.getMicrophone();

if(!mic) {

Alert.show("No microphone available");

debug("No microphone available");

Security.showSettings("microphone");

}

Otherwise, check whether the microphone is accessible or not, if it is muted, prompt the privacy dialogue to ask user to allow the microphone access:

if(mic.muted) {

debug("Microphone muted!");

Security.showSettings("privacy");

}

With these testing during the initialization stage of the Flash recorder, it can allow users to enable the microphone access at the early beginning. One interesting thing is that after doing this, the "Setting" option of the Flash object now is clickable. 

Now, looking back to the code solving the problem, which is so apparent, however, before you know the answer, it is really hard to predict. 

2) Cross-browser Flash recorder compatibility

As the Flash recorder problem was finally solved using the above method, I was so happy to update the trunk and our server and hoped to see the site working nicely. But the browser shows that the Flash recorder cannot load, the only information I got is "Error 2046". ....

Although rather depressed, I still have to solve the problem. Googled a bunch of pages and tried several suggestions, the one that first clear the browser cache and then set the Flash player not to save local cache and then re-enable its local cache (some kind of clear Flash player local cache), gives some progress by changing the "Error 2046" to "Error 2032". 

For "Error 2032", there are mainly two groups of explanations, one said there are something wrong with the URLs in your Actionscript's HTTPRequests, which is not my problem as those URLs are definitely correct and are under the same folder as the player. The other is the RSL problem of Flash compiler. To solve the RSL linkage problem, go to the "Flex Build Path" properties page, "Library path" tab and change the framework linkage to "merged into code".

3) Adding password change page

4) Refine the user extra information update page to reflect the existing user information if available, instead of always showing the default values.

Till now, the website for exemplary recordings are finally come to a usable stage. 

In this week, I will try to accomplish followings:

1) Prompts adding page for administrators;

2) Design recording prompts to start our exemplary recording data collection;

3) Bug fixing and system testing;

4) Study the Amazon Mechanical Turk and start thinking how to incorporate our recording onto that platform.

Troy: GSoC 2012 Pronunciation Evaluation Week 4

Finally, the data collection website now can provide the basic capabilities! Anyone who are interested, check out our website at http://talknicer.net/~li-bo/datacollection/login.php and have a try. If you encountered any problems, do let us know. Following are the stuff I have done during the last week:

1) Discussed with my mentor James to finalize the schema design and created the whole database with MySQL. The whole database design could be found http://talknicer.net/w/Database_schema .  During the development of the website, slightly modifications were carried out to refine the database design. Such as the age field for the users table, only when I try to insert user values into the table did I realize the age value is dependent on the registration date which may not be a good idea to store age. Storing birthdate would be much better. Similar changes like that are updated. What I learnt from these is that a good design comes from practice not purely imagination. 

2) Implement the two types of user registration page: one for student and one for exemplary. As we don't want to constrain the two types of users to be exclusive and to avoid redundant work, the registration involves two steps: one basic registration and one extra information update. For student, only the basic one is compulsory, but for the exemplary they have to finish both the two forms. 

3) Adding extra supporting functionalities for user management. These including: password reset, mode selection for users with multiple types. 

4) Incorporating the audio recorder with the website for recording and uploading to servers. 

Things to do this week:

1) Prompts adding page;

2) Testing the system; 

3) Design the pronunciation learning game for student users.

Troy: GSoC 2012 Pronunciation Evaluation Week 3

Due to some personal stuff, I haven't made much progress during the 3rd week of GSoC. I will try my best to catch up in this week.

Things done:

1. Tailor the previous audio recorder to provide only the recording and playback functionality and leave interfaces for Javascript to communicate with the web site pages. 

2. Discuss with my mentor regarding the database design. 

Things to do this week:

1. Fix schema for prompts to handle word lists with pronunciation and parts of speech (along with a separate text string for display which might not have as clear word boundaries because of punctuation--such as this--etc.)

2. Get separate registration interface for exemplar uploaders

  

3. Get an interface to add prompts

  

4. Do the interface to upload recordings for prompts

5. Think about the game play and do its schema once the basic features are decided

Troy: GSoC 2012 Pronunciation Evaluation Week 2

Following are things done in the second week of GSoC 2012:

1. Setup the server rtmplite to automatically check whether the process is still running or not. If it is stopped, restart it. 

To accomplish this, first of all, create a .process file under my home folder and put the current rtmplite process id as the only content of this .process file. You can use 'top' or 'ps' to find out the current process id of your application. 

Then I created following script file to do the status checking:

if [ -e "$pidfile" ]  then	  	# check whether the process is running  	rtmppid=`/usr/bin/head -n 1 ${pidfile} | /usr/bin/awk '{print $1}'`;  	  	# restart the process if not running  	if [ ! -d /proc/${rtmppid} ]  	then  		/usr/bin/python ${exefile} -r ${dataroot} &  		rtmppid=$!  		echo "${rtmppid}" > ${pidfile}  		echo `/bin/date` "### rtmp process restarted with pid: ${rtmppid}"  	fi  fi

In this script, first we will check whether the .process files ( i.e. the $pidfile) exists or not. If we don't want the server to check for this process for now (maybe when we apply patches to the program), we could simply delete this file and it won't check the process again. And after the maintenance, recreate the file with the new process id. The checking will automatically going on.

The checking itself is quite simple: getting the process id from the file and see whether the process exists by looking into the /proc system folder where each running process will have a folder. Goolge the '/proc linux' you will get more information about this mystery folder which contains quite a lot information about your system. 

2. Implement the login and registration pages using HTML5.

First for user information storage, we use MySQL database, thus a user table is designed and created in the server's mysql database:

      Field
Type
      
Comments
      
userid
      INTEGER
Compulsory, automatically increased, primary key
      
email
      
VARCHAR(200)
      Compulsory, users are identified by emails
      password
VARCHAR(50)
      
Compulsory, encrypted using SHA1, at least 8 alphanumeric characters
      
name
      
VARCHAR(100)
      Not compulsory, default 'NULL'
      age
INTEGER
      
Not compulsory, default 'NULL', accepted values [0,150]
      
sex
      
CHAR(1)
      Not compulsory, default 'NULL', accepted values {'M', 'F'}
      native
CHAR(1)
      
Not compulsory, default 'NULL', accepted values {'Y', 'N'}. Indicating the user is a native English speaker or not.
      
place
      
VARCHAR(1000)
      Not compulsory, default 'NULL'. Indicating the place when the user lived at the age between 6 and 8.
      
accent
      CHAR(1)
Not compulsory, default 'NULL', accepted values {'Y', 'N'}. Indicating the user has a self-reported accent or not.  
  

The creation of the database:

CREATE TABLE users   (  	userid INTEGER NOT NULL AUTO_INCREMENT,  	email VARCHAR(200) NOT NULL,  	password VARCHAR(50) NOT NULL,  	name VARCHAR(100),  	age INTEGER,  	sex SET('M', 'F'),  	native SET('Y', 'N') DEFAULT 'N',  	place VARCHAR(1000),  	accent SET('Y', 'N'),  	CONSTRAINT PRIMARY KEY (userid),  	CONSTRAINT chk_age CHECK (age>=0 AND age<=150)  );
Secondly, the login and simple registration pages are implemented in HTML5 which I have to learn in practice. Follows are the screenshots of the pages:
      

  

Also if you are interested, you can go to this page to help us test the system: http://talknicer.net/~li-bo/datacollection/login.php . On the server side we use PHP to retrive information from the page and do the query in mysql database and finally sending the data back to the page. 
    
The recording interface, has also been ported to use HTML instead of pure Flex as I did before. The page current shows up OK but no event interaction between HTML and Flash yet. 
    

3. Database design for the whole project. 
A bunch of tables are designed to store various information for this project. Detailed table information could be found on our wiki page: http://talknicer.net/w/Database_schema . Here I will give a brief discussion. First the user table created in the previous step will be augmented to keep two kind of user information: one for normal student user and one for exemplar recordings. The reason to put them into one table instead of two is that student users, when they can do an excellent job in pronunciation, should also be allowed to contribute to the exemplar recording. Also for exemplar recorders, if they register through the website, they have to show they are proficient enough to contribute a qualified exemplar recording. 

Beside the user table, there are several other tables to for necessary information such as languages for list of languages defined by ISO in case we may extend our project to other languages; region table to have an idea of the user's accent; prompts table for the list of text resources will be used for pronunciation evaluation. 

Then are the tables to log the recordings the users do and tables for set of tests designed in the system. 

Things to do in the coming week:

1. Discuss more regarding the game part to finish the last part of schema design. 

2. Figure out how to integrated the Flash audio recorder nicely with the HTML5 interface. 

3. Implement the student recording interface.

4. Further tasks could be found on this list: http://talknicer.net/w/To_do_list

First use of Bazaar and LaunchPad.net

1. Check the launchpad brunch:

bzr branch lp:~troy-lee2008/pronunciationeval/branch-troy

2. Update to launchpad:

bzr push lp:~troy-lee2008/pronunciationeval/branch-troy

This should be executed everytime you want your local changes be reflected in the server.

3. Useful operations

bzr merge

bzr add [file/directory]

bzr mkdir [directory]

bzr mv [file/directory...]

bzr rm [file/directory]

bzr commit -m "message"

Most of the time, the operations go in following steps:

a) bzr merge: to sync to the most recent revision with the server

b) bzr add, mv, rm etc. to make necessary changes

c) bzr commit to commit changes to your local repository

d) bzr push to submit your changes to the server repository

The initial bunch creation, I mainly reference from http://doc.bazaar.canonical.com/latest/en/mini-tutorial/index.html and https://help.launchpad.net/Code/UploadingABranch

Some other helps:

http://doc.bazaar.canonical.com/latest/en/tutorials/using_bazaar_with_launchpad.html

[GSoC 2012: Pronunciation Evaluation #Troy] Week 1

The first week of GSoC 2012 has already been a busy summer. Following are the stuff I managed to accomplish:

1. The server side rtmplite is now configured to save the recordings into the folder [path_to_webroot]/data on the server. And for the audioRecorder app, all the data will be under the [path_to_webroot]/data/audioRecorder folder and for each user there will be a separate folder (e.g. [path_to_webroot]/data/audioRecorder/user1). For each recording utterance, the file name is now in the format of [sentence name]_[quality level].flv
2. Till now the conversion from FLV to WAV is done purely on the server side inside rtmplite with Python's subprocess.Popen() function calling FFMPEG. After the rtmplite close the FLV file, the conversion will be carried out immediately and the converted WAV file has exactly the same path and name except the suffix, which is WAV instead of FLV. Thanks very much for Guillem to help me test "sox" to do the conversion. However, I failed to use sox directly to convert FLV to WAV in the terminal, which said "unknown format flv". If needed I will try to figure out whether it is the problem of the sox I have.  James then pointed out we can do it inside rtmplite. This really helps. Yes, why must send an extra HTTP request to invoke a PHP process for the conversion? 
3. To verify the recording parameter, i.e. the quality for speex encoding, I tried to record the same utterance ("NO ONE AT THE STATE DEPARTMENT WANTS TO LET SPIES IN") with the quality varying from 0 to 10. Apparently, the higher the quality, the larger the FLV file will be. From my own listening, the better the quality. However, it is hard to notice the differences above level 7. I also tried to generate alignment scores to see whether the quality affects the alignment. However, from the results shown in the following graph, the acoustic scores seems incomparable among different recordings. However, we will for now set the recording quality to 8. 
4. For the audioRecorder, only when the NetConnection event and NetStream open and close events successfully finished will the UI and other events carry on. Also 0.5s delay is inserted into the starting and ending of the recording button click event to avoid clipping. 

For the 2nd week,

1. Solve the problem encountered in converting FLV to WAV using FFMEPG with Python's Popen(). If the main Python script (call it test.py for now) is run in the terminal with "python test.py", then no problems, everything works great. However, if I want to put it in background and log off the server by doing "python test.py &", everytime when Popen() is invoked, the whole process hangs there with a "Stopped + test.py &" error message. I have tried different approaches found from web without success for two days. I will try to figure out a way to work around it. Otherwise, maybe turn to Guillem's suggestion and figure out whether sox works.
2. Finish the upload interface. There will be two kinds of interfaces: one for students and one for exemplar pronunciations. For the students': we want to display one to five phrases below space for a graphic or animation, assuming the smallest screen possible but with HTML which also looks good in a large window. For the exemplar, we just need to display one phrase but we should also have per-upload form fields (name, age, sex, native English speaker?, where lived ages 6-8 (determines accent), self-reported accent, etc.) which should persist across multiple uploads by the same user (with cookies?)
3. Testing the rtmplite for multiple users and the same user's multiple recording sessions.

First use of SVN

Due to the requirement of GSoC 2012, we need to check in our code in the cmusphinx subversion. Here is what I tried. 

Step 1: Creating a new brunch:

svn mkdir https://cmusphinx.svn.sourceforge.net/svnroot/cmusphinx/branches/speecheval

Then a log message will be opened with vi, just add some comments to it, save and quit. Next press 'c' and Enter, user authentication will be prompted, if the username is not your SVN username, just press Enter and then it will ask for both username and password. 

After that, seeing the message "Committed revision 11369" would probably indicating a successful operation. 

However, I did receive an email saying that the operation "Is being held until the list moderator can review it for approval.". Browsing the online SVN branches, the folder is already there. 

Step 2: Check out the brunch:

svn co https://cmusphinx.svn.sourceforge.net/svnroot/cmusphinx/branches/speecheval speecheval

Some other commands useful:

svn add [filename/folder]

svn delete [filename/folder]

svn commit -m "message"

[GSoC 2012] Before Week 1

GSoC 2012 has already officially started since this Monday (21 May). Although the expected weekly report should be starting next Monday, it would be better to have a brief overview of the preparations we have done during the bonding period.

The projects starts with the group chat with our mentor James and the other student Ronanki. From the chat together with the following email communications, the project is becoming more and more clear for me. For my project, the major focuses would be:

1) a web portal for automatic pronunciation evaluation audio collection;

2) Android based mobile automatic pronunciation evaluation app.

The core of these two applications is the edit distance grammar based automatic pronunciation evaluation using cmusphinx3, which would server as both the foundation and prior. 

Following are the preparations I have done during the bonding period:

1. Trying out the basic wami-recorder demo on my school's server;
2. Change to the rtmplite for audio recording. rtmplite is a Python implementation of the Flash RTMP server with minimum support needed for real-time streaming and recording using AMF0. On the server side, the daemon RTMP server process will by default listen on the TCP 1935 port for connection and streaming. On the client side, the user needs to use NetConnection to setup a session with the server and using NetStream for audio and video streaming and also recording. The demo application has been set up at: http://talknicer.net/~li-bo/testClient/bin-debug/testClient.html
3. Based on the understanding of the demo application, which does the real time streaming and recording of both audio and video, I started to write my own audio recorder which is the key component for both the web-based audio data collection and the evaluation app. The basic version of the recorder was hosted at: http://talknicer.net/~li-bo/audioRecorder/audioRecorder.html . The current implementation includes:
1. Distinguish recordings from different users with user id;
2. Pre-defined text sentences loading for recording, which may be useful for the data collection;
3. Real-time audio recording;
4. Playback the recordings from the server.
5. Basic event control logic, such as prevent users from recording and playing in the same time etc.
4. Besides, I have also learnt from http://cmusphinx.sourceforge.net/wiki/sphinx4:sphinxthreealigner on how to do alignment using cmusphinx3. To generate the phoneme alignment scores, two steps of alignments are needed. Regarding the details of how to carry out the alignment can be found on my more tech-oriented posts (http://troylee2008.blogspot.com/2012/05/testing-cmusphinx3-alignment.html and http://troylee2008.blogspot.com/2012/05/cmusphinx3-phoneme-alignment.html) on my personal blog.

Currently, there are followings thing ongoing:

1. Setup the server side process to well manage the user recordings, i.e. distinguishing between users and different utterances.
2. Figuring out how to automatically convert the recorded server side FLV files to WAV files after the user stop the recording. 
3. Verify the recording parameters against the recording quality and also taking the network bandwidth into consideration.
4. Incorporating delays between network events in the recorder. The current version does not wait for the network events (such as connection set up, data package transmission etc) to successfully finish to process next user event, which usually cause the recordings to be clipped out. 

Configure Ubuntu12.04 to boot without a monitor

Normally, the Ubuntu 12.04 desktop system is for PC which is assumed to be connected with a monitor, a keyboard and a mouse. As I just want to remotely login to the computer, I just removed those devices and leave the machine with only power and network. Everything works fine before I need to reboot the machine remotely. Then I can never connected it again before I reconnect it with a monitor and reboot it.

To solve the problem, I finally find this post: http://ubuntuforums.org/showthread.php?t=1452600&page=3 . The solution is:

Step 1. Back up the original xorg.conf to xorg.conf.bk just in case. Create a new xorg.conf in /etc/X11 with the following. 

Section "Device"
Identifier "VNC Device"
Driver "vesa"
EndSection

Section "Screen"
Identifier "VNC Screen"
Device "VNC Device"
Monitor "VNC Monitor"
SubSection "Display"
Modes "1024x768"
EndSubSection
EndSection

Section "Monitor"
Identifier "VNC Monitor"
HorizSync 30-70
VertRefresh 50-75
EndSection

Step 2. Disable KMS for your video card

The list is to know which video card manufacturer you have and use the command line entry below it to create the appropriate kms.conf file with the line "options...modeset=0" line in it. If you have access to the GUI you could just are easily create/modify the file and put the "options...modeset=0" in as appropriate.

The following are input into the terminal windows as a line command.

# ATI Radeon:
echo options radeon modeset=0 > /etc/modprobe.d/radeon-kms.conf

# Intel:
echo options i915 modeset=0 > /etc/modprobe.d/i915-kms.conf

# Nvidia (this should revert you to using -nv or -vesa):
echo options nouveau modeset=0 > /etc/modprobe.d/nouveau-kms.conf

As for my case, mine is Intel, I add "option i915 modeset=0 " to /etc/modprobe.d/dkms.conf .

Step 3. Reboot