Archive

Archive for November, 2011

Ubuntu 11.10 performance tuning

November 19, 2011 4 comments

I recently upgraded my netbook’s OS to Ubuntu 11.10 and noticed that it’s taking longer than usual to launch programs that I use quite often. For example, it takes 7-8 seconds to launch Google Chrome, about 4 seconds to open terminal. Here’s what I did to make the os run faster without investing with a SSD disk (My root partition is using ext4 fs, skip step 1 through 3 if you are using file systems other than ext3 or ext4):
1. Add the following bold part to the root partition line in /etc/fstab
errors=remount-ro,noatime,data=writeback

2.

tune2fs -o journal_data_writeback /dev/sda5

/dev/sda5 is the root partition.

3. change the following line in /etc/default/grub
GRUB_CMDLINE_LINUX=”rootflags=data=writeback
(bold part is added)

4. Run update-grub
5. Add the following line to /etc/sysctl.conf
vm.swappiness=10

NOTE: all the above changes require root privileges.

6. Reboot

The difference? It now takes about 3-4 seconds to launch Google Chrome from cold start.

Categories: howto, ubuntu

A simple node.js rss parser using sax-js

November 15, 2011 Leave a comment

The xml parser sax-js written by Issacs (the creator of npm, the de facto package manager of node.js) comes with a few examples that deal with local xml files. I couldn’t find one that can parse xml data from remote host (think RSS) therefore I decided to write one. In this example I borrowed codes heavily from both sax-js’s example code and node-rss source code.

The codes

cat saxrss.js

var sax=require('sax');
var http=require('http');

var callback=function(){};

exports.get_rss=function(host,port,path, cb) {
	callback=cb
	var parser = sax.parser(true)
	var item = null
	var currentTag = null
	var items=[]
	var cnt=0

	parser.onclosetag = function (tagName) {
		var tag_name=tagName.toLowerCase();
		if (tag_name === 'item' || tag_name === 'entry') {
			currentTag = item = null
			cnt++
			return
		}
		if (currentTag && currentTag.parent) {
			var p = currentTag.parent
			delete currentTag.parent
			currentTag = p
		}
	}

	parser.onopentag = function (tag) {
		var tag_name=tag.name.toLowerCase()
		if (tag_name !== 'item' && tag_name !== 'entry' && !item) return
		if (tag_name === 'item') {
			item = tag
				items[cnt]={}
		}
		tag.parent = currentTag
		tag.children = []
		tag.parent && tag.parent.children.push(tag)
		currentTag = tag
	}

	parser.ontext = function (text) {
		if (currentTag) {
			items[cnt][currentTag.name.toLowerCase()]=text
		}
	}

	parser.onend = function () {
		callback(items)
	}

	var body='';
	http.get( { host:host, path:path, port:port }, function(res) {
		res.addListener('end', function() {
			parser.write(body).end()
		});
		res.setEncoding('utf8');
		res.on('data', function(d) {
			body+=d;
		});
	});
}

cat test1.js

var rss=require('./saxrss.js');
var host='feeds.finance.yahoo.com';
// to get finance headlines about stock AAPL
var path='/rss/2.0/headline?s=aapl&region=US&lang=en-US';

rss.get_rss(host, 80, path, function(items) {
	console.log(items);
});
To run

node test1.js

Required node modules:

sax

References:

node-rss
sax-js

[ UPDATE 2/20/2012 ]
With xml-simple module, the above example can be written as

// getting xml and convert to json object using xml-simple example
var http=require('http'), simplexml=require('xml-simple'), config= {host:'feeds.finance.yahoo.com', path:'/rss/2.0/headline?s=aapl&region=US&lang=en-US', port:80}, body='';

http.get( config, function( res ) {
	res.addListener('end', function() {
		simplexml.parse(body, function(e, parsed) {
			console.log(parsed.channel.item);
			//console.log(JSON.stringify(parsed));
		});
	});
	res.setEncoding('utf8');
	res.on('data', function(d) {
		body+=d;
	});
});

To install xml-simple, simply npm install -g xml-simple.

tmux techniques by example

November 14, 2011 2 comments

I am a big fan of tmux – a terminal multiplexer. Think of it as a text version of vnc client, with many more powerful features. In this post I will demo some of the tmux techniques that I use quite often.

Assumptions:

1) GNU version of tmux
2) Default shell is BASH

Preparation:

For demo’s purpose I make up a dummytask.sh to simulate the task(s) that we will be running in tmux windows:

#!/bin/bash
taskname=$@
ans=''
if [ -n "$taskname" ]; then
	while [ ! "$ans" == "q" -a ! "$ans" == "Q" ]; do
		read -e -n 1 -p "Running task $taskname, to exit, press q(Q). " ans
	done
	echo "Done task $taskname."
else
	echo "Usage: $0 task."
	echo "Example1: $0 debugging"
	echo "Example2: $0 importing data"
fi
Example: create a tmux session with session name mysess
tmux new-session -s mysess -d

If the option -d (detached) is omitted, you will be taken directly to the first window titled “0:bash” once the command is executed and any commands afterwards will be entered into that window. Therefore it’s a good habit to use option -d whenever creating a new session.

Example: create a new tmux session and change the first default window title to task1

[ type q, Enter, exit, Enter if the sess tmux session is currently attached ]
In the first example, tmux will create a first window titled “0:bash” (could be ksh, csh etc depending on default shell setting) by default, to change to something else, simply using -n (name) option:

tmux new-session -s sess -d -n task1
Example: create tmux session mysess if it has not been created yet
tmux list-session 2>&1 | grep -q "^mysess:" || tmux new-session -s sess -d

Notes: 2>&1 is to suppress error output “failed to connect to server: Connection refused”, which occurs when there are no tmux sessions running. -q is used to suppress the normal output of of grep. It won’t affect the result but using it makes the commands less distracting. Regular expression ^sess: is used to make sure it won’t match session name such as “sessionabc” by mistake. Logical operator || is just a shorthand form of if [ ! condititon ]; then … fi.

Example: create a new window title mywin in an existing tmux session mysess, if the window has not existed yet, create the session first if it hasn’t existed yet
#!/bin/bash
sess=mysess
wn=mywin

tmux list-session 2>&1 | grep -q "^$sess" || tmux new-session -s $sess -d
tmux list-window -t $sess 2>&1 | grep -q ": $wn \[" || tmux new-window -t $sess -n $wn

Example: run a script in mysess:mywin in the above example

#!/bin/bash
sess=mysess
wn=mywin

tmux list-session 2>&1 | grep -q "^$sess" || tmux new-session -s $sess -d
tmux list-window -t $sess 2>&1 | grep -q ": $wn \[" || tmux new-window -t $sess -n $wn
tmux send-keys -t $sess:$wn "./dummytask.sh cooking" Enter

How do we know the above script is doing what we intended to do? Check out the next example.

Example: attach tmux session mysess
tmux a -t mysess
Example: run a script in the first window of the newly created tmux session
tmux new-session -s mysess -n mywin "bash dummytask.sh cooking"

This works but there’s a problem, once you exit the program by pressing q or Q, tmux session also terminates. This gets more annoying when a program crashes and you don’t get any debug info, a better handling of the task will be provided in the following example.

Example: run program in a tmux window and exit to bash shell inside the window if the program exits or crashes
#!/bin/bash
sess=mysess
wn=mywin

# duplicate session or window handling code here
# ...
tmux send-keys -t $sess:$wn "./dummytask.sh cooking" Enter
Example: how to check if a tmux session is attached

Sometimes it’s desired to run certain command not in a tmux window, use the following code to detect if attempt is made to run some command inside a tmux window:

if [ "$TERM" = "screen" -a -n "$TMUX" ]; then
    echo "This command should be run when tmux is not attached"
fi
Example: attach a tmux session with a specific window selected
#!/bin/bash
sess=mysess

tmux list-session 2>&1 | grep -q "^$sess" || tmux new-session -s $sess -d


wn=win0
tmux list-window -t $sess 2>&1 | grep -q ": $wn \[" || tmux new-window -t $sess -n $wn
tmux send-keys -t $sess:$wn "./dummytask.sh task 0" Enter

wn=winX
tmux list-window -t $sess 2>&1 | grep -q ": $wn \[" || tmux new-window -t $sess -n $wn
tmux send-keys -t $sess:$wn "./dummytask.sh important mission" Enter

wn=win1
tmux list-window -t $sess 2>&1 | grep -q ": $wn \[" || tmux new-window -t $sess -n $wn
tmux send-keys -t $sess:$wn "./dummytask.sh another thing" Enter


# here's the meat of this script, select window winX before attaching the session
tmux select-window -t $sess:winX && tmux a -t $sess
Categories: Programming, tmux

Javascript function chaining how to – a simple demo

November 12, 2011 5 comments
<html>
<head>
<title>Rico's Appbox Demos | A simple javascript function chaining demo</title>


<script type="text/javascript">
(function() {
	if(!window['Foo']) window['Foo']={};
	var self=window['Foo'];
	var name=null;
	var description=null;
	var income=-999;

	function init(options) {
		name=options.name||'no name';	
		description=options.description||'default description';
		return self;
	}

	function bark() {
		alert( 'Hi my name is '+name+' and I do ' +description+'.' );
		return self;
	}

	function showincome() {
		alert( "Shhhh don't tell anybody that my income is "+income+'.' );
		return self;
	}

	window['Foo']['init']=init;
	window['Foo']['bark']=bark;
	window['Foo']['showincome']=showincome;
})();
function say_it() {
	var 	thename=document.getElementById('id-name').value,
		thedesc=document.getElementById('id-desc').value;
	return Foo.init({ name:thename, description:thedesc }).bark();
}
function say_it_more() {
	say_it().showincome();
}
function start_over() {
	Foo.init({}).bark().showincome();
}
</script>

</head>
<body>

<p><input id="id-name" value="the one and only"></input></p>
<p><input id="id-desc" value="a lot of things"></input></p>
<p><a href="#" onclick="say_it();">Say It</a></p>
<p><a href="#" onclick="say_it_more();">Say It More</a></p>
<p><a href="#" onclick="start_over();">Start Over</a></p>

</body>
</html>

For a live demo, please visit ricosappbox.appspot.com/demos/jschain.

Categories: javascript

So long, MySQL

November 9, 2011 Leave a comment

I’ve been playing around with PostgreSQL recently and I am total impressed by the features that PostgreSQL supports. I would like to point out the create domain statement (which confirms with SQL standard) that instantly makes me a MySQL betrayer: (from http://www.postgresql.org/docs/8.4/interactive/sql-createdomain.html)

CREATE DOMAIN us_postal_code AS TEXT
CHECK(
   VALUE ~ '^\\d{5}$'
OR VALUE ~ '^\\d{5}-\\d{4}$'
);

CREATE TABLE us_snail_addy (
  address_id SERIAL PRIMARY KEY,
  street1 TEXT NOT NULL,
  street2 TEXT,
  street3 TEXT,
  city TEXT NOT NULL,
  postal us_postal_code NOT NULL
);

The main reason: Data validation is made easy and powerful. With MySQL, data validation relies heavily on the application that interacts with the tables.

Categories: database, postgresql

A mistake to avoid when calling jQuery.ajax

November 4, 2011 Leave a comment

I can’t think of a reason why jQuery.ajax accepts both object and query string as the valid format the data parameter, for example,

$.ajax( {
    url:'/someurl',
    type:'POST',
    data: { key1: val1, key2: val2 },
    // alternatively, the following format is also accepted
    // data: 'key1='+val1+'&key2='+val2,
    success: function() {
        alert(d);
    }
} );

In the above example, the query string format is also accepted but should be discouraged to use, reason of which will be demonstrated with the following example. We need to submit data in the text fields as is to the server. When query string format is used, data is passed to the server without being encoded properly, as you can see from the first screenshot. For demonstration purpose, password field uses “text” as its input type.

index.html

<html>
<head>
<title>Rico's Appbox Demos | A mistake to avoid when using jQuery.ajax() function</title>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js"></script>

<script type="text/javascript">
	function do_post( data ) {
		var io='Sent from client:\nusername: '+$('#username').val()+'\npassword='+$('#password').val();
		$.ajax( {
			url: '/demos/jpost',
			type:'POST',
			data: data,
			success: function(d) {
				io+='\n\nReceived by server:\n'+d
				alert(io);
			}
		});
		return false;
	}

        function post_qs() {
		do_post('username='+$('#username').val()+'&password='+$('#password').val());
	}

	function post_json() {
		do_post({
			username: $('#username').val(),
			password: $('#password').val()
		});
	}
</script>

</head>
<body>

<p>
	Username: <input id="username" type="text" value="This?is+ok">
</p>
<p>
	Password: <input id="password" type="text" value="asdf%20test">
</p>
<p>
	<a href="#" onclick="post_qs()">Post via query string format (BAD)</a><br />
	<a href="#" onclick="post_json()">Post via json format (GOOD)</a> 
</p>

</body>
</html>

Live demo is available at http://ricosappbox.appspot.com/demos/jpost.
The screenshots below are slightly different to the live demo result because they were created when I used php as the back-end to print the post data.

Screenshots:
When first link (using query string format) is clicked:

When the second link (using json format) is clicked:

Categories: javascript, jQuery

Node.js: improve mysql query performance with mysql-pool and cluster

November 1, 2011 1 comment

Prior testing:

npm install mysql -g
npm install mysql-pool -g
npm install express -g
npm install cluster -g

Code that archives best result:

require.paths.push('/usr/local/lib/node_modules');
var express=require('express');
var cluster=require('cluster');
var MySQLPool = require("mysql-pool").MySQLPool;
var pool = new MySQLPool({
  poolSize: 10,
  user:     'node',
  password: 'node',
  database: 'testdb'
});

function symsearch(q, res) {
	var query=pool.query(
		   "select symbol,company_name from symbols where symbol like ? limit 10",
			['%'+q+'%'],
			function selectcb(err, results, fields) {
				if(err) {throw err;}
				console.log("Searching Result for "+q);
				res.send( results );
			}
	);

}

var app=express.createServer();
app.get('/sym/:q', function( req, res) {
	var q=req.params.q || '';
	if(q.length>0) symsearch(q, res);
});

cluster(app).listen(3000);

Benchmark command:

ab -n 10000 -c 250 http://localhost:3000/sym/<pattern>

Results:
With the js code above, I am able to achieve 1300/sec on a dual-core 2.9GHz Virtualbox vm (ubuntu 10.10 32bit). When the same test is conducted on Apache2 + php5, I am getting about 800/sec. If I opt-for the single connection mode (use ‘mysql’ instead of ‘mysql-pool’), the result is only a little better than apache + php combo. The use of node module cluster is to take advantage of the dual-core cpu (node would run with in single-core mode due to its single-threaded nature).

Categories: benchmarking, mysql, node.js
Follow

Get every new post delivered to your Inbox.