Batch-processing script

I have developed this batch-processing script that’ll chop down files under a directory and process each group of files. Currently this bash script has the following features:
1) preview mode (option -p) lets you preview what commands will be executed
2) group files based on either
a) fixed number of files, for example, every 20 files
b) fixed bytes of file size sum, for example, every 10 MB
3) allow file list generation (which defaults to command find) to be customized by user
4) supports environment variable $VOL when the processing program is shell script (and is able to use environment variables)

For detail usage and examples, simply run bp.sh without any parameters.

View the code in (pretty) html format
View the raw codes in pure text format
[ Rename bp.sh.txt to bp.sh after downloading, the same step should be done for the following example scripts. ]
Example code 1: testbp.sh
Example code 2: testbp2.sh

Feedbacks or comments are welcome.

Watch out these bash wildcards

I would like to list a couple of mistakes I used to make often when using Linux commands:

1. Remove contents of a folder with rm
Let’s say I have a folder ~/data/docs and I want to empty it. There are hidden files (files that start with . ) along with files with regular names.
The wrong way:

$ rm -rf ~/data/docs/*

Reason:
the above command will remove all files and directories under ~/data/docs except the hidden ones.

The right way:

$ rm -rf ~/data/docs/* ~/data/docs/.* 2>/dev/null

or

$ rm -rf ~/data/docs/{*,.*} 2>/dev/null

Note: the trailing 2>/dev/null suppresses the following error messages

rm: cannot remove `.' directory `/home/rico/data/docs/.'
rm: cannot remove `..' directory `/home/rico/data/docs/..'


An even better way

$ cd ~/data/docs
$ find . -delete

Note: using only the find command

find ~/data/docs -delete

will delete the folder ~/data/docs as well, hence the cd command.

 

2. Rsync contents of folder src with dest
The wrong way: (unless copying only non-hidden items is the goal)

$ rsync -av /path/to/src/* /path/to/dest/

Reason:
Similar to the rm mistake. Using rsync this way will only copy non-hidden files and folders from src/ to dest/

The right way:

$ rsync -av /path/to/src/ /path/to/dest/

Be careful not to type this instead [ without the trailing slash after src ]:

$ rsync -av /path/to/src /path/to/dest/

as that will put folder src instead of the content of it under /path/to/dest, that is, after running the above command, the destination folder structure will have the following folder:
/path/to/dest/src

Extract ip address(es) from standard input using perl and regular expression

Method 1:

$ cat file | perl -nle 'print "$1" if
/(?!0+\.0+\.0+\.0+$)(([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5]))/'

Method 2:

$ cat file | perl -nle 'print $1 if /(([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+))/g && $2<256 && $3<256 && $4<256 && $5<256'


I prefer using method 2 because it’s simpler to construct and for ip addresses such as 1.2.3.1234 (obvious an invalid one), using method 1 results in1.2.3.123 being matched. But method 2 will not mistakingly pick up that ip address.

Reference: Mastering Regular Expressions by Jeffrey E.F. Friedl.

Top reasons I don’t like Grub2

Recently I replaced my ubuntu 9.04 with LinuxMint 8 which uses Grub2 as the boot loader and I found it quite annoying for the following reasons:

1) There is no more /boot/grub/menu.lst to edit, instead, any new boot entries should be inserted into /etc/grub.d/40_custom

2) Adding a new boot entry or modifying boot settings (such as default timeout value) requires an additional command in order for the new settings to become effective:
update-grub2
[ update-grub also works ]

3) Because of the above reasons, if the LinuxMint can’t boot due to some misconfiguration in the grub setting, it becomes harder to get it repaired since you run into the egg-and-chicken dilemma: you have to boot into LinuxMint in order to get grub2 repaired and updated [ which is exactly the reason I don't like using boot loader lilo ]

4) The syntax has changed in grub2 for boot entries: among other changes, kernel is now replaced with linux, initrd is replaced with module (and module supports loading other things, this seems to be the only + point for grub 2)

All in all I was not impressed with the new “features” introduced by grub2 and as a result I downgraded it to grub using the other OS (Centos 5.4) on the same hard drive.

Make use of bash variable $_

I’ve discovered recently that you can use $_ to access the last argument in the last command that you type in bash. For example, you run this command

$ cat /usr/local/bin/myscript.sh

the screen will display the content of the file /usr/local/bin/myscript.sh, which you wish to edit now. The old (and I believe most of us still do) way to do is to scroll up with the up arrow key, move cursor right after cat and replace it with your editor (say, vi), now the new command becomes

$ vi /usr/local/bin/myscript.sh

Hit enter and you will be able to edit the file (assuming you have write access to myscript.sh)

It’s working well but not that well when you can do it in a better way:
right after the cat command, try the following instead:

$ vi $_

this will substitute $_ with the last argument (which is /usr/local/bin/myscript.sh), clean and efficient. [ I am not saying cat a file then edit it is the most efficient way, but the above is to demonstrating the usage of $_ and I hope you find this tip useful.]

Sprint HTC Hero rooted and flashed

Yesterday I helped one of my colleagues to get wifi tether installed onto her Sprint HTC Hero phone under Ubuntu 9.10. After some initial research I thought it’s gonna be an easy job since all I need is to root the phone and install wifi tether from http://code.google.com/p/android-wifi-tether/, as it turned out the rooting and the installation of wifi tether is pretty easy but I could not get a connected laptop to ping the gateway (which is the phone) or browse the Internet. Further googling (http://code.google.com/p/android-wifi-tether/issues/detail?id=188&q=sprint%20hero%20no%20internet) indicated that putting a custom ROM to the phone is the way to get it work. So basically I follow the link at http://android.modaco.com/content/htc-hero-hero-modaco-com/295568/cdma-08-11-1-1-modaco-custom-rom-core-featuring-wavesecure/ and did some modification to the steps as some did not work as the post instructed. And here’s what I did to the phone:
1) backup sd card [ this step is optional but recommended ]
2) download and extract Android SDK [ http://developer.android.com/sdk/download.html?v=android-sdk_r3-linux.tgz , I added this step because I had problem in the following step installing recovery image to the phone ]
3) Follow instructions at http://android.modaco.com/content/htc-hero-hero-modaco-com/291486/gsm-cdma-09-11-installing-the-patched-recovery-image-on-your-device/ and run commands:

chmod +x ./install-recovery-cdma-linux.sh
./install-recovery-cdma-linux.sh


since the phone is CDMA and it’s done under Linux.
the above commands is supposed to put recovery-RA-heroc-v1.2.3.img and flash to the phone but strangely after running ./install-recovery-cdma-linux.sh I didn’t see recovery-RA-heroc-v1.2.3.img on the /sdcard/ folder of the phone. So did the following to make it happen:

export PATH=/path/to/android-sdk-linux_86/tools:$PATH
adb push /path/to/recovery-RA-heroc-v1.2.3.img /sdcard/
adb shell
su
flash_image recovery /sdcard/recovery-RA-heroc-v1.2.3.img


Note: under the recovery zip file folder there’s another image file recovery-RA-hero-v1.2.3.img [ without "c" after "hero" ], be careful not to use it because I made a mistake and ended up with not being able to boot the recovery image.

After the above the restart the phone while pressing “Home” button to get to the recovery screen.
4) Following the rest of the instructions on http://android.modaco.com/content/htc-hero-hero-modaco-com/295568/cdma-08-11-1-1-modaco-custom-rom-core-featuring-wavesecure/
Note: I didn’t rename any of the .zip files downloaded into update.zip as I could use the recovery utility to select the .zip file I want to install.

After done all the steps above the wifi tether that comes with the modaco rom works like a charm and my colleague also loves the business card scanner app.

Compiling my first DirectFB program

I have to admit I am a total DirectFB newbie and I had a hard time even compiling my first DirectFB program (the official DirectFB or tutorials on the Internet described procedures how to build DirectFB but not how to even compile the simplest demo program, forgive my poor google skill if somebody already posted the compiling instruction). Anyway I use the following command to compile successfully:

gcc -o simple simple.c `directfb-config --cflags` `directfb-config --libs`

OS: fresh installation of Gentoo 09132009
DirectFB version: 1.2.7
Source code of the demo program: http://www.directfb.org/docs/DirectFB_Tutorials/simple.html

Splash Screen with GTK+

The code:

#include <gtk/gtk.h>
static void destroy (GtkWidget*, gpointer);
static gboolean delete_event (GtkWidget*, GdkEvent*, gpointer);
int main (int argc,
          char *argv[])
{
  GtkWidget *window, *image;
  gtk_init (&argc, &argv);
  window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
  gtk_window_set_title (GTK_WINDOW (window), "A Splash Screen Demo");
  gtk_container_set_border_width (GTK_CONTAINER (window), 0);
  gtk_widget_set_size_request (window, 800, 480);
  gtk_window_set_decorated(GTK_WINDOW (window), FALSE);
  gtk_window_set_position(GTK_WINDOW(window), GTK_WIN_POS_CENTER);
  gtk_window_set_resizable(GTK_WINDOW(window), FALSE);

  /* Connect the main window to the destroy and delete-event signals. */
  g_signal_connect (G_OBJECT (window), "destroy",
                    G_CALLBACK (destroy), NULL);
  g_signal_connect (G_OBJECT (window), "delete_event",
                    G_CALLBACK (delete_event), NULL);

  image=gtk_image_new_from_file("wait.png");
  gtk_container_add(GTK_CONTAINER(window), image);

  gtk_widget_show_all (window);
  gtk_main ();
  return 0;
}

/* Stop the GTK+ main loop function when the window is destroyed. */
static void
destroy (GtkWidget *window,
          gpointer data)
{
   gtk_main_quit ();
}
/* Return FALSE to destroy the widget. By returning TRUE, you can cancel
  * a delete-event. This can be used to confirm quitting the application. */
static gboolean delete_event (GtkWidget *window,
               GdkEvent *event,
               gpointer data)
{
    return FALSE;
}

And here’s the included “Please wait” image file in .png format:

The png file used by the splash screen demo program

To compile:

gcc -Wall -g demo.c -o splash `pkg-config --cflags gtk+-2.0` `pkg-config --libs gtk+-2.0`

Now test it:

./splash
# use Alt + F4 to close and exit.

A quick(er) way to view config files

Quite often we need to check the (effective) settings in a configuration file and the most common way is to use grep

grep -v ^# config_file

But this does not work well because
1) it does not filter out empty lines
2) it does not filter out lines start with space(s) or tab(s) and are followed by # or ; (openvpn uses both # and ; as comment characters)

As a solution to the solve the above limitations, I use the following egrep command quite often

egrep -v "^[[:cntrl:] ]*[#;]|^[[:cntrl:] ]*$ config_file

But the above command is a lot of typing therefore I add the following alias to my ~/.bashrc and it filters out all comment lines and empty lines [ including lines containing only space(s) and/or tab(s) ] nicely.

alias cf='egrep -v "^[[:cntrl:] ]*[#;]|^[[:cntrl:] ]*$"'

With this alias I just need to type the following to view my config files:

cf config_file

Reference:
Inspired by http://www.linuxjournal.com/content/tech-tip-view-config-files-without-comments, I also left a comment there (rc3) but that’s not the final version that I am using here.

arp cache poisoning: it’s for real

Last week I helped one of our clients solve their web server being inserted with iframe problem, which started a few weeks back when they told me that their customers complained accessing to their product order status page resulting in some unexpected pop-up windows. As a result this client’s website is marked unsafe by anti-virus programs such as mcfee. My initial investication showed that accessing the problematic page results in an iframe line being inserted to the top of the page in the html source code:
<iframe src=”h t t p://w w w.h a c k i n g s.c n”></iframe>
(I intentionally add some spaces in the url as I don’t want you to accidentally hit that link going to the hacker’s site.)

A bit more detail about the setup of the webserver:
1) ASP.NET + IIS
2) runs behind a Linux firewall accessible through DNAT

My investigation continued and my accessing the same page from the firewall resulted in the same result — the hacking line being inserted. But when I accessed the page from any other workstation including the webserver itself, I didn’t see the line, which led to my wrong conclusion that there must be some virus in the webserver that is smart enough to tell where the traffic is from — when it’s from the gateway, the request is most likely made from the Internet; when it’s from the LAN, it stops inserting the line to fool the web administrator that the web server is fine. My conclusion is not coincident: I also used a Linux laptop to set up a temporary web server providing the exact same urls to replace the original webserver and the hacking line disappeared. I recommended the client to bring down the IIS webserver immediately and replace it with a Linux server. I thought the problem was solved.

A few days later, my client reported that the problem came back, with a Linux server being the web server. My hunting for the real cause of the problem went on and somehow arp cache poisoning got into my head. I heard it a lot before but never seen it in action. To get started I installed and run arpwatch on the firewall and before long I saw a lot of flip-flops scrolling up on the screen (mac addresses change back-and-forth) for a bunch of ip addresses in the client’s LAN. The client doesn’t use dhcp therefore there shouldn’t be flip-flops — At least that tells me that I was on the right direction to solve the problem. From the arpwatch log I saw something like this

...
Jul 29 13:21:18 linux-gw arpwatch: report: pausing (cdepth 3)
Jul 29 13:21:18 linux-gw arpwatch: flip flop 172.16.0.1 18:8b:41:72:b1 (0:13:20:3d:5e:53)
Jul 29 13:21:18 linux-gw arpwatch: report: pausing (cdepth 3)
Jul 29 13:21:18 linux-gw arpwatch: changed MAC address 172.16.0.199 0:50:da:7b:c6:60 (0:13:20:3d:5e:53)
...

I could see from the log entries that those flip flop lines share the same pattern — the mac address for a specific ip gets changed to the same mac address 0:13:20:3d:5e:53 then back to its original mac address. At that point I concluded whichever machine that has 0:13:20:3d:5e:53 as the mac address is the one that initializes arp poisoning attack. As the client didn’t have a ip to mac address mapping, the hunting for that machine took a bit while but luckily it was found on one of the workstations. I instructed the client to disconnect that workstation from the LAN and I conducted the test again, the hacking line was gone — either from the Internet or the gateway. What a relief! arpwatch continued to report a lot of flip-flops, but they are just for changing back to their original mac addresses from the poisoned mac address. When arpwatch stopped reporting flip-flops I generated an IP-Mac address map for the client so it will a piece of cake to ID which machine generates arp cache poisoning attack when problem like this happens again.