Home > linux, Tip > Words filtering with simple unix commands

Words filtering with simple unix commands

Let’s say I have a dictionary file named allwords containing sorted words with one word per line and I also have a collection of “offensive” words in a file call badwords (also one word per line but not sorted). Now I want to take out all the offensive words listed in badwords from the original dictionary file. With Unix this task becomes a piece of cake

$ sort -u badwords > badwords.sort
$ comm -23 allwords badwords.sort > goodwords

The first command can be omitted if badwords is already sorted. In order to get the result as expected, both files need to have the same line terminators (which can be found out easily with command file). There are a number of ways to convert CRLF terminated text to *nix format, here’s my favorite:

$ perl -pi -e 's/\r\n/\n/g' input.file
[ credit goes to http://www.cyberciti.biz/faq/howto-unix-linux-convert-dos-newlines-cr-lf-unix-text-format/ ]

Advertisements
Categories: linux, Tip
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: