Category: Shell

The TERM environment variable is used by your terminal emulator to indicate which type of control string it supports. For the most part you should not override this variable, but there is a narrow set of circumstances where the default value can be incorrect or at least not optimal.

For example in most distributions the default value for a console terminal is linux and for a graphical terminal xterm. That’s fine, but don’t allow you to enjoy vim or screen 256 color mode in graphical mode.

A traditional xterm only supports 16 colors and this value is specify into the terminfo database.
As you can image, changing the xterm entry would have made users of the newer versions happy but also broken the configuration for others. Instead a new value xterm-256color was declared. That may seems fine but when you log in remotely to machines with an older terminal database, the value isn’t recognized.

In this case the only viable solution left is to override the TERM value in order to unset the 256 color mode. You can do it by adding something like that to the .bash_profile on the remote machine:

if echo $TERM | grep -q -- '-256color'; then
    export TERM=`echo -n $TERM | sed 's/-256color//'`
fi

One thing that mystify unix newbies is the result difference between top and ps commands about CPU usage:

# ps -o %cpu 21162
%CPU
5.5

but a top -p 21162 give us 18%. There is something wrong right ?
Nope. But the confusion rise from the fact that ps and top don’t have the same definition of what constitute CPU usage.

top give you an average value of CPU consumption per core on a short period of time (by default 3 seconds). A value of 200% means that the process “hogged” in average two core during the last 3 seconds.

On the other hand ps calculate it value on the process lifetime, and don’t take into account how many core have been hog. So a for example a value of 15.3% means that since the process is running 15.3% of it lifetime it has been bugging on the CPU. The other 84.7% of the time the process was doing nothing, probably waiting for some input to append.

So as you can see this two commands have a very different definition of what CPU usage is, and both value are relevant in their own.

Let’s said you have a filesystem with a directory containing a huge number of files, something like half a million. What is the fastest method to delete so many files ?

Never use shell expansion

Before answering the question let’s state the obvious: you should never use shell expansion when dealing with a huge number of files.

cd huge_directory
rm -f *
-bash: /bin/rm: Argument list too long

As you can see rm didn’t even start because shell expansion produce a command that exceeds the ARG_MAX limit (128kb since kernel 2.6.23). So if you insist to use rm for the job (and you shouldn’t) at least do it the right way:

rm -rf huge_directory

It will internally list the files and directories it’s going to delete.

Using find

“If you can’t do it in one setting, divide and make a loop”. That’s the strategy of the find command. First let’s use it with the -exec parameter:

time find ./ -type f -exec rm {} \;
real    14m51.735s
user    2m24.330s
sys     9m48.743s

Not so great. In fact using find this way is very inefficient because it spawn an external rm process for each file ! Luckily, find have it own built-in -delete command:

time find ./ -type f -delete
real    5m11.937s
user    0m1.259s
sys     0m28.441s

That’s much better, but we can do better.

Using rsync

Using rsync for this task may seem a little strange in the first place, but it work really really well:

time rsync -a --delete emptydir/ huge_directory/
real    2m52.502s
user    0m2.772s
sys     0m32.649s

Clearly rsync is the winner. But why ? It’s a little tricky.

When deleting a file you invoke the unlink system call. This call removes a ‘link’ to the file’s inode. Once there is no more ‘link’ the system free the associated space by zeroing the inode. Pretty simple stuff. But how do it know which ‘link’ to unlink ? By using another system call that list the content for a given directory: readdir.

Now here the thing. readdir doesn’t list files in-order but randomly (in really not so randomly because it’s depend on inode number) and by packet of 32Kb. When there is a reasonable number of files that perfectly fine and quite efficient, but not in an over-crowded directory.

This behavior is the main reason why using ls or rm in a directory containing millions of files is such a pain in the ass. Each operation make hundred of readdir calls.

On the other hand rsync doesn’t rely on readdir but use it own implementation. It use a single huge buffer and list all files in-reverse order. This way rsync can unlink files after files without glancing a second time at the directory structure. Huge gain of time when dealing with millions of files.

zsync is a file transfer program that allows you to synchronize a local file with a remote server version. zsync is very efficient as it only downloads new parts of the file, using the same algorithm as rsync.

Server usage

Use zsyncmake to build a control file for zsync client, like this:

zsyncmake -z myiso.iso

Client usage

zsync http://foobar.com/daily-live/current/myiso.iso.zsync

Let say you have a file with multiple sections, delimited by the character sequence -|, and you need to create multiple files, one for each section. How you do that ?

Basically you can use three tools for the job.

Use csplit

Csplit is a very useful and not well know utility, present into coreutils package.

csplit --quiet --prefix=outfile infile.txt  "/-|/+1" "{*}"

Use awk

awk '{print $0 " -|"> "output" NR}' RS='-\\|' infile.txt

Use bash/perl/python

Here i give you the bash one-liner version, but you can also use perl, python, ruby or any other scripting language:

cat infile.txt | ( I=0; echo -n "" > output0; while read line; do echo $line >> output$I; if [ "$line" == '-|' ]; then I=$[I+1]; echo -n "" > output$I; fi; done )