Sending email using telnet

Every admin knows as to use telnet to test remote port connectivity. But did you know that you can use this command in more ‘creative’ manners ?

For example to send mail in a fully manual mode:

# telnet localhost smtp
Connected to localhost.
Escape character is '^]'.
220 sd-35049.dedibox.fr ESMTP Exim 4.72 Mon, 16 Jan 2012 12:56:06 +0100
HELO sd-35049.dedibox.fr
250 sd-35049.dedibox.fr Hello localhost [::1]
mail from: megamind@secretbase.com
250 OK
rcpt to: foobar@yopmail.com
250 Accepted
data
354 Enter message, ending with "." on a line by itself
Hey genius buddy
This is a test!
.
250 OK id=1cT5uM-0001oK-Dm
quit
221 sd-35049.dedibox.fr closing connection
Connection closed by foreign host.

[Varnish] Basic performance tuning

Storage engine

Varnish has two storage engine: file and malloc. Both need you to specify (inside the /etc/default/varnish file on Debian) the amount of memory used for caching objects. The difference between the two is how varnish access the cache content.

With file storage varnish create a mapper file on the disk and bind it to virtual memory using the mmap system call. All operations are done via the mapper file. This mode is pretty simple, very ‘posix-compliant’ and therefore portable but… not so great for high performance, even if you use very fast drives.

Now that doesn’t mean file storage is completely useless. It’s a good choice for particular case when you can’t have enough RAM to cache all needed content (and adding SSD-like card like a Fusion-io isn’t an option) because the speed of your IO subsystem will always overcomes content generation. But that shouldn’t be your first pick.

Next we have the malloc storage where varnish directly reserve chunk of memory for each cache’s object, using the system call of the same name. This is the fastest type of cache as memory access latency and throughput are a few orders of magnitude lower/faster than even the fastest SSD drives. My recommendation is always to choose this storage first, and only change in last resort.

Worker threads

Worker threads are the ones that deliver cached objects to clients. You can tune them by playing on three parameters: thread_pool, thread_pool_min and thread_pool_max.

thread_pool is the number of pools that threads will be grouped into. The default value is 2. You can increase the number of pool but you shouldn’t exceed the number of CPU cores.

thread_pool_min defines the minimum amount of threads that always need to be alive per pool. As creating threads is a time-consuming operation it better to always keep at least 50 threads alive. In case of a sudden spike in traffic, you will have enough threads to handle the first wave of requests while varnish spawn new threads.

thread_pool_max defines the maximum amount of threads per pool. This parameter is trickier, as the ideal value depends on available resources and the traffic you want to handle. Usually you don’t want to go over 5000 threads as specified in the documentation.

Varnish shared memory log

The VSL file is used to log most traffic data. It operates on a circular non-persistent buffer of 80MB by default. This file is used by tools like varnishlog or varnishtop. On a Debian system, you can find it into the /var/lib/varnish directory. To improve performance you can move the whole directory inside a tmpfs.

[OpenVZ] NFS file locking problem

My coworkers have found a very annoying bug. Debian’s OpenVZ kernel superior to 2.6.32-squeeze5 have a regression that make NFS exclusive file locking mechanism broken.

Therefore concurrency problems could arise when several fronts try to write into the same file. To test your setup you can do this on an NFS client:

# strace flock -x -w10 "/NFS-MOUNTPOINT/myfile" -c "sleep 10&"

If it’s hang, you have a locking problem. For the moment the only solution is to downgrade your kernel. A ticket have been created on OpenVZ bugtracker.

[OpenBSD] pf tips

Find a rule match

Let say you think a port is filtered by IP but it’s not. You have a somewhere a rule to open for your taste. How to find it ?

First establish a connection from a given IP:

test-host ~ $ nc -v 91.216.209.157 21

Then on the firewall list state connection from this IP:

fw5:~# pfctl -vvv -ss | grep -A3 78.192.224.148
all tcp 91.216.209.157:21 <- 78.192.224.148:56272       ESTABLISHED:ESTABLISHED
   [899608849 + 29200] wscale 7  [1508798648 + 5889] wscale 4
   age 00:00:41, expires in 04:59:19, 3:2 pkts, 164:164 bytes, rule 72
   id: 5565c647ffd1505e creatorid: 410e4752

The matched rule has the number 72.

Find a rule by its number

When doing a tcpdump on the pflog interface, matches are displayed using a number. To find the corresponding rule list them all and filter them:

fw3:~# pfctl -sr -g | grep '@72'

Purge a 'stuck' rule

When a relayd failover goes wrong, you can find yourself with a 'stuck' anchor rule. To list them all:

fw3:~# pfctl -a relayd/* -s rules

Find the outdated rule, then remove it manually:

fw3:~# pfctl -a relayd/foobar_https -F rules

Restrict SSH login to a single command

One of OpenSSH coolest features is the ability to force a command for a given SSH access. You can use this to create limited access for demos, monitoring tools, administration tasks and backup process.

Command without arguments

This is the simplest case, you want to restrict the user to a single command without any arguments. This can be achieved with the command= option into the authorized_keys file like this:

command="/usr/bin/command",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

Note that in addition to enforcing the command, i also disabled terminal and a number of advanced SSH features, such as TCP and X11 forwarding.

Command with fixed arguments

Same as before, just add the complete chain into the command= block. For example with rsync :

command="rsync --server --delete -logDtpre.iLsf . ~/foobar",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

Or if you use rsync in daemon mode:

command="rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

Command with arbitrary arguments

Authorize arbitrary arguments is a little more complicated. This is where the environment variable $SSH_ORIGINAL_COMMAND comes in. When forcing the user command, OpenSSH store the original user command into this variable. By forcing the user to a custom script shell parsing the content of the $SSH_ORIGINAL_COMMAND, you can authorize a specific command with arbitrary arguments:

#!/bin/sh
# /usr/local/bin/limitssh.sh

set -- $SSH_ORIGINAL_COMMAND
case "$1" in
  /usr/bin/command *)
    ;;
  *)
    exit 1
    ;;
esac

exec "/usr/bin/command" "$@"

Then modify your authorized_keys file like this:

command="/usr/local/bin/limitssh.sh $SSH_ORIGINAL_COMMAND",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

It is important to be aware of potential security issues here, such as user escaping to a shell prompted from the listed command. Therefore i recommend you to always disable user terminal (no-pty option) and never use this technique with complex ‘command’ like rsync.

Multiple commands

You can modify your bash script to authorize multiple commands instead of one. Keep in mind this solution is kind of a patch-up job and you should really consider using a more robust restricted shell solution instead:

#!/bin/sh
# /usr/local/bin/limitssh.sh

set -- $SSH_ORIGINAL_COMMAND
case "$1" in
	"ps")
		ps -ef
		;;
	"vmstat")
		vmstat 1 100
		;;
	"apache stop")
		/etc/init.d/apache2 stop
		;;
	"apache start")
		/etc/init.d/apache2 start
		;;
	*)
		exit 1
		;;
esac

Erase partition content with dd

If you don’t want or can’t use shred, it’s possible to use dd to erase the content of a partition.

Quick and unsafe method

The first method is to write 0 on all sectors of the disk in a single pass:

# dd if=/dev/zero of=/dev/sdX bs=512 conv=notrunc

Slow and moderately safe method

The second method, a bit more secure than the previous one, is to fill sector with data from the kernel pseudorandom number generator:

# dd if=/dev/urandom of=/dev/sdX bs=512 conv=notrunc

Slow and safe method

Pretty much the same method then the previous one, but this time we will make multiple pass (here 8) :

# for n in `seq 8`; do dd if=/dev/urandom of=/dev/sdX bs=512 conv=notrunc; done

[Bash] Bash history tips

Changing bash history length

To increase the history length of your command history simply redefine the HISTFILESIZE variable into your .bash_profile. For example:

HISTFILESIZE=2000

will quadruple the default value.

Checking bash history

Everybody know the standard history command, but do you know you this command can take options ?

For example for displaying only the last 5 commands:

history 5

For writing the current display to a file:

history -w /tmp/foobar.txt

There is a ton of other options for modify or filter the ouput, built-in into the history command.

Disable bash history

Setting the HISTFILESIZE variable to zero will disable bash command history completely. Having the history file disabled does not effect command recall for the current session.

Executing commands from bash history

Just give the command number. For example for the 51th command:

!51

For the last command use the shortcut: !!.

Dealing with multiple bash session

By default, bash writes its history at the end of each session, overwriting the existing file with an updated version. This means that if you are logged in with multiple sessions, only the last one to exit will have its history saved.

To bypass this limitation you can add the following into your .bashrc:

shopt -s histappend

Unlimited bash history

Beware: unlimited bash history could slow down your session.
If you want an unlimited bash history you can modify your .bash_profile like this:

# Unlimited bash history setting
# http://stackoverflow.com/questions/9457233/unlimited-bash-history
export HISTFILESIZE=
export HISTSIZE=
export HISTTIMEFORMAT="[%F %T] "
# Change the file location because certain bash sessions truncate .bash_history file upon close.
export HISTFILE=~/.bash_unlimited_history

What is the difference between HISTSIZE and HISTFILESIZE ?

HISTSIZE is the number of lines or commands that are stored in memory in a history list while your bash session is ongoing.

HISTFILESIZE is the number of lines or commands that are allowed in the history file at startup time of a session, and also are stored in the history file at the end of your bash session.

Varnish

varnish is a reverse-proxy build exclusively for web acceleration. It speed up websites by caching their contents in memory and serve the requested objects (HTML pages, images, css, js, etc…) directly instead of the HTTP server.

It not only reduce user-perceived page load time, but also increase platform load capacity. Varnish is licensed under the two-clause BSD license, but commercial support is also available from Varnish-software.

Architecture / Performance

The big plus of Varnish compared to using a classic proxy, like Squid, in reverse mode, is it ability to store data in virtual memory and lets the kernel decide what to do with it (place the object in RAM or swap or anything else). This behavior make Varnish very memory efficient.

Varnish is also massively multi-threaded, queuing incoming connection only when it reaches its limit or when multiple users ask for the same non-cacheable resource. Also in order to reduce system calls, varnish logs requests in shared memory. In a nutshell: varnish is clearly build for performance.

Configuration

Configuration is done via a specific ‘pseudo-language’ called VCL. VCL is trans-coded into C code and compiled as a shared object when reloading the configuration. If you know what you do, you can add C code into a VCL setting file to increase varnish functionalities.

Speaking of functionalities, varnish supports round-robin load balancing and most ESI standard tags, which simplify a lot the inclusion of dynamic content into ‘static’ cachable pages.

Saidar – an ncurses tool for viewing system statistics

saidar is a curses-based tool for viewing the system statistics available through libstatgrab. As libstatgrab provides an abstraction layer for retrieving information, saidar can be use on multiple OS (Solaris 2.x, GNU/Linux, FreeBSD, NetBSD, OpenBSD, HP-UX).

Statistics include CPU, processes, load, memory, swap, network I/O, disk I/O and file system information. For launching saidar simply type it name:

# saidar

The refresh delay is 3 seconds by default and can be changed using the -d parameter. You can also add color to the output using the -c option.