[OpenVZ] NFS file locking problem

My coworkers have found a very annoying bug. Debian’s OpenVZ kernel superior to 2.6.32-squeeze5 have a regression that make NFS exclusive file locking mechanism broken.

Therefore concurrency problems could arise when several fronts try to write into the same file. To test your setup you can do this on an NFS client:

# strace flock -x -w10 "/NFS-MOUNTPOINT/myfile" -c "sleep 10&"

If it’s hang, you have a locking problem. For the moment the only solution is to downgrade your kernel. A ticket have been created on OpenVZ bugtracker.

[OpenBSD] pf tips

Find a rule match

Let say you think a port is filtered by IP but it’s not. You have a somewhere a rule to open for your taste. How to find it ?

First establish a connection from a given IP:

test-host ~ $ nc -v 91.216.209.157 21

Then on the firewall list state connection from this IP:

fw5:~# pfctl -vvv -ss | grep -A3 78.192.224.148
all tcp 91.216.209.157:21 <- 78.192.224.148:56272       ESTABLISHED:ESTABLISHED
   [899608849 + 29200] wscale 7  [1508798648 + 5889] wscale 4
   age 00:00:41, expires in 04:59:19, 3:2 pkts, 164:164 bytes, rule 72
   id: 5565c647ffd1505e creatorid: 410e4752

The matched rule has the number 72.

Find a rule by its number

When doing a tcpdump on the pflog interface, matches are displayed using a number. To find the corresponding rule list them all and filter them:

fw3:~# pfctl -sr -g | grep '@72'

Purge a 'stuck' rule

When a relayd failover goes wrong, you can find yourself with a 'stuck' anchor rule. To list them all:

fw3:~# pfctl -a relayd/* -s rules

Find the outdated rule, then remove it manually:

fw3:~# pfctl -a relayd/foobar_https -F rules

Restrict SSH login to a single command

One of OpenSSH coolest features is the ability to force a command for a given SSH access. You can use this to create limited access for demos, monitoring tools, administration tasks and backup process.

Command without arguments

This is the simplest case, you want to restrict the user to a single command without any arguments. This can be achieved with the command= option into the authorized_keys file like this:

command="/usr/bin/command",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

Note that in addition to enforcing the command, i also disabled terminal and a number of advanced SSH features, such as TCP and X11 forwarding.

Command with fixed arguments

Same as before, just add the complete chain into the command= block. For example with rsync :

command="rsync --server --delete -logDtpre.iLsf . ~/foobar",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

Or if you use rsync in daemon mode:

command="rsync --server --daemon .",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

Command with arbitrary arguments

Authorize arbitrary arguments is a little more complicated. This is where the environment variable $SSH_ORIGINAL_COMMAND comes in. When forcing the user command, OpenSSH store the original user command into this variable. By forcing the user to a custom script shell parsing the content of the $SSH_ORIGINAL_COMMAND, you can authorize a specific command with arbitrary arguments:

#!/bin/sh
# /usr/local/bin/limitssh.sh

set -- $SSH_ORIGINAL_COMMAND
case "$1" in
  /usr/bin/command *)
    ;;
  *)
    exit 1
    ;;
esac

exec "/usr/bin/command" "$@"

Then modify your authorized_keys file like this:

command="/usr/local/bin/limitssh.sh $SSH_ORIGINAL_COMMAND",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB...YrAFnVw== foobar_public_key

It is important to be aware of potential security issues here, such as user escaping to a shell prompted from the listed command. Therefore i recommend you to always disable user terminal (no-pty option) and never use this technique with complex ‘command’ like rsync.

Multiple commands

You can modify your bash script to authorize multiple commands instead of one. Keep in mind this solution is kind of a patch-up job and you should really consider using a more robust restricted shell solution instead:

#!/bin/sh
# /usr/local/bin/limitssh.sh

set -- $SSH_ORIGINAL_COMMAND
case "$1" in
	"ps")
		ps -ef
		;;
	"vmstat")
		vmstat 1 100
		;;
	"apache stop")
		/etc/init.d/apache2 stop
		;;
	"apache start")
		/etc/init.d/apache2 start
		;;
	*)
		exit 1
		;;
esac

Erase partition content with dd

If you don’t want or can’t use shred, it’s possible to use dd to erase the content of a partition.

Quick and unsafe method

The first method is to write 0 on all sectors of the disk in a single pass:

# dd if=/dev/zero of=/dev/sdX bs=512 conv=notrunc

Slow and moderately safe method

The second method, a bit more secure than the previous one, is to fill sector with data from the kernel pseudorandom number generator:

# dd if=/dev/urandom of=/dev/sdX bs=512 conv=notrunc

Slow and safe method

Pretty much the same method then the previous one, but this time we will make multiple pass (here 8) :

# for n in `seq 8`; do dd if=/dev/urandom of=/dev/sdX bs=512 conv=notrunc; done

Pound

pound is a tiny reverse proxy load balancer and SSL offloader. It’s not a caching proxy like Varnish, but its simplicity and lightweight make it a good choice for making an HTTPS front-end on a moderate traffic platform.

Create a PEM file

pound use the PEM format. A single PEM file can contain all the needed files (public certificate, intermediate certificate, root certificate and private key).

To convert your SSL files certificate to a PEM file usable for Pound:

# cat server.key > cert.pem
# cat your.domain.tld.crt >> cert.pem
# cat intermediate.crt >> server.pem

Disable SSLv3

To improve security you can disable the SSLv3 protocol. You need at least the patched version 2.6 to do that. Add the DisableSSLv3 directive inside your ListenHTTPs block.

Improve ciphers selection

To improve security you can also disable old/weak ciphers. Redefine the ciphers selection like this:

Ciphers    "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+aRSA:-RC4:EECDH+aRSA+RC4:EECDH+RC4:EDH+aRSA+RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:RC4+SHA"

Further Reading and sources

Shred

Wipe entire hard drives

The primary usage of the shred command is to wipe entire partition by overwriting the content. For example, to wipe /dev/sda5:

shred -vfz -n 10 /dev/sda5

-v: show progress
-f: change permissions to allow writing if necessary
-z: add a final overwrite with zeros to hide shredding
-n: overwrite N times instead of the default three time

Here we will overwrite /dev/sda5 ten times, enough to ensure that data can’t be retrieve without very special and complicated method.

Shred individual files

shred can also be use to overwrite and delete a given file, but it maybe not so efficient in that case. The man page warm you about:

CAUTION: Note that shred relies on a very important assumption: that the file system overwrites data in place. This is the traditional way to do things, but many modern file system designs do not satisfy this assumption.

It’s hard to evaluate if recovery of a “shredded” file could be possible, as it depend on the filesystem (and the mount options for ext3/4) and how data is ordered on the device. But keep this limitations in mind. For the command simply do:

shred -u foobar.txt

By default shred overwrites the file 25 times. You can customize this value using the --iterations=n parameter.

Restricted SFTP access

FTP is an old protocol created when network was a ‘new thing’ and everybody was a ‘care bears’, therefore it’s insecure by design and you shouldn’t even propose it to yours customers. Instead always push the SFTP option first.

For practical usage, there is three little downsides to SFTP use:

  • SFTP doesn’t have it own dedicated port. Personally i like to reuse the ‘Simple File Transfer Protocol’ port (TCP 115) but this ideas is enough to trigger heart attacks to network ‘ayatollah’
  • each user must have an unix account
  • an SFTP access isn’t by default chrooted inside the user directory

SFTP on a dedicated port

Until we have SFTP support inside ProFTPD, the only solution is to spawn a second OpenSSH daemon on a separated port.

Here a snippet of my new setting file /etc/ssh/sftp-115:

Port 115

# Protocol
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UsePrivilegeSeparation yes

# Restriction
AcceptEnv no
AllowAgentForwarding no
AllowTcpForwarding no
Banner no
MaxAuthTries 3

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO
UseDNS no

# Authentication
LoginGraceTime 120
PermitRootLogin no
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile /home/sftp/%u/.ssh/authorized_keys
PasswordAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Disable some options
UsePAM no
X11Forwarding no
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes

# Use sftp-internal (allow chroot)
Subsystem sftp internal-sftp
ForceCommand internal-sftp

Before OpenSSH 4.8 there is no internal-sftp module, therefore you must use the external command sftp-server instead:

Subsystem sftp /usr/lib/openssh/sftp-server

For launching the daemon you can use xinetd, clone and modify your distribution OpenSSH init script or simply add a one-liner inside the /etc/rc.local:

/usr/sbin/sshd -f /etc/ssh/sftp-115

Do as you wish.

Unix user account

Not much to say here, you must create a proper unix account for each SFTP. I suggest you to set the home directory to something like /home/sftp/<user> to distinguish easily theses users from regular ones.

In case you use sftp-server you must also change the user shell value. First add sftp-server as a valid shell :

# echo '/usr/lib/stfp-server' >> /etc/shells

Then for each SFTP unix account:

# usermod -s /usr/lib/sftp-server <user>

Chroot user

If you use internal-sftp simply add the following snippet to /etc/ssh/sftp-115:

ChrootDirectory /home/sftp/%u

For sftp-server, bad news, you simply can’t chroot users.

To give user access to a directory use the mount --bind command:

mount --bind /var/www/foobar /home/user/foobar/www

Don’t forget to add this line into your SFTP daemon init script.

Reverse SSH tunnel

Have you ever wanted to SSH to an host that sits behind a firewall doing NAT ?
That possible with some trickery via a reverse SSH tunnel.

First, let say to our nated-host’s administrator to SSH our public-host relay server like this:

ssh -R 2222:localhost:22 user@<public-host>

This command open the port 2222 on the public-host and redirect it to the nated-host port 22.

Then we simply connect to the public-host as usual :

ssh user@<public-host>

and then access to the SSH tunnel:

ssh localhost -p 2222