Posts

Posted on

The heartbleed vulnerability

What is heartbleed ?

Heartbleed is the name given to a critical vulnerability in the OpenSSL library. The official designation of this bug is CVE-2014-0160. Theoretically this vulnerability could be use by an attacker to gain access to data transmitted between clients and server, by random chunk of 64ko. Retrieved content could be pretty much anything: session content, password, even the server private key.

How did this happen ?

The vulnerability is due to a missing validation on a variable size. The bug was introduced into OpenSSL 1.0.1 and fixed into version 1.0.1g.

How does it work ?

The following XKCD comic does a pretty good job at explaining the issue in simple terms.

To test if your server is vulnerable

Use this: https://filippo.io/Heartbleed

What to do ?

First don’t panic and make security updates. Check you have an OpenSSL version 1.0.1g or higher. Because of the buzz around the vulnerability most SSL providers offer free certificate regeneration. Jump on the offer, regenerate your server private key and CSR.

Further Reading and sources

Posted on

TCP/IP tuning for scalability

Increase max files limits

Sockets *are* files, so in order to improve TCP scalability the first step is to verify you have enough file descriptors available. You can check the current values with the ulimit -a command. If needed increase them.

Increase port range

By default ports between 32768 and 61000 are kept for ‘ephemeral’ usage. That plenty for clients, but could be too tight for busy web servers.

You can increase the port range like this:

# sysctl -w net.ipv4.ip_local_port_range="12800 65535"

Don’t forget to add your setting inside the /etc/sysctl.conf file.

Enable TCP reuse

The TCP_TW_REUSE flag allow the kernel to reuse TCP connection in TIME_WAIT state for a new outgoing connection if the new timestamp is strictly bigger than the most recent timestamp recorded for the previous connection.

This optimization is pretty great for web server that deal with many short TCP connections. You can enable it like this:

# sysctl -w net.ipv4.tcp_tw_reuse=1

Don’t forget to add your setting inside the /etc/sysctl.conf file.

There is also another parameter, called net.ipv4.tcp_tw_recycle. Do NOT enable it !

Fast-recycling of every TIME-WAIT sockets sound like a good idea but this option will cause problems with NAT clients. For more information on the subject, check this excellent article.

Posted on

[Varnish] Debugging VCL configuration

Dry-run compiling

The first step to debug your varnish configuration is to simply ‘dry-run’ compiling it:

varnishd -Cf /etc/varnish/myconf.vcl

The output of this command will either be a successfully-compiled VCL or an error message telling you on what line the error occurs. Most of the time that enough to fix syntax error.

Add message into syslog

Since branch 3.X varnish can load and use additional module. One of the most useful is the std module. You can use it to add “trace” message into the syslog:

import std;

vcl_recv {
  ...
  std.syslog(0, "Cookie content: " + req.http.Cookie);
}

Use varnishlog

varnishlog is a command line utility that display all varnishd ‘internal’ activities: request handling, request processing, backend checks/request, hash information, objects caching/retrieving, etc… As you can imagine, the output is extremely verbose and kind of frightening. But you can add parameters, to filter it.

For example, to display only hashes:

varnishlog -c -i Hash

To display only ‘User-Agent’ strings:

varnishlog -c -i RxHeader -I User-Agent

To display only POST requests:

varnishlog -c -m RxRequest:POST

To display only request with a 404 response:

varnishlog -c -m TxStatus:404

To filter requests for a specific URL:

varnishlog -c -m RxURL:"/foo/bar.php"

To filter URLs asked to the backend:

varnishlog -b -i TxURL

To filter for a particular IP:

varnishlog -c -o ReqStart 192.168.1.16

Monitoring activities with varnishtop

Another interesting command line tool is varnishtop. Like varnishlog it reads varnishd shared memory logs but instead of displaying current entries, it presents a continuously updated list of the most commonly occurring entries. Like varnishlog you can add additional parameters to filter it output.

Posted on

Top, ps and cpu usage

One thing that mystify unix newbies is the result difference between top and ps commands about CPU usage:

# ps -o %cpu 21162
%CPU
5.5

but a top -p 21162 give us 18%. There is something wrong right ?
Nope. But the confusion rise from the fact that ps and top don’t have the same definition of what constitute CPU usage.

top give you an average value of CPU consumption per core on a short period of time (by default 3 seconds). A value of 200% means that the process “hogged” in average two core during the last 3 seconds.

On the other hand ps calculate it value on the process lifetime, and don’t take into account how many core have been hog. So a for example a value of 15.3% means that since the process is running 15.3% of it lifetime it has been bugging on the CPU. The other 84.7% of the time the process was doing nothing, probably waiting for some input to append.

So as you can see this two commands have a very different definition of what CPU usage is, and both value are relevant in their own.

Posted on

[Varnish] Separate cache for mobile device

Let’s say you have a website with two version, one intended for ‘desktop’ browser the other for ‘mobile’ device. Both use the same domain name. How can you get two separate caches ?

The wrong answer

Add the value of the User-Agent field into your vcl_hash function. That simple enough, right ? Wrong ! Very wrong. The User-Agent field is a total mess and it has so many possible values that this setting will make Varnish completely useless. Never do that !

The good answer

To ensure two separate cache you must add into the vcl_hash an header with only two possible value: one for all the ‘desktop’ User-Agent, the other for all the ‘mobile’ ones. That means add a custom header which value will depend on a whole bunch of regex, parsing and sanitizing the horrendous User-Agent value.

Fortunately for you, this annoying task has already been done. You can find a VCL for this specific task here. Import it into your main varnish configuration file and then adjust your vcl_hash for non-static content like this:

sub vcl_hash {
  if (req.url !~ "(?i)\.png|gif|jpeg|jpg|ico|gz|tgz|bz2|tbz|mp3|ogg|zip|rar|otf|ttf|eot|woff|svg|pdf)$") {
     hash_data(req.http.X-UA-Device);
  }
}
Posted on

Fastest method to delete a huge number of files

Let’s said you have a filesystem with a directory containing a huge number of files, something like half a million. What is the fastest method to delete so many files ?

Never use shell expansion

Before answering the question let’s state the obvious: you should never use shell expansion when dealing with a huge number of files.

cd huge_directory
rm -f *
-bash: /bin/rm: Argument list too long

As you can see rm didn’t even start because shell expansion produce a command that exceeds the ARG_MAX limit (128kb since kernel 2.6.23). So if you insist to use rm for the job (and you shouldn’t) at least do it the right way:

rm -rf huge_directory

It will internally list the files and directories it’s going to delete.

Using find

“If you can’t do it in one setting, divide and make a loop”. That’s the strategy of the find command. First let’s use it with the -exec parameter:

time find ./ -type f -exec rm {} \;
real    14m51.735s
user    2m24.330s
sys     9m48.743s

Not so great. In fact using find this way is very inefficient because it spawn an external rm process for each file ! Luckily, find have it own built-in -delete command:

time find ./ -type f -delete
real    5m11.937s
user    0m1.259s
sys     0m28.441s

That’s much better, but we can do better.

Using rsync

Using rsync for this task may seem a little strange in the first place, but it work really really well:

time rsync -a --delete emptydir/ huge_directory/
real    2m52.502s
user    0m2.772s
sys     0m32.649s

Clearly rsync is the winner. But why ? It’s a little tricky.

When deleting a file you invoke the unlink system call. This call removes a ‘link’ to the file’s inode. Once there is no more ‘link’ the system free the associated space by zeroing the inode. Pretty simple stuff. But how do it know which ‘link’ to unlink ? By using another system call that list the content for a given directory: readdir.

Now here the thing. readdir doesn’t list files in-order but randomly (in really not so randomly because it’s depend on inode number) and by packet of 32Kb. When there is a reasonable number of files that perfectly fine and quite efficient, but not in an over-crowded directory.

This behavior is the main reason why using ls or rm in a directory containing millions of files is such a pain in the ass. Each operation make hundred of readdir calls.

On the other hand rsync doesn’t rely on readdir but use it own implementation. It use a single huge buffer and list all files in-reverse order. This way rsync can unlink files after files without glancing a second time at the directory structure. Huge gain of time when dealing with millions of files.

Posted on

[Magento] Various tips

Disable log tables usage

If you already have an external “profiling” tool you can trash the content of several magento table like:

  • log_customer
  • log_quote
  • log_summary
  • log_summary_type
  • log_url
  • log_url_info
  • log_visitor
  • log_visitor_info
  • log_visitor_online

and replace the engine for them by ‘blackhole’. This will significantly improves performance.
You can find more info on the subject here.

Directories and NFS

The following directories can be shared inside NFS share:

  • var/cache
  • var/locks
  • var/media
  • var/exports

It recommended to not put var/report and var/logs inside.

SQL request for finding the number of order

mysql> select count(*), date_format(updated_at, '%Y%m%d %H') as datum from sales_flat_order where created_at > '2013-11-20' group by datum ;
Posted on

Mosh

I think no body reading this blog need an introduction to SSH, the standard of remote access terminal since the end of the 90’s. We all know and love this protocol, and its main implementation openssh. But sometimes SSH strict and clean design can be a pain in the ass. During my on-call duty i sometime have no other choice than to work using only a poor 3G/EDGE mobile access. High latency and intermittent connectivity don’t play well with SSH. Even with a GNU screen session on the remote server that never an enjoyable moment.

It’s in such situations that a tool like mosh become interesting.

What’s Mosh ?

Mosh stand for Mobile Shell. Like SSH that a remote-terminal protocol, but designed with mobile access in mind. It allows roaming, support intermittent connectivity, predictive echoing and local buffering for line typing/editing/deleting (yep openssh waits for the server’s reply before showing you your own typing, now you understand the typing latency). All of these features make it way more convenient to use on a high latency and/or unreliable links than a standard SSH session.

Installing Mosh

Mosh need to be installed on both the client and the server. For Debian, there is only one package simply call mosh. It’s available in the official repository since Debian Wheezy.

Using Mosh

It’s much simpler than what you think. Just type:

mosh username@server

and the mosh command will take care of everything. First it will log you using the ssh command, then start the mosh-daemon on the remote server. After that it close the ssh session and reconnect you to the mosh one. Note that by default the mosh-daemon chose a random UDP port between 60000 and 61000. If like me, you’re not a fan of subnet opening, you can use the -p parameter to force a specific port of your choice.

Posted on

lsof – a more powerful command than you think

lsof is a command used to find out which files are open by which process. But as the Unix way of life is that everything is a file, lsof can be use for a lot more then that:

List processes which opened a specific file

# lsof /var/log/syslog
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
rsyslogd 13315 root    7w   REG   0,36   246213 15074839 /var/log/syslog

List opened files under a directory

# lsof +D /var/log/
COMMAND    PID          USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
php5-fpm  1968          root    2w   REG   0,36      434 15074914 /var/log/php5-fpm.log
php5-fpm  1968          root    5w   REG   0,36      434 15074914 /var/log/php5-fpm.log
apache2   7466          root    2w   REG   0,36      279 15076913 /var/log/apache2/error.log
...

List all open files by a specific process

# lsof -p 1968
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF      NODE NAME
php5-fpm 1968 root  cwd    DIR               0,36     4096  15073281 /
php5-fpm 1968 root  rtd    DIR               0,36     4096  15073281 /
php5-fpm 1968 root  txt    REG               0,36  9110296  15081382 /usr/sbin/php5-fpm
php5-fpm 1968 root  mem    REG              253,0           15081382 /usr/sbin/php5-fpm (path dev=0,36)
...

List opened files based on process names

# lsof -c ssh
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME
sshd     8463 root  cwd    DIR               0,36     4096   15073281 /
sshd     8463 root  rtd    DIR               0,36     4096   15073281 /
sshd     8463 root  txt    REG               0,36   787080   15076801 /usr/sbin/sshd
sshd     8463 root  mem    REG              253,0            15077206 /lib/x86_64-linux-gnu/libnss_files-2.19.so (path dev=0,36)
...

You can use the -c parameter, multiple time.

Show network connections

# lsof -i
COMMAND     PID  USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
ssh         636 daber    3u  IPv4 1381573      0t0  TCP 10.10.32.54:54188->b1.vpn.ti.smile.fr:ssh (ESTABLISHED)
ssh         834 daber    3u  IPv4 1385285      0t0  TCP 10.10.32.54:60902->b3.vpn.ti.smile.fr:ssh (ESTABLISHED)
ssh         892 daber    3u  IPv4 1386338      0t0  TCP 10.10.32.54:39496->b2.vpn.ti.smile.fr:ssh (ESTABLISHED)
chromium   1476 daber   87u  IPv4 1429223      0t0  TCP zapan.dhcp.mpl.intranet:49404->par10s09-in-f35.1e100.net:https (ESTABLISHED)
...

You can also add additional parameters to filter on the port number. For example to only show SSH connections: lsof -i TCP:22
You can also specify a range: lsof -i TCP:1-1024

Show all files opened by a specific user

# lsof -u daber
COMMAND     PID  USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
ssh         636 daber  cwd       DIR               0,35     28672   13238276 /home/daber
ssh         636 daber  rtd       DIR                8,1      4096          2 /
ssh         636 daber  txt       REG                8,1    666088     659561 /usr/bin/ssh
ssh         636 daber  mem       REG                8,1     22952     524315 /lib/x86_64-linux-gnu/libnss_dns-2.19.so
...

Note that you can use ^ to inverse the command (exclude only a particular user).

Kill all process of particular user

# kill `lsof -t -u daber`
Posted on

hping2

hping2 is a network tool able to send custom TCP/UDP/ICMP packets and display target replies. It work pretty much like ping but with far more options. It can be used among other things to:

  • test firewall rules
  • make port scanning
  • test net performance using different protocols, packet size, etc…
  • Path MTU discovery
  • traceroute like under different protocols

Testing port state

One of the most useful use-case of hping is to test if a TCP port is open or not:

# hping -S -p 22 192.70.106.78
HPING 192.70.106.78 (eth0 192.70.106.78): S set, 40 headers + 0 data bytes
len=40 ip=192.70.106.78 ttl=64 DF id=0 sport=22 flags=RA seq=0 win=0 rtt=0.2 ms

The RA flag indicate that the TCP port 22 is closed. The remote host have sent a RST/ACK in response to our SYN packet. If the port was open the flag would have been SA instead.

Not that you can also use the ++ parameter to automatically increase the port number:

# hping -S -p ++80 192.168.10.1

Port scanning

hping can be use as a lightweight port scanner:

# hping -S --scan 20-22,80,8080 -V 192.168.100.1
using eth0, addr: 192.168.100.18, MTU: 1500
Scanning 192.168.100.1 (192.168.100.1), port 20,21,22,80,8080
5 ports to scan, use -V to see all the replies
+----+-----------+---------+---+-----+-----+
|port| serv name |  flags  |ttl| id  | win |
+----+-----------+---------+---+-----+-----+
   20 ftp-data   : ..R.A...  64     0     0
   21 ftp        : ..R.A...  64     0     0
   22 ssh        : .S..A...  64     0  5840
   80 www        : .S..A...  64     0  5840
 8080 http-alt   : .S..A...  64     0  5840
All replies received. Done.
Not responding ports:

Firewall mapping

traceroute is usually the first utility people use for this task but it’s limited to UDP “probe” packets (on port 53 by default). hping can use any protocol:

# hping -z -t 6 -S mail.test.com -p 143
TTL 0 during transit from ip=10.1.5.3
7: TTL 0 during transit from ip=10.1.5.3
8: TTL 0 during transit from ip=10.2.5.3
9: TTL 0 during transit from ip=10.3.5.3
10: TTL 0 during transit from ip=10.4.5.3
11: TTL 0 during transit from ip=10.6.5.3
....
len=46 ip=10.5.5.3 flags=SA DF seq=33 ttl=47 id=0 win=5840 rtt=4341.3ms

Doing a SYN attack

hping can forge packets with a spoofed IP address using the -a parameter. Together with the -i (for interval) option, you can use it to make a SYN attack :

# hping -a 192.168.10.99 -S 192.168.10.33 -p 80 -i u1000

Transferring file

hping can be use in very creative manners, for example to transfer a file between two hosts you have access to, through a very ‘closed’ firewall.

On the receiving end we need to start hping in listener mode, and specify a ‘signature’ string that indicate the beginning of the file content:

# hping 192.168.10.66 --listen signature --safe --icmp > myfile

On the sending side, you must ‘sign’ the packet, with the signature used at the receiving site, and indicate the file to read:

# hping 192.168.10.44 --icmp -d  100 --sign signature
--file myfile

ICMP, TCP or UDP can be use indiscriminately.

Further Reading and sources