Webserver – Page 2 – Just another Sys Admin blog… wait really ?

2014-09-142017-02-14

[Varnish] Testing ESI support

Since version 2.1 varnish support, at least partially, the ESI protocol. ESI is very useful to improve the cache hit ratio because it allow the splitting of uncachable dynamic pages into an ‘heavy’ static page and one or several ‘light’ dynamic page bloc.

Enable ESI support

For Varnish 2.x add into the vcl_recv:

esi;

For Varnish 3.x the syntax is:

set beresp.do_esi = true;

Testing ESI

To test ESI support, you need to create two files. The first will be a simple HTML ‘template’ file. We will call it frontpage.html:

<html>
<head>
<title>ESI Test Page</title>
</head>
<body>
<p><esi:include src="/backpage.php"/></p>
</body>
</html>

Next we need a file to generate the ‘dynamic’ content. A very simple PHP script like this we be enough:

<?php
echo "ESI is correctly interpreted";
?>

If ESI support is correctly enabled, calling the first page will result into calling the second file as well, displaying the proper “ESI is correctly interpreted” string.

2014-09-042016-09-13

GoAccess

GoAccess is a real-time ncurse weblog analyzer and interactive viewer. Contrary to Awstats and other similar product, GoAccess doesn’t keep any “history”, but in exchange it’s way more faster. Having ‘on the fly’ HTTP statistics is extremely useful when load suddenly increase on a front webserver.

For analyzing a given log file use the -f option, like this:

goaccess -f /var/log/apache/access.log

If you don’t use standard NCSA or common log format, you can specify the log format with --log-format=. For a more permanent solution you can redefine the log-format value inside the /etc/goaccess.conf setting file.

You can also generate an html report like this:

goaccess -f /var/log/apache/access.log -a -o report.html

2014-07-132017-02-14

[Varnish] Purge objects from command line

Purge all content

For Varnish 2.x:

varnishadm -S /etc/varnish/secret -T :6082 'purge.url .*'

For Varnish 3.x:

varnishadm -S /etc/varnish/secret -T :6082 'ban.url .'

Purge objects from a specific URL

For Varnish 2.x:

varnishadm -S /etc/varnish/secret -T :6082 'purge.url /foobar'

For Varnish 3.x:

varnishadm -S /etc/varnish/secret -T :6082 'ban.url == /foobar'

Note that ban accepts regex pattern. By modifying the expression you can create more specific bans. For example, to purge for all CSS files only for a specific domain:

varnishadm -S /etc/varnish/secret -T :6082 'ban req.http.host ~ www.mydomain.com && req.url ~ .css'

Further Reading and sources

2014-06-172016-10-17

[Apache] CORS header

What is CORS ?

CORS or cross-origin resource sharing is a mechanism that allows resources on a web page to be requested from a different domain than the page origin. To do that the CORS mecanism use a specific header: Access-Control-Allow-Origin.

For security reasons, browsers restrict cross-origin HTTP requests initiated from within scripts. For example XMLHttpRequest and Fetch follow the same-origin policy. This header allow you to customize this behavior.

Authorize a domain

Just add to your apache configuration:

Header set Access-Control-Allow-Origin "https://www.domain.com"

Authorize multiple domains

Access-Control-Allow-Origin can take only one value. You could use the value * but that ugly and defeat the whole purpose of filtering request domain origin.

But with a little trickery you can do that:

SetEnvIfNoCase Origin "https?://(domain\.com|staging\.domain\.com)(:\d+)?$" ACAO=$0
Header set Access-Control-Allow-Origin %{ACAO}e env=ACAO

This regex use the content of the Origin header and define a matching Access-Control-Allow-Origin value. Here we authorize both the http and https versions of domain.com and staging.domain.com to load ressources from our domain.

Further Reading and sources

2014-04-222016-11-22

The heartbleed vulnerability

What is heartbleed ?

Heartbleed is the name given to a critical vulnerability in the OpenSSL library. The official designation of this bug is CVE-2014-0160. Theoretically this vulnerability could be use by an attacker to gain access to data transmitted between clients and server, by random chunk of 64ko. Retrieved content could be pretty much anything: session content, password, even the server private key.

How did this happen ?

The vulnerability is due to a missing validation on a variable size. The bug was introduced into OpenSSL 1.0.1 and fixed into version 1.0.1g.

How does it work ?

The following XKCD comic does a pretty good job at explaining the issue in simple terms.

To test if your server is vulnerable

Use this: https://filippo.io/Heartbleed

What to do ?

First don’t panic and make security updates. Check you have an OpenSSL version 1.0.1g or higher. Because of the buzz around the vulnerability most SSL providers offer free certificate regeneration. Jump on the offer, regenerate your server private key and CSR.

Further Reading and sources

2014-03-202017-02-20

[Varnish] Debugging VCL configuration

Dry-run compiling

The first step to debug your varnish configuration is to simply ‘dry-run’ compiling it:

varnishd -Cf /etc/varnish/myconf.vcl

The output of this command will either be a successfully-compiled VCL or an error message telling you on what line the error occurs. Most of the time that enough to fix syntax error.

Add message into syslog

Since branch 3.X varnish can load and use additional module. One of the most useful is the std module. You can use it to add “trace” message into the syslog:

import std;

vcl_recv {
  ...
  std.syslog(0, "Cookie content: " + req.http.Cookie);
}

Use varnishlog

varnishlog is a command line utility that display all varnishd ‘internal’ activities: request handling, request processing, backend checks/request, hash information, objects caching/retrieving, etc… As you can imagine, the output is extremely verbose and kind of frightening. But you can add parameters, to filter it.

For example, to display only hashes:

varnishlog -c -i Hash

To display only ‘User-Agent’ strings:

varnishlog -c -i RxHeader -I User-Agent

To display only POST requests:

varnishlog -c -m RxRequest:POST

To display only request with a 404 response:

varnishlog -c -m TxStatus:404

To filter requests for a specific URL:

varnishlog -c -m RxURL:"/foo/bar.php"

To filter URLs asked to the backend:

varnishlog -b -i TxURL

To filter for a particular IP:

varnishlog -c -o ReqStart 192.168.1.16

Monitoring activities with varnishtop

Another interesting command line tool is varnishtop. Like varnishlog it reads varnishd shared memory logs but instead of displaying current entries, it presents a continuously updated list of the most commonly occurring entries. Like varnishlog you can add additional parameters to filter it output.

2014-02-112017-02-20

[Varnish] Separate cache for mobile device

Let’s say you have a website with two version, one intended for ‘desktop’ browser the other for ‘mobile’ device. Both use the same domain name. How can you get two separate caches ?

The wrong answer

Add the value of the User-Agent field into your vcl_hash function. That simple enough, right ? Wrong ! Very wrong. The User-Agent field is a total mess and it has so many possible values that this setting will make Varnish completely useless. Never do that !

The good answer

To ensure two separate cache you must add into the vcl_hash an header with only two possible value: one for all the ‘desktop’ User-Agent, the other for all the ‘mobile’ ones. That means add a custom header which value will depend on a whole bunch of regex, parsing and sanitizing the horrendous User-Agent value.

Fortunately for you, this annoying task has already been done. You can find a VCL for this specific task here. Import it into your main varnish configuration file and then adjust your vcl_hash for non-static content like this:

sub vcl_hash {
  if (req.url !~ "(?i)\.png|gif|jpeg|jpg|ico|gz|tgz|bz2|tbz|mp3|ogg|zip|rar|otf|ttf|eot|woff|svg|pdf)$") {
     hash_data(req.http.X-UA-Device);
  }
}

2013-10-042017-02-09

Maximum HTTP header size

The HTTP specification (RFC2616 for HTTP/1.1) doesn’t define a maximum header size.
However, in practice, all servers have limits, for header numbers and header field size:

Apache 2.x: 8K
Nginx: 8K
IIS: 8K-16K (depending version)

If a request line exceed the limit a 414 Request-URI Too Large error is returned. If a request header field exceed the limit a 400 Bad Request error is returned. In order to be sure that a request will be proceed by all HTTP server, it’s better to limit the request size to not exceed 8190 Bytes (and yes that include cookie data).

If you can’t do that, the only remaining solution is to increase the limits values. For apache you can play with the LimitRequestFieldSize and LimitRequestLine parameters. For nginx take a look at the large_client_header_buffers parameter.

Keep in mind that increasing theses values to much will seriously degrade performance.

2013-09-262016-09-26

Verify a SSL certificate installation with openssl

Usually people use their browser or a dedicated website like ssltest to test a server certificate installation.

But you can do it directly with the openssl command:

# openssl s_client -CApath /etc/ssl/certs/ -connect www.foobar.com:443
...
...
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: 9E01CD86FA9F1483AD505F17E34C0B9BF99F57BBF9B5E6A5F2946F8858A86807
    Session-ID-ctx: 
    Master-Key: 8ED5443DCD5F6706A0DF5C0196E1B3AFBAAD3FB0B5B680EB212D4FC3F2BCC24209D0E241FBA746D85559CFA8539D99F4
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1474905181
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

The line Verify return code: 0 indicate a correct installation. Note that the -CApath option tells openssl where to look for the CA certificates. On Debian the directory is /etc/ssl/certs/.

2013-05-082017-03-08

[Magento] Session and zend_mm_heap corrupted

Since version 1.3 Magento can use multiple layer of cache, for example a memcached for content cache and session, and a database for the rest. This mechanism work pretty well except when you have a connectivity problem.

If Magento is unable to load a session, the request processing will end immediately preventing Apache from logging it. The only information inside the error log will be an zend_mm_heap corrupted entry.

To ‘fix’ this behavior, you must increase the default timeout for the PHP session handler for a value greater than the TCP re-transmission delay. A 5 seconds value is good enough.