[Varnish] Beware of Vary header

Vary is one of the most powerful HTTP response headers. Used correctly it’s possible to do wonderful things with it. Unfortunately most people used it so badly, that today most CDN don’t use it anymore and simply avoid caching if it used for anything else than HTTP compression.

Using it for HTTP compression

Most people use the Vary header exclusively with the Accept-Encoding value in order to cache several version of the same URI depending on the Accept-Encoding field value. Usual values for Accept-Encoding are gzip, deflate and more rarely sdch, but many more exist. Also keep in mind that any combination of these individual values is valid !

To keep the number of cache version at minimum, you can use the following VCL snippet to normalize the Accept-Encoding header, only keeping gzip and deflate as acceptable values:

if (req.http.Accept-Encoding) {
   if (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
   } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
   } else {
      remove req.http.Accept-Encoding;

Using it with other field

It possible to specify any other field into the Vary value and also multiple fields by separating their names with a comma. But i don’t recommend this setup because of how CDN treat it.

Anyways, if you want to use Vary correctly you must always keep in mind these two cardinal rules: never use the raw values of the indicated field and never choose a field with too much variant.

For example the following fields should NEVER be used into a Vary value:

  • Referer: because that utterly stupid
  • Cookie: because that probably the most unique request header ever
  • User-Agent: because with more then 8000 usual variants, you pretty to sure to never use your cache anymore

NAT traversal

In a typical IPSec setup tunnels are established between equipment having public IPs. But what happen if one or both gateway are behind firewalls doing NAT ? As you can imagine this setup simply can’t work. There is two reasons for that.

First IKE exchange can’t be successful because of how NAT translations work. The embedded address of the source computer within the IP payload can’t match the source address of the IKE packet as it is replaced by the address of the NAT device.

Secondly ESP traffic can’t be NATed because there is no port in ESP. Therefore when a client attempts to initiate an ESP connection behind a network device doing NAT, the device is unable to maintain a unique translation state with these packets.

NAT-Traversal to the rescue !

To bypass these limitations, NAT-T was created. The idea is simply to a friendly NAT protocol (UDP) to encapsulate ESP exchange. During phase 1, if NAT-T is used, IKE negotiations switch to UDP port 4500.

After the tunnel is establish, instead of sending plain ESP packets, each one is encapsuled in an UDP packets. This trick allow the device doing NAT to maintain the connection state in its table.

Further Reading and sources