Self-contained maintenance page

Sometimes we need to take our web apps offline temporarily. It could be to do a major upgrade or in an emergency situation. Anyways, you should at least set a static page to notify regular users that the site is currently unavailable. And it’s even better if you can maintain access for you and your fellow developers in the meantime.

Create a maintenance page

First step is to create a good maintenance page. Good means clean, simple, visually attractive and the most important thing: “self-contained” ! So any required resource should be embedded inside a “simple” HTML file.

How to do that ?

For CSS use the tag style to integrate the content into the HTML like this:

<style media="screen" type="text/css">
 ... Add CSS data here ...
</style>

For images encode them in base64 using the base64 command and add the resulting string into the HTML like this:

<img src="data:image/gif;base64,BASE64STRING" alt="my_image" />

You have a good maintenance page ? Perfect, now let talk how to serve it.

Serve the maintenance page

The first component that receive the users requests should be serving the page. Depending your stack that means either the:

  • HTTP server should serve the page
  • HTTP reverse-proxy cache (aka. Varnish) should serve the page

In the first case i highly suggest you to make a dedicated maintenance VirtualHost on a separate port.

In the second case you must modify your vcl_error section with something like this:

if (obj.status == 703) {
    set obj.status = 503;
    set obj.http.Content-Type = "text/html; charset=utf-8";
    synthetic {"
           ... Add HTML content here ...
    "};
    return(deliver);
}

Note the fake HTTP code 703. We will use it later 😉

Maintain access for you, display maintenance page for others

If your maintenance page is served by Varnish use an acl for IP filtering like this:

acl me_and_dev {
"1.2.3.4"/32;
}

sub vcl_recv {
    if (!client.ip ~ me_and_dev) {
            error 703 "Service unavailable";
    }
}

If your maintenance page is served by Apache use an iptable rule to redirect traffic to the maintenance VirtualHost‘s port except for your IP :

iptables -t NAT -A PREROUTING -i eth0 -p tcp --dport 80 \! -s '1.2.3.4' --to-ports 81

Note that if you can’t use IP filtering, other solution are possible like using a special maintenance cookie set after requesting a “secret url”.

[Exim] Smarthost configuration

Let say we already have a functional SMTP server that is responsible for routing all mail for a given domain. Now we add a new server into this network that must be able to send its own emails, for example for administrative purposes (alerts, crontab, logs, emails generated by the applications, etc …). This new MTA doesn’t have to manage mail for users.

The clean way to do this is to configure our exim daemon to send mail via the “official” SMTP server. For this, just modify the /etc/exim4/update-exim4.conf.conf file like this:

dc_eximconfig_configtype='satellite'
dc_other_hostnames=''
dc_local_interfaces='127.0.0.1 ; ::1'
dc_readhost='mydebian.mydomain.com'
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='myrelay.mydomain.com::25'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='true'
dc_mailname_in_oh='true'
dc_localdelivery='mail_spool'

Now the host mydebian.mydomain.com can send emails using myrelay.mydomain.com‘s MTA.

Restricted SFTP access

FTP is an old protocol created when network was a ‘new thing’ and everybody was a ‘care bears’, therefore it’s insecure by design and you shouldn’t even propose it to yours customers. Instead always push the SFTP option first.

For practical usage, there is three little downsides to SFTP use:

  • SFTP doesn’t have it own dedicated port. Personally i like to reuse the ‘Simple File Transfer Protocol’ port (TCP 115) but this ideas is enough to trigger heart attacks to network ‘ayatollah’
  • each user must have an unix account
  • an SFTP access isn’t by default chrooted inside the user directory

SFTP on a dedicated port

Until we have SFTP support inside ProFTPD, the only solution is to spawn a second OpenSSH daemon on a separated port.

Here a snippet of my new setting file /etc/ssh/sftp-115:

Port 115

# Protocol
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UsePrivilegeSeparation yes

# Restriction
AcceptEnv no
AllowAgentForwarding no
AllowTcpForwarding no
Banner no
MaxAuthTries 3

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO
UseDNS no

# Authentication
LoginGraceTime 120
PermitRootLogin no
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile /home/sftp/%u/.ssh/authorized_keys
PasswordAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Disable some options
UsePAM no
X11Forwarding no
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes

# Use sftp-internal (allow chroot)
Subsystem sftp internal-sftp
ForceCommand internal-sftp

Before OpenSSH 4.8 there is no internal-sftp module, therefore you must use the external command sftp-server instead:

Subsystem sftp /usr/lib/openssh/sftp-server

For launching the daemon you can use xinetd, clone and modify your distribution OpenSSH init script or simply add a one-liner inside the /etc/rc.local:

/usr/sbin/sshd -f /etc/ssh/sftp-115

Do as you wish.

Unix user account

Not much to say here, you must create a proper unix account for each SFTP. I suggest you to set the home directory to something like /home/sftp/<user> to distinguish easily theses users from regular ones.

In case you use sftp-server you must also change the user shell value. First add sftp-server as a valid shell :

# echo '/usr/lib/stfp-server' >> /etc/shells

Then for each SFTP unix account:

# usermod -s /usr/lib/sftp-server <user>

Chroot user

If you use internal-sftp simply add the following snippet to /etc/ssh/sftp-115:

ChrootDirectory /home/sftp/%u

For sftp-server, bad news, you simply can’t chroot users.

To give user access to a directory use the mount --bind command:

mount --bind /var/www/foobar /home/user/foobar/www

Don’t forget to add this line into your SFTP daemon init script.

Monty Python Unix Joke

- Stop! Whoever crosseth the bridge of Death, must answer first these questions three, ere the other side he see:
  What is your name?
- Sir Brian of Bell
- What is your quest ?
- I seek the Holy Grail
- What are four lowercase letters that are not legal flag arguments to the Berkeley UNIX version of 'ls' ?
- I, er…. AIIIEEEEEE!

Check TCP/UDP port with netcat

netcat, usually abbreviated in nc, is a network tool able to reads and writes data across network connections using TCP or UDP protocol. It’s a feature-rich tool, that every admin should know.

Check TCP port

nc -zv 192.168.0.10 80
Connection to 192.168.0.10 80 port [tcp/http] succeeded!

Check UDP port

nc -zuv 192.168.0.10 123
Connection to 192.168.0.10 123 port [udp/ntp] succeeded!

Not that contrary to TCP, UDP is a connectionless protocol. In theory if an UDP port is formally “closed” (for example via an iptables REJECT rule) the destination host should respond with an ICMP Port unreachable packet. But a lot of firewall simply silently drop the message, resulting in a wrong ‘succeed’ result.

Scan a range of ports

To find all the open ports in a given range:

nc -zv 192.168.0.10 1-500 2>&1 | grep succeeded
Connection to 192.168.0.10 21 port [tcp/ftp] succeeded!
Connection to 192.168.0.10 22 port [tcp/ssh] succeeded!
Connection to 192.168.0.10 80 port [tcp/http] succeeded!

To run a pseudo-server on a given port

nc -l -p 3873

[Apache] Reload vs restart

People are often confused by the difference between the reload and the restart operation. That lead to a lot of questions like “Can i do a simple reload when changing a parameters into the modphp php.ini setting file ?” or “Does a reload interrupt the service ?

First of all it’s important to understand that the reload operation doesn’t really exist, not in a former sense. When doing a reload the parent process do reload its configuration, but as it doesn’t do anything by himself except spawning child process, the new configuration isn’t effective right away. Then the parent process send a graceful signal to each of its child to exit after finishing their current request (or to exit immediately if they’re not serving anything). As each child dies off the parent replaces it with a new child.

So a reload operation, doesn’t interrupt the service, but it effect isn’t immediate. If you need an immediate effect you must do a restart operation which “violently” kill all the child process. The reload operation also have one big limitation : it doesn’t take into account new files. For enabling a new module or after changing a SSL certificates, you must use the restart operation.

[OpenBSD] relayd

relayd is an open source load balancer which is able to handle protocol layers 3, 4 and 7. It’s the standard “in-house” load-balancing solution of OpenBSD. It can be setup as a forward, reverse or TCP port redirector and/or SSL/TLS ‘terminator’.

Basic commands

Check relayd configuration:

# relayd -n -f /etc/relayd.conf

Show detailed status of hosts and tables:

# relayctl show host

Show detailed status of redirections including the current and average access statistics:

# relayctl show redirects

Show detailed status of relays including the current and average access statistics:

# relayctl show relays

Dump the complete list of running relay sessions:

# relayctl show sessions

Display a list of all relays, redirections, routers, tables, and hosts:

# relayctl show summary

Schedule an immediate check of all hosts:

# relayctl poll

Reloading configuration

You can reload the current configuration file with the command:

# relayctl reload

Beware, this command will flush the current ‘state’ and provoke a short downtime. Also it seems that if you have SSL ressources, reload make relayd crash. So instead of using this command you should make a CARP failover “switch” and restart the process like this :

# pkill relayd && sleep 1 && relayd -f /etc/relayd.conf && relayctl poll

[OpenBSD] Forcing CARP failover

To force a CARP failover from a master firewall/router to its slave, you can manually change the value of the demotion counter like this:

fw1:~# ifconfig -g carp carpdemote 50

This command will increase the advskew value of all CARP interface on the host by the demotion counter value. The interface with the lowest advskew value will be promoted ‘MASTER’. I use a base advskew value of 80 on the master host and 120 on the slave. The failover should be transparent to end users.

You can check the current interfaces states and their advskew values like this:

fw2:~# ifconfig carp | grep adv
        carp: MASTER carpdev em1 vhid 150 advbase 8 advskew 80
        carp: MASTER carpdev em1 vhid 151 advbase 8 advskew 80
        [...]

To switch back simply decrease the demotion counter in order to have a lower value on the master again:

fw1:~# ifconfig -g carp -carpdemote 50

Guru meditation

If you use the HTTP reverse-proxy Varnish or the VirtualBox hypervisor, you probably already encounter a funny Guru meditation error message.

But who is this guru and what is the origin of this message ?

In the 1980s the Amiga computer system was a very popular personal computer brand. Originally intended as a videogame machine but latter reconceived as a general purpose computer, the development of the first Amiga computer took a considerable time. In the interim the Amiga corporation released a number of other products.

One of them was the Joyboard, a balance board peripheral for the Atari 2600. The Joyboard was conceived by installing the four directional latches of a joystick on the bottom of a plastic board. Leaning in a certain direction engaged these latches, controlling the game pretty much like the modern Nintendo Wiiboard does.

According to the legend, in the early development of the AmigaOS, developers became so frustrated with the system’s frequent crashes that, as a relaxation technique, they attempted to sit cross-legged perfectly still on a Joyboard. In this position they look like Indian gurus.

Quickly they developed a little video game where the winner was the one who stayed still the longest without engaging any of Joyboard’s latches. If a player moved too much a guru meditation game-over screen occurred. As an easter-egg this guru meditation was integrated into the AmigaOS as a general error message, and since them became quite popular.