Just another Sys Admin blog… wait really ?

2010-12-072017-02-09

Parallel SSH is a useful tool designed to run a command on multiple servers in parallel. An advantage of PSSH over some alternatives (like DSH) is the manner outputs are stored: all outputs are sent to a folder containing an output file per server.

Using PSSH

First create your servers list, using the syntax host:port user :

vi /tmp/hosts_list.txt
server1
server2
server3:2222 foobar

Then create two directories: one for standard outputs, the second for error outputs.

mkdir /tmp/output
mkdir /tmp/error

Then execute your command:

parallel-ssh -h /tmp/hosts_list.txt -o /tmp/output/ -e /tmp/error/ uptime

Make a file immutable

Native linux partition format (like ext2/3/4) have a pretty cool functionality: file attribute. Theses attributes give you some control on how the kernel treat a particular file. The command chattr and lsattr allow you to change and check theses attributes.

One attribute that can be a life saver for system administrator is the ‘immutable’ one. When flagged ‘immutable’ no one, not even root, can change a file. This allow you to prevent any accidental modification. For example let say we want to prevent any change on the /etc/shadow file:

# chattr +i /etc/shadow

When modification are required, we simply remove the flag:

# chattr -i /etc/shadow

There is a lot of other attributes to play with. Check chattr man page for more detail on them.

2010-10-312016-08-31

[OpenBSD] ftp-proxy and ephemeral port

As you may already know, FTP is a pain in the a** for firewall configuration and doesn’t work well through NAT. Hopefully OpenBSD propose an elegant solution: diverting FTP traffic through a proxy server while dynamically modify Packet Filter’s rules on the fly 🙂

This proxy is very simple to enable. Just add something like this in your PF setting file:

anchor "ftp-proxy/*"
pass in quick on $int_if proto tcp to port 21 divert-to 127.0.0.1 port 8021

Then start the ftp-proxy daemon. By default it’s bound on TCP 8021.

Now you may encounter a connection issue with some ‘old’ FTP client in active mode. The reason for that is that ftp-proxy doesn’t strictly follow the RFC 959. In order to avoid port collisions ftp-proxy use an ephemeral port as a source port instead of the port 20. To force a very ‘RFC-compliant’ behaviour add the option -r to startup like this:

vi /etc/rc.conf.local
ftpproxy_flags="-r"

2010-10-232017-06-23

Rsnapshot

rsnapshot is a utility for making local and remote backup. It is written in perl and use rsync for data transfert, and hard-link for deduplication. You can find more information about it, on the official website.

2010-09-182016-08-23

[Debian] Network bonding

Network bonding is a technique that ‘bind’ multiple network interfaces into a single channel/NIC in order to increase redundancy or throughput.

Depending the operating system other term are used like :

OpenBSD: network trunking
Cisco: Etherchannel
HP-UX: Auto-port
AIX: Link-aggregation

Don’t be fooled by marketing term, in the end the concept is always to aggregate multiple interfaces into a single virtual one.

If you want to increase redundancy your ideal setup should be aggregating two interfaces using two different network cards connected to two separate switches. The bonding will be in active / passive mode, only using one port a time. This setup not only ensures high availability operation, but more importantly, allows for maintenance of network switches without service interruption.

Here’s how to make the bonding on Debian Lenny :

Prerequisites

install ifenslave
kernel module ‘bonding’ loaded

Modify network setting

lenny:~# vi /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto bond0
iface bond0 inet static
      slaves eth0 eth1
      bond_mode active-backup
      bond_miimon 100
      bond_downdelay 200
      bond_updelay 200
      address xxx.xxx.xxx.xxx
      netmask xxx.xxx.xxx.xxx
      network xxx.xxx.xxx.xxx
      gateway xxx.xxx.xxx.xxx
lenny:~# /etc/init.d/networking restart

The slaves command specify which interfaces to aggregate, and bond_mod how. Here we use active-backup but if you setup bonding in order to increase throughput check the balance_rr mode instead.

Further Reading and sources

2010-09-172017-02-09

[OpenVZ] FTP inside containers

To enable FTP inside containers, you must first make sure the proper modules are loaded on the host:

modprobe ip_conntrack
modprobe ip_conntrack_ftp

Don’t forget to add them to /etc/modules
Them inside the /etc/vz/vz.conf setting file add/modify the following line:

IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length iptable_nat ipt_state ipt_conntrack ip_conntrack_ftp"

Then restart the vz service.

2010-08-162017-02-09

IPMI on dedibox server

Check if your hardware is compatible

Most middle and all high-end models are IPMI compatible. To check if your server is, log into the dedibox website then: Quick access -> Server selection -> Hardware -> IPMI information and configuration. If the hardware is compatible an IP is already assigned to the IPMI controller.

Into the Access list -> Add IP address you can modify the IPs authorized to connect to the IPMI controller. The modification isn’t taken into account immediately, so maybe you have to wait a couple of hours.

Dedibox configuration

Under Debian the ipmitools package provide the appropriate modules and tools to send and receive IPMI orders. But before using it you have to load two kernel modules:

# modprobe ipmi_si
# modprobe ipmi_devintf

Don’t forget to add them into the /etc/modules file.

Then you must create an IPMI user:

# ipmitool -U root user set name 8 foobar
# ipmitool -U root user set password 8 foofoo
# ipmitool -U root user enable 8

Test your setup

You can check your setup using a simple status command:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis status

Serial Over Lan

SOL is a useful feature that give you access to the server’s serial port over its IPMI network address. If you configure the server OS to have a tty on this physical interface, then you can do a lot of interesting stuff, like gathering data for troubleshoot a crashed server or live-checking the server boot process.

First modify the grub setting file /boot/grub/menu.lst and add BEFORE the “AUTOMAGIC KERNELS LIST” section this line:

serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

Then replace the line

# defoptions=quiet

# defoptions=quiet console=tty0 console=ttyS0,9600n8

Don’t forget to make an update-grub to apply the modifications.
Then modify the /etc/inittab file to add the serial interface to the getty entries:

T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100

Reboot the server. Now that you have a working tty on the serial port you can enable SOL like this:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX sol activate

Usefull IPMI commands

Here a short list of some of the most useful IPMI commands:

– Do a complete electrical reset (stop, wait the start):

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis power cycle

– Do an hardware power reset:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis power reset

– Do an ACPI reset:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis  power soft

– Shutdown the server:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis  power off

– Start the server:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis  power on

– Get the server status:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis status

– Make an hardware self-check:

# ipmitool -I lanplus -U foobar -P foofoo -H XX.XX.XX.XX chassis selftest

Further Reading and sources

http://documentation.dedibox.fr/doku.php?id=gestion:ipmi

2010-08-142016-08-23

[Postgres] Pg8.3 – Warm stand-by replication

Unlike Mysql, Postgres doesn’t have a native replication process. Nevertheless it’s possible to built a warm stand-by replicate using asynchronous log transfer.

Prerequisites

SSH access between your master and slave nodes
rsync
pg_standby

Enable archive log and log shipping

First create a directory for shipped log on the slave node :

mkdir /var/lib/postgresql/alog/

Then enable archive log and log shipping into the master setting file :

master:~# vi /etc/postgresql/8.3/main/postgresql.conf
archive_mode = on
archive_command = 'rsync -a %p slave:/var/lib/postgresql/alog/%f'
archive_timeout = 60
master:~# /etc/init.d/postgresql-8.3 restart

The archive_command will be executed each time the archive_timeout is reached or after the log file grow by 16Mo. After setting these parameters and restarting the postgresql service you can watch the archive directory and you should start seeing WAL files being transmitted on your slave node.

Enable replication

Now you need to configure your stand-by server for “continous recovery mode” using archive log has data source. The ‘pg_standby’ tool will be usefull for that.

slave:~$ vi /var/lib/postgresql/8.3/main/recovery.conf
restore_command = '/usr/lib/postgresql/8.3/bin/pg_standby -d /var/lib/postgresql/alog/%f %p %r >> /var/log/postgresql/pg_standby.log 2>&1'
slave:~$ /etc/init.d/postgresql-8.3 start

Ta-dah your warm stand-by replicate should be ready. You can check the /var/log/postgresql/pg_standby.log for more info. Note that you can’t connect to the slave postgres anymore, this behaviour is perfectly normal.

Further Reading and sources

http://www.postgresql.org/docs/8.3/interactive/warm-standby.html

2010-08-112017-02-09

Memory scanner and editor

If you know the Windows program ‘ArtMoney’ you already have an idea what a memory scanner/editor like Scanmem could be used for.

For other people, basically this kind of tools allow you to search for variables in memory of a running process, generally using one initial value and then a second. When you have the precise offset, you can set the value as you wish (and then have a toon of money, lives or ammo).

This kind of tool could also be used to analyze an hostile process on a compromised machine but cheating in games is way more cool ^^

Just another Sys Admin blog… wait really ?

Posts

Reverse SSH tunnel

Parallel SSH

Make a file immutable

[OpenBSD] ftp-proxy and ephemeral port

Rsnapshot

[Debian] Network bonding

[OpenVZ] FTP inside containers

IPMI on dedibox server

[Postgres] Pg8.3 – Warm stand-by replication

Memory scanner and editor