DNS Workshop

Saturday Jun 16, 2012

Supervising BIND 9

BIND 9 is a mature piece of software. Compared with its predecessors BIND 4 and BIND 8, it is more stable and has less security problems. One reason for this is the "Design by contract" programming style used by the BIND 9 team. BIND 9 is very paranoid about data it consumes, and about its internal data structures. Once BIND 9 finds an unexpected state in its internal data-structures, it terminates the DNS server process instead of continue to run with wrong data (and risking a security vulnerability).

While this behavior is good for security, it is very bad for service uptime. The DNS server process terminates, and with it the DNS service. Users (Customers) do not not so much care about security if they cannot reach Facebook. BIND 9 had several incidents in the past years where BIND 9 terminated because of issues inside the code or data-structures, like "BIND 9 Resolver crashes after logging an error in query.c".

BIND 10 is aiming to solve this, as project lead Shane Kerr writes in "Software Robustness and BIND 10". But until BIND 10 arrives, a work-around is needed for BIND 9.

The real issue for the DNS service is not BIND 9 terminating on bad data, but that BIND 9 cannot restart after the fact. There is no "supervisor" process in BIND 9.

Some operating systems have a build-in solution: MacOS X has launchd, and the BIND 9 version Apple delivers with the OS is automatically restarted should it terminate unexpected. Solaris has SMF (Service Management Facility), and BIND 9 can be integrated into SMF. Unbuntu Linux now has upstart, and Fedora systemd, which can also monitor processes and restart them if needed.

For Unix and Linux operating systems that do not ship with a process supervisor solution, supervisord is a nice and easy to setup solution. Supervisord comes as a package with many Linux distributions, and it also works on the BSD Unixes. The configuration below is for OpenBSD, but should require only minor tweaks to run on other Unix systems as well.

Installation

Supervisord is written in Python (2.4 - 2.7) and can be installed from source (where we have to download and install all dependencies) or with the help of setuptools, which takes care of downloading and installing all dependencies. Below I use setuptools:

bash# sh setuptools-0.6c11-py2.7.egg
bash# easy_install supervisor

A basic configuration file for BIND 9 "named"

Below is my basic /etc/supervisord.conf configuration file for one service, the BIND 9 DNS Server:

[unix_http_server]
file = /tmp/supervisor.sock
chmod = 0777
chown= nobody:nogroup

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock

[supervisord]
logfile = /var/log/supervisord.log
logfile_maxbytes = 10MB
logfile_backups=10
loglevel = info
pidfile = /var/run/supervisord.pid
identifier = supervisor
directory = /tmp

[program:named]
command=/usr/sbin/named -f
process_name=%(program_name)s
numprocs=1
directory=/var/named
priority=100
autostart=true
autorestart=unexpected
startsecs=5
startretries=3
exitcodes=0,2
stopsignal=TERM
stopwaitsecs=10
redirect_stderr=false
stdout_logfile=/var/log/named_supervisord.log
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB

starting "supervisord"

Once the configuration file is in place, we can start supervisord. Make sure that BIND 9 is not started, else you will end up with two instances of the BIND 9 server running, which is not a good idea. Also make sure that supervisord will be started on reboot of the server, either trough a startscript or other means. The Supervisord packages coming with Linux distributions install a startscript.

bash# supervisord
bash# tail /var/log/supervisord.log
2012-06-16 16:59:48,812 INFO Increased RLIMIT_NOFILE limit to 1024
2012-06-16 16:59:48,949 INFO RPC interface 'supervisor' initialized
2012-06-16 16:59:48,953 INFO RPC interface 'supervisor' initialized
2012-06-16 16:59:48,963 INFO daemonizing the supervisord process
2012-06-16 16:59:48,964 INFO set current directory: '/tmp'
2012-06-16 16:59:48,967 INFO supervisord started with pid 14724
2012-06-16 16:59:49,976 INFO spawned: 'named' with pid 16701
2012-06-16 16:59:55,020 INFO success: named entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)

Great, supervisord has started, and it also started the BIND 9 process "named". DNS is working now.

simulating a BIND 9 crash

to simulate a BIND 9 crash, we "kill" the BIND 9 named process:

bash# ps aux | grep named
_syslogd 32633  0.0  0.1   512   648 ??  I     17Apr12    2:28.76 syslogd -a /var/named/dev/log -a /var/empty/dev/log
root     16701  0.0  0.8  5684  6500 ??  I      4:59PM    0:00.50 /usr/sbin/named -f
bash# bash-3.2# kill -9 16701
bash# tail supervisord.log 
2012-06-16 17:03:29,192 INFO exited: named (terminated by SIGKILL; not expected)
2012-06-16 17:03:30,201 INFO spawned: 'named' with pid 9832
bash#

Works as a expected. Supervisord has detected that the BIND 9 process has terminated, and has restarted a new one. DNS is still up and running.

Controlling supervisord

The supervisord can be controlled from the commandline using the supervisorctl command. A list of all a control commands can be found with "help", and a description of each command with "help command":

bash# supervisorctl help

default commands (type help ):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update 
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

bash# supervisorctl help status
status                  Get all process status info.
status            Get status on a single process by name.
status      Get status on multiple named processes.

bash# supervisorctl status
named                            RUNNING    pid 25770, uptime 0:00:12

bash# supervisorctl stop named
named: stopped

bash# supervisorctl start named
named: started

Now, whenever there is an assertion error in the code triggered, BIND 9 will terminate, but supervisord will bring it back from the dead. Your DNS service stays up, and the users and customers happy.

Read the supervisord documentation on how to setup event notifications, so that you get an E-Mail on the event that BIND 9 has been restarted. There might be a security vulnerability nontheless, which you would like to report to bind9-bugs@isc.org.

Of course supervisord can be used to restart other processes as well, including other types of DNS Servers (NSD, Unbound, dnsmasq ...).

Comments:

Post a Comment:
  • HTML Syntax: Allowed

Calendar

Feeds

Search

Links

Navigation