Odynia.org blog – 2011

Typically when you’re building a new server, you need to know what you’re going to use it for. As mentioned in a previous post, Shana will be replacing the existing Tyrande server. So she will need to take on all the roles that Tyrande already fulfils. After that Tyrande will be rebuilt as Taiga, then the two will share the roles, with a bias towards Shana as the more powerful server.

The current roles running on Tyrande:

Web Server (Apache)
Database Server (PostgreSQL)
Database Server (MySQL)
SMTP Mail Server (Postfix)
POP/IMAP Mail Server (Dovecot)
Domain Name Server (DNS – Bind)

In addition to typical services that I run on all my servers:

Time Server (NTPD)
Firewall (pf)
Secure Remote Shell (OpenSSH)
Custom Server Monitoring (Meidon)
Nightly backups using zfs snapshots and rsync
Nightly security updates and vulnerability checks

And the synchronisation, load balancing and failover between Shana and Taiga.

These are the things I’ll be dedicating the following blog posts to. Some are obviously going to take multiple posts. Lets give a bit of an overview here though:

Web Server

I use Apache as my web server. I always have. It hosts blogs and custom projects, and everything in between. Typically I have a preference for PHP over perl, python or ruby. Again as with my other posts, these blogs aren’t to debate my software preferences. You can fight that elsewhere

I prefer the OpenBSD Apache layout too, hence the /var/www/ filesystems in the ZFS setup post. I’ll go into detail on installing Apache, PHP, Subversion and its dependencies. We’ll be running the web root of Apache mainly on the SSD, so we’ll need to look at keeping a copy on the normal disk too.

Database Server (PostgreSQL)

PostgreSQL is my SQL database of choice. All of my projects use it for the SQL data storage. I wouldn’t touch MySQL with a 50 foot poll. I’ll go into detail installing it too, and putting its data in /var/pgsql. As a bonus we’ll be running PostgreSQL mainly on the SSD and configuring asynchronous replication to a second instance of PostgreSQL running on Shana that writes to the HDD. This way we can keep a constant backup on less volatile disk that we can manually failover to in an instant.

Database Server (MySQL)

I run MySQL only because of WordPress. That is all.

SMTP Mail Server / POP3/IMAP Mail Server

I use Postfix as my SMTP server of choice and Dovecot for POP3/IMAP. Its fairly lightly utilised but I prefer the flexibility of running my own mail server. Note: I always use my ISPs server for outgoing SMTP, so I don’t have to deal with the difficulties of running my own server – namely blacklists. I’ll detail installing and configuring them to use a SQL database (PostgreSQL) as its list of valid user accounts with aliasing that works at the account level (so an alias can login, even), catch-alls and a bunch of fun stuff.

Domain Name Server

I host my own DNS too using BIND. I have accounts over at easyDNS to have global redundant DNS servers, but they all slave off my primary server. Then I can use normal zone files to manage my domains instead of a web interface (I like the flexibility). I’ll cover setting all that up too.

Time Server

I configure the built in ntpdate and ntpd to keep my local machine time synchronised with an appropriate time server.

Firewall

I use OpenBSD’s Packet Filter (pf) for my firewalling. It is above and beyond the best open source firewall package I’ve found. I think even OS X Lion incorporates pf now. I don’t do too much fancy stuff in pf, just some normalisation, default block, country-based blacklists and bruteforce blacklisting on SSH. (You try to connect to port 22 on my server more than 5 times in 60 seconds and you’re blacklisted).

Secure Remote Shell

Standard config of the built in OpenSSH sshd.

Custom Server Monitoring

I wrote a small PHP-based package a few years ago that runs some commands every minute and saves the output to a PostgreSQL database. It can alert via push notification if something goes wrong, and display a little dashboard thing:

Meidon Dashboard

Nightly Backups using ZFS and Rsync

I set this up a while ago. Its an awesome set of scripts called rsbackup that basically take a list of servers with individual configurations, connects to them, runs any pre-backup commands (like taking ZFS snapshots), then rsyncs any changes down and runs some post-backup commands.

So nightly, my media centre Mac Mini calls out to Tyrande (just as it will do for Shana and Taiga just by adding a configuration file), asks her to take snapshots of the MySQL database, run a pg_dump of the PostgreSQL database and then copies it all back to my Drobo. Neat eh?

I initially did ZFS snapshots for PostgreSQL too but stopped for reasons that I can’t remember. I’ll investigate whether PostgreSQL 9.1 plays more friendly with snapshots and re-evaluate the best way forward for Shana.

Nightly security updates and vulnerability checks

How to configure the FreeBSD to check for security/operating system updates nightly, and ensure that the built in nightly security checks are emailed to you! Can’t stress how important this is.

Sychronisation, Load Balancing and Failover

As mentioned in the original server overview post, I’ll be using CARP for failover and load balancing between Shana and Taiga. So I’ll cover off how to configure all of that too.

So that’s all of the stuff that I’ll be setting up for Shana, and then for Taiga. Should be fun!

There are some base things that I do to every FreeBSD box I build; regardless of what it is going to run. Software packages that I’ve gotten used to having around.

Semantics: I’ve prefixed any line where I’m running something in a shell with a dollar symbol ($). Typically your root shell would be prefixed by a hash (#) but the syntax highlighting plugin I use doesn’t like that. Later on once I switch to bash it becomes the full [username@hostname:/path/to/current/directory]$ style. Everything is run as root in this post.

Let’s start by updating the ports collection. If you’re not sure what the ports collection is, go read this; note that I’ll always install from the port, never the package (personal preference).

To update ports we use the portsnap utility. It can also initialise your ports collection if you didn’t install it from the DVD.

portsnap fetch will download the latest copy of the ports collection.

$ portsnap fetch
# Looking up portsnap.FreeBSD.org mirrors... 5 mirrors found.
# Fetching public key from portsnap2.FreeBSD.org... done.
# Fetching snapshot tag from portsnap2.FreeBSD.org... done.
# Fetching snapshot metadata... done.
# Fetching snapshot generated at Mon Oct 31 11:21:14 EST 2011:
# bca06dba4618d31623f9268301a48429df5c16ca546159100% of   64 MB  411 kBps 00m00s
# Extracting snapshot... done.
# Verifying snapshot integrity... done.
# Fetching snapshot tag from portsnap2.FreeBSD.org... done.
# Fetching snapshot metadata... done.
# Updating from Mon Oct 31 11:21:14 EST 2011 to Mon Oct 31 21:19:34 EST 2011.
# Fetching 4 metadata patches... done.
# Applying metadata patches... done.
# Fetching 0 metadata files... done.
# Fetching 59 patches.....10....20....30....40....50.... done.
# Applying patches... done.
# Fetching 9 new ports or files... done.

$ portsnap fetch

# Looking up portsnap.FreeBSD.org mirrors... 5 mirrors found.

# Fetching public key from portsnap2.FreeBSD.org... done.

# Fetching snapshot tag from portsnap2.FreeBSD.org... done.

# Fetching snapshot metadata... done.

# Fetching snapshot generated at Mon Oct 31 11:21:14 EST 2011:

# bca06dba4618d31623f9268301a48429df5c16ca546159100% of 64 MB 411 kBps 00m00s

# Extracting snapshot... done.

# Verifying snapshot integrity... done.

# Fetching snapshot tag from portsnap2.FreeBSD.org... done.

# Fetching snapshot metadata... done.

# Updating from Mon Oct 31 11:21:14 EST 2011 to Mon Oct 31 21:19:34 EST 2011.

# Fetching 4 metadata patches... done.

# Applying metadata patches... done.

# Fetching 0 metadata files... done.

# Fetching 59 patches.....10....20....30....40....50.... done.

# Applying patches... done.

# Fetching 9 new ports or files... done.

Then we use portsnap update to bring our collection up to scratch. Of course, portsnap doesn’t like it if the /usr/ports collection was not created by it; so you’ll need to run portsnap extract command first to let it rebuild it. Subsequent updates can use portsnap update. You can also skip the ports.txz file from the install then.

$ portsnap extract

1	$ portsnap extract

If you’re a stickler for being always up to date you can always through those into a cron job to update nightly too.

Now to install software! I tend to install:

portaudit – provides a system to check if install ports are listed in a database of published security vulnerabilities.
bash – my preferred shell.
nano – my preferred command line text editor (an update to the pico project).
sudo – a way to run commands as other users.
wget – a non-interactive network downloader.
screen – a full screen window manager that multiplexes a physical terminal between several processes (typically shells). i.e. it provides virtual terminals inside a single terminal.

I’m not here to debate my shell/text editor preferences either, there are plenty of blogs around that do that

So lets install those.

Port Audit

We do portaudit first so that it can check other ports as they are installed.

$ cd /usr/ports/ports-mgmt/portaudit
$ make install clean
# ===>  Vulnerability check disabled, database not found
# ===>  License check disabled, port has not defined LICENSE
# ===>  Extracting for portaudit-0.5.17
# ===>  Patching for portaudit-0.5.17
# ===>  Configuring for portaudit-0.5.17
# ===>  Building for portaudit-0.5.17
# ===>  Installing for portaudit-0.5.17
# ===>   Generating temporary packing list
# ===>  Checking if ports-mgmt/portaudit already installed
# 
# ===>  To check your installed ports for known vulnerabilities now, do:
# 
#       /usr/local/sbin/portaudit -Fda
# 
# ===>   Compressing manual pages for portaudit-0.5.17
# ===>   Registering installation for portaudit-0.5.17
# ===>  Cleaning for portaudit-0.5.17

$ cd /usr/ports/ports-mgmt/portaudit

$ make install clean

# ===> Vulnerability check disabled, database not found

# ===> License check disabled, port has not defined LICENSE

# ===> Extracting for portaudit-0.5.17

# ===> Patching for portaudit-0.5.17

# ===> Configuring for portaudit-0.5.17

# ===> Building for portaudit-0.5.17

# ===> Installing for portaudit-0.5.17

# ===> Generating temporary packing list

# ===> Checking if ports-mgmt/portaudit already installed

# ===> To check your installed ports for known vulnerabilities now, do:

# /usr/local/sbin/portaudit -Fda

# ===> Compressing manual pages for portaudit-0.5.17

# ===> Registering installation for portaudit-0.5.17

# ===> Cleaning for portaudit-0.5.17

Like it says, run the check. This will also download the latest copy of the vulnerability database.

$ /usr/local/sbin/portaudit -Fda
# auditfile.tbz                                 100% of   70 kB   56 kBps
# New database installed.
# Database created: Mon Oct 31 21:55:01 EST 2011
# 0 problem(s) in your installed packages found.

$ /usr/local/sbin/portaudit -Fda

# auditfile.tbz 100% of 70 kB 56 kBps

# New database installed.

# Database created: Mon Oct 31 21:55:01 EST 2011

# 0 problem(s) in your installed packages found.

A clean system! You’d hope so given thats the first port we installed..

Bash

Installing bash is easy, change to the port directory and run make install clean. (The clean merely means that ports will clean up after itself.)

$ cd /usr/ports/shells/bash
$ make install clean
#  <lots of downloading and compilation that I'll skip>
#  Accept the defaults when it gives you configuration options

$ cd /usr/ports/shells/bash

$ make install clean

# <lots of downloading and compilation that I'll skip>

# Accept the defaults when it gives you configuration options

(NB. In the original attempt to do this I received an “Access Denied” message while trying to run configure. Turns out I had misconfigured the zroot/usr/ports filesystem! You definitely need to be able to execute files on it. I’ve gone back and updated that post to correctly specify exec=on for zroot/usr/ports. To turn that on after the fact use zfs set exec=on zroot/usr/ports.)

So finally we’re on a better shell! Lets update our normal user account to use it

$ chsh bok

#  The passwd file entry will appear. Change this line:
#  Shell: /bin/sh
#  to
#  Shell: /usr/local/bin/bash

$ chsh bok

# The passwd file entry will appear. Change this line:

# Shell: /bin/sh

# to

# Shell: /usr/local/bin/bash

Then we can install some other stuff.

Nano

Nano is an awesome little text editor that I find much more friendly than vi (though, I am decently proficient in vi).

We’ll want to do something slightly different for nano though. There are two features that I really really really hate in nano. It’s text wrapping and text justification. The former I never want but always happens, the latter I keep activating by accident.

We’re going to disable those two features at compilation time.

$ cd /usr/ports/editors/nano
$ make configure

#  Now we need to edit the config.h file in the build directory
$ vi work/nano-2.2.6/config.h

#  After this line
#  /* #undef DISABLE_JUSTIFY */
#  add
#  #define DISABLE_JUSTIFY

#  After this line
#  /* #undef DISABLE_ROOTWRAPPING */
#  add
#  #define DISABLE_ROOTWRAPPING

#  After this line
#  /* #undef DISABLE_WRAPPING */
#  Add
#  #define DISABLE_WRAPPING

#  and save. We can then continue with the install
$ make install clean

$ cd /usr/ports/editors/nano

$ make configure

# Now we need to edit the config.h file in the build directory

$ vi work/nano-2.2.6/config.h

# After this line

# /* #undef DISABLE_JUSTIFY */

# add

# #define DISABLE_JUSTIFY

# After this line

# /* #undef DISABLE_ROOTWRAPPING */

# add

# #define DISABLE_ROOTWRAPPING

# After this line

# /* #undef DISABLE_WRAPPING */

# Add

# #define DISABLE_WRAPPING

# and save. We can then continue with the install

$ make install clean

You can set your default editor to nano now if you wish (I’ve changed over to the bash shell here):

[root@shana /usr/ports/editors/nano]$ nano ~bok/.profile

#  Change this line
#  EDITOR=vi;     export EDITOR
#  to
#  EDITOR=nano;   export EDITOR

#  and save.

[root@shana /usr/ports/editors/nano]$ nano ~bok/.profile

# Change this line

# EDITOR=vi; export EDITOR

# to

# EDITOR=nano; export EDITOR

# and save.

Whats next?

Sudo

Ah sudo! Another easy one.

[root@shana /usr/ports/editors/nano]$ cd /usr/ports/security/sudo
[root@shana /usr/ports/security/sudo]$ make install clean
#  <skippy>
# 
# ==> SECURITY REPORT: 
#       This port has installed the following binaries which execute with
#       increased privileges.
# /usr/local/bin/sudo
# /usr/local/bin/sudoedit
# 
#       If there are vulnerabilities in these programs there may be a security
#       risk to the system. FreeBSD makes no guarantee about the security of
#       ports included in the Ports Collection. Please type 'make deinstall'
#       to deinstall the port if this is a concern.
# 
#       For more information, and contact details about the security
#       status of this software, see the following webpage: 
# http://www.courtesan.com/sudo/
# ===>  Cleaning for sudo-1.8.3_1

[root@shana /usr/ports/editors/nano]$ cd /usr/ports/security/sudo

[root@shana /usr/ports/security/sudo]$ make install clean

# <skippy>

# ==> SECURITY REPORT:

# This port has installed the following binaries which execute with

# increased privileges.

# /usr/local/bin/sudo

# /usr/local/bin/sudoedit

# If there are vulnerabilities in these programs there may be a security

# risk to the system. FreeBSD makes no guarantee about the security of

# ports included in the Ports Collection. Please type 'make deinstall'

# to deinstall the port if this is a concern.

# For more information, and contact details about the security

# status of this software, see the following webpage:

# http://www.courtesan.com/sudo/

# ===> Cleaning for sudo-1.8.3_1

We’ll configure sudo later.

Wget

Another easy one! You see a pattern here yet? Ports is easy

If you forget which port folder something is in just ask whereis:

[root@shana /usr/ports/security/sudo]$ whereis wget
wget: /usr/ports/ftp/wget

1 2	[root@shana /usr/ports/security/sudo]$ whereis wget wget: /usr/ports/ftp/wget

Now we can do the rest! Note: wget requires perl. This might be a long install.

[root@shana /usr/ports/security/sudo]$ cd /usr/ports/ftp/wget
[root@shana /usr/ports/ftp/wget]$ make install clean
# <lots of snippage>

[root@shana /usr/ports/security/sudo]$ cd /usr/ports/ftp/wget

[root@shana /usr/ports/ftp/wget]$ make install clean

# <lots of snippage>

Wget is awesome for grabbing stuff. Just use:

$ wget -c "<pasted URL>"

1	$ wget -c "<pasted URL>"

and it will grab it to the current directory. The -c is for resuming/continuing if you happen to interrupt the download.

Screen

You know the process by now!

[root@shana /usr/ports/ftp/wget]$ cd `whereis -q screen`
[root@shana /usr/ports/sysutils/screen]$ make install clean
# <snippage>
# ===> SECURITY REPORT: 
#       This port has installed the following binaries which execute with
#       increased privileges.
# /usr/local/bin/screen
#
#       If there are vulnerabilities in these programs there may be a security
#       risk to the system. FreeBSD makes no guarantee about the security of
#       ports included in the Ports Collection. Please type 'make deinstall'
#       to deinstall the port if this is a concern.
#
#       For more information, and contact details about the security
#       status of this software, see the following webpage: 
# http://www.gnu.org/software/screen/
# ===>  Cleaning for screen-4.0.3_12

[root@shana /usr/ports/ftp/wget]$ cd `whereis -q screen`

[root@shana /usr/ports/sysutils/screen]$ make install clean

# <snippage>

# ===> SECURITY REPORT:

# This port has installed the following binaries which execute with

# increased privileges.

# /usr/local/bin/screen

# If there are vulnerabilities in these programs there may be a security

# risk to the system. FreeBSD makes no guarantee about the security of

# ports included in the Ports Collection. Please type 'make deinstall'

# to deinstall the port if this is a concern.

# For more information, and contact details about the security

# status of this software, see the following webpage:

# http://www.gnu.org/software/screen/

# ===> Cleaning for screen-4.0.3_12

All done!

Now that thats all done, lets check to see what was actually installed.

[root@shana ~]$ pkg_info
# bash-4.1.11         The GNU Project's Bourne Again SHell
# bison-2.4.3,1       A parser generator from FSF, (mostly) compatible with Yacc
# gettext-0.18.1.1    GNU gettext package
# gmake-3.82          GNU version of 'make' utility
# libiconv-1.13.1_1   A character set conversion library
# libidn-1.22         Internationalized Domain Names command line tool
# libtool-2.4_1       Generic shared library support script
# m4-1.4.16,1         GNU m4
# nano-2.2.6          Nano's ANOther editor, an enhanced free Pico clone
# perl-5.12.4_2       Practical Extraction and Report Language
# pkg-config-0.25_1   A utility to retrieve information about installed libraries
# portaudit-0.5.17    Checks installed ports against a list of security vulnerabi
# screen-4.0.3_12     A multi-screen window manager
# sudo-1.8.3_1        Allow others to run commands as root
# wget-1.13.4_1       Retrieve files from the Net via HTTP(S) and FTP

[root@shana ~]$ pkg_info

# bash-4.1.11 The GNU Project's Bourne Again SHell

# bison-2.4.3,1 A parser generator from FSF, (mostly) compatible with Yacc

# gettext-0.18.1.1 GNU gettext package

# gmake-3.82 GNU version of 'make' utility

# libiconv-1.13.1_1 A character set conversion library

# libidn-1.22 Internationalized Domain Names command line tool

# libtool-2.4_1 Generic shared library support script

# m4-1.4.16,1 GNU m4

# nano-2.2.6 Nano's ANOther editor, an enhanced free Pico clone

# perl-5.12.4_2 Practical Extraction and Report Language

# pkg-config-0.25_1 A utility to retrieve information about installed libraries

# portaudit-0.5.17 Checks installed ports against a list of security vulnerabi

# screen-4.0.3_12 A multi-screen window manager

# sudo-1.8.3_1 Allow others to run commands as root

# wget-1.13.4_1 Retrieve files from the Net via HTTP(S) and FTP

That’s quite a bit for 5 small software utilities, but thats the beauty of ports. It will go and download, compile and install all dependencies. When the time comes it will update them all for you too. And don’t worry – the dependencies here after re-used in a lot of other open source software packages, so there is not much waste.

Just to be sure we can even check them for vulnerabilities:

[root@shana ~]$ portaudit -a
# 0 problem(s) in your installed packages found.

1 2	[root@shana ~]$ portaudit -a # 0 problem(s) in your installed packages found.

Beautiful!

Monthly archives for November, 2011

FreeBSD Install: ZFS Filesystems (Take 2)

Server Build: SSDs are in!

FreeBSD Install: Disk Setup (Take 2)

Server Build: Shana’s Role

FreeBSD Install: Initial software install