FreeBSD Install: ZFS Filesystems (Take 2)

November 5th, 2011 No comments

Note: This is a repost of FreBSD Install: ZFS Filesystems from a couple of days ago. I’m reposting because I’ve discovered that you can’t use a separate log device on the root zpool. So I’ve had to split the main pool into two, and the SSDs arrived also, so we now have three pools:

  • zroot: The root filesystem (with no /usr, /home, /tmp or /var)
  • zapps: Looks after the main /usr, /home, and /var mountpoints. Has 2GB of mirrored space on the SSDs for the ZIL, and 2x8GB for the L2ARC cache.
  • zflash: Will house /tmp, plus miscellaneous other mount points directly on the flash for ultimate performance.

Now that the partitions are setup and the initial zpool has been created, its time to get going on the filesystem setup.

ZFS is slightly different from a normal filesystem per partition style approach. You can have many filesystems with different settings all sharing the same partition, so you don’t need to worry about running out of space on one area over another. (If you need to limit space for particular areas you can use quotas and reservations).

If you don’t have a three pool setup like this thats fine, the mount points and settings themselves are still relevant, just put everything under zroot (or whatever you named your root pool). Where possible too don’t specify mount points explicitly, its more flexible to let zfs base it on the parent mount point (like a parent directory).

I’m aiming to set it up something like this:

Under zroot

Path Compress? Exec Set UID? Mount Point
zroot No Yes Yes legacy (/)

Under zapps

Path Compress? Exec Set UID? Mount Point
zapps/archive Yes (lzjb) Yes No /archive
zapps/usr No Yes Yes /usr
zapps/home No Yes No /home
zapps/usr/ports Yes (lzjb) Yes No
zapps/usr/ports/distfiles No No No
zapps/usr/ports/packages No No No
zapps/usr/src Yes (lzjb) No No
zapps/var No Yes Yes /var
zapps/var/crash Yes (lzjb) No No
zapps/var/db No No No
zapps/var/db/pkg Yes (lzjb) Yes No
zapps/var/empty No No No
zapps/var/log Yes (lzjb) No No
zapps/var/mail Yes (gzip) No No
zapps/var/vmail Yes (gzip) No No
zapps/var/run No No No
zapps/var/pgsql No Yes Yes
zapps/var/pgsql/data_mirror No Yes Yes
zapps/var/mysql No Yes Yes
zapps/var/www No Yes Yes
zapps/var/www/vhosts_mirror No Yes No
zapps/var/www/logs Yes (lzjb) No No

Under zflash

Path Compress? Exec Set UID? Other
zflash/archive_compile No Yes No /archive/compile
zflash/usr_obj No Yes No /usr/obj
zflash/tmp No Yes No /tmp
zflash/var_tmp Yes (lzjb) Yes No /var/tmp
zflash/pgsql_data No Yes Yes /var/pgsql/data
zflash/www_vhosts No Yes No /var/www/vhosts

What does all that mean?

Compression

ZFS supports compression natively. That means that while a file may appear to be uncompressed and stored in plain text, its actually compressed before it is stored on the disk. This means that a normal text file on the disk takes up less space, naturally there is a performance penalty as the file needs to be uncompressed every time it is accessed. Typically you try and balance the performance vs space argument to get the best result – files that are not normally compressed and not accessed too often are best for compressed filesystems. Frequently accessed files should remain uncompressed.

Some, like /var/log store only text-based log files, so compression fits perfectly. Others, like databases and web roots are frequently accessed and are best left uncompressed for optimum performance.

Execution

The exec option stores whether or not processes are allowed to be executed from inside that filesystem. This is useful from a security perspective – places that should not store executable binaries can be denied them – like /var/log.

Set UID

Another useful security option. The Set UID bit allows an executable process to run as another user. This is useful for privilege escalation (such as processes that need access to root-only files like password utilities). Much like the execution option, you turn this off for filesystems that don’t have executable processes, and especially those that do have executable processes but should not allow for privilege escalation – such as web roots.

Considerations

Why would you have separate filesystems for different directories you ask? Differing setup options and tuning are one reason, and there are many more options available to you (the ZFS section of the FreeBSD Handbook is an awesome resource here). Another is for snapshotting. ZFS allows for live snapshots of the filesystems that can be restored later on.

You don’t have to get this right later on either, you can always create extras later on.

Note also that we only specify the mount point where necessary. That is; whenever there is no parent mount point, or that parent mount point is in another zpool. If you have to change mount points later on, a mount point that is inherited from its parent will change too. Otherwise you have to change each explicitly.

Setup

Alright, now that we know what we’re looking to do, its get it done!

Notes: remember, we named the zfs root filesystem zroot. Compression defaults to off, exec defaults to on and setuid defaults to on. You will also get errors about being unable to mount these new filesystems, you can safely ignore those.

# We left some default mount points, so lets fix that.
$ zfs set mountpoint=/mnt zroot

$ zfs create -o mountpoint=/mnt/usr zapps/usr
$ zfs create -o mountpoint=/mnt/var zapps/var
$ zfs create -o mountpoint=/mnt/archive -o compression=lzjb -o setuid=off zapps/archive
$ zfs create -o mountpoint=/mnt/archive/compile -o setuid=off zflash/archive_compile
$ zfs create -o mountpoint=/mnt/home -o setuid=off zapps/home
$ zfs create -o mountpoint=/mnt/tmp -o setuid=off zflash/tmp
$ zfs create -o compression=lzjb -o exec=on -o setuid=off zapps/usr/ports
$ zfs create -o compression=off -o exec=off -o setuid=off zapps/usr/ports/distfiles
$ zfs create -o compression=off -o exec=off -o setuid=off zapps/usr/ports/packages
$ zfs create -o mountpoint=/mnt/usr/obj -o setuid=off zflash/usr_obj
$ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/usr/src
$ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/var/crash
$ zfs create -o exec=off -o setuid=off zapps/var/db
$ zfs create -o compression=lzjb -o setuid=off zapps/var/db/pkg
$ zfs create -o exec=off -o setuid=off zapps/var/empty
$ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/var/log
$ zfs create -o compression=gzip -o exec=off -o setuid=off zapps/var/mail
$ zfs create -o compression=gzip -o exec=off -o setuid=off zapps/var/vmail
$ zfs create zapps/var/pgsql
$ zfs create -o mountpoint=/mnt/var/pgsql/data -o recordsize=8k zflash/pgsql_data
$ zfs create -o recordsize=8k zapps/var/pgsql/data_mirror
$ zfs create -o recordsize=8k zapps/var/mysql
$ zfs create -o mountpoint=/mnt/var/tmp -o exec=off -o setuid=off zflash/var_tmp
$ zfs create zapps/var/www
$ zfs create -o mountpoint=/mnt/var/www/vhosts -o setuid=off zflash/www_vhosts
$ zfs create -o setuid=off zapps/var/www/vhosts_mirror
$ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/var/www/logs

So now that thats done, lets verify:

zfs list -o name,used,availablecompression,exec,setuid
NAME                         USED  AVAIL  COMPRESS  EXEC  SETUID  MOUNTPOINT
zapps                        994K  65.0G       off    on      on  none
zapps/archive                 31K  65.0G      lzjb    on     off  /mnt/archive
zapps/home                    31K  65.0G       off    on     off  /mnt/home
zapps/usr                    155K  65.0G       off    on      on  /mnt/usr
zapps/usr/ports               93K  65.0G      lzjb    on     off  /mnt/usr/ports
zapps/usr/ports/distfiles     31K  65.0G       off   off     off  /mnt/usr/ports/distfiles
zapps/usr/ports/packages      31K  65.0G       off   off     off  /mnt/usr/ports/packages
zapps/usr/src                 31K  65.0G      lzjb   off     off  /mnt/usr/src
zapps/var                    434K  65.0G       off    on      on  /mnt/var
zapps/var/crash               31K  65.0G      lzjb   off     off  /mnt/var/crash
zapps/var/db                  62K  65.0G       off   off     off  /mnt/var/db
zapps/var/db/pkg              31K  65.0G      lzjb   off     off  /mnt/var/db/pkg
zapps/var/empty               31K  65.0G       off   off     off  /mnt/var/empty
zapps/var/log                 31K  65.0G      lzjb   off     off  /mnt/var/log
zapps/var/mail                31K  65.0G      gzip   off     off  /mnt/var/mail
zapps/var/mysql               31K  65.0G       off    on      on  /mnt/var/mysql
zapps/var/pgsql               62K  65.0G       off    on      on  /mnt/var/pgsql
zapps/var/pgsql/data_mirror   31K  65.0G       off    on      on  /mnt/var/pgsql/data_mirror
zapps/var/vmail               31K  65.0G      gzip   off     off  /mnt/var/vmail
zapps/var/www                 93K  65.0G       off    on      on  /mnt/var/www
zapps/var/www/logs            31K  65.0G      lzjb   off     off  /mnt/var/www/logs
zapps/var/www/vhosts_mirror   31K  65.0G       off    on     off  /mnt/var/www/vhosts_mirror
zflash                       409K  41.1G       off    on      on  none
zflash/archive_compile        31K  41.1G       off    on     off  /mnt/archive/compile
zflash/pgsql_data             31K  41.1G       off    on      on  /mnt/var/pgsql/data
zflash/tmp                    31K  41.1G       off    on     off  /mnt/tmp
zflash/usr_obj                31K  41.1G       off    on     off  /mnt/usr/obj
zflash/www_vhosts             31K  41.1G       off    on     off  /mnt/var/www/vhosts
zflash/var_tmp                31K  41.4G       off    on     off  /mnt/var/tmp
zroot                        116K  1.95G       off    on      on  /mnt

All done! We need to export and re-import the setup now (export makes the ZFS pool available for importing somewhere else). When we do this we’ll save a copy of the zpool.cache file. I think order is important here to prevent missing parents/conflicts:

$ zpool export zflash
$ zpool export zapps
$ zpool export zroot

$ zpool import -o cachefile=/tmp/zpool.cache zroot
$ zpool import -o cachefile=/tmp/zpool.cache zapps
$ zpool import -o cachefile=/tmp/zpool.cache zflash

You shouldn’t get any errors this time, and your entire filesystem should now be available under /mnt.

We need to set the correct permissions on the tmp filesystems, and symlink the home directory around the place. Most guides recommend you store your home directory under the /usr partition and symlink it to /home; but as we only have the one ZFS partition it really doesn’t matter. I prefer to put it directly under /home, and symlink it to /usr/home for compatibility.

chmod 1777 /mnt/tmp
cd /mnt/usr ; ln -s ../home home
chmod 1777 /mnt/var/tmp

Lets leave this post here; I install FreeBSD back in this post. (out of order posting/missing something important fail)

-bok

Categories: Server Build Tags: , ,

Server Build: SSDs are in!

November 4th, 2011 No comments

I got a little tired of waiting for the caddies to arrive so I installed the SSDs in a temporary kind of way. By that I mean I pulled out the SATA brackets from the slots and have connected them all manually for the moment :)

Anyhow, so I now have two SSDs that are all ready for configuration! In this post I’ll partition them, get some mirrored swap setup and the ZFS mirror.

This is mostly the same process as last time; only we now need to be really careful about which disks we use – we don’t want to blow away our existing setup. The new SSDs are /dev/da2 and /dev/da3.

Partitioning

Let’s start by creating GEOM partitions!

$ gpart create -s GPT da2
da2 created
$ gpart create -s GPT da3
da3 created

You’ll notice that this is all the same as last time :)

Lets add a swap partition to each disk. 4GB sounds right (same as our physical memory):

$ gpart add -s 4g -t freebsd-swap -l swap0 da2
da2p1 added
$ gpart add -s 4g -t freebsd-swap -l swap1 da3
da3p1 added

Same as when we were adding the freebsd-boot and freebsd-zfs partitions as last time. Indeed, I’ll partition the remainder of the disk for ZFS later on in the same way.

But first, lets add a device for our ZFS Intent Log. The ZFS Best Practices Guide suggests a ZIL of about half your physical memory, and the ZIL can be mirrored; so lets make a 2GB one:

$ gpart add -s 2g -t freebsd-zfs -l zil0 da2
da2p2 added
$ gpart add -s 2g -t freebsd-zfs -l zil1 da3
da3p2 added

And add some a separate cache device for L2ARC. Calculations vary for how much L2ARC you should have, but it seems to come down to how much data do you want to have available as cache on your SSD. The consensus seems to be about 10 seconds of data at your typical/maximum throughput. Given that Shana isn’t a particularly high performance server, I might just go with 2x 8GB of L2ARC. Lets create the partitions for that:

$ gpart add -s 8g -t freebsd-zfs -l zl2arc0 da2
da2p3 added
$ gpart add -s 8g -t freebsd-zfs -l zl2arc1 da3
da3p3 added

Then setup the rest of the disks for normal ZFS filesystem usage:

$ gpart add -t freebsd-zfs -l flashdisk0 da2
da2p2 added
$ gpart add -t freebsd-zfs -l flashdisk1 da3
da3p2 added

Calling them flashdisk0 and flashdisk1 this time, so we don’t ever confuse them with disk0 and disk1.

Swap Setup

With that done, we can use gmirror to create a mirrored swap device, but first we should load gmirror properly:

$ gmirror load

We’ll need to add geom_mirror_load=”YES” to our /boot/loader.conf after we’ve installed FreeBSD. Don’t forget, ok?

Then we can create (label) the mirror, and add the secondary:

# Create the new mirror, called "swap", add the swap0 partition from da2
$ gmirror label -F -h -b round-robin swap /dev/gpt/swap0

# Add the secondary, which was swap1 on da3
$ gmirror insert -h -p 1 swap /dev/gpt/swap1

You should see nothing if all goes well; but you can check the status with:

$ gmirror status
       Name    Status  Components
mirror/swap  DEGRADED  gpt/swap0 (ACTIVE)
                       gpt/swap1 (SYNCHRONIZING, 10%)

Ok so we’re a bit impatient, but when it goes good:

$ gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gpt/swap0 (ACTIVE)
                       gpt/swap1 (ACTIVE)

And our swap lives! But to actually make use of it you need to tell the system to use the new /dev/mirror/swap device as swap. Add the following to your /etc/fstab (after we’ve installed FreeBSD!):

# Device                Mountpoint      FSType  Options Dump    Pass#
/dev/mirror/swap        none            swap    sw      0       0

Our swap setup is complete!

ZFS Intent Log and Cache Devices

Now we can add the ZIL and L2ARC to our existing zapps zpool. This is pretty straight forward:

# Remember we said that the ZFS log devices can be mirrored

$ zpool add zapps log mirror /dev/gpt/zil0 /dev/gpt/zil1

# And to check it out:
$ zpool status zapps
  pool: zapps
 state: ONLINE
 scan: resilvered 94K in 0h0m with 0 errors on Thu Nov  3 20:01:33 2011
config:

        NAME               STATE      READ WRITE CKSUM
        zapps              ONLINE        0     0     0
          mirror-0         ONLINE        0     0     0
            gpt/appsdisk0  ONLINE        0     0     0
            gpt/appsdisk1  ONLINE        0     0     0
        logs
          mirror-1         ONLINE        0     0     0
            gpt/zil0       ONLINE        0     0     0
            gpt/zil1       ONLINE        0     0     0

errors: No known data errors

That wasn’t so difficult now was it? Its the same again to add the L2ARC:

# L2ARC can't be mirrored; but ZFS can look after itself

$ zpool add zapps cache /dev/gpt/zl2arc0 /dev/gpt/zl2arc1

$ zpool status zapps
  pool: zapps
 state: ONLINE
 scan: resilvered 94K in 0h0m with 0 errors on Thu Nov  3 20:01:33 2011
config:

        NAME               STATE      READ WRITE CKSUM
        zapps              ONLINE        0     0     0
          mirror-0         ONLINE        0     0     0
            gpt/appsdisk0  ONLINE        0     0     0
            gpt/appsdisk1  ONLINE        0     0     0
        logs
          mirror-1         ONLINE        0     0     0
            gpt/zil0       ONLINE        0     0     0
            gpt/zil1       ONLINE        0     0     0
        cache
          gpt/zl2arc0      ONLINE        0     0     0
          gpt/zl2arc1      ONLINE        0     0     0

errors: No known data errors

All done! Looks nice eh?

ZFS Mirror

Before calling it a day, we’ll add the ZFS mirror. I’m going to call the zpool on this one zflash, and we’ll mount the zflash root temporarily under /mnt. Remember what we called the flash disks above.

$ zpool create zflash /dev/gpt/flashdisk0
$ zfs set mountpoint=/mnt zflash
$ zpool attach zflash /dev/gpt/flashdisk0 /dev/gpt/flashdisk1

All done! We now have a nice mirrored flash disk mounted under /mnt.

Summary

So we now have three zpools:

  • zroot: The root filesystem (with no /usr, /home, /tmp or /var)
  • zapps: Looks after the main /usr, /home, and /var mountpoints. Has 2GB of mirrored space on the SSDs for the ZIL, and 2x8GB for the L2ARC cache.
  • zflash: Will house /tmp, plus miscellaneous other mount points directly on the flash for ultimate performance.

Of which, we can confirm the status before closing:

$ zpool status
  pool: zapps
 state: ONLINE
 scan: resilvered 94K in 0h0m with 0 errors on Thu Nov  3 20:01:33 2011
config:

        NAME               STATE      READ WRITE CKSUM
        zapps              ONLINE        0     0     0
          mirror-0         ONLINE        0     0     0
            gpt/appsdisk0  ONLINE        0     0     0
            gpt/appsdisk1  ONLINE        0     0     0
        logs
          mirror-1         ONLINE        0     0     0
            gpt/zil0       ONLINE        0     0     0
            gpt/zil1       ONLINE        0     0     0
        cache
          gpt/zl2arc0      ONLINE        0     0     0
          gpt/zl2arc1      ONLINE        0     0     0

errors: No known data errors

  pool: zflash
 state: ONLINE
 scan: resilvered 94K in 0h0m with 0 errors on Thu Nov  3 20:26:29 2011
config:

        NAME               STATE      READ WRITE CKSUM
        zflash             ONLINE        0     0     0
          mirror-0         ONLINE        0     0     0
            gpt/flashdisk0 ONLINE        0     0     0
            gpt/flashdisk1 ONLINE        0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
 scan: resilvered 86.5K in 0h0m with 0 errors on Thu Nov  3 19:59:18 2011
config:

        NAME               STATE      READ WRITE CKSUM
        zroot              ONLINE        0     0     0
          mirror-0         ONLINE        0     0     0
            gpt/rootdisk0  ONLINE        0     0     0
            gpt/rootdisk1  ONLINE        0     0     0

errors: No known data errors

-bok

Categories: Server Build Tags: , , , , ,

FreeBSD Install: Disk Setup (Take 2)

November 3rd, 2011 No comments

Note: This is a repost of FreBSD Install: Disk Setup from a couple of days ago. I’m reposting because I’ve discovered that you can’t use a separate log device on the root zpool. If you don’t care about that sort of thing, go read the first post – it’ll walk you through partitioning for a single large zpool – which is a perfectly good way to go. Otherwise, read on!

So at this stage I’m installing FreeBSD 9.0-RC1. I figure that by the time I get to actually deploy the server the release version of 9.0 should be out and I can update to that.

So ultimately I’m aiming to have the two 72GB SAS drives in a mirror/RAID1 setup, and two 60GB SSD in a second mirror/RAID1 setup.

I’m also going with ZFS, as the most awesomest filesystem in existence (awesomest is a word, or at least its passing spellcheck!). Given the benefits of allowing ZFS full access to the drives themselves, I’m going to let ZFS take care of the drive mirroring instead of the built in P400i hardware RAID controller.

I’ll be using these instructions to install FreeBSD 9.0-RC1 on a full ZFS system.

So I’ve downloaded and booted off the Live CD! Lets get going.

My two hard disks are da0 and da1 (so thats /dev/da0 and /dev/da1). The SSDs will be on da2 and da3. The hard disks themselves have been configured as individual RAID0 devices in the RAID setup utility.

We’re going to put a boot partition and two ZFS partitions. One for / (root) and the system binaries, and everything else onto a secondary “apps” pool. The secondary will have space on the SSD for ZIL and L2ARC (see the next post).

So we need a writable /tmp filesystem so that we can save the zpool.cache:

umount /dev/md1
mdmfs -s 512M md1 /tmp

Then we create the GUID Partition Table on the first disk (da0):

gpart create -s gpt da0

# Add the boot volume
gpart add -b 34 -s 64k -t freebsd-boot da0

# Add the root partition, 2gb should be enough
gpart add -s 2g -t freebsd-zfs -l rootdisk0 da0

# Add remaining space for ZFS, labelled appsdisk0
gpart add -t freebsd-zfs -l appsdisk0 da0

# Install the GPT/ZFS boot code into the first partition
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

And then the same again for the second disk (da1), except we label it disk1 (see what I did there?):

gpart create -s gpt da1
gpart add -b 34 -s 64k -t freebsd-boot da1
gpart add -s 2g -t freebsd-zfs -l rootdisk1 da1
gpart add -t freebsd-zfs -l appsdisk1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

You’ll probably notice I haven’t configured a swap volume. This will be going on the SSD so we’ll live without it for now.

Let’s move on to creating our mirrored ZFS root pool volume.

Our new drives under the GPT are available at /dev/gpt/rootdisk0 and /dev/gpt/rootdisk1.

zpool create zroot /dev/gpt/rootdisk0
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot

Now all we need to do is create the mirror is attach the second drive:

zpool attach zroot /dev/gpt/rootdisk0 /dev/gpt/rootdisk1

# Now we can check the status of the mirror
zpool status zroot

#   pool: root
#  state: ONLINE
# config:
#
#         NAME               STATE     READ WRITE CKSUM
#         zroot              ONLINE       0     0     0
#           mirror-0         ONLINE       0     0     0
#             gpt/rootdisk0  ONLINE       0     0     0
#             gpt/rootdisk1  ONLINE       0     0     0
#
# errors: No known data errors

Easy!

And to create the second pool:

zpool create zapps /dev/gpt/appsdisk0
zfs set mountpoint=/mnt zapps
zfs set canmount=off zapps
zfs attach zapps /dev/gpt/appsdisk0 /dev/gpt/appsdisk1

We’ll continue with the filesystem layout and setup tomorrow, including details on what I’ll let ZFS compress and what I’m offloading to the SSD.

-bok

Categories: Server Build Tags: , , ,

Server Build: Shana’s Role

November 2nd, 2011 No comments

Typically when you’re building a new server, you need to know what you’re going to use it for. As mentioned in a previous post, Shana will be replacing the existing Tyrande server. So she will need to take on all the roles that Tyrande already fulfils. After that Tyrande will be rebuilt as Taiga, then the two will share the roles, with a bias towards Shana as the more powerful server.

The current roles running on Tyrande:

  • Web Server (Apache)
  • Database Server (PostgreSQL)
  • Database Server (MySQL)
  • SMTP Mail Server (Postfix)
  • POP/IMAP Mail Server (Dovecot)
  • Domain Name Server (DNS – Bind)

In addition to typical services that I run on all my servers:

  • Time Server (NTPD)
  • Firewall (pf)
  • Secure Remote Shell (OpenSSH)
  • Custom Server Monitoring (Meidon)
  • Nightly backups using zfs snapshots and rsync
  • Nightly security updates and vulnerability checks

And the synchronisation, load balancing and failover between Shana and Taiga.

These are the things I’ll be dedicating the following blog posts to. Some are obviously going to take multiple posts. Lets give a bit of an overview here though:

Web Server

I use Apache as my web server. I always have. It hosts blogs and custom projects, and everything in between. Typically I have a preference for PHP over perl, python or ruby. Again as with my other posts, these blogs aren’t to debate my software preferences. You can fight that elsewhere :)

I prefer the OpenBSD Apache layout too, hence the /var/www/ filesystems in the ZFS setup post. I’ll go into detail on installing Apache, PHP, Subversion and its dependencies. We’ll be running the web root of Apache mainly on the SSD, so we’ll need to look at keeping a copy on the normal disk too.

Database Server (PostgreSQL)

PostgreSQL is my SQL database of choice. All of my projects use it for the SQL data storage. I wouldn’t touch MySQL with a 50 foot poll. I’ll go into detail installing it too, and putting its data in /var/pgsql. As a bonus we’ll be running PostgreSQL mainly on the SSD and configuring asynchronous replication to a second instance of PostgreSQL running on Shana that writes to the HDD. This way we can keep a constant backup on less volatile disk that we can manually failover to in an instant.

Database Server (MySQL)

I run MySQL only because of WordPress. That is all.

SMTP Mail Server / POP3/IMAP Mail Server

I use Postfix as my SMTP server of choice and Dovecot for POP3/IMAP. Its fairly lightly utilised but I prefer the flexibility of running my own mail server. Note: I always use my ISPs server for outgoing SMTP, so I don’t have to deal with the difficulties of running my own server – namely blacklists. I’ll detail installing and configuring them to use a SQL database (PostgreSQL) as its list of valid user accounts with aliasing that works at the account level (so an alias can login, even), catch-alls and a bunch of fun stuff.

Domain Name Server

I host my own DNS too using BIND. I have accounts over at easyDNS to have global redundant DNS servers, but they all slave off my primary server. Then I can use normal zone files to manage my domains instead of a web interface (I like the flexibility). I’ll cover setting all that up too.

Time Server

I configure the built in ntpdate and ntpd to keep my local machine time synchronised with an appropriate time server.

Firewall

I use OpenBSD’s Packet Filter (pf) for my firewalling. It is above and beyond the best open source firewall package I’ve found. I think even OS X Lion incorporates pf now. I don’t do too much fancy stuff in pf, just some normalisation, default block, country-based blacklists and bruteforce blacklisting on SSH. (You try to connect to port 22 on my server more than 5 times in 60 seconds and you’re blacklisted).

Secure Remote Shell

Standard config of the built in OpenSSH sshd.

Custom Server Monitoring

I wrote a small PHP-based package a few years ago that runs some commands every minute and saves the output to a PostgreSQL database. It can alert via push notification if something goes wrong, and display a little dashboard thing:

Meidon Dashboard

Nightly Backups using ZFS and Rsync

I set this up a while ago. Its an awesome set of scripts called rsbackup that basically take a list of servers with individual configurations, connects to them, runs any pre-backup commands (like taking ZFS snapshots), then rsyncs any changes down and runs some post-backup commands.

So nightly, my media centre Mac Mini calls out to Tyrande (just as it will do for Shana and Taiga just by adding a configuration file), asks her to take snapshots of the MySQL database, run a pg_dump of the PostgreSQL database and then copies it all back to my Drobo. Neat eh?

I initially did ZFS snapshots for PostgreSQL too but stopped for reasons that I can’t remember. I’ll investigate whether PostgreSQL 9.1 plays more friendly with snapshots and re-evaluate the best way forward for Shana.

Nightly security updates and vulnerability checks

How to configure the FreeBSD to check for security/operating system updates nightly, and ensure that the built in nightly security checks are emailed to you! Can’t stress how important this is.

Sychronisation, Load Balancing and Failover

As mentioned in the original server overview post, I’ll be using CARP for failover and load balancing between Shana and Taiga. So I’ll cover off how to configure all of that too.

So that’s all of the stuff that I’ll be setting up for Shana, and then for Taiga. Should be fun!

FreeBSD Install: Initial software install

November 1st, 2011 No comments

There are some base things that I do to every FreeBSD box I build; regardless of what it is going to run. Software packages that I’ve gotten used to having around.

Semantics: I’ve prefixed any line where I’m running something in a shell with a dollar symbol ($). Typically your root shell would be prefixed by a hash (#) but the syntax highlighting plugin I use doesn’t like that. Later on once I switch to bash it becomes the full [username@hostname:/path/to/current/directory]$ style. Everything is run as root in this post.

Let’s start by updating the ports collection. If you’re not sure what the ports collection is, go read this; note that I’ll always install from the port, never the package (personal preference).

To update ports we use the portsnap utility. It can also initialise your ports collection if you didn’t install it from the DVD.

`portsnap fetch` will download the latest copy of the ports collection.

$ portsnap fetch
# Looking up portsnap.FreeBSD.org mirrors... 5 mirrors found.
# Fetching public key from portsnap2.FreeBSD.org... done.
# Fetching snapshot tag from portsnap2.FreeBSD.org... done.
# Fetching snapshot metadata... done.
# Fetching snapshot generated at Mon Oct 31 11:21:14 EST 2011:
# bca06dba4618d31623f9268301a48429df5c16ca546159100% of   64 MB  411 kBps 00m00s
# Extracting snapshot... done.
# Verifying snapshot integrity... done.
# Fetching snapshot tag from portsnap2.FreeBSD.org... done.
# Fetching snapshot metadata... done.
# Updating from Mon Oct 31 11:21:14 EST 2011 to Mon Oct 31 21:19:34 EST 2011.
# Fetching 4 metadata patches... done.
# Applying metadata patches... done.
# Fetching 0 metadata files... done.
# Fetching 59 patches.....10....20....30....40....50.... done.
# Applying patches... done.
# Fetching 9 new ports or files... done.

Then we use `portsnap update` to bring our collection up to scratch. Of course, portsnap doesn’t like it if the /usr/ports collection was not created by it; so you’ll need to run `portsnap extract` command first to let it rebuild it. Subsequent updates can use `portsnap update`. You can also skip the ports.txz file from the install then.

$ portsnap extract

If you’re a stickler for being always up to date you can always through those into a cron job to update nightly too.

Now to install software! I tend to install:

  • portaudit – provides a system to check if install ports are listed in a database of published security vulnerabilities.
  • bash – my preferred shell.
  • nano – my preferred command line text editor (an update to the pico project).
  • sudo – a way to run commands as other users.
  • wget – a non-interactive network downloader.
  • screen – a full screen window manager that multiplexes a physical terminal between several processes (typically shells). i.e. it provides virtual terminals inside a single terminal.

I’m not here to debate my shell/text editor preferences either, there are plenty of blogs around that do that :-)

So lets install those.

Port Audit

We do portaudit first so that it can check other ports as they are installed.

$ cd /usr/ports/ports-mgmt/portaudit
$ make install clean
# ===>  Vulnerability check disabled, database not found
# ===>  License check disabled, port has not defined LICENSE
# ===>  Extracting for portaudit-0.5.17
# ===>  Patching for portaudit-0.5.17
# ===>  Configuring for portaudit-0.5.17
# ===>  Building for portaudit-0.5.17
# ===>  Installing for portaudit-0.5.17
# ===>   Generating temporary packing list
# ===>  Checking if ports-mgmt/portaudit already installed
#
# ===>  To check your installed ports for known vulnerabilities now, do:
#
#       /usr/local/sbin/portaudit -Fda
#
# ===>   Compressing manual pages for portaudit-0.5.17
# ===>   Registering installation for portaudit-0.5.17
# ===>  Cleaning for portaudit-0.5.17

Like it says, run the check. This will also download the latest copy of the vulnerability database.

$ /usr/local/sbin/portaudit -Fda
# auditfile.tbz                                 100% of   70 kB   56 kBps
# New database installed.
# Database created: Mon Oct 31 21:55:01 EST 2011
# 0 problem(s) in your installed packages found.

A clean system! You’d hope so given thats the first port we installed..

Bash

Installing bash is easy, change to the port directory and run `make install clean`. (The clean merely means that ports will clean up after itself.)

$ cd /usr/ports/shells/bash
$ make install clean
#  <lots of downloading and compilation that I'll skip>
#  Accept the defaults when it gives you configuration options

(NB. In the original attempt to do this I received an “Access Denied” message while trying to run configure. Turns out I had misconfigured the zroot/usr/ports filesystem! You definitely need to be able to execute files on it. I’ve gone back and updated that post to correctly specify exec=on for zroot/usr/ports. To turn that on after the fact use `zfs set exec=on zroot/usr/ports`.)

So finally we’re on a better shell! Lets update our normal user account to use it

$ chsh bok

#  The passwd file entry will appear. Change this line:
#  Shell: /bin/sh
#  to
#  Shell: /usr/local/bin/bash

Then we can install some other stuff.

Nano

Nano is an awesome little text editor that I find much more friendly than vi (though, I am decently proficient in vi).

We’ll want to do something slightly different for nano though. There are two features that I really really really hate in nano. It’s text wrapping and text justification. The former I never want but always happens, the latter I keep activating by accident.

We’re going to disable those two features at compilation time.

$ cd /usr/ports/editors/nano
$ make configure

#  Now we need to edit the config.h file in the build directory
$ vi work/nano-2.2.6/config.h

#  After this line
#  /* #undef DISABLE_JUSTIFY */
#  add
#  #define DISABLE_JUSTIFY

#  After this line
#  /* #undef DISABLE_ROOTWRAPPING */
#  add
#  #define DISABLE_ROOTWRAPPING

#  After this line
#  /* #undef DISABLE_WRAPPING */
#  Add
#  #define DISABLE_WRAPPING

#  and save. We can then continue with the install
$ make install clean

You can set your default editor to nano now if you wish (I’ve changed over to the bash shell here):

[root@shana /usr/ports/editors/nano]$ nano ~bok/.profile

#  Change this line
#  EDITOR=vi;     export EDITOR
#  to
#  EDITOR=nano;   export EDITOR

#  and save.

Whats next?

Sudo

Ah sudo! Another easy one.

[root@shana /usr/ports/editors/nano]$ cd /usr/ports/security/sudo
[root@shana /usr/ports/security/sudo]$ make install clean
#  <skippy>
#
# ==> SECURITY REPORT:
#       This port has installed the following binaries which execute with
#       increased privileges.
# /usr/local/bin/sudo
# /usr/local/bin/sudoedit
#
#       If there are vulnerabilities in these programs there may be a security
#       risk to the system. FreeBSD makes no guarantee about the security of
#       ports included in the Ports Collection. Please type 'make deinstall'
#       to deinstall the port if this is a concern.
#
#       For more information, and contact details about the security
#       status of this software, see the following webpage:
# http://www.courtesan.com/sudo/
# ===>  Cleaning for sudo-1.8.3_1

We’ll configure sudo later.

Wget

Another easy one! You see a pattern here yet? Ports is easy :)

If you forget which port folder something is in just ask whereis:

[root@shana /usr/ports/security/sudo]$ whereis wget
wget: /usr/ports/ftp/wget

Now we can do the rest! Note: wget requires perl. This might be a long install.

[root@shana /usr/ports/security/sudo]$ cd /usr/ports/ftp/wget
[root@shana /usr/ports/ftp/wget]$ make install clean
# <lots of snippage>

Wget is awesome for grabbing stuff. Just use:

$ wget -c "<pasted URL>"

and it will grab it to the current directory. The -c is for resuming/continuing if you happen to interrupt the download.

Screen

You know the process by now!

[root@shana /usr/ports/ftp/wget]$ cd `whereis -q screen`
[root@shana /usr/ports/sysutils/screen]$ make install clean
# <snippage>
# ===> SECURITY REPORT:
#       This port has installed the following binaries which execute with
#       increased privileges.
# /usr/local/bin/screen
#
#       If there are vulnerabilities in these programs there may be a security
#       risk to the system. FreeBSD makes no guarantee about the security of
#       ports included in the Ports Collection. Please type 'make deinstall'
#       to deinstall the port if this is a concern.
#
#       For more information, and contact details about the security
#       status of this software, see the following webpage:
# http://www.gnu.org/software/screen/
# ===>  Cleaning for screen-4.0.3_12

All done!

Now that thats all done, lets check to see what was actually installed.

[root@shana ~]$ pkg_info
# bash-4.1.11         The GNU Project's Bourne Again SHell
# bison-2.4.3,1       A parser generator from FSF, (mostly) compatible with Yacc
# gettext-0.18.1.1    GNU gettext package
# gmake-3.82          GNU version of 'make' utility
# libiconv-1.13.1_1   A character set conversion library
# libidn-1.22         Internationalized Domain Names command line tool
# libtool-2.4_1       Generic shared library support script
# m4-1.4.16,1         GNU m4
# nano-2.2.6          Nano's ANOther editor, an enhanced free Pico clone
# perl-5.12.4_2       Practical Extraction and Report Language
# pkg-config-0.25_1   A utility to retrieve information about installed libraries
# portaudit-0.5.17    Checks installed ports against a list of security vulnerabi
# screen-4.0.3_12     A multi-screen window manager
# sudo-1.8.3_1        Allow others to run commands as root
# wget-1.13.4_1       Retrieve files from the Net via HTTP(S) and FTP

That’s quite a bit for 5 small software utilities, but thats the beauty of ports. It will go and download, compile and install all dependencies. When the time comes it will update them all for you too. And don’t worry – the dependencies here after re-used in a lot of other open source software packages, so there is not much waste.

Just to be sure we can even check them for vulnerabilities:

[root@shana ~]$ portaudit -a
# 0 problem(s) in your installed packages found.

Beautiful!

FreeBSD Install: Installing FreeBSD

October 31st, 2011 No comments

So when we left our install yesterday, we’d pretty much done everything as far as pre-installation setup is concerned, we just needed to copy the files.

This actually is the simplest bit! Its just a matter of extracting the .txz archives onto the new /mnt filesystem. We’ll copy/paste the script from the guide and be done with it.

$ sh
$ cd /usr/freebsd-dist
$ export DESTDIR=/mnt
$ for file in base.txz lib32.txz kernel.txz doc.txz ;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

So in my case I’m installing the base OS, lib32, kernel, document. I didn’t install the source as I have no plans at this stage to rebuild from source. You can always download and install that later. I also didn’t install the ports package as we’ll be fetching and updating that from portsnap anyway.

That’s it! All installed. Some reconfiguration is needed before rebooting onto our shiny new OS.

Save the zpool.cache into the new boot directory (this is where I went wrong the first time – don’t forget this step or it won’t boot!)

$ cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache

And you’re almost good to go – we need to tell the boot loader configuration file to load ZFS and boot from zroot (or we’ll have some issues..), we also need to tell it to load gmirror or our swap won’t work.

$ echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
$ echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
$ echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf

Then load ZFS as part of the boot and daemon starting process:

$ echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf

And configure our swap partition in the /etc/fstab:

$ cat > /mnt/etc/fstab
# Device                Mountpoint      FSType  Options Dump    Pass #
/dev/mirror/swap        none            swap    sw      0       0
EOF (ctrl+d)

And we can change the work prefix directory for ports, so it does all its compilation on the SSDs.

echo 'WRKDIRPREFIX=/usr/obj' >> /mnt/etc/make.conf

Then prepare the filesystem for the reboot.

$ zfs set readonly=on zapps/var/empty
$ zfs umount -a

If you get an error here its because you cd’ed to the /mnt filesystem, cd back out to root and try again. Then we update our mount points.

$ zfs set mountpoint=legacy zroot
$ zfs set mountpoint=/tmp zflash/tmp
$ zfs set mountpoint=/usr zapps/usr
$ zfs set mountpoint=/var zapps/var
$ zfs set mountpoint=/home zapps/home
$ zfs set mountpoint=/archive zapps/archive
$ zfs set mountpoint=/archive/compile zflash/archive_compile
$ zfs set mountpoint=/var/pgsql/data zflash/pgsql_data
$ zfs set mountpoint=/usr/obj zflash/usr_obj
$ zfs set mountpoint=/var/tmp zflash/var_tmp
$ zfs set mountpoint=/var/www/vhosts zflash/www_vhosts

If you’ve added any directories at the root level don’t forget to update those too. Check with `zfs list` to ensure everything has a proper mountpoint. Its important that you do this! Nothing else should be mounted under /mnt, otherwise it will be missing when we reboot. If its a lower-level mount point thats fine; but forgetting /usr could prove complicated at this point.

And on that note!

# Don't forget to pop the DVD out after you hit enter

$ reboot

While we’re waiting, treat yourself to a good cup of tea/coffee/beverage of choice.

You should get a lovely booting system.

When you get to the login screen, login as root (no password needed). The very first thing you should do is set a root password! Do this now. Go on, I’ll wait.

$ passwd
Changing local password for root
New Password:
Retype New Password:

All good. Now you should set the timezone of your machine. You can do this using `sysinstall` if you’re familiar with the old FreeBSD installer, or else `tzsetup`. Likewise add user accounts:

$ adduser
Username: bok
Full name: bok
Uid (Leave empty for default):
Login group [bok]:
Login group is bok. Invite bok into other groups? []: wheel
Login class [default]:
Shell (sh csh tcsh nologin) [sh]:
Home directory [/home/bok]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]:
Enter password:
Enter password again:
Lock out the account after creation? [no]:
Username   : bok
Password   : *****
Full Name  : bok
Uid        : 1001
Class      :
Groups     : bok wheel
Home       : /home/bok
Home Mode  :
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes
adduser: INFO: Successfully added (test) to the user database.
Add another user? (yes/no): no
Goodbye!

From now on you should be using your user account for everything, and only `su`ing to root when you really need the extra privileges.

You also need to configure your hostname, network settings and initial services. I always turn on sshd ASAP so I can stop using the console! Network settings and hostname you can always do through `sysinstall`. I prefer doing it by hand though.

(Note: bce0 is my network card, the secondary NIC in this machine is bce1)

Edit your /etc/rc.conf:

hostname="shana.itransit.com.au"
ifconfig_bce0="inet 172.16.0.4/24"
defaultrouter="172.16.0.1"
zfs_enable="YES"
sshd_enable="YES"

You’ll notice the use of private IP addresses here. That’s because we’re building this machine at home and not directly on the internets. I use the 172.16.0.0/24 subnet here.

Update your /etc/resolv.conf:

domain itransit.com.au
search itransit.com.au odynia.org
nameserver 172.16.0.1

Again temporary name server settings.

At this point you should be able to reboot and have a fully functional system that you can ssh in to from your desktop; it saves you lying on the floor next to the server.

-bok

PS. Because I can, here’s the dmesg for Shana:

Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-RC1 #0: Tue Oct 18 18:51:43 UTC 2011
root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz (2000.12-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f7 Family = 6 Model = f Stepping = 7
Features=0xbfebfbff
Features2=0x4e33d
AMD Features=0x20000800
AMD Features2=0x1
TSC: P-state invariant, performance statistics
real memory = 4294967296 (4096 MB)
avail memory = 4091486208 (3901 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
ACPI Warning: Invalid length for Pm1aControlBlock: 32, using default 16 (20110527/tbfadt-638)
ACPI Warning: Invalid length for Pm2ControlBlock: 32, using default 8 (20110527/tbfadt-638)
ioapic0 irqs 0-23 on motherboard
ioapic1 irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0
cpu0: on acpi0
cpu1: on acpi0
cpu2: on acpi0
cpu3: on acpi0
pcib0: on acpi0
pci0: on pcib0
pcib1: at device 2.0 on pci0
ACPI Warning: For \\_SB_.PCI0.PT02._PRT: Return Package has no elements (empty) (20110527/nspredef-500)
pci9: on pcib1
pcib2: at device 0.0 on pci9
pci10: on pcib2
pcib3: at device 0.0 on pci10
pci11: on pcib3
pcib4: at device 1.0 on pci10
pci14: on pcib4
pcib5: at device 2.0 on pci10
pci15: on pcib5
pcib6: at device 0.3 on pci9
pci16: on pcib6
pcib7: at device 3.0 on pci0
pci6: on pcib7
ciss0: port 0x4000-0x40ff mem 0xfde00000-0xfdefffff,0xfddf0000-0xfddf0fff irq 16 at device 0.0 on pci6
ciss0: PERFORMANT Transport
pcib8: at device 4.0 on pci0
pci19: on pcib8
pcib9: at device 5.0 on pci0
pci22: on pcib9
pcib10: at device 6.0 on pci0
pci2: on pcib10
pcib11: at device 0.0 on pci2
pci3: on pcib11
bce0: mem 0xf8000000-0xf9ffffff irq 18 at device 0.0 on pci3
bce0: /usr/src/sys/dev/bce/if_bce.c(1253): Management firmware enabled but not running!
miibus0: on bce0
brgphy0: PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Ethernet address: 00:1c:c4:42:98:78
bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (1.9.6); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NOT RUNNING!)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib12: at device 7.0 on pci0
pci4: on pcib12
pcib13: at device 0.0 on pci4
pci5: on pcib13
bce1: mem 0xfa000000-0xfbffffff irq 19 at device 0.0 on pci5
bce1: /usr/src/sys/dev/bce/if_bce.c(1253): Management firmware enabled but not running!
miibus1: on bce1
brgphy1: PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Ethernet address: 00:1c:c4:42:28:72
bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (1.9.6); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (NOT RUNNING!)
Coal (RX:6,6,18,18; TX:20,20,80,80)
uhci0: port 0x1000-0x101f irq 16 at device 29.0 on pci0
usbus0: on uhci0
uhci1: port 0x1020-0x103f irq 17 at device 29.1 on pci0
usbus1: on uhci1
uhci2: port 0x1040-0x105f irq 18 at device 29.2 on pci0
usbus2: on uhci2
uhci3: port 0x1060-0x107f irq 19 at device 29.3 on pci0
usbus3: on uhci3
ehci0: mem 0xf7df0000-0xf7df03ff irq 16 at device 29.7 on pci0
usbus4: EHCI version 1.0
usbus4: on ehci0
pcib14: at device 30.0 on pci0
pci1: on pcib14
vgapci0: port 0x3000-0x30ff mem 0xd8000000-0xdfffffff,0xf7ff0000-0xf7ffffff irq 23 at device 3.0 on pci1
pci1: at device 4.0 (no driver attached)
pci1: at device 4.2 (no driver attached)
uhci4: port 0x3800-0x381f irq 22 at device 4.4 on pci1
usbus5: on uhci4
pci1: at device 4.6 (no driver attached)
isab0: at device 31.0 on pci0
isa0: on isab0
atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x500-0x50f irq 17 at device 31.1 on pci0
ata0: on atapci0
acpi_tz0: on acpi0
attimer0: port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
atkbdc0: port 0x60,0x64 irq 1 on acpi0
atkbd0: irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sc0: at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atrtc0: at port 0x70 irq 8 on isa0
Event timer "RTC" frequency 32768 Hz quality 0
ppc0: cannot reserve I/O port range
uart1: at port 0x2f8-0x2ff irq 3 on isa0
p4tcc0: on cpu0
p4tcc1: on cpu1
p4tcc2: on cpu2
p4tcc3: on cpu3
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
usbus5: 12Mbps Full Speed USB v1.0
ugen0.1: at usbus0
uhub0: on usbus0
ugen1.1: at usbus1
uhub1: on usbus1
ugen2.1: at usbus2
uhub2: on usbus2
ugen3.1: at usbus3
uhub3: on usbus3
ugen4.1: at usbus4
uhub4: on usbus4
ugen5.1: <0x103c> at usbus5
uhub5: <0x103c UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
bce0: bce_pulse(): Warning: bootcode thinks driver is absent! (bc_state = 0x00000006)
ugen5.2: at usbus5
ukbd0: on usbus5
kbd2 at ukbd0
ums0: on usbus5
ums0: 3 buttons and [XY] coordinates ID=0
ugen5.3: at usbus5
uhub6: on usbus5
uhub6: 7 ports with 7 removable, self powered
uhub4: 8 ports with 8 removable, self powered
da0 at ciss0 bus 0 scbus0 target 0 lun 0
da0: Fixed Direct Access SCSI-5 device
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 69973MB (143305920 512 byte sectors: 255H 32S/T 17562C)
da1 at ciss0 bus 0 scbus0 target 1 lun 0
da1: Fixed Direct Access SCSI-5 device
da1: 135.168MB/s transfers
da1: Command Queueing enabled
da1: 69973MB (143305920 512 byte sectors: 255H 32S/T 17562C)
da2 at ciss0 bus 0 scbus0 target 2 lun 0
da2: Fixed Direct Access SCSI-5 device
da2: 135.168MB/s transfers
da2: Command Queueing enabled
da2: 57207MB (117161280 512 byte sectors: 255H 32S/T 14358C)
da3 at ciss0 bus 0 scbus0 target 3 lun 0
da3: Fixed Direct Access SCSI-5 device
da3: 135.168MB/s transfers
da3: Command Queueing enabled
da3: 57207MB (117161280 512 byte sectors: 255H 32S/T 14358C)
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
cd0 at ata0 bus 0 scbus2 target 0 lun 0
cd0: Removable CD-ROM SCSI-0 device
SMP: AP CPU #2 Launched!
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray open
Timecounter "TSC-low" frequency 15625908 Hz quality 1000
GEOM_MIRROR: Device mirror/swap launched (2/2).
Trying to mount root from zfs:zroot []...
bce0: link state changed to UP
bce0: Gigabit link up!

Categories: Server Build Tags: ,

FreeBSD Install: ZFS Filesystems

October 30th, 2011 No comments

Note: It seems you cannot use a separate log device on a root pool with zfs. I’ll need to re-do my layout here in a newer post; it should be identical except most of it will move to a separate zpool. If you’re not worried about a separate ZIL; you can safely follow this post.

Now that the partitions are setup and the initial zpool has been created, its time to get going on the filesystem setup.

ZFS is slightly different from a normal filesystem per partition style approach. You can have many filesystems with different settings all sharing the same partition, so you don’t need to worry about running out of space on one area over another. (If you need to limit space for particular areas you can use quotas and reservations).

I’m aiming to set it up something like this:

Path Compress? Exec Set UID? Other
zroot No Yes Yes
zroot/archive Yes (lzjb) Yes No
zroot/archive/compile No Yes No
zroot/usr No Yes Yes
zroot/home No Yes No
zroot/tmp No Yes No
zroot/usr/ports Yes (lzjb) Yes No
zroot/usr/ports/distfiles No No No
zroot/usr/ports/packages No No No
zroot/usr/src Yes (lzjb) No No
zroot/var No Yes Yes
zroot/var/crash Yes (lzjb) No No
zroot/var/db No No No
zroot/var/db/pkg Yes (lzjb) Yes No
zroot/var/empty No No No
zroot/var/log Yes (lzjb) No No
zroot/var/mail Yes (gzip) No No
zroot/var/vmail Yes (gzip) No No
zroot/var/run No No No
zroot/var/tmp Yes (lzjb) Yes No
zroot/var/pgsql No Yes Yes
zroot/var/pgsql/data No Yes Yes recordsize=8k
zroot/var/mysql No Yes Yes recordsize=8k
zroot/var/www No Yes Yes
zroot/var/www/vhosts No Yes No
zroot/var/www/logs Yes (lzjb) No No

What does all that mean?

Compression

ZFS supports compression natively. That means that while a file may appear to be uncompressed and stored in plain text, its actually compressed before it is stored on the disk. This means that a normal text file on the disk takes up less space, naturally there is a performance penalty as the file needs to be uncompressed every time it is accessed. Typically you try and balance the performance vs space argument to get the best result – files that are not normally compressed and not accessed too often are best for compressed filesystems. Frequently accessed files should remain uncompressed.

Some, like /var/log store only text-based log files, so compression fits perfectly. Others, like databases and web roots are frequently accessed and are best left uncompressed for optimum performance.

Execution

The exec option stores whether or not processes are allowed to be executed from inside that filesystem. This is useful from a security perspective – places that should not store executable binaries can be denied them – like /var/log.

Set UID

Another useful security option. The Set UID bit allows an executable process to run as another user. This is useful for privilege escalation (such as processes that need access to root-only files like password utilities). Much like the execution option, you turn this off for filesystems that don’t have executable processes, and especially those that do have executable processes but should not allow for privilege escalation – such as web roots.

Considerations

Why would you have separate filesystems for different directories you ask? Differing setup options and tuning are one reason, and there are many more options available to you (the ZFS section of the FreeBSD Handbook is an awesome resource here). Another is for snapshotting. ZFS allows for live snapshots of the filesystems that can be restored later on.

You don’t have to get this right later on either, you can always create extras later on.

When I get the extra caddies for my SSDs, we’ll move some of these filesystems over to the faster storage.

Setup

Alright, now that we know what we’re looking to do, its get it done!

Notes: remember, we named the zfs root filesystem zroot. Compression defaults to off, exec defaults to on and setuid defaults to on. You will also get errors about being unable to mount these new filesystems, you can safely ignore those.

# We left the zroot with the default mount point, so lets fix that.
zfs set mountpoint=/mnt zroot
zfs create zroot/usr
zfs create zroot/var
zfs create -o compression=lzjb -o setuid=off zroot/archive
zfs create -o setuid=off zroot/archive/compile
zfs create -o setuid=off zroot/home
zfs create -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/vmail
zfs create zroot/var/pgsql
zfs create -o recordsize=8k zroot/var/pgsql/data
zfs create -o recordsize=8k zroot/var/mysql
zfs create zroot/var/www
zfs create -o setuid=off zroot/var/www/vhosts
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/www/logs

So now that thats done, lets verify:

zfs list -o name,used,availablecompression,exec,setuid

# NAME                        USED  AVAIL  COMPRESS  EXEC  SETUID
# zroot                      1.09M  66.9G       off    on      on
# zroot/archive                62K  66.9G      lzjb    on     off
# zroot/archive/compile        31K  66.9G       off    on     off
# zroot/home                   31K  66.9G       off    on     off
# zroot/tmp                    31K  66.9G       off    on     off
# zroot/usr                   155K  66.9G       off    on      on
# zroot/usr/ports              93K  66.9G      lzjb   off     off
# zroot/usr/ports/distfiles    31K  66.9G       off   off     off
# zroot/usr/ports/packages     31K  66.9G       off   off     off
# zroot/usr/src                31K  66.9G      lzjb   off     off
# zroot/var                   496K  66.9G       off    on      on
# zroot/var/crash              31K  66.9G      lzjb   off     off
# zroot/var/db                 62K  66.9G       off   off     off
# zroot/var/db/pkg             31K  66.9G      lzjb    on     off
# zroot/var/empty              31K  66.9G       off   off     off
# zroot/var/log                31K  66.9G      lzjb   off     off
# zroot/var/mail               31K  66.9G      gzip   off     off
# zroot/var/mysql              31K  66.9G       off    on      on
# zroot/var/pgsql              62K  66.9G       off    on      on
# zroot/var/pgsql/data         31K  66.9G       off    on      on
# zroot/var/run                31K  66.9G       off   off     off
# zroot/var/tmp                31K  66.9G      lzjb    on     off
# zroot/var/vmail              31K  66.9G      gzip   off     off
# zroot/var/www                93K  66.9G       off    on      on
# zroot/var/www/logs           31K  66.9G      lzjb   off     off
# zroot/var/www/vhosts         31K  66.9G       off    on     off

All done! We need to export and re-import the setup now (export makes the ZFS pool available for importing somewhere else). When we do this we’ll save a copy of the zpool.cache file.

zpool export zroot
zpool import -o cachefile=/tmp/zpool.cache zroot

You shouldn’t get any errors this time, and your entire filesystem should now be available under /mnt.

We need to set the correct permissions on the tmp filesystems, and symlink the home directory around the place. Most guides recommend you store your home directory under the /usr partition and symlink it to /home; but as we only have the one ZFS partition it really doesn’t matter. I prefer to put it directly under /home, and symlink it to /usr/home for compatibility.

chmod 1777 /mnt/tmp
cd /mnt ; ln -s usr/home home
chmod 1777 /mnt/var/tmp

Lets leave this post here; in the next one we’ll actually install FreeBSD onto our shiny ZFS filesystem.

-bok

Categories: Server Build Tags: , ,

Server Build Overview

October 28th, 2011 No comments

I thought it’d be a good time to do up a quick overview of what I’m aiming to deliver, and the process for getting there.

Currently I have one server (tyrande.itransit.com.au) that hosts almost all of my stuff. She has served me faithfully for several years now with almost no issues to speak of (except one notable temporary failure that triggered a dash to Sydney and back to Melbourne in the same day). This of course leaves me with issues though should she ever die, and if I want to do anything really serious with her it blows my business continuity options to almost zero (I do have nightly backups).

So it looks something like this:

Current Situation

Tyrande is a little bit outdated these days too; she’s a HP DL145r02 with an AMD Opteron 252 running at 2.6GHz with 2GB of PC3200 DDR RAM and 2x80GB SATA HDDs (3.5″). She’s still quite useful though, but she’ll mainly be used as a backup or load balanced server (33% load) going forward.

So I purchased a new server I’m naming Shana. She’s a HP DL360 G5 with a Quad-core Xeon E5335 running at 2.00GHz with 4GB of DDR2 ECC RAM and 2x73GB 15k SAS drives (2.5″), and she has dual power supplies also. To that I’m adding a pair of Patriot PYRO Sanforce SF-2200 60GB SATA3 Solid State Drives to offload some of the work to.

Let's build a new server!

Once Shana has been built, she will be installed into the Rack Unit above Tyrande at the data centre, and I’ll start migrating things over to her. I anticipate the migration taking a couple of weeks.

Migrate data to the new hotness

Once that’s done I’m going to take Tyrande and give her a bit of a makeover. If I can I’ll add another SSD (or two) though that might be difficult as she only has the two drive bays. Then she’ll be wiped and re-christened as Taiga. The Tyrande name will live on as one of the load balanced virtual hosts.

Taiga is born!

During that time, Tyrande/Taiga will be taken from the data centre back to my apartment. Once she’s been rebuilt though she’ll obviously need to be reinstalled. At that time I’ll be setting up the redundancy and load balancing using the Common Address Redundancy Protocol (CARP) over a direct connection between the servers. This direct connection (on the secondary NICs) will allow synchronisation and communication between the servers. I’ll explain all that in another post.

Let's go fishing for CARP

I expect that would take another couple of weeks to build. The ultimate goal is two servers setup and working in parallel. Shana would take 66% of the load and Taiga 33%. Though these numbers can be adjusted as appropriate. In the event of hardware failure the remaining server would take 100% of the load and no one would be the wiser. The same should occur for software failure, but I’ll have to build something to monitor the internal state and demote itself as appropriate.

The Final Goal: Time for a Beer!

After that, the real fun can begin :D

-bok

Categories: Server Build Tags: ,

FreeBSD Install: Disk Setup

October 27th, 2011 No comments

Note: It seems you cannot use a separate log device on a root pool with zfs. I’ll need to redo my layout here a bit in a new post, which is pretty much just splitting the freebsd-zfs partition into two. If you’re not concerned about a separate ZIL, then this post is still relevant.

So at this stage I’m installing FreeBSD 9.0-RC1. I figure that by the time I get to actually deploy the server the release version of 9.0 should be out and I can update to that.

So ultimately I’m aiming to have the two 72GB SAS drives in a mirror/RAID1 setup, and two 60GB SSD in a second mirror/RAID1 setup.

I’m also going with ZFS, as the most awesomest filesystem in existence (awesomest is a word, or at least its passing spellcheck!). Given the benefits of allowing ZFS full access to the drives themselves, I’m going to let ZFS take care of the drive mirroring instead of the built in P400i hardware RAID controller.

I’ll be using these instructions to install FreeBSD 9.0 on a full ZFS system.

So I’ve downloaded and booted off the Live CD! Lets get going.

My two hard disks are da0 and da1 (so thats /dev/da0 and /dev/da1). I’m waiting on the caddies for the SSDs, so we’ll install and configure those later. The hard disks themselves have been configured as individual RAID0 devices in the RAID setup utility.

So we need a writable /tmp filesystem so that we can save the zpool.cache:

umount /dev/md1
mdmfs -s 512M md1 /tmp

Then we create the GUID Partition Table on the first disk (da0):

gpart create -s gpt da0

# Add the boot volume
gpart add -b 34 -s 64k -t freebsd-boot da0

# Add remaining space for ZFS, labelled disk0
gpart add -t freebsd-zfs -l disk0 da0

# Install the GPT/ZFS boot code into the first partition
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

And then the same again for the second disk (da1), except we label it disk1 (see what I did there?):

gpart create -s gpt da1
gpart add -b 34 -s 64k -t freebsd-boot da1
gpart add -t freebsd-zfs -l disk1 da1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1

You’ll probably notice I haven’t configured a swap volume. This will be going on the SSD so we’ll live without it for now.

Let’s move on to creating our mirrored ZFS pool volume.

Our new drives under the GPT are available at /dev/gpt/disk0 and /dev/gpt/disk1.

zpool create zroot /dev/gpt/disk0
zpool set bootfs=zroot zroot
zfs set checksum=fletcher4 zroot

Now all we need to do is create the mirror is attach the second drive:

zpool attach root /dev/gpt/disk0 /dev/gpt/disk1

# Now we can check the status of the mirror
zpool status zroot

#   pool: root
#  state: ONLINE
# config:
#
#         NAME           STATE     READ WRITE CKSUM
#         zroot          ONLINE       0     0     0
#           mirror-0     ONLINE       0     0     0
#             gpt/disk0  ONLINE       0     0     0
#             gpt/disk1  ONLINE       0     0     0
#
# errors: No known data errors

Easy!

We’ll continue with the filesystem layout and setup tomorrow (or probably Saturday, as I’ll be drinking), including details on what I’ll let ZFS compress and what I’m offloading to the SSD.

-bok

Categories: Server Build Tags: , ,

Server Build

October 27th, 2011 No comments

Well its been a while!

Recently I’ve purchased a new server to update and augment the existing server that hosts this blog and all of my other internet endeavours.

The server in question is a HP DL360 G5. It’s a 1RU server with a Quad-core 2.00GHz CPU and 4GB of memory. It came with 2x 73GB 15K 2.5″ SAS drives.

The intention is to install this new server into the rack unit above my existing server as the new primary server, then I can rebuild the existing (older) server as a secondary. I generally have a preference for FreeBSD as my unix variety of choice. I’ll not debate the merits of that here; there are plenty of comparisons out there already.

Over the next several days or weeks I figured I’d blog about the process of setting it up. If only so the next time I have to rebuild this thing I have a record.

Categories: FreeBSD, Server Build Tags: