FreeBSD Install: ZFS Filesystems (Take 2)
Note: This is a repost of FreBSD Install: ZFS Filesystems from a couple of days ago. I’m reposting because I’ve discovered that you can’t use a separate log device on the root zpool. So I’ve had to split the main pool into two, and the SSDs arrived also, so we now have three pools:
- zroot: The root filesystem (with no /usr, /home, /tmp or /var)
- zapps: Looks after the main /usr, /home, and /var mountpoints. Has 2GB of mirrored space on the SSDs for the ZIL, and 2x8GB for the L2ARC cache.
- zflash: Will house /tmp, plus miscellaneous other mount points directly on the flash for ultimate performance.
Now that the partitions are setup and the initial zpool has been created, its time to get going on the filesystem setup.
ZFS is slightly different from a normal filesystem per partition style approach. You can have many filesystems with different settings all sharing the same partition, so you don’t need to worry about running out of space on one area over another. (If you need to limit space for particular areas you can use quotas and reservations).
If you don’t have a three pool setup like this thats fine, the mount points and settings themselves are still relevant, just put everything under zroot (or whatever you named your root pool). Where possible too don’t specify mount points explicitly, its more flexible to let zfs base it on the parent mount point (like a parent directory).
I’m aiming to set it up something like this:
Under zroot
| Path | Compress? | Exec | Set UID? | Mount Point |
|---|---|---|---|---|
| zroot | No | Yes | Yes | legacy (/) |
Under zapps
| Path | Compress? | Exec | Set UID? | Mount Point |
|---|---|---|---|---|
| zapps/archive | Yes (lzjb) | Yes | No | /archive |
| zapps/usr | No | Yes | Yes | /usr |
| zapps/home | No | Yes | No | /home |
| zapps/usr/ports | Yes (lzjb) | Yes | No | |
| zapps/usr/ports/distfiles | No | No | No | |
| zapps/usr/ports/packages | No | No | No | |
| zapps/usr/src | Yes (lzjb) | No | No | |
| zapps/var | No | Yes | Yes | /var |
| zapps/var/crash | Yes (lzjb) | No | No | |
| zapps/var/db | No | No | No | |
| zapps/var/db/pkg | Yes (lzjb) | Yes | No | |
| zapps/var/empty | No | No | No | |
| zapps/var/log | Yes (lzjb) | No | No | |
| zapps/var/mail | Yes (gzip) | No | No | |
| zapps/var/vmail | Yes (gzip) | No | No | |
| zapps/var/run | No | No | No | |
| zapps/var/pgsql | No | Yes | Yes | |
| zapps/var/pgsql/data_mirror | No | Yes | Yes | |
| zapps/var/mysql | No | Yes | Yes | |
| zapps/var/www | No | Yes | Yes | |
| zapps/var/www/vhosts_mirror | No | Yes | No | |
| zapps/var/www/logs | Yes (lzjb) | No | No |
Under zflash
| Path | Compress? | Exec | Set UID? | Other |
|---|---|---|---|---|
| zflash/archive_compile | No | Yes | No | /archive/compile |
| zflash/usr_obj | No | Yes | No | /usr/obj |
| zflash/tmp | No | Yes | No | /tmp |
| zflash/var_tmp | Yes (lzjb) | Yes | No | /var/tmp |
| zflash/pgsql_data | No | Yes | Yes | /var/pgsql/data |
| zflash/www_vhosts | No | Yes | No | /var/www/vhosts |
What does all that mean?
Compression
ZFS supports compression natively. That means that while a file may appear to be uncompressed and stored in plain text, its actually compressed before it is stored on the disk. This means that a normal text file on the disk takes up less space, naturally there is a performance penalty as the file needs to be uncompressed every time it is accessed. Typically you try and balance the performance vs space argument to get the best result – files that are not normally compressed and not accessed too often are best for compressed filesystems. Frequently accessed files should remain uncompressed.
Some, like /var/log store only text-based log files, so compression fits perfectly. Others, like databases and web roots are frequently accessed and are best left uncompressed for optimum performance.
Execution
The exec option stores whether or not processes are allowed to be executed from inside that filesystem. This is useful from a security perspective – places that should not store executable binaries can be denied them – like /var/log.
Set UID
Another useful security option. The Set UID bit allows an executable process to run as another user. This is useful for privilege escalation (such as processes that need access to root-only files like password utilities). Much like the execution option, you turn this off for filesystems that don’t have executable processes, and especially those that do have executable processes but should not allow for privilege escalation – such as web roots.
Considerations
Why would you have separate filesystems for different directories you ask? Differing setup options and tuning are one reason, and there are many more options available to you (the ZFS section of the FreeBSD Handbook is an awesome resource here). Another is for snapshotting. ZFS allows for live snapshots of the filesystems that can be restored later on.
You don’t have to get this right later on either, you can always create extras later on.
Note also that we only specify the mount point where necessary. That is; whenever there is no parent mount point, or that parent mount point is in another zpool. If you have to change mount points later on, a mount point that is inherited from its parent will change too. Otherwise you have to change each explicitly.
Setup
Alright, now that we know what we’re looking to do, its get it done!
Notes: remember, we named the zfs root filesystem zroot. Compression defaults to off, exec defaults to on and setuid defaults to on. You will also get errors about being unable to mount these new filesystems, you can safely ignore those.
# We left some default mount points, so lets fix that. $ zfs set mountpoint=/mnt zroot $ zfs create -o mountpoint=/mnt/usr zapps/usr $ zfs create -o mountpoint=/mnt/var zapps/var $ zfs create -o mountpoint=/mnt/archive -o compression=lzjb -o setuid=off zapps/archive $ zfs create -o mountpoint=/mnt/archive/compile -o setuid=off zflash/archive_compile $ zfs create -o mountpoint=/mnt/home -o setuid=off zapps/home $ zfs create -o mountpoint=/mnt/tmp -o setuid=off zflash/tmp $ zfs create -o compression=lzjb -o exec=on -o setuid=off zapps/usr/ports $ zfs create -o compression=off -o exec=off -o setuid=off zapps/usr/ports/distfiles $ zfs create -o compression=off -o exec=off -o setuid=off zapps/usr/ports/packages $ zfs create -o mountpoint=/mnt/usr/obj -o setuid=off zflash/usr_obj $ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/usr/src $ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/var/crash $ zfs create -o exec=off -o setuid=off zapps/var/db $ zfs create -o compression=lzjb -o setuid=off zapps/var/db/pkg $ zfs create -o exec=off -o setuid=off zapps/var/empty $ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/var/log $ zfs create -o compression=gzip -o exec=off -o setuid=off zapps/var/mail $ zfs create -o compression=gzip -o exec=off -o setuid=off zapps/var/vmail $ zfs create zapps/var/pgsql $ zfs create -o mountpoint=/mnt/var/pgsql/data -o recordsize=8k zflash/pgsql_data $ zfs create -o recordsize=8k zapps/var/pgsql/data_mirror $ zfs create -o recordsize=8k zapps/var/mysql $ zfs create -o mountpoint=/mnt/var/tmp -o exec=off -o setuid=off zflash/var_tmp $ zfs create zapps/var/www $ zfs create -o mountpoint=/mnt/var/www/vhosts -o setuid=off zflash/www_vhosts $ zfs create -o setuid=off zapps/var/www/vhosts_mirror $ zfs create -o compression=lzjb -o exec=off -o setuid=off zapps/var/www/logs
So now that thats done, lets verify:
zfs list -o name,used,availablecompression,exec,setuid NAME USED AVAIL COMPRESS EXEC SETUID MOUNTPOINT zapps 994K 65.0G off on on none zapps/archive 31K 65.0G lzjb on off /mnt/archive zapps/home 31K 65.0G off on off /mnt/home zapps/usr 155K 65.0G off on on /mnt/usr zapps/usr/ports 93K 65.0G lzjb on off /mnt/usr/ports zapps/usr/ports/distfiles 31K 65.0G off off off /mnt/usr/ports/distfiles zapps/usr/ports/packages 31K 65.0G off off off /mnt/usr/ports/packages zapps/usr/src 31K 65.0G lzjb off off /mnt/usr/src zapps/var 434K 65.0G off on on /mnt/var zapps/var/crash 31K 65.0G lzjb off off /mnt/var/crash zapps/var/db 62K 65.0G off off off /mnt/var/db zapps/var/db/pkg 31K 65.0G lzjb off off /mnt/var/db/pkg zapps/var/empty 31K 65.0G off off off /mnt/var/empty zapps/var/log 31K 65.0G lzjb off off /mnt/var/log zapps/var/mail 31K 65.0G gzip off off /mnt/var/mail zapps/var/mysql 31K 65.0G off on on /mnt/var/mysql zapps/var/pgsql 62K 65.0G off on on /mnt/var/pgsql zapps/var/pgsql/data_mirror 31K 65.0G off on on /mnt/var/pgsql/data_mirror zapps/var/vmail 31K 65.0G gzip off off /mnt/var/vmail zapps/var/www 93K 65.0G off on on /mnt/var/www zapps/var/www/logs 31K 65.0G lzjb off off /mnt/var/www/logs zapps/var/www/vhosts_mirror 31K 65.0G off on off /mnt/var/www/vhosts_mirror zflash 409K 41.1G off on on none zflash/archive_compile 31K 41.1G off on off /mnt/archive/compile zflash/pgsql_data 31K 41.1G off on on /mnt/var/pgsql/data zflash/tmp 31K 41.1G off on off /mnt/tmp zflash/usr_obj 31K 41.1G off on off /mnt/usr/obj zflash/www_vhosts 31K 41.1G off on off /mnt/var/www/vhosts zflash/var_tmp 31K 41.4G off on off /mnt/var/tmp zroot 116K 1.95G off on on /mnt
All done! We need to export and re-import the setup now (export makes the ZFS pool available for importing somewhere else). When we do this we’ll save a copy of the zpool.cache file. I think order is important here to prevent missing parents/conflicts:
$ zpool export zflash $ zpool export zapps $ zpool export zroot $ zpool import -o cachefile=/tmp/zpool.cache zroot $ zpool import -o cachefile=/tmp/zpool.cache zapps $ zpool import -o cachefile=/tmp/zpool.cache zflash
You shouldn’t get any errors this time, and your entire filesystem should now be available under /mnt.
We need to set the correct permissions on the tmp filesystems, and symlink the home directory around the place. Most guides recommend you store your home directory under the /usr partition and symlink it to /home; but as we only have the one ZFS partition it really doesn’t matter. I prefer to put it directly under /home, and symlink it to /usr/home for compatibility.
chmod 1777 /mnt/tmp cd /mnt/usr ; ln -s ../home home chmod 1777 /mnt/var/tmp
Lets leave this post here; I install FreeBSD back in this post. (out of order posting/missing something important fail)
-bok
