Building ZFS Root Ubuntu AMIs With Packer

 Posted on March 2, 2017  |  12 minutes  |  2502 words  |  

For all applications of importance or significance, we recommend using ZFS. On bare metal servers, ZFS is king of the hill, but on AWS and Linux it is still gaining traction. Data integrity guarantees as well as features such as “instantaneous” snapshots, compression, quotas, and the ability to send/receive datasets make ZFS very compelling. In this post, we’re going to demonstrate how to build from-scratch AMIs booting Ubuntu Linux with a ZFS root file system.

One of the precipitating events that spurred our decision to use root ZFS in AWS was a blog post by Scott Emmons who detailed how to build Amazon AMIs running with a root ZFS filesystem for both the Ubuntu and Debian Linux distributions.

We were already using ZFS on data volumes, but not for the root filesystem, so when I was looking to build trusted base images for a recent AWS infrastructure project, I decided to use Scott’s detailed post of the manual steps, cross referenced with Canonical’s own documentation as the basis for an automated build using HashiCorp’s Packer - in codified build processes we trust!

The reasons for wanting a ZFS root file system are well described in Scott’s post and across the internet (as is why ZFS is preferable to BTRFS, for example!). Similarly, the reasons for wanting to scratch-build AMIs instead of using the prebuilt ones are well understood at this point - so I’m not going to rehash either argument here.

In this example, we’re building images for Ubuntu “Yakkety” (16.10), though it’s easy enough to adapt for 16.04 - the latest LTS Ubuntu release - or Debian. The Packer template and accompanying scripts for both “Yakkety” and “Xenial” are available on GitHub.

Note that you’ll need version 0.12.3 - released March 2nd 2017 - of Packer to run builds using these templates!

Packer’s EBS Surrogate Builder

Packer has a variety of different builders for AWS. The most common of these is the amazon-ebs builder, which launches a nominated source image, connects over SSH in order to run a set of provisioning steps, and then creates an image from the launched machine. Since we’re installing an operating system from scratch rather than specializing an existing machine, the amazon-ebs builder won’t work for us in this case.

In fact, of the builders available in Packer 0.12.2, only the amazon-chroot builder would work for our purposes, has a number of important disadvantages though:

  • Builds must be run from inside a running EC2 instance
  • Templates using the amazon-chroot builder cannot be run via Terraform Enterprise.

In order to overcome these limitations, a new builder was added to Packer: amazon-ebssurrogate. The new builder follows steps similar to those of the existing amazon-ebs, with an additional EBS volume attached. The additional volume is used as the root volume for our soon-to-be-created AMI. As a result, instead of creating the image from the source machine root volume, the new builder snapshots the attached EBS volume and creates an AMI with a root volume derived from that snapshot - ZFS batteries included out of the box.

Packer Template

Packer defines templates using a JSON file configuring the various options available for a given builder, and the provisioning steps to be taken. We can use the following template to build our ZFS root AMI - since JSON doesn’t allow commenting, I’ll call out the important options below:

{
	"variables": {
		"aws_access_key_id": "{{ env `AWS_ACCESS_KEY_ID` }}",
		"aws_secret_access_key": "{{ env `AWS_SECRET_ACCESS_KEY` }}",
		"region": "us-west-2",
		"buildtime": "{{ isotime \"2006-01-02-1504\" }}"
	},
	"builders": [{
		"type": "amazon-ebssurrogate",
		"access_key": "{{ user `aws_access_key_id` }}",
		"secret_key": "{{ user `aws_secret_access_key` }}",
		"region": "{{ user `region` }}",
		"spot_price_auto_product": "Linux/UNIX (Amazon VPC)",

		"ssh_pty": true,
		"instance_type": "m4.large",
		"associate_public_ip_address": true,
		"ssh_username": "ubuntu",
		"ssh_timeout": "5m",

		"source_ami_filter": {
			"filters": {
				"virtualization-type": "hvm",
				"name": "*ubuntu-yakkety-16.10-amd64-server-*",
				"root-device-type": "ebs"
			},
			"owners": ["099720109477"],
			"most_recent": true
		},

		"launch_block_device_mappings": [
			{
				"device_name": "/dev/xvdf",
				"delete_on_termination": true,
				"volume_size": 8,
				"volume_type": "gp2"
			}
		],

		"run_tags": {
			"Name": "Packer Builder - ZFS Root Ubuntu",
			"Project": "operator-error.com"
		},
		"run_volume_tags": {
			"Name": "Packer Builder - ZFS Root Ubuntu",
			"Project": "operator-error.com"
		},

		"ami_name": "ubuntu-yakkety-16.10-amd64-zfs-server-{{ user `buildtime` }}",
		"ami_description": "Ubuntu Yakkety (16.10) with ZFS Root Filesystem",
		"ami_virtualization_type": "hvm",
		"ami_regions": ["eu-west-2"],
		"ami_root_device": {
			"source_device_name": "/dev/xvdf",
			"device_name": "/dev/xvda",
			"delete_on_termination": true,
			"volume_size": 8,
			"volume_type": "gp2"
		},
		"tags": {
			"Name": "Ubuntu Yakkety (16.10) with ZFS Root Filesystem",
			"BuildTime": "{{ user `buildtime` }}"
		}
	}],
	"provisioners": [
		{
			"type": "file",
			"source": "sources-us-west-2.list",
			"destination": "/tmp/sources.list"
		},
		{
			"type": "file",
			"source": "chroot-bootstrap.sh",
			"destination": "/tmp/chroot-bootstrap.sh"
		},
		{
			"type": "shell",
			"start_retry_timeout": "5m",
			"script": "surrogate-bootstrap.sh",
			"skip_clean": true
		}
	]
}
  • The variables section maps the standard AWS CLI environment variables through to Packer variables. Note that region is hardcoded - our AMI will be copied to other regions later, but the apt sources list we use during bootstrapping is region-specific.
  • We capture the current date and time in the buildtime variable so it can be used consistently as part of the AMI name and tags.
  • We define one builder: an amazon-ebssurrogate (as described above).
  • Packer will automatically calculate a bid price for an instance on the spot market - in this case the spot_price_auto_product is based on the history for the “Linux/UNIX (Amazon VPC)” product. If a spot instance isn’t available, Packer will use an on-demand instance.
  • We’re using a fairly beefy instance - I originally tried using a much smaller one, but the debootstrap tool would often fail to download packages correctly. It’s possible this can be tuned down to a smaller instance though.
  • We use a source_ami_filter to find the most recent pre-built HVM AMI for EBS - in this case Ubuntu Yakkety (16.10). This is the AMI used to launch the source machine.
  • Packer will automatically create a temporary key pair for use building this image, so we only need to pass in the ssh_username of the default user - ubuntu in this case.
  • We need to attach a new EBS volume to the source instance. In this case, we choose an 8GB GP2 volume, and mount it at /dev/xvdf. In order to keep the AWS account tidy, we set delete_on_termination to true - by the point the source instance is terminated a snapshot will already exist.
  • We name the AMI using a similar pattern to Canonical, adding zfs, and append the build time.
  • We specify the root device of the new AMI, including the mount point on the source instance, and the parameters for the root volume on the AMI.
  • The AMI we produce is region-agnostic thanks to CloudInit, so we can use Packer to copy it to whichever regions we may wish to use it. In this case, we are building in us-west-2 (Oregon) and copying to eu-west-2 (London), so the image will be available in both regions, albeit with a different ID.

Finally, we specify a list of provisioners to run on the source instance once it is available.

Preparing the Surrogate Instance

The first phase of provisioning is to prepare the source machine. We need to do the following:

  • Install ZFS, debootstrap and a partitioning tools
  • Partition the new EBS volume
  • Create a zpool consisting only of the new EBS volume
  • Create and mount the various ZFS datasets for our install with appropriate attributes set
  • debootstrap the target distribution into our new filesystems
  • Copy in an apt sources.list appropriate for the region in which we are building
  • Mount the /dev, /proc and /sys directories from the surrogate instance into the new root
  • chroot into the new root and run the next script
  • Remove the region-specific sources.list - CloudInit will generate a region-appropriate version when instances based on the new AMI are booted.
  • Quiesce the new root volume and export the zpool

This is all performed by the first provisioning script, surrogate-bootstrap.sh. Note that this script calls the second provisioning script chroot-bootstrap.sh inside the chroot - this script is described in the next section.

The surrogate-bootstrap.sh script looks like this:

#!/bin/bash

set -ex

# Update apt and install required packages
DEBIAN_FRONTEND=noninteractive sudo apt-get update
DEBIAN_FRONTEND=noninteractive sudo apt-get install -y \
	zfs-zed \
	zfsutils-linux \
	zfs-initramfs \
	zfs-dkms \
	debootstrap \
	gdisk

# Partition the new root EBS volume
sudo sgdisk -Zg -n1:0:4095 -t1:EF02 -c1:GRUB -n2:0:0 -t2:BF01 -c2:ZFS /dev/xvdf

# Create zpool and filesystems on the new EBS volume
sudo zpool create \
	-o altroot=/mnt \
	-o ashift=12 \
	-o cachefile=/etc/zfs/zpool.cache \
	-O canmount=off \
	-O compression=lz4 \
	-O atime=off \
	-O normalization=formD \
	-m none \
	rpool \
	/dev/xvdf2

# Root file system
sudo zfs create \
	-o canmount=off \
	-o mountpoint=none \
	rpool/ROOT

sudo zfs create \
	-o canmount=noauto \
	-o mountpoint=/ \
	rpool/ROOT/ubuntu

sudo zfs mount rpool/ROOT/ubuntu

# /home
sudo zfs create \
	-o setuid=off \
	-o mountpoint=/home \
	rpool/home

sudo zfs create \
	-o mountpoint=/root \
	rpool/home/root

# /var
sudo zfs create \
	-o setuid=off \
	-o overlay=on \
	-o mountpoint=/var \
	rpool/var

sudo zfs create \
	-o com.sun:auto-snapshot=false \
	-o mountpoint=/var/cache \
	rpool/var/cache

sudo zfs create \
	-o com.sun:auto-snapshot=false \
	-o mountpoint=/var/tmp \
	rpool/var/tmp

sudo zfs create \
	-o mountpoint=/var/spool \
	rpool/var/spool

sudo zfs create \
	-o exec=on \
	-o mountpoint=/var/lib \
	rpool/var/lib

sudo zfs create \
	-o mountpoint=/var/log \
	rpool/var/log

# Display ZFS output for debugging purposes
sudo zpool status
sudo zfs list

# Bootstrap Ubuntu Yakkety into /mnt
sudo debootstrap --arch amd64 yakkety /mnt
sudo cp /tmp/sources.list /mnt/etc/apt/sources.list

# Copy the zpool cache
sudo mkdir -p /mnt/etc/zfs
sudo cp -p /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

# Create mount points and mount the filesystem
sudo mkdir -p /mnt/{dev,proc,sys}
sudo mount --rbind /dev /mnt/dev
sudo mount --rbind /proc /mnt/proc
sudo mount --rbind /sys /mnt/sys

# Copy the bootstrap script into place and execute inside chroot
sudo cp /tmp/chroot-bootstrap.sh /mnt/tmp/chroot-bootstrap.sh

# This script is described in the following section
sudo chroot /mnt /tmp/chroot-bootstrap.sh
sudo rm -f /mnt/tmp/chroot-bootstrap.sh

# Remove temporary sources list - CloudInit regenerates it
sudo rm -f /mnt/etc/apt/sources.list

# This could perhaps be replaced (more reliably) with an `lsof | grep -v /mnt` loop,
# however in approximately 20 runs, the bind mounts have not failed to unmount.
sleep 10 

# Unmount bind mounts
sudo umount -l /mnt/dev
sudo umount -l /mnt/proc
sudo umount -l /mnt/sys

# Export the zpool
sudo zpool export rpool

Inside the chroot

The last steps of the surrogate provisioning script above execute the script chroot-bootstrap.sh with the root set to our new EBS volume. That script is copied by Packer onto the source instance in one of the earlier provisioners, and performs the following actions:

  • Install varous packages required for a booting system, including ZFS and grub, which are not installed as part of the debootstrap process
  • Set the locale of the target operating system
  • Install OpenSSH Server
  • Install and configure grub with the settings recommended for AWS
  • Configure eth0 to use DHCP
  • Install the other packges making up Ubuntu Server, by installing the ubuntu-standard metapackage
  • Return control to the surrogate-bootstrap.sh script

The script to perform these actions, chroot-bootstrap.sh, looks like this:

#!/bin/bash

set -ex

# Update APT with new sources
apt-get update

# Do not configure grub during package install
echo 'grub-pc grub-pc/install_devices_empty select true' | debconf-set-selections
echo 'grub-pc grub-pc/install_devices select' | debconf-set-selections

# Install various packages needed for a booting system
DEBIAN_FRONTEND=noninteractive apt-get install -y \
	linux-image-generic \
	linux-headers-generic \
	grub-pc \
	zfs-zed \
	zfsutils-linux \
	zfs-initramfs \
	zfs-dkms \
	gdisk

# Set the locale to en_US.UTF-8
locale-gen --purge en_US.UTF-8
echo -e 'LANG="en_US.UTF-8"\nLANGUAGE="en_US:en"\n' > /etc/default/locale

# Install OpenSSH
apt-get install -y --no-install-recommends openssh-server

# Install GRUB
# shellcheck disable=SC2016
sed -ri 's/GRUB_CMDLINE_LINUX=.*/GRUB_CMDLINE_LINUX="boot=zfs \$bootfs"/' /etc/default/grub
grub-probe /
grub-install /dev/xvdf

# Configure and update GRUB
mkdir -p /etc/default/grub.d
{
	echo 'GRUB_RECORDFAIL_TIMEOUT=0'
	echo 'GRUB_TIMEOUT=0'
	echo 'GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 ip=dhcp tsc=reliable net.ifnames=0"'
	echo 'GRUB_TERMINAL=console'
} > /etc/default/grub.d/50-aws-settings.cfg
update-grub

# Set options for the default interface
{
	echo 'auto eth0'
	echo 'iface eth0 inet dhcp'
} >> /etc/network/interfaces

# Install standard packages
DEBIAN_FRONTEND=noninteractive apt-get install -y ubuntu-standard cloud-init

Running a build

We can run Packer with the template described above using the following command line:

$ export AWS_ACCESS_KEY_ID=<redacted>
$ export AWS_SECRET_ACCESS_KEY=<redacted>
$ export AWS_REGION=us-west-2
$ packer build template.json
amazon-ebssurrogate output will be in this color.

==> amazon-ebssurrogate: Prevalidating AMI Name...
    amazon-ebssurrogate: Found Image ID: ami-a49b1bc4
==> amazon-ebssurrogate: Creating temporary keypair: packer_58b440e9-ec4d-4f96-6696-45206ac7dc7b
==> amazon-ebssurrogate: Creating temporary security group for this instance...
==> amazon-ebssurrogate: Authorizing access to port 22 the temporary security group...
==> amazon-ebssurrogate: Launching a source AWS instance...
    amazon-ebssurrogate: Instance ID: i-0d0e3a4ed4197df61
==> amazon-ebssurrogate: Waiting for instance (i-0d0e3a4ed4197df61) to become ready...
==> amazon-ebssurrogate: Adding tags to source instance
==> amazon-ebssurrogate: Adding tags to source EBS Volumes
==> amazon-ebssurrogate: Waiting for SSH to become available...
==> amazon-ebssurrogate: Connected to SSH!
==> amazon-ebssurrogate: Uploading sources-us-west-2.list => /tmp/sources.list
==> amazon-ebssurrogate: Uploading chroot-bootstrap.sh => /tmp/chroot-bootstrap.sh
==> amazon-ebssurrogate: Provisioning with shell script: surrogate-bootstrap.sh
# Output omitted for brevity
==> amazon-ebssurrogate: Stopping the source instance...
==> amazon-ebssurrogate: Waiting for the instance to stop...
==> amazon-ebssurrogate: Creating snapshot of EBS Volume vol-0037800b3245db2e2...
    amazon-ebssurrogate: Snapshot ID: snap-01e214eddcfc79191
==> amazon-ebssurrogate: Registering the AMI...
==> amazon-ebssurrogate: AMI: ami-a032b1c0
==> amazon-ebssurrogate: Waiting for AMI to become ready...
==> amazon-ebssurrogate: Copying AMI (ami-a032b1c0) to other regions...
    amazon-ebssurrogate: Copying to: eu-west-2
    amazon-ebssurrogate: Waiting for all copies to complete...
==> amazon-ebssurrogate: Modifying attributes on AMI (ami-a032b1c0)...
    amazon-ebssurrogate: Modifying: description
==> amazon-ebssurrogate: Modifying attributes on AMI (ami-fddacf99)...
    amazon-ebssurrogate: Modifying: description
==> amazon-ebssurrogate: Modifying attributes on snapshot (snap-01e214eddcfc79191)...
==> amazon-ebssurrogate: Modifying attributes on snapshot (snap-88c78eef)...
==> amazon-ebssurrogate: Adding tags to AMI (ami-a032b1c0)...
==> amazon-ebssurrogate: Tagging snapshot: snap-01e214eddcfc79191
==> amazon-ebssurrogate: Creating AMI tags
==> amazon-ebssurrogate: Creating snapshot tags
==> amazon-ebssurrogate: Adding tags to AMI (ami-fddacf99)...
==> amazon-ebssurrogate: Tagging snapshot: snap-88c78eef
==> amazon-ebssurrogate: Creating AMI tags
==> amazon-ebssurrogate: Creating snapshot tags
==> amazon-ebssurrogate: Terminating the source AWS instance...
==> amazon-ebssurrogate: Deleting temporary security group...
==> amazon-ebssurrogate: Deleting temporary keypair...
Build 'amazon-ebssurrogate' finished.

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebssurrogate: AMIs were created:

eu-west-2: ami-fddacf99
us-west-2: ami-a032b1c0

The process takes a while - but out the other end you should receive a functioning AMI for Ubuntu Yakkety, with a ZFS root file system!

Running an instance with the new AMI

Following our build, we can see in the AWS Console that we have an AMI available in the us-west-2 region:

There is also an image available in the eu-west-2 region:

We can now launch instances running one of our new images and ensure all is as we expect.

# Default subnet, security group permits SSH from 0.0.0.0/0
$ aws ec2 run-instances --region "eu-west-2" \
	--associate-public-ip-address \
	--image-id "ami-fddacf99" \
	--instance-type "t2.micro" \
	--key-name "jen20" \
	--security-group-ids "sg-5e518b37" \
	--subnet-id "subnet-0bf90a70"

Finally, running a series of commands on the instance shows we are indeed booting from ZFS, and that CloudInit has populated a sources list appropriate for the region we launched in.

ZFS Root Linux in AWS

The source code for the Packer templates are available on GitHub - please feel free to open issues or pull requests as you see fit!

Credits

A number of people helped either inspire or review this article. Thanks go to:

  • Scott Emmons for this post about the steps required to build Linux AMIs with a ZFS root filesystem.
  • Sean Chittenden for reviewing the template and proof-reading this post prior to publication.
  • Zachary Schneider for reviewing the template and proof-reading this post prior to publication.
zfs  aws  packer