Buildbot Archives - XpresServers

How To Build and Deploy Packages for Your FreeBSD Servers Using Buildbot and Poudriere

The author selected the Free and Open Source Fund to receive a donation as part of the Write for DOnations program.

Introduction

The FreeBSD ports and packages collection, hereafter called ports tree, is FreeBSD’s build system for external software. It offers a Makefile-based, consistent way of building packages. The port refers to the build recipe, that is the Makefile and related files; while package is the output of building one port into a binary (compressed) archive of the package files and its meta information.

Manually building and installing a subset or all of the over 30,000 ports is possible with make install. However, the builds would run on one of your servers—not a clean environment. For production use cases, manual builds would also mean that each host needs the same revision of the ports tree, and needs to compile all packages for itself. This means repeated, error-prone work by humans and the servers. It is preferable to retrieve and use identical, pre-built binary packages on each host and serve them from a central, secure package repository.

To achieve this, Poudriere is the standard tool on FreeBSD to build, test, and audit packages as well as maintain the package repositories. Each build is run isolated in a fresh jail, running the desired version of FreeBSD, and starting with no packages installed. Only the base system, plus any explicitly specified dependencies, are available to the clean build. Poudriere takes care of rebuilding packages when necessary as well as updating the package repository after a build has finished. The poudriere command line tool is central to administering different ports trees, FreeBSD versions, port build options, and lastly, running the builds.

In this tutorial you’ll configure Poudriere, build a set of desired packages, set up HTTP-based package hosting, and automate the build using Buildbot as a continuous integration platform. Finally, you will securely access the packages from a client machine.

Note: To cover production-like use cases, the tutorial examples use the quarterly stable branches of the ports tree. Staying on one such branch protects you from breaking changes and provides security and build fixes where necessary—if you regularly update the tree from upstream (Subversion, or its GitHub mirror). You can choose to stay on one branch for an extended period of time, depending on the pace at which your system updates can be handled by developer/infrastructure teams. The ports collection supports FreeBSD releases until they become end-of-life (EOL)—see Supported FreeBSD releases—so that OS and package updates can be handled independently. Alternatively, you could consider a local version control repository cloned from the upstream tree. That way, you can manage patches and only merge upstream changes at the time you desire.

Prerequisites

Before you begin this guide, you will need:

A server running FreeBSD 11.2. If you’re new to working with FreeBSD, you may find it helpful to customize this server by following our guide on How to Get Started with FreeBSD.
Note: FreeBSD 12.0 currently has an issue with nested jails, which first needs to be fixed before 12.x can be used for this tutorial.
10 GB free disk space or more to have enough capacity to store packages and logs.
A basic Buildbot setup by completing the How To Set Up Buildbot on FreeBSD tutorial.
Another server running FreeBSD, the same version, which you are going to use as a client to fetch and install the packages that you’re going to automatically build and host in a HTTP/HTTPS-based package repository.

Step 1 — Installing Poudriere for Use in Buildbot Worker

After completing the prerequisite tutorial, you’ll have a working Buildbot master and worker jail plus Nginx setup. You will build upon this existing setup in the following steps. In this first step, you’re going to install the build tool Poudriere inside the worker jail, since that is where the Buildbot worker process will trigger builds later on.

Connect to your server hosting Buildbot and open a root shell in the worker jail with the following command:

sudo jexec buildbot-worker0 csh

Install Poudriere as a package:

Then confirm installation by pressing y and then ENTER.

Note: It is preferable to use the official FreeBSD package repository for installing Buildbot, Poudriere, and so on. If you build those tool packages yourself, you start off in a chicken-and-egg situation: wanting to install external software, but requiring Poudriere installed to get cleanly built packages. Since Poudriere is a very stable and backward-compatible tool, nothing speaks against updating it regularly and independently from your production packages.

If you followed the prerequisite tutorial, this is already the case and you can continue without following this note.

You’ve successfully installed the latest Poudriere tool and dependencies. In the next several steps, you will go through preparations to configure Poudriere.

`Step 2 — Creating a Package Signing Key (Optional)`

It’s recommended to set up digital signatures for built packages in order to provide more security. Skip this step if you want to secure your installation later, or in a different way. Otherwise, let’s go ahead and create a key pair used to sign packages (using the private key) and verify packages (using the public part).

Packages, by default, are built as .txz files, which are strongly compressed tarballs of the package contents. The compressed files’ checksums, together with serving the files via HTTP/HTTPS (TCP checksums), already provide some protection against corrupted data. Package contents typically comprise files and directories plus meta information such as the package name, version, and miscellaneous options. Files may even include setuid-able programs (as seen in the sudo package—though sudo is not built into FreeBSD), and the installation-time scripts run as root user. Installing from unverified sources therefore poses a security risk.

By serving the packages over HTTPS, you cannot detect whether someone tampered with the packages on-disk. Integrity and authenticity of your packages can be added by configuring Poudriere to sign the package repository with an RSA private key. Signed digests and the corresponding public key are thereby stored in the package repository’s digests.txz file. The required key pair (RSA private and public key) can be kept unchanged for a long time unless the private key was lost or compromised.

In this step you’ll create the key pair where the builds run (worker jail) and download the public part for later use on package clients (discussed in a later step).

Ensure you’re still in the worker jail root shell.

Create a new RSA private key:

openssl genrsa -out /usr/local/etc/poudriere.key 4096

The private key file only needs to be accessible by root—the user that runs Poudriere. Protect its access permissions:

chmod 0600 /usr/local/etc/poudriere.key

Later, you’ll need the public key part available on clients for verifying package signatures. Let’s extract the public key now:

openssl rsa -in /usr/local/etc/poudriere.key -pubout -out /tmp/poudriere.pub

Lastly, download the public key file from your own computer:

scp your-server:/usr/jails/buildbot-worker0/tmp/poudriere.pub /tmp/poudriere.pub

This concludes the optional creation of a key pair for package signing. You will later configure the actual signing with Poudriere and use the downloaded public key file on clients for the verification.

Another optional step follows: if you use the ZFS filesystem, Poudriere can make use of it to speed up builds. Otherwise, you can skip to Step 4 to configure Poudriere in order to get ready for running the first build.

`Step 3 — Setting Up ZFS (Optional)`

This step only applies if you run a FreeBSD system on top of the ZFS filesystem. For instance if you’re using a DigitalOcean Droplet the image is labeled 11.2 x64 zfs (for FreeBSD 11.2). In this step, you’re going to create the filesystems that Poudriere can use to create and manage jails faster, potentially speeding up your builds.

You can find out whether you’re using ZFS by listing pools. Make sure you’re on the server’s shell, not inside a jail.

Run the following command to list the zpools:

If any pool is available, it will print information about it:

OutputNAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot   148G  94.4G  54.1G        -         -    66%    63%  1.00x  ONLINE  -

Otherwise if ZFS support is not available the tool will print no pools available, or failed to initialize ZFS library. This means that none of your system is using ZFS; in this case, skip to the next step. If you have decided to use another disk or storage type, such as the UFS filesystem, you can also move on to the next step.

If you plan to use ZFS, remember the printed pool name on which you want to store build-related data. You should plan for several gigabytes of storage.

ZFS is helpful to separate the various datasets of Poudriere, like build jails, ports trees, logs, packages, and other data. These are stored independently and as a result can be deleted quickly with the certainty of not leaving free space or traces behind.

For Poudriere to make use of ZFS, you need to do three things: create a parent ZFS dataset, allow for the creation and deletion of ZFS datasets (which the Buildbot worker jail, or any other jail, by default cannot do), and edit Poudriere’s configuration accordingly.

In the prerequisite tutorial, you configured the Buildbot worker jail in /etc/jail.buildbot-worker0.conf. Open this file with your preferred text editor and add the following highlighted lines to delegate a parent dataset to allow the jail to administer ZFS datasets beneath the parent. Remember to replace zroot with your desired pool name:

sudo ee /etc/jail.buildbot-worker0.conf

/etc/jail.buildbot-worker0.conf

buildbot-worker0 {
    host.hostname = buildbot-worker0.localdomain;
    ip4.addr = "lo1|10.0.0.3/24";
    path = "/usr/jails/buildbot-worker0";
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    mount.devfs; # need /dev/*random for Python
    persist;

    exec.poststart = "/sbin/zfs jail buildbot-worker0 zroot/pdr/w0";
}

In this article we will store build-related data on the ZFS pool zroot—please adapt this ZFS-related configuration here and throughout the rest of the article if you chose a pool of a different name.

After adding this content, save and exit the editor. If you’re using ee, do this by pressing CTRL+C, typing exit, and pressing ENTER.

Create the parent ZFS dataset mentioned in the configuration file:

sudo zfs create zroot/pdr
sudo zfs create zroot/pdr/w0

This deliberately assumes that you may want to add more workers in the future and therefore creates a sub-dataset for your first worker. The dataset name is short on purpose, since older versions of FreeBSD (before 12.0) had a mount name limit of 88 characters.

In order for a jail to take control of a parent dataset and administer any children, the dataset must be marked with the following flag:

sudo zfs set jailed=on zroot/pdr/w0

With the preconditions now met, the jail will start correctly with the new configuration:

sudo service jail restart buildbot-worker0

With these instructions, you successfully created the required filesystems—ZFS datasets—and allowed the jail to manage the parent dataset. In the next step, you will configure Poudriere, which involves specifying the chosen zpool and dataset used to store build-related data.

`Step 4 — Configuring Poudriere, the Build Jail, and the Ports Tree`

Until this point, you’ve installed Poudriere and optionally covered requirements for package signing and ZFS. For Poudriere to be able to run in a “jailed” fashion—that is, functioning correctly from within the Buildbot worker jail—you need to provide certain permissions to the jail. For example, if you use ZFS, you have already delegated a parent dataset for use and administration by the jail.

Let’s first configure the loopback IP and all of the permissions, and then step through the respective meaning following the changes.

Poudriere wants to start two build jails per build: one with loopback-only networking and one with internet access. Only build stages that are supposed to reach the internet will use the latter. For example, the fetch may download source tarballs, but the build phase is not allowed internet access. The existing configuration of the worker jail has ip4.addr = "lo1|10.0.0.3/24" that allows internet access. In order to allow Poudriere to assign a loopback address to freshly started build jails, the IP must also be passed to its parent (the worker jail). For this to work, please ensure you have applied the latest version of the firewall configuration file /usr/local/etc/ipfw.rules from the prerequisite tutorial, which will block the loopback interface lo0 from opening outgoing connections through NAT.

Add the highlighted lines to your worker jail configuration:

sudo ee /etc/jail.buildbot-worker0.conf

/etc/jail.buildbot-worker0.conf

buildbot-worker0 {
    host.hostname = buildbot-worker0.localdomain;
    ip4.addr = "lo1|10.0.0.3/24";
    ip4.addr += "lo0|127.0.0.3";
    path = "/usr/jails/buildbot-worker0";
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    mount.devfs; # need /dev/*random for Python
    persist;

    # If you followed the ZFS setup step, you have this line
    # already (keep it). For non-ZFS setup, this line must be absent.
    exec.poststart = "/sbin/zfs jail buildbot-worker0 zroot/pdr/w0";

    allow.chflags;
    allow.mount;
    allow.mount.devfs;
    allow.mount.nullfs;
    allow.mount.procfs;
    allow.mount.tmpfs;
    allow.mount.zfs; # only needed if you use ZFS
    allow.raw_sockets; # optional
    allow.socket_af; # optional
    allow.sysvipc; # optional
    children.max=16;
    enforce_statfs=1;
}

Here you’ve added the following (also see the jail(8) manpage):

ip4.addr += "lo0|127.0.0.3" adds another IPv4 address to the jail. You will later configure Poudriere’s LOIP4 variable in order to assign this loopback address to build jails that are not supposed to talk to the internet or other machines in your network, such as during the build phase. If you ever have a build that requires internet access during build, Poudriere supports a variable ALLOW_NETWORKING_PACKAGES as a workaround. However, it is preferable to follow best practice and perform downloads, and other internet-facing tasks earlier, in the fetch phase for which Poudriere permits internet access.
allow.chflags allows Poudriere to render certain system files like /bin/sh immutable in the build jail.
allow.mount and the other allow.mount.* options enable Poudriere to mount certain required filesystems into the build jails.
allow.raw_sockets which permits use of raw sockets, and allow.socket_af which permits use of any socket address family, are both applied to the internet-capable build jails. This is helpful so that you can run tools like ping in interactive mode, like when entering a build jail to debug problems.
allow.sysvipc is deprecated in favor of three separate settings sysvmsg/sysvsem/sysvshm to restrict jails to only see their own shared memory objects (via “SYS V” IPC primitives). However, Poudriere can only pass on allow.sysvipc to build jails because it cannot read the relevant sysctl information for the three separate parameters (as of FreeBSD 11.2). With this deprecated configuration, the jail could read shared memory of processes outside the jail. This is only relevant for certain software that depends on IPC features, like PostgreSQL, so chances are small for this to affect security. You can remove this configuration unless you depend on a port that requires it during build.
children.max=16 allows 16 sub-jails below the worker jail. You can raise this number later if you have a lot of CPUs and Poudriere tries to create more build jails than permitted. Each Poudriere build will try to create a reference jail and two build jails per “job”, and its default is to use the number of CPUs (as output by sysctl -n hw.ncpu) as the job count.
enforce_statfs=1 is required together with allow.mount in order to mount certain filesystems.

Save and exit the configuration file.

Restart the jail for its configuration to take affect immediately:

sudo service jail restart buildbot-worker0

The respective kernel modules must be loaded so that Poudriere can perform mounts. Run the following commands to load the modules at boot time and immediately:

sudo sysrc -f /boot/loader.conf nullfs_load=YES
sudo kldload -n nullfs
sudo sysrc -f /boot/loader.conf tmpfs_load=YES
sudo kldload -n tmpfs

You already installed the Poudriere package earlier, which has copied the sample file /usr/local/etc/poudriere.conf.sample to /usr/local/etc/poudriere.conf. Next, you will make edits to the configuration file. All possible configuration variables already exist in the sample, so uncomment or adapt the respective line in the file to set a variable to a certain value.

For the following commands, please ensure you are still in a root shell in the worker jail:

sudo jexec buildbot-worker0 csh

Open the file with the following command:

ee /usr/local/etc/poudriere.conf

If you have decided to use ZFS, please fill in your desired zpool and parent dataset: