Big releases are a big risk

In many embedded software projects, releases are rare and stressful events. They often involve a lot of manual work, coordination between teams, and a high risk of things going wrong. The alternative is to make releases small, frequent and boring. Instead of bundling months of development into a single firmware version, deliver incremental improvements continuously. Each change should be small enough that its impact is well understood and reversible.

When releases are small and routine, the feedback loop with customers becomes much shorter and product development becomes more agile. While embedded software probably never will be able to ship as fast as pure software products, there are still many ways to speed up the delivery process and make it more reliable. However, moving from multiple months between releases to a few weeks or even days can have a huge impact on your product and your customers. It allows you to respond to customer feedback more quickly, fix bugs faster, and deliver new features sooner.

Making Releases Routine

To enable faster releases for embedded devices, you need to invest in a few key capabilities that reinforce each other. Some of these are technical foundations, others are organizational habits or cultural shifts, but most can be introduced incrementally without turning the whole company upside down. The goal is to make releasing feel like just another part of everyday development work rather than a nerve‑wracking special event.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

The first step is to treat releases as a routine, not a ceremony. If you ship only once or twice a year, every release feels exceptional, with custom scripts, ad‑hoc decisions and lots of manual coordination. This naturally increases risk, because nobody gets enough practice to truly master the process. By contrast, when releases happen regularly, the team gains muscle memory. You can get there by deliberately increasing the frequency of internal releases first: ship every sprint to a staging environment, then to a small set of internal devices, and only then to customers. Over time, the release pipeline becomes so familiar that pushing a new version out is as unremarkable as merging a pull request.

Shifting the attitude inside the team towards releases is one thing, but to go to full power you also need the technical capabilities to deliver the releases. Over‑The‑Air (OTA) updates are the technical backbone that makes this routine possible at scale. Without a solid OTA mechanism, delivering embedded software means physical access to devices, manual interventions, or complex field procedures. This makes releases cumbersome and risky. A robust OTA solution lets you update devices in the field reliably, with support for progressive rollouts, version tracking and rollbacks. There are many mature OTA-solutions out there, and most can be conveniently integrated with your build system and CI/CD pipeline. With OTA in place, you can decouple the release process from physical logistics and manual steps. The outcome is that “shipping” becomes an automated action triggered from your pipeline rather than a logistical project involving manual coordination.

The ability to update devices in the field is a big enabler, but to profit the most from it automating the delivery pipeline is a must. Automated testing via a powerful CI/CD pipeline is what turns frequent updates from reckless to safe. If every change can be built, tested and validated automatically, then the risk of each individual release drops dramatically. The testing possibility and complexity for embedded software is often huge, but even automating a subset of tests can have a big impact on confidence. Build up an efficient and automated testing pipeline that covers the critical paths of your software. Start by putting the entire build under CI so that every commit produces a reproducible artifact. Then layer on unit tests, static analysis, and hardware‑in‑the‑loop or end‑to‑end tests where feasible. Over time, your pipeline becomes a safety net: if something slips through, the pipeline catches it before customers ever see it. This confidence is what allows you to ship more often without losing sleep.

Finally, shipping faster requires shifting product management away from chasing big, “heroic” releases and towards delivering features continuously. Large, monolithic releases encourage long planning cycles, heavy upfront design and a tendency to cram in “just one more feature” which delays getting real feedback. Shift from delivering major versions that “make everying better” to delivering single features or small improvements that can be released independently. Product managers play a crucial role here by setting expectations with stakeholders about incremental releases, and measuring success by customer impact rather than marketing the next big release “that makes everything better”. You can support this shift by aligning roadmaps with short delivery cadences, encouraging experimentation, and using metrics such as lead time and value delivered instead of just counting features.

When you combine these four elements - releases as routine, OTA as plumbing, automated testing as your safety net and product management focused on incremental value - you create a delivery system that is both faster and more resilient. Releases stop being terrifying, once‑in‑a‑while events and become a steady, predictable flow of improvements. That, in turn, lets your teams spend less time orchestrating risky upgrades and more time building products your customers actually care about.

Encryption at rest in embedded Linux in a nutshell

On a high level, encryption at rest means that all data stored on the device is encrypted when the device is powered off. This ensures that if an attacker gains physical access to the device, they cannot read the data without the encryption key. In practice, this is typically achieved by encrypting the root filesystem and any other sensitive partitions using a strong encryption algorithm such as AES. When the device boots up, the encryption key is retrieved from a secure location (e.g. One-Time Programmable memory (OTP)) and used to unlock the encrypted partitions, allowing the operating system to access the data as needed. While this adds some complexity to the boot process and comes with a slight performance overhead for I/O operations, the security benefits far outweigh these drawbacks for most applications.

In practice for embedded Linux devices, encryption at rest can be implemented using LUKS (Linux Unified Key Setup), which is a widely used disk encryption specification for Linux. The tool to manage LUKS encrypted partitions is cryptsetup and is well supported in the Linux kernel through the dm-crypt kernel module.

Let’s look at how these steps can be implemented in detail with yocto.

Creating a LUKS-encrypted root filesystem with yocto

To implement encryption at rest using LUKS on embedded Linux devices based on the Raspberry Pi Compute Module 4 with the yocto Project, the following steps are necessary:

• Create an initramfs image that includes the cryptsetup tool and any necessary dependencies.
• Modify the bootloader configuration to load the initramfs image during the boot process.
• Create an init script within the initramfs that handles the unlocking of the LUKS-encrypted root filesystem using the encryption key stored in OTP memory.
• Implement a provisioning process during the first boot to create the LUKS-encrypted root filesystem and store the encryption key in OTP memory.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

Since the encryption of the partitions is device specific and needs to be done only once, a provisioning process is required during the first boot of the device. As a consequence, the device image created from yocto is unencrypted initially and will only be encrypted during the provisioning process. A convenient way to achieve this is to use an A/B partition scheme, where one partition is used for provisioning and the other for normal operation. During the first boot, the init script generates a unique encryption key for the device, creates the LUKS-encrypted root filesystem, copies the unencrypted root filesystem into the encrypted container, and stores the encryption key in OTP memory. On subsequent boots, the init script retrieves the encryption key from OTP memory and unlocks the LUKS-encrypted root filesystem for normal operation. So the init script needs to handle two scenarios: the first boot (provisioning) and subsequent boots (normal operation).

But first of all, the initramfs image needs to be created and the bootloader configuration modified.

Create initramfs with cryptsetup

Since the initramfs is just a minimal Linux image, we can create it using yocto by defining a custom image recipe. The image recipe should include the cryptsetup package and any other necessary dependencies. Here is an example of how to create an initramfs image with cryptsetup. Let’s call this recipe initramfs-cryptsetup.bb:

inherit core-image

PACKAGE_INSTALL = "provisioning-initramfs-init cryptsetup e2fsprogs rsync userland openssl rpi-eeprom"

IMAGE_FEATURES = ""

PACKAGE_EXCLUDE = "kernel-image-*"

IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"

INITRAMFS_MAXSIZE = "262144"

do_install_bootfiles() {
    
    install -m 644 ${IMGDEPLOYDIR}/${IMAGE_BASENAME}-${MACHINE}.cpio.gz ${DEPLOY_DIR_IMAGE}/bootfiles/initramfs.cpio.gz
    
}

addtask do_install_bootfiles after do_image_cpio before do_image_complete

If we look at this a bit closer, we can see that the image recipe inherits from core-image, which provides a basic image structure. The PACKAGE_INSTALL variable specifies the packages to be included in the initramfs image, including cryptsetup for LUKS support provisioning-initramfs-init, which is our custom package that contains the init script for handling the unlocking of the LUKS-encrypted root filesystem. Additionally, other necessary packages such as e2fsprogs, rsync, userland, openssl, and rpi-eeprom are included to support various functionalities required for the provisioning process, create the encryption keys and store them in the OTP memory of the Raspberry Pi CM4. Since the initramfs is intended to only run the provisioning and unlocking process, we exclude the kernel image packages by setting the PACKAGE_EXCLUDE variable to kernel-image-*. The IMAGE_FSTYPES variable is set to ${INITRAMFS_FSTYPES} to specify the desired initramfs file types (e.g., cpio.gz). The INITRAMFS_MAXSIZE variable is set to 262144 (256MB) to define the maximum size of the initramfs image. IMAGE_FEATURES is deliberately left empty to avoid including unnecessary features in the initramfs image. The do_install_bootfiles function installs the generated initramfs image to the appropriate location for the bootloader to access it during the boot process.

Next we need to make sure that this initramfs image is loaded by the bootloader during the boot process.

Modify bootloader configuration to load initramfs

To modify the bootloader configuration to load the initramfs image during the boot process, we need to update the bootloader settings to include the initramfs image. For Raspberry Pi devices, this is typically done by modifying the config.txt file located in the boot partition. In the case of the Raspberry Pi recipes this can be achieved by modifying the RPI_EXTRA_CONFIG variable in the machine configuration file (e.g., raspberrypi4.conf or raspberrypi-cm4.conf). Here is an example of how to modify the bootloader configuration to load the initramfs image:

RPI_EXTRA_CONFIG += "
...
initramfs initramfs.cpio.gz\n\
..."

This tells the bootloader to load the specified initramfs image (initramfs.cpio.gz) during the boot process, but it is only half the story. To make this accessible, we also need to make sure that the initramfs image is copied to the boot.img container during the image creation process. If your distribution is not yet using secure boot, I suggest to implement this first. Here is a step by step guide for secure boot using yocto.

In the recipe creating the boot.img we need to add the dependency to our initramfs image and copy it into the boot image. Here is an example of how to do this:

inherit deploy nopackages

DEPENDS += "rpi-bootfiles rpi-cmdline rpi-config linux-raspberrypi dosfstools-native mtools-native initramfs-cryptsetup"

# The raspberry pi bootloader files are located in the following directories
BOOTFILES_DIR = "${WORKDIR}/bootfiles" 
DTBO_DIR = "${DEPLOY_DIR_IMAGE}"

S = "${WORKDIR}/boot-img-container"

do_compile() {
    ...
   # Create boot.img file code, omitted for brevity here
   ...
}

do_compile[depends] += "initramfs-cryptsetup:do_install_bootfiles"

By adding the initramfs-cryptsetup dependency to the DEPENDS variable, we ensure that the initramfs image is built before the boot.img is created. The do_compile[depends] line ensures that the initramfs image is installed to the bootfiles directory before creating the boot.img. With these modifications and together with the modifications of the RPI_EXTRA_CONFIG variable, the initramfs will be included in the boot.img and loaded by the bootloader during the boot process. At this point we have everything in place to start looking at how the unlocking of the LUKS-encrypted root filesystem during boot and the provisioning process during the first boot can be implemented.

Init script for unlocking LUKS-encrypted root filesystem

Inside the initramfs there is an init script, which is a shell script that runs during the initramfs stage of the boot process. In our case, the init script is responsible for handling both the provisioning process during the first boot and the normal unlocking process during subsequent boots. First we create a new recipe for placing the init script inside the initramfs-image, let’s call it provisioning-initramfs-init.bb:

SRC_URI = "file://provisioning-initramfs-init"

# Installs shell and binaries required for the initscript
RDEPENDS:${PN} += "${VIRTUAL-RUNTIME_base-utils}"

do_install() {
    # Install the init script itself
    install -m 0755 ${WORKDIR}/provisioning-initramfs-init ${D}/init
    # Kernel will panic if /dev doesn't exist
    install -d ${D}/dev
    # Required for logging to work
    mknod -m 622 ${D}/dev/console c 5 1

    # Add the mount points
    install -d ${D}/mnt
}

FILES:${PN} = "/init /dev/console /proc /sys /mnt"

This recipe is included in the initramfs image by adding it to the PACKAGE_INSTALL variable in the initramfs-cryptsetup.bb recipe as shown earlier. The actual init script (provisioning-initramfs-init) contains the logic for both the provisioning and normal unlocking processes. Additionally, it creates the /dev directory and the /dev/console device to allow for logging over uart, but this is mainly for debug purpose.

So far we have set up the initramfs image with the necessary tools and created a recipe to include the init script. But this does not yet cover the actual logic for unlocking the LUKS-encrypted root filesystem and the provisioning process. Let’s look at how this can be implemented in the init script.

The init script

The init script is a shell script that runs during the initramfs stage of the boot process. In our case it it checks whether the device has been provisioned (i.e., whether the LUKS-encrypted root filesystem has been created and the encryption key stored in OTP memory). If not, it performs the provisioning process; otherwise, it retrieves the encryption key from OTP memory and unlocks the LUKS-encrypted root filesystem for normal operation.

The following flowchart illustrates the provisioning and boot process:

flowchart TD
    
    %% Boot chain
    INITRAM[Load initramfs from boot.img] --> KEYCHECK{"does a private key in the OTP exist?"}
    KEYCHECK -- No (do provisioning) --> GENKEY[Generate key and store in OTP]
    GENKEY --> ENCRYPTROOTFSB[create a LUKS container for <span style='color:orange'>rootfsB</span> using the key from OTP]
    ENCRYPTROOTFSB --> POPULATEB[Populate <span style='color:orange'>rootfsB</span> with a copy of <span style='color:green'>rootfsA</span>]
    POPULATEB --> ENCRYPTROOTFSA[create a LUKS container for <span style='color:green'>rootfsA</span> using key from OTP]
    ENCRYPTROOTFSA --> POPULATEA[Populate <span style='color:green'>rootfsA</span> with a copy from <span style='color:orange'>rootfsB</span>]
    POPULATEA --> MAP[Map all LUKS containers using key from OTP]
    MAP --> MOUNT[Mount all LUKS containers]
    MOUNT --> BOOT[Boot into selected rootfs]
    KEYCHECK -- Yes --> MAP
    

The path for already provisioned devices is quite straight forward: Retrieve the encryption key from OTP memory, map the LUKS containers using cryptsetup, mount the root filesystem, and call switch_root to boot into the LUKS-encrypted root filesystem.

The provisioning process is more complex and requires several steps to create the LUKS-encrypted root filesystem and store the encryption key in OTP memory. Our script will perform the following steps during the provisioning process:

1. Detect if the device is already provisioned by checking if the encryption key is present in OTP memory and if the root filesystem partitions are already LUKS encrypted.
2. Generate a unique encryption key for the device using a secure random number generator.
3. Store the generated encryption key in the One-Time Programmable (OTP) memory of the Raspberry Pi CM4 using the appropriate tools (e.g., rpi-eeprom).
4. Create a LUKS-encrypted container for the root filesystem using the generated encryption key.
5. Copy the unencrypted root filesystem into the LUKS-encrypted container.
6. Boot into the newly created LUKS-encrypted root filesystem.

One question to clarify is where to store the unencrypted root filesystem before the provisioning process. Since LUKS does not support in-place encryption of existing filesystems, we need to have the unencrypted root filesystem available somewhere during the first boot. If you are using an A/B partition scheme, you can store the unencrypted root filesystem in one of the partitions (e.g., partition A) and use the other partition (e.g., partition B) for the LUKS-encrypted root filesystem. During the provisioning process, the init script can copy the unencrypted root filesystem from partition A into the LUKS-encrypted container on partition B and then do the same vice versa from the now encrypted B partition to A. While this complicates the init script somewhat, it streamlines the provisioning process significantly, as no external media is required to provide the unencrypted root filesystem during the first boot and the entire image can be flashed at once using .wic images or similar.

So let’s look at the key commands that need to be executed in the init script for both the provisioning and normal unlocking processes.

Provisioning and boot process

First we need to check if the device is already provisioned. The simplest way to do this is to check if all partitions are already LUKS encrypted and if there is a non-zero key stored in OTP memory. If the OTP key is not set, we need to generate a new encryption key and store it in OTP memory. The RPI eeprom tools provide a convenient way to do this:

rpi-otp-private-key -w $(openssl rand -hex 32) -y

If the key is either generated or already present in OTP memory, we read it out and store it in a temporary key file for use with cryptsetup:

rpi-otp-private-key -r > /keyfile.bin
# in case the private key is only zeroes, the file is not created
if [ ! -f /keyfile.bin ] || [ ! -s /keyfile.bin ]; then
    echo "[initramfs] FATAL ERROR: OTP private key file /keyfile.bin missing or empty. This is fatal!"
    exec /bin/sh
fi

Note, that on error we drop into a shell for debugging purposes, but in a production system you might want to reboot -f or take halt the system entirely to avoid leaving the device in an inconsistent or vulnerable state.

⚠️ Dropping into a shell on error is dangerous for production systems. Use with caution and only for debugging purposes. ⚠️

Next we check if the root filesystem partitions are already LUKS encrypted or need to be provisioned. Here is an example of how to do this:

ROOTFS_A="/dev/mmcblk0p5"   # rootfs_A
ROOTFS_B="/dev/mmcblk0p6"   # rootfs_B
OTP_KEY_SET=false
rpi-otp-private-key -c && OTP_KEY_SET=true
cryptsetup isLuks "${ROOTFS_B}" || { [ "${OTP_KEY_SET}" = false ] && ROOTFS_B_NEEDS_SETUP=true; }
cryptsetup isLuks "${ROOTFS_A}" || { [ "${OTP_KEY_SET}" = false ] && ROOTFS_A_NEEDS_SETUP=true; }

The rpi-otp-private-key -c command stems from the rpi-eeprom yocto recipe and checks if a private key is stored in OTP memory. If the OTP is non-zero, it returns 0 else 1. We use this to set the OTP_KEY_SET variable accordingly. Next we check if the rootfs partitions are LUKS containers with the cryptsetup isLuks command. If either of the rootfs partitions is not a LUKS container and the OTP key is not set, we mark that partition as needing setup.

For this we create two helper functions: set_up_luks_partition and provision_luks_partition. The first function creates the LUKS container on the specified partition, while the second function copies the unencrypted root filesystem into the LUKS container and stores the encryption key in OTP memory. Here is an example implementation of these functions:

set_up_luks_partition(){
    target_partition="$1"
    mapping_name="$2"

    echo "[initramfs] Setting up LUKS on ${target_partition}..."
    cryptsetup luksFormat "${target_partition}" --batch-mode --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to format LUKS partition ${target_partition}."
        exec /bin/sh
    }

    echo "[initramfs] formatting ${target_partition} to ext4."

    cryptsetup luksOpen ${target_partition} ${mapping_name} --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to open LUKS partition ${target_partition}."
        exec /bin/sh
    }   

    mkfs.ext4 /dev/mapper/${mapping_name} || {
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to format LUKS partition ${target_partition}."
        exec /bin/sh
    }

    
    cryptsetup luksClose ${mapping_name}
    echo "[initramfs] LUKS partition ${target_partition} is set up and ready."
}

First, the set_up_luks_partition function takes two arguments: the target partition to be encrypted and the mapping name for the LUKS container. It uses cryptsetup luksFormat to create the LUKS container on the specified partition, using a predefined key file (/keyfile.bin) for simplicity. After creating the LUKS container, it opens it with cryptsetup luksOpen, formats it to ext4 using mkfs.ext4, and then closes the LUKS container. If any of these steps fail, an error message is printed, and the system reboots. However depending on where things go wrong this might leave you with a bricked device, so be careful when testing this!

Once the LUKS container is set up, we can proceed to the provisioning process with the provision_luks_partition function:

provision_luks_partition() {
    target_partition="$1"
    mapping_name="$2"
    source_partition="$3"

    echo "[initramfs] Opening LUKS partition ${target_partition}..."
    cryptsetup luksOpen ${target_partition} ${mapping_name} --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to open LUKS partition ${target_partition}."
        exec /bin/sh
    }   

    echo "[initramfs] mounting ${mapping_name} to /mnt/target."
    mkdir -p /mnt/target
    mount /dev/mapper/${mapping_name} /mnt/target || {
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to mount LUKS partition ${target_partition}."
        exec /bin/sh
    }
    mkdir -p /mnt/src
    mount ${source_partition} /mnt/src || {
        
        umount /mnt/target
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to mount source partition ${source_partition}."
        exec /bin/sh
    }

    echo "[initramfs] Copy everything from ${source_partition} (/mnt/src) to /mnt/target using rsync."
    rsync -aHAXx /mnt/src/ /mnt/target/ || {
        umount /mnt/target
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to copy data from /mnt/src to /mnt/target."
        exec /bin/sh
    }

    umount /mnt/target
    umount /mnt/src
    cryptsetup luksClose ${mapping_name}

}

The provision_luks_partition function takes three arguments: the target partition to be provisioned, the mapping name for the LUKS container, and the source partition containing the unencrypted root filesystem. It opens the LUKS container on the target partition, mounts it to /mnt/target, and mounts the source partition to /mnt/src. It then uses rsync to copy all data from the source partition to the target LUKS-encrypted partition. After the data transfer is complete, it unmounts both partitions and closes the LUKS container. If any step fails, an error message is printed, and the system reboots.

We then can use these functions to handle the provisioning process during the first boot. Here is an example of how to use these functions in the init script:

if [ "${ROOTFS_B_NEEDS_SETUP}" = true ]; then
    set_up_luks_partition ${ROOTFS_B} "rootfs_b" 
    provision_luks_partition ${ROOTFS_B} "rootfs_b" ${ROOTFS_A}
    
else
    echo "[initramfs] Rootfs_B partition ${ROOTFS_B} is already LUKS formatted. Skipping setup and provisioning."
fi

# Create a LUKS container for mmcblk0p5 (rootfs_A) if needed
if [ "${ROOTFS_A_NEEDS_SETUP}" = true ]; then
    echo "[initramfs] Rootfs_A partition ${ROOTFS_A} is not LUKS formatted. Setting up LUKS."
    set_up_luks_partition ${ROOTFS_A} "rootfs_a"
    # map rootfsb to copy data from it
    cryptsetup luksOpen ${ROOTFS_B} "rootfs_b" --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to open LUKS partition ${ROOTFS_B}."
        exec /bin/sh
    }
    provision_luks_partition ${ROOTFS_A} "rootfs_a" "/dev/mapper/rootfs_b"
    cryptsetup luksClose "rootfs_b"
else
    echo "[initramfs] Rootfs_A partition ${ROOTFS_A} is already LUKS formatted. Skipping setup and provisioning."
fi

Note that while the first provisioning is done from an unencrypted partition to an encrypted one, the second provisioning is done from an already encrypted partition to another encrypted partition. For this we need to open the source LUKS container first and use the mapped device as source for the provision_luks_partition function.

Once the provisioning process is complete, we can proceed to unlock the LUKS-encrypted root filesystem for normal operation during subsequent boots. Here is an example of how to do this in the init script, for simplicity let’s say always into rootfs_A:

cryptsetup luksOpen ${ROOTFS_A} rootfs_a --key-file /keyfile.bin || {
    echo "[initramfs] ERROR: Failed to open LUKS partition ${ROOTFS_A}."
    exec /bin/sh
}
mount /dev/mapper/rootfs_a /mnt/rootfs || {
    cryptsetup luksClose rootfs_a
    echo "[initramfs] ERROR: Failed to mount LUKS partition ${ROOTFS_A}."
    exec /bin/sh
}

At this point, the LUKS-encrypted root filesystem is unlocked and mounted to /mnt/rootfs, allowing the normal boot process to continue.

Finally, we need to switch to the unlocked root filesystem and continue the normal boot process. This can be done using the switch_root command:

exec switch_root /mnt/rootfs /sbin/init

At this point, the init script has completed its tasks, and the normal boot process continues from the unlocked LUKS-encrypted root filesystem.

Conclusion

Getting an embedded device to support encryption at rest may look daunting at first, but as illustrated yocto makes it relatively straightforward to create the necessary initramfs image and modify the bootloader configuration. The more complex part is the provisioning process during the first boot, which requires careful handling of encryption keys and data transfer. While such a self-provisioning process may be comfortable to use, it might come with some security implications. First of all, until the provisioning is done, the unencrypted root filesystem is present on the device, which means that an attacker with physical access to the device could potentially read and modify the unencrypted data before the provisioning process is complete. Additionally, the provisioning process itself involves copying data from an unencrypted partition to an encrypted one, which could be vulnerable to interception or tampering if not properly secured. Therefore, it is crucial to ensure that the provisioning process is executed in a secure environment and that appropriate measures are taken to protect the unencrypted data during this phase. Using an external source to pull the unencrypted root filesystem during provisioning could mitigate some of these risks, but it also adds complexity to the provisioning process.

Another concern is that the provisioning code is still present in the initramfs even after the device has been provisioned. An attacker who gains access to the device could potentially exploit vulnerabilities in the provisioning code to compromise the security of the encrypted root filesystem. To mitigate this risk, it is advisable to implement a mechanism to disable or remove the provisioning code from the initramfs after the first successful boot and provisioning process. If your device supports update mechanisms, you could push an updated initramfs without the provisioning code after the first boot.

Despite these considerations, implementing encryption at rest using LUKS on embedded Linux devices based on the Raspberry Pi Compute Module 4 with the yocto Project provides a robust solution for protecting sensitive data and will make exploiting the device significantly harder, even if an attacker has physical access.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

What is secure boot?

There are many aspects to consider when building a secure embedded device. One of the first steps towards security is making sure that a device runs only trusted software. This is generally called Secure boot and it is preventing attackers from loading malicious code on the device during the boot process. Secure boot is typically implemented using a chain of trust, where each stage of the boot process verifies the integrity and authenticity of the next stage before executing it.

In practice, this means that the device’s bootloader is cryptographically signed, and the hardware verifies this signature before executing the bootloader. If the signature is invalid, the boot process is halted, preventing the execution of potentially malicious code. For a more in-depth documentation on secure boot for raspberry pi devices, check out the Raspberry Pi Secure Boot documentation.

Implementing secure boot with Yocto

On a high level, implementing secure boot for an embedded Linux device using Yocto involves the following steps:

• Enable trusted boot support on the device itself - In the case of the Raspberry Pi CM4, this involves configuring the EEPROM firmware to enable secure boot features.
• Provide the public key used for verifying the signature of the bootloader to the device and store it on the EEPROM.
• Create a boot.img container that includes the bootloader, kernel, and device tree blobs (DTBs) - This container needs to be signed using a public key.
• Configure Yocto to build and sign the boot.img - This involves creating custom Yocto layers and recipes to handle the signing process and ensure that the boot.img is correctly formatted.
• Replace existing boot files with the signed boot.img and pack it into the boot partition of the device’s storage medium.

Before we dive into the details, we enable trusted boot on the Raspberry Pi CM4.

Enabling trusted boot on the Raspberry Pi CM4

Trusted boot on the Raspberry Pi CM4 needs to be enabled in the boot EEPROM by setting the SIGNED_BOOT flag in the boot.conf and by providing the public key to verify the signature. This is done using the rpiboot tool, which allows us to write the necessary configuration to the EEPROM. The usbboot repository provides the necessary tools and documentations to update the EEPROM for Raspberry Pi.

For illustration purposes, this article shows how to enable secure boot using a self-signed key pair. In a production environment, you would typically use a key pair issued by a trusted certificate authority (CA).

First, we generate a public/private key pair using OpenSSL:

openssl genrsa 2048 > keypair.pem

Next, we set up the boot.conf file to enable signed boot by setting SIGNED_BOOT=1. Then we use the update-pieeprom tool from rpiboot to update the EEPROM with the public key and flash it to the device:

update-pieeprom -k  keypair.pem
rpiboot -d /path/to/bootconf/

From this point on, the device will expect a signed boot.img during the boot process, the signature of it is expected to be present in a boot.sig file. If we try to boot an unsigned image, the device will halt the boot process with a Bad signature boot.sig error. To disable secure boot, we can overwrite the EEPROM again with SIGNED_BOOT=0.

Now that secure boot is enabled on the device, we can move on to configuring Yocto to build and sign the boot.img.

Configuring Yocto to build and sign the boot.img

Now that secure boot is enabled on the device and the key pair is generated, we can configure Yocto to build and sign the boot.img. This involves creating a custom Yocto recipe to create the boot container and to handle the signing process.

By default the raspberrypi layers in yocto store the boot files directly in the boot partition, so once we have a signed boot.img, we need to override the default behavior to use our signed image instead.

Let’s start with creating the boot.img container

Creating the boot.img container

The boot.img container is a fat16 formatted file that contains the bootloader, kernel, and device tree blobs (DTBs). We can create a custom Yocto recipe to build this container, let’s call this boot-img-container.bb:

The recipe will do the following things:

• Create a staging directory to hold the boot files
• Copy the bootloader, kernel, and DTBs to the staging directory
• Create a fat16 formatted boot.img
• Copy the boot files from the staging directory to the boot.img

Let’s look at the recipe code:

inherit deploy nopackages

DEPENDS += "rpi-bootfiles rpi-cmdline rpi-config linux-raspberrypi dosfstools-native mtools-native"

# The raspberry pi bootloader files are located in the following directories
BOOTFILES_DIR = "${WORKDIR}/bootfiles" 
DTBO_DIR = "${DEPLOY_DIR_IMAGE}"

S = "${WORKDIR}/boot-img-container"

do_compile() {
    # Create staging and target directory
    mkdir -p ${B}/boot-img-container-staging/overlays
    mkdir -p ${S}

    #collect all bootfiles from BOOTFILES_DIR and DTBO_DIR and and put it into the staging dir
    find ${BOOTFILES_DIR} -maxdepth 1 -type f ! -name 'cmdline.txt' -exec install -m 0644 -t ${B}/boot-img-container-staging/ {} +

    # copy all resolved .dtbo symlinks (the image is vfat formatted and symlinks are not supported)
    find ${DTBO_DIR} -type f -name '*.dtbo' -exec install -m 0644 -t ${B}/boot-img-container-staging/overlays {} +

    # copy all dtb files except overlay_map.dtb
    find ${DEPLOY_DIR_IMAGE} -type f -name '*.dtb' ! -name 'overlay_map.dtb' -exec install -m 0644 -t ${B}/boot-img-container-staging/ {} +

    # the overlay map file needs to be placed into the overlays folder too
    install -m 0644 ${DEPLOY_DIR_IMAGE}/overlay_map.dtb ${B}/boot-img-container-staging/overlays/overlay_map.dtb

    # copy the kernel image
    install -m 0644 ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE_DIRECT} ${B}/boot-img-container-staging/${SDIMG_KERNELIMAGE}

    BOOT_STAGING_DIR="${B}/boot-img-container-staging"
    BOOT_STAGING_SIZE_BYTES=$(find "$BOOT_STAGING_DIR" -type f -printf '%s\n' 2>/dev/null | awk '{s+=$1} END{print s+0}')
    
    # Create a FAT image sized to fit the staged files (+ padding), format it, copy files.
    BOOT_IMG="${S}/boot.img"

    # Add padding and round up to nearest MiB for filesystem overhead use awk because shell arithmetic is poorly supported in bitbake
    PADDING=$(awk 'BEGIN{printf "%d\n", 8*1024*1024}')
    PADDED_SIZE=$(awk -v a="$BOOT_STAGING_SIZE_BYTES" -v b="$PADDING" 'BEGIN{printf "%d\n", a+b}')
    MIN_SIZE=$(awk 'BEGIN{printf "%d\n", 1*1024*1024}')
    if [ "$PADDED_SIZE" -lt "$MIN_SIZE" ]; then
        PADDED_SIZE="$MIN_SIZE"
    fi
    PADDED_SIZE=$(awk -v n="$PADDED_SIZE" 'BEGIN{s=1024*1024; printf "%d\n", int((n + s - 1) / s) * s }')

    # Create an empty file of the desired size
    rm -f "$BOOT_IMG"
    truncate -s "$PADDED_SIZE" "$BOOT_IMG"

    # Create FAT16 filesystem
    mkfs.vfat -F 16 -n BOOT "$BOOT_IMG"

    # Copy files to the FAT image
    mcopy -i ${BOOT_IMG} -s ${BOOT_STAGING_DIR}/* ::/

}
do_compile[depends] += "rpi-bootfiles:do_deploy"
do_compile[depends] += "rpi-cmdline:do_deploy"
do_compile[depends] += "rpi-config:do_deploy"
# make do_compile depend on the boot files and kernel image
do_compile[filedeps] += " \
    ${BOOTFILES_DIR}/* \
    ${DEPLOY_DIR_IMAGE}/*.dtb \
    ${DEPLOY_DIR_IMAGE}/*.dtbo \
    ${DEPLOY_DIR_IMAGE}/overlay_map.dtb \
    ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE_DIRECT} \

First of all, we define the recipe as a deployment recipe by inheriting the deploy class. This allows us to place the generated boot.img in the deploy directory for later use. Since the recipe does not produce any packages to be put in the root filesystem, we also inherit the nopackages class. As for dependencies, we need the rpi-bootfiles and linux-raspberrypi recipes to provide the necessary boot files and kernel image. Additionally, we need dosfstools-native and mtools-native to create and manipulate the FAT filesystem.

Next we define the directories where the boot files are located. The BOOTFILES_DIR variable points to the directory where the Raspberry Pi bootloader files are located, while the DTBO_DIR variable points to the deploy directory where the DTBs are located.

And now begins the actual work. While we do not generate packages, we still compile an artifact, so we implement the do_compile function.

After creation of the staging and the target directory, we collect all necessary boot files from the defined directories and copy them to the staging directory. We make sure to copy the resolved .dtbo files instead of the symlinks, since the FAT filesystem does not support symlinks. These are all the find ... -exec install ... commands in the recipe. We also copy the kernel image to the staging directory.

Next comes the actual creation of the boot.img. We first calculate the size needed for the image by summing up the sizes of all files in the staging directory. We then add some padding and round up to the nearest MiB to account for filesystem overhead. This is a bit of an optimization to avoid wasting space, for a first implementation you could also just pick a fixed size that is large enough to hold all files. Since shell arithmetic is poorly supported in bitbake, I used awk for the calculations.

With the calculated size, we create an empty file of that size using truncate, format it as a FAT16 filesystem using mkfs.vfat, and finally copy all files from the staging directory to the boot.img using mcopy.

And with that, we have our boot.img container created!

Lastly, we make sure that the do_compile task depends on the deployment of the boot files and kernel image, so that it is executed after those are available. We also add the boot files and kernel image as file dependencies to ensure that any changes to these files will trigger a rebuild of the boot.img.

Next up is signing the boot.img.

Signing the boot.img

After creating the boot.img, we need to sign it using our private key. For debugging purposes, it is convenient to also be able to create unsigned images, so let’s add the signing as a separate recipe that depends on the boot-img-container recipe. This way we can choose whether to build a signed or unsigned image by including the appropriate recipe in our build.

Signing is relatively straightforward:

• Find the generated boot.img from the boot-img-container recipe
• Generate a sha256 hash of the boot.img to create a digest
• Sign the digest using the private key to create the signature
• Copy the boot.sig file to the deploy directory

The resulting recipe could look like this:

inherit nopackages

DEPENDS += "boot-img-container openssl-native xxd-native"

BOOT_IMG_PATH = "${DEPLOY_DIR_IMAGE}/boot.img"
SIG_FILE_PATH = "${DEPLOY_DIR_IMAGE}/boot.sig"
# Path to the .pem file containing the private key for signing 
BOOT_IMG_CERTIFICATE_PEM = "${THISDIR}/files/development-1.key.pem"

do_compile() {
    if [ -z "${BOOT_IMG_CERTIFICATE_PEM}" ]; then
        bbfatal "BOOT_IMG_CERTIFICATE_PEM is not set. Please specify the path to the .pem file."
    fi

    if [ -f "${BOOT_IMG_PATH}" ]; then
        # Generate a signature file containing the sha256 hash of boot.img
        sha256sum "${BOOT_IMG_PATH}" | awk '{print $1}' > "${SIG_FILE_PATH}"
        # next is the time of signing 
        printf 'ts: %s\n' "$(date -u +%s)" >> "${SIG_FILE_PATH}"

        # Generate RSA signature of the boot.img using openssl and hash it using sha256
        # append at the end of the sig file

        sig=$(openssl dgst -sha256 -sign "${BOOT_IMG_CERTIFICATE_PEM}" "${BOOT_IMG_PATH}" | xxd -c 4096 -p )
        printf 'rsa2048: %s\n' "${sig}" >> "${SIG_FILE_PATH}"
    else
        bbwarn "boot.img not found, skipping signature generation."
    fi
}

do_deploy() {
    if [ -f "${SIG_FILE_PATH}" ]; then
        install -D -m 0644 "${SIG_FILE_PATH}" "${D}${DEPLOYDIR}/boot.sig"
    else
        bbwarn "Signature file not found, skipping installation."
    fi
}

addtask do_compile after do_fetch before do_install
addtask do_deploy after do_compile before do_build
do_compile[depends] += "boot-img-container:do_deploy"

This recipe inherits the nopackages class since it does not produce any packages. It depends on the boot-img-container recipe to ensure that the boot.img is available for signing. Additionally, it depends on openssl-native and xxd-native to perform the signing operation.

The BOOT_IMG_PATH variable points to the location of the generated boot.img, while the SIG_FILE_PATH variable defines where the signature file will be created. The BOOT_IMG_CERTIFICATE_PEM variable should point to the private key file used for signing, this should be overridden with the actual path to the private key in a secure manner (i.e. by pulling it from an AWS Secrets Manager, Azure Key Vault, or similar secure storage).

The do_compile function first checks if the BOOT_IMG_CERTIFICATE_PEM variable is set and if the boot.img file exists. If both checks pass, it generates a SHA256 hash of the boot.img and writes it to the signature file. It also appends a timestamp to the signature file for reference. Then, it generates an RSA signature of the boot.img using the private key and appends it to the signature file.

And with that, we have our signed boot.img and boot.sig files ready for deployment! The do_deploy function installs the boot.sig file to the deploy directory, so the image creation command automatically packs it.

Lastly, we need to tell bitbake to execute the do_compile and do_deploy tasks at the appropriate times in the build process by adding them to the task graph. We do this by placing the do_compile task before the do_install task and the do_deploy task before the do_build task. We also make sure that the do_compile task depends on the deployment of the boot-img-container recipe to ensure that the boot.img is available for signing.

With that we have the recipes ready to create and sign the boot.img. The last step is to make sure that the signed image is used during the build process instead of the default boot files.

Using the signed boot.img in the build

Since the recipes are not packages, we add them to the machine part of the image configuration file (i.e. raspberrypi-cm4.conf):

IMAGE_BOOT_FILES = "boot.img config.txt boot.sig"
EXTRA_IMAGEDEPENDS += "boot-img-container sign-boot-img"
do_image_wic[depends] += "sign-boot-img:do_compile"

First the existing IMAGE_BOOT_FILES variable is overridden to include the boot.img and boot.sig files instead of the default boot files produced by the raspberrypi layer. Next, we add the boot-img-container and sign-boot-img recipes to the EXTRA_IMAGEDEPENDS variable to ensure that they are built before the image is created.

The recipes to create and sign the boot image need to be executed before the image creation process, so we add a dependency from the do_image_wic task to the do_compile task of the sign-boot-img recipe. If you are using a different image creation method (i.e. do_image_sdcard), you need to adjust this accordingly.

And with that, we have successfully integrated secure boot into our Yocto build process for the Raspberry Pi CM4. You can now build your image as usual using bitbake, and the resulting image will include the signed boot.img and boot.sig files, ensuring that only trusted software is executed during the boot process.

What about further improvements?

The recipes and approach described in this post provide a basic implementation of secure boot for an embedded Linux device using Yocto. For production use one improvement that needs to be made is to securely manage the private key used for signing the boot.img. Storing the private key directly in the source code or build environment is not secure and can lead to compromise of the key. Instead, consider using a secure key management solution, such as a hardware security module (HSM) or a cloud-based key management service (KMS), to store and manage the private key.

The Cyber Resilience Act at a glance

At the highest level, the Cyber Resilience Act (CRA) is a regulation by the European Union that aims to enhance the cybersecurity of products with digital elements, including software and connected hardware devices. The CRA establishes baseline security requirements for these products throughout their lifecycle, from design and development to deployment and maintenance. The term “products with digital elements” is defined broadly and includes any product that relies on software to function, such as IoT devices, medical devices, industrial control systems, and general-purpose software applications.

At the core, the CRA focuses on three main areas:

• Security-by-design: Integrate security measures into the product design and development process from the outset.
• Vulnerability management and incident response: Establish processes for identifying, reporting, and remediating vulnerabilities throughout the product lifecycle. Develop and maintain an incident response plan to address security incidents effectively.
• Compliance documentation: Maintain comprehensive documentation to demonstrate compliance with CRA requirements.

Let’s look at some of the key requirements of the CRA in more detail.

Security-by-design

Security by design is a fundamental principle of the CRA and is often considered the most technical part. This means that security considerations must be part of the design and development process of software products. Companies need to implement secure coding practices, conduct regular security testing (e.g., static and dynamic analysis, fuzzing, penetration testing), and ensure that products are resilient against common cyber threats. Additionally, products must be designed to minimize attack surfaces and protect sensitive data. But what about existing products? Do they need to be redesigned from scratch? Not necessarily. The CRA recognizes that many products are already in the market and allows for a risk-based approach to security improvements.

The CRA recognizes that many products are already in the market and allows for a risk-based approach to security improvements.

This means companies need to assess the security and perform a threat analysis of their existing products first. After this assessment, they can prioritize security enhancements based on the identified risks and start implementing necessary measures to improve security over time. An important point here is that many security issues might be mitigated by means other than code changes, e.g., by limiting exposure to threat vectors or improving user guidance.

Vulnerability management and incident response

The authors of the CRA understand that no software is perfect and vulnerabilities will inevitably be discovered over time. Therefore, the CRA mandates that companies establish vulnerability management and incident response processes. At the core, vulnerability management means setting up a way to report and triage vulnerabilities, a way to inform customers about the existence of a vulnerability, and ensuring timely remediation of identified vulnerabilities. Again, this sounds like a lot of work, but in many cases existing processes for receiving user feedback or bug reports can be extended to cover security vulnerabilities.

While proactively addressing vulnerabilities is important, companies must also be prepared to respond effectively when security incidents occur. This is closely tied to vulnerability management and may even use some of the same facilities.

Incident response means to have a clear plan in place that outlines roles, responsibilities, and communication channels during a security incident.

The most important part of incident response is to have a clear plan in place that outlines roles, responsibilities, and communication channels during a security incident. Make it explicit who is taking the lead, who needs to sit in the crisis room, and what response times are expected. Who can make decisions to enable containing the incident, mitigating its impact, and recovering normal operations. The third part to be ready for the CRA is comprehensive compliance documentation.

Compliance documentation

Chances are that if you are affected by the CRA, you already have some level of compliance documentation in place, because many other regulations (e.g., GDPR, industry-specific regulations) also require documentation of security practices. If so, good news: you can build upon your existing documentation efforts. If you have not yet started, don’t panic; creating the necessary documentation is not an unmanageable task. So what kind of documentation is required? First, the efforts described in the previous sections need to be documented. This includes documenting the security-by-design practices, vulnerability management processes, and incident response plans discussed earlier.

Don’t panic, creating the necessary documentation for the CRA is not an unmanageable task.

Additionally, companies must maintain records of security testing results, risk assessments, and any security-related decisions made during the product lifecycle. In practice this means nothing more than keeping the CI/CD and testing results of your releases ready and creating a comprehensive changelog. Additionally, the CRA requires companies to keep track of third-party components used in their products, including open-source libraries. This means maintaining a Software Bill of Materials (SBOM) that lists all components, their versions, and any known vulnerabilities associated with them. This can be automated to a large extent using existing tools that generate SBOMs as part of the build process.

The compliance documentation is the final piece of the puzzle to be ready for the CRA. While security-by-design and vulnerability management are the more practical parts, compliance documentation ensures that companies can demonstrate their adherence to CRA requirements. So where to start?

Get started with the Cyber Resilience Act

Preparing for the CRA may seem overwhelming at first, but breaking it down into manageable steps can make the process more approachable. An incremental approach is often the best way to build up the necessary capabilities to pass an audit and get your products certified.

Start with a high-level gap assessment to identify where your current practices stand in relation to the CRA requirements. Then implement a “CRA-Light” process that covers the most critical aspects of the CRA without overwhelming your teams. Start with the “who” regarding vulnerability management and incident response. Do a high-level STRIDE threat model for your main products to identify the biggest risks. Automate SBOM generation in your build process and start collecting compliance documentation as you go along.

While this might not cover all aspects of the CRA right away, it establishes a foundation that can be built upon over time. Building up CRA compliance incrementally allows teams to adapt and improve their processes without feeling overwhelmed by the full scope of the regulation from the start. And remember, passing audits on the first try is nice, but often not the case—so build a process that allows you to continuously improve over time.

After all, the CRA is not just about compliance; it’s about enhancing the overall security posture of products and protecting users from cyber threats. By taking proactive steps to meet CRA requirements, companies can not only ensure market access in the EU but also contribute to a safer digital ecosystem.

std::expected in a nutshell

At the core, std::expected<T, E> is a returnable type that can either hold a value of type T (indicating success) or an error of type E (indicating failure). This makes it clear to the caller that a function can fail and provides a structured way to handle that failure.

Let’s look at a simple example of a function that computes the square root of a number, returning an error if the input is negative:

#include <expected>
#include <cmath>

std::expected<double, std::string> safe_sqrt(double x) {
    if (x < 0) {
        return std::unexpected("Negative input");
    }
    return std::sqrt(x);
}

...

// Usage
const auto result = safe_sqrt(-1);
if (result) {
    std::cout << "Square root: " << *result << '\n';
} else {
    std::cout << "Error: " << result.error() << '\n';
}

In this example, safe_sqrt returns an std::expected<double, std::string>. If the input is valid, it returns the square root; otherwise, it returns an error message. The caller can then check if the result is valid and handle the error accordingly. So how does this compare to traditional error handling methods?

Comparison to Traditional Error Handling

Before std::expected, there were typically two main approaches to error handling in C++: exceptions and error codes. While exceptions can be powerful, they typically bring with them more complexity in control flow and then there is the discussion which errors should cause an exception to be thrown and which should not. The benefit of exceptions is that they allow for clean separation of error handling code and for propagation of errors up the call stack. Error codes on the other hand tend to either clutter the code by requiring out-parameters or have the problem of being either ignored or misunderstood by the caller. While nodiscard can help with ignored return values, it still does not solve the problem that the caller has to semantically understand the meaning of the return value.

std::expected provides a middle ground. It makes error handling explicit in the type system, allowing to pass semantic information about the error back to the caller. The beauty of std::expectedis also, that it can help to discern between expected or recoverable errors (e.g. file not found, invalid input) and unexpected or unrecoverable errors (e.g. out of memory, logic errors) which should still be handled via exceptions.

Tip: Use std::expected for recoverable errors where the caller can take action based on the error, and reserve exceptions for truly exceptional situations.

Let’s look at a more complex example that demonstrates how std::expected can be used in a real-world scenario.

Real world example: Reading a QR code from an image

Let’s suppose we want to write a function that reads a QR code from binary image data. The function generally has three paths:

1. The image contains a valid QR code and we can return the decoded string.
2. The image does not contain a QR code and we want to return an error indicating that.
3. The image data is unreadable (e.g. corrupted or unrecognizable format) and we want to throw an exception.

While the first two paths are expected and recoverable errors, the third path is an unexpected error that should be handled via exceptions. So the implementation could look like this:

#include <expected>
#include <string>
#include <stdexcept>
#include <vector>   

std::expected<std::string, std::string> read_qr_code(std::vector<uint8_t> const& image_data) {
    
    if (image_data.empty() || check_if_corrupted(image_data)) {
        throw std::invalid_argument("Invalid image data");
    }

    // Assume parse_image_data is a function that parses the image data and returns the QR code string or throws on failure
    std::string parsed_data = parse_image_data(image_data); // May throw exceptions on failure

    if (parsed_data.empty()) {
        return std::unexpected("No QR code found");
    }

    return parsed_data;
}

Note that in this example, both the success and error type are strings, but they could be any type. In a lot of cases, it might still make sense to use an enum or a custom error type for the error case to make it more structured. However, by using std::expected, we already are able to add a lot more context to the function without cluttering the code. This already is a big improvement over returning, but there is more. std::expected also provides a set of powerful combinators for composing operations that may fail, allowing for more elegant error handling.

Monadic chaining with and_then

One of the benefits of std::expected is that it allows for monadic chaining, A technique widely used in functional programming. Monadic chaining lets you compose a sequence of operations that may fail, without deeply nested if/else. With std::expected, you chain:

• and_then when the next step itself may fail and returns another expected.
• transform when the next step cannot fail and just maps the value.
• or_else to act on or recover from an error.

Below is a continuation of the QR example, showing a pipeline that: 1) reads the QR payload, 2) validates it, 3) parses it as a URI, 4) extracts the host (pure mapping), 5) adds context to the error if anything failed.

// Reuse the earlier read_qr_code signature:
// std::expected<std::string, std::string> read_qr_code(const std::vector<uint8_t>&);

#include <expected>
#include <string>
#include <vector>

struct Uri {
    std::string scheme;
    std::string host;
    std::string path;
};

std::expected<std::string, std::string>
validate_payload(std::string const& s) {
    if (s.empty()) {
        return std::unexpected("Empty QR payload");
    }
    if (s.size() > 4096) { // arbitrary sanity limit
        return std::unexpected("QR payload too large");
    }
    return s; // valid as-is
}

std::expected<Uri, std::string>
parse_uri(std::string const& s) {
    auto starts_with = [&](std::string const& p) {
        return s.rfind(p, 0) == 0;
    };

    Uri u;
    if (starts_with("https://")) {
        u.scheme = "https";
    } else if (starts_with("http://")) {
        u.scheme = "http";
    } else {
        return std::unexpected("Not an http(s) URI");
    }

    // very naive split: scheme://host/path
    auto pos = s.find("://");
    auto rest = s.substr(pos + 3);
    auto slash = rest.find('/');
    if (slash == std::string::npos) {
        u.host = rest;
        u.path = "/";
    } else {
        u.host = rest.substr(0, slash);
        u.path = rest.substr(slash);
    }
    if (u.host.empty()) {
        return std::unexpected("Missing host");
    }
    return u;
}

std::string host_from(Uri const& u) {
    return u.host; // pure mapping, cannot fail
}

std::expected<std::string, std::string>
annotate_error(std::string const& err) {
    return std::unexpected(std::string{"QR processing failed: "} + err);
}

// Usage: linear, early-exiting pipeline
std::expected<std::string, std::string>
extract_qr_host(std::vector<uint8_t> const& image) {
    return read_qr_code(image)
        .and_then(validate_payload)        // may fail -> expected<Payload, E>
        .and_then(parse_uri)               // may fail -> expected<Uri, E>
        .transform(host_from)              // cannot fail -> expected<std::string, E>
        .or_else(annotate_error);          // act on error path, keep E the same
}

As we see, the resulting code in extract_qr_host is linear and easy to read. Each step is clearly defined, and error handling is centralized without deeply nested conditionals. The use of and_then, transform, and or_else makes the intent of each operation explicit.

There are some pitfalls and good practices to keep in mind when using monadic chaining with std::expected:

• Keep the error type E consistent across and_then/or_else steps. If you must change it, use transform_error.
• Use and_then only with functions that return expected<..., E>. Use transform for pure mappings returning plain values.
• The chain short-circuits on the first error, returning that error downstream. So having a catch-all or_else at the end is a good practice.
• If a step can throw, those exceptions still propagate unless caught and converted to std::unexpected.
• Prefer passing by const& in chain steps to avoid copies or use move semantics. However the guaranteed copy elision in C++17 and later often makes this less of a concern.

With these practices in mind, std::expected and its combinators can greatly enhance the clarity and robustness of error handling in your C++ code.

final thoughts

With the arrival of std::expected in C++23 there is another powerful tool in C++ to allow more expressive code in a functional programming style. This can make applications that do a lot of data processing and have many recoverable failure paths much cleaner and easier to maintain. While it does not replace exceptions for unrecoverable errors, it nicely complements them by providing a structured way to handle expected errors. And the beauty of it is, that it still works seamlessly with existing C++ code and libraries - So no need to go all in and change it everywhere. So give it a try in your next C++ project and see how it can improve your error handling!

While std::expected became part of the C++ standard with C++23, it has been around for quite some time as an open-source library by Sy Brand before that.

Why build your own AI agent?

While tapping into an existing online service to build your custom AI agent might be convenient, there are several reasons why you might want to build your own. There might be privacy concerns, or you might need offline access to the agent or maybe you want to customize the agent to your specific needs. Building your own AI agent gives you full control over the model, the data and the functionality. You can tailor it to your specific use case and integrate it seamlessly into your existing systems. A great place to start building your own AI agent is the llama.cpp project.

Llama.cpp is a C/C++ implementation of Facebook’s LLaMA model that allows you to run LLMs on your own hardware. It is designed to be efficient and lightweight, making it suitable for running on a wide range of devices, from high-end servers to low-power edge devices. Llama.cpp supports various LLaMA models, including the smaller ones that can run on consumer-grade hardware. It offers a wide range of cross-platform support including various GPU backends for acceleration, but also runs on CPU only systems. So let’s see how to get started with llama.cpp.

Building a simple AI agent with llama.cpp

So let’s build a simple executable that uses llama.cpp to answer simple prompts by using a pre-trained model. You can find the complete code on on GitHub it the bernedom/LlamaPlayground repository which is inspired by the simple example provided in the llama.cpp repository. The executable will be called LLamaPlayground and can be used like this:

> LLamaPlayground -m ./models/mymodel.gguf What is the capital of Switzerland?

The capital of Switzerland is Bern.

To get started we need a few things:

• A C++17 compatible compiler and CMake
• llama.cpp as library, which we can get by using CMakes FetchContent module. (Unfortunately the conan module for llama.cpp is outdated at the time of writing)
• A LLM-Model as .gguf file which can be obtained from various sources, e.g. Huggingface

Setting up the project

Let’s start by setting up a simple CMake project, that creates our executable and pulls in llama.cpp as a dependency. To kickstart your project with a basic CMake and dependency handling already set up, consider this CMake/Conan/Catch2 Template.

In the CMakeLists.txt we need to pull in llama.cpp using FetchContent and link it to our executable like this

...
include(FetchContent)
FetchContent_Declare(
        llama
        GIT_REPOSITORY https://github.com/ggml-org/llama.cpp.git
        GIT_TAG master
)
FetchContent_MakeAvailable(llama)
...

This gives us access to the llama target that we can link to our executable like this target_link_libraries(LLamaPlayground PRIVATE llama).

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Next we need to obtain a model. You can find various models on Huggingface. For this example I downloaded the ggml-org/gemma-3-1b-it-GGUF model and placed it in a models folder next to the executable. This is a fairly lightweight model that accepts text or images as input and generates text as output.

The easiest way to get the model is to use the huggingface CLI. You can install it using pip:

pip install huggingface_hub[cli]

To download the model, you can use the following command:

hf download ggml-org/gemma-3-1b-it-GGUF --local-dir ./models

And with that we are ready to start coding.

Coding the AI agent

Our simple AI agent will have to do three basig things:

1. Model Loading: Loading the pre-trained model file.
2. Prompt Tokenization: Converting the input prompt into tokens.
3. Text Generation: Using the model to predict and generate tokens.

I’ll skip the boilerplate code to parse the arguments and focus on the llama.cpp specific parts, for the full code head over to the Github repository containing the example code

Model Loading

First let’s load the model and initialize it.

llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = ngl;

llama_model *model = llama_model_load_from_file(model_path.c_str(), model_params);

For simplicity we use the default model parameters, but you can customize them as needed. To illustrate how this works, the n_gpu_layers parameter specifies how many layers of the model should be offloaded to the GPU for acceleration. If you don’t have a GPU or want to run on CPU only, you can set this to 0. That is already all the magic needed to load the model, lets go to the next step Prompt Tokenization.

Prompt Tokenization

To convert the user-supplied prompt into tokens we need to convert it from a string into tokens that the model can understand. The prompt is tokenized using the model’s vocabulary. Tokenization involves splitting the input string into smaller units (tokens) that the model can process, these are no longer human readable strings but rather integer values that represent words or subwords.

const llama_vocab *vocab = llama_model_get_vocab(model);
const int n_prompt = -llama_tokenize(vocab, prompt.c_str(), prompt.size(), nullptr, 0, true, true);

std::vector<llama_token> prompt_tokens(n_prompt);
llama_tokenize(vocab, prompt.c_str(), prompt.size(), prompt_tokens.data(), prompt_tokens.size(), true, true);

The first call to llama_tokenize with a nullptr for the output tokens is used to determine how many tokens are needed for the prompt. It’s a particularity of llama.cpp that the first call returns a negative value, because it failed to write the tokens to a nullptr. The absolute value of the returned integer indicates the number of tokens required.

The second call actually performs the tokenization and fills the prompt_tokens vector with the resulting tokens. With this we can initialize the context for the model and prepare it for text generation.

llama_context_params ctx_params = llama_context_default_params();
ctx_params.n_ctx = n_prompt + n_predict - 1;
ctx_params.n_batch = n_prompt;

llama_context *ctx = llama_init_from_model(model, ctx_params);

The context defines the model’s working environment, including the number of tokens to process and the batch size. For our small example we set the context size to the sum of the prompt tokens and the number of tokens we want to predict. The batch size is set to the number of prompt tokens, which means that the model will process all prompt tokens in one go.

Now that we have the prompt tokenized, we can move on to the final step: Text Generation.

Text Generation

The main loop evaluates the model and generates tokens iteratively. Each token is sampled, converted back to text, and appended to the output. for this we loop until we reach the desired number of predicted tokens or encounter an end-of-generation token.

llama_batch batch = llama_batch_get_one(prompt_tokens.data(), prompt_tokens.size());

for (int n_pos = 0; n_pos + batch.n_tokens < n_prompt + n_predict;) {
    llama_decode(ctx, batch);

    llama_token new_token_id = llama_sampler_sample(smpl, ctx, -1);
    if (llama_vocab_is_eog(vocab, new_token_id)) break;

    char buf[128];
    int n = llama_token_to_piece(vocab, new_token_id, buf, sizeof(buf), 0, true);
    std::cout << std::string(buf, n);

    batch = llama_batch_get_one(&new_token_id, 1);
}

First we fetch a batch of tokens to process, in our case this might be the entire prompt, but since we might want to extend to larger prompts, we enter a loop.

Inside the loop the current batch is decoded using llama_decode, which processes the tokens and updates the model’s internal state. Then we sample a new token using llama_sampler_sample, which selects the next token based on the model’s predictions. If the sampled token is an end-of-generation token, we break out of the loop.

The sampled token is then converted back to a human-readable string using llama_token_to_piece and printed to the console. Finally, we prepare the next batch containing just the newly generated token for the next iteration of the loop. And that is all the code we need to build a simple AI agent using llama.cpp. You can find the complete code on on GitHub it the bernedom/LlamaPlayground repository.

To run the AI agent, you need to compile the code using CMake and your C++ compiler. Make sure you have the model file in the specified path. Beware that even the smaller models can be quite memory/hungry, so make sure you have enough RAM available.

Whats next?

This is a very simple example to get you started with llama.cpp and building your own AI agent. As the talk of edge AI and on-device AI is getting more and more popular, llama.cpp might be a great starting point to explore this field. There are of course many more things to explore be it towards more efficiency or hardware acceleration through CUDA or Vulkan or more advanced prompt handling and context management. Whether you find any practical use for it or just play around with it, I hope this post helped you to get started.

What is artifact based CI?

Let’s look at a typical CI/CD pipeline: When a developer pushes code changes, the CI system triggers a build process that compiles the code, runs tests, and produces artifacts (e.g., binaries, libraries, documentation). In a traditional setup, every time a change is pushed, the entire build process is executed from scratch, which can be time-consuming and resource-intensive.

For a simple and small project such a straight forward pipeline is all what it takes. However if projects grow larger and contain multiple components, the build process can become complex and time-consuming. This is where artifact based CI comes into play. A slightly more complicated example might be an application that consists of several modules, such as an executable and several libraries, where the executable depends on the libraries. A trivial approach to CI/CD would be to build all the libraries from scratch on every change, then deploy all together to a test-environment, run the tests and if successfully, package them and publish them somewhere or deploy them to the target systems.

Since the libraries and the executable should be buildable individually, building everything from scratch, even if only a part of the code has changed is wasteful. Also, if for example only code in the executable changes, running the full library build and test process is not necessary. So to optimize this, we can introduce artifact based CI.

Artifact based CI is a practice where the output of your build process (the artifacts) are stored and reused in subsequent builds. This means that instead of rebuilding everything from scratch every time, we can leverage previously built artifacts, which can save a lot of time and resources.

If done right, artifact based CI can not just speed up your builds and increase the reliability of your CI/CD pipelines, but they are also a great way to test deployment processes on the go. By deploying the same artifacts that will be used in production, you can catch deployment issues early in the development process.

The same pipeline as before, but now with artifact based CI could look like this:

Wow, this looks a lot more complicated, but let’s break it down a bit. The first thing introduced here is some form of decision making to determine whether a component has changed or not. This can often be done by checking the version control systems for changes in the relevant files or directories. What we also need is a way to store the artifacts produced by the build process. Many CI/CD systems such as github actions or gitlab pipelines offer a simple way to store and retrieve build outputs, but for more complex applications a dedicated artifact repository, such as JFrog Artifactory, Nexus Repository might be better suited. The key is to ensure that the artifacts are versioned and easily retrievable. The last thing to notice is that this already gives us a way to speed up the build process, even if we need to build everything from scratch, because we can parallelize the jobs for building the libraries.

When moving to artifact based CI, the central problem usually revolves around how to efficiently store and retrieve the artifacts, how to version them properly, and how to ensure that the right artifacts are used in the right places.

Finding the right artifact

The core concept of artifact based CI rotates around storing and retrieving build artifacts efficiently. Looking a bit closer this the main problem to be solved is that we need to find the right artifact for the current build. So putting up a good versioning and naming scheme is crucial. In a typical workflow we have three kind of versions of artifacts that we need to consider:

• Development builds: These are the artifacts produced during the development process, often for branches or pull requests. Typically, these builds are only used by a small group of developers (or even a single one) working on a specific feature or bug fix. They are usually stored temporarily and can be discarded after a certain period of time or when the feature is merged into the main branch.

• Released builds: These are the artifacts that are deployed to production environments. They should be stored permanently and should be easily retrievable.

• Staged builds: These are the artifacts used for testing purposes, such as staging or QA environments. They can be a mix of development and production builds, depending on the testing requirements. If all goes well, these artifacts might be promoted to production builds or discarded if newer artifacts are available.

The naming and versioning of the artifacts should reflect these different stages. A common approach is to use semantic versioning, where the version number consists of three parts: major, minor, and patch. For example, a development build might be named myapp-1.0.0-dev-123, where 123 is a build number, a commit hash, or a branch name (or a combination of all of them). A released build might be named myapp-1.0.0, and a staged build might be named myapp-1.0.0-staging.

This way it is easy to identify where an artifact comes from and what it is used for. If building for multiple platforms or configurations, the naming scheme can be extended to include the platform and configuration, such as myapp-1.0.0-linux-x86_64, myapp-1.0.0-windows-x86_64, etc.

For a start, a simple storage solution for these artifacts might be enough, but as the project grows, it is worth considering a dedicated artifact repository and a package manager to handle the artifacts. If you have the versioning and naming scheme covered, you are already a huge step further towards efficient artifact-based CI, and further improvements can usually be made incrementally.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Further improvements

Moving from a build-everything-all-the-time approach to an artifact-based CI approach can be a bit of a paradigm shift, but it can lead to significant improvements in build times and reliability. Once the CI pipeline is set up to handle artifacts, the door is open to further optimizations, such as caching dependencies, parallelizing builds, and more. Depending on the team’s setup, it can even be beneficial to make the artifacts available to developers locally, so they can test their changes against the same artifacts that will be used in production.

As always, the key is to start somewhere and iterate and improve over time. Shifting to artifact-based CI is a great step towards more efficient and reliable CI/CD pipelines, and it can help teams to focus on delivering value rather than dealing with long build times and waiting for feedback.

Let’s fix that.

1. Vendoring everything instead of using find_package

One of the worst time-wasters: bundling dependencies directly into your project (aka “vendoring”), either as source code or worse, as precompiled binaries.

It bloats your repo, makes updates a hassle, and creates fragile builds. Modern CMake projects should use find_package to locate and integrate dependencies cleanly. Many libraries now provide CMake config files out of the box. If they don’t, writing your own Find<Package>.cmake is a good step toward future-proofing.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Even better, tools like Conan or vcpkg can take care of dependency management for you. Stop hardcoding paths - let CMake and your package manager handle it.

2. Not using CMakePresets

CMake Presets are a game-changer - use them! Since their introduction, they’ve made managing multiple build configurations so much easier. Instead of fiddling with command-line flags or custom scripts, you can define builds cleanly in CMakePresets.json.

CMake presets are one of the most impactful features introduced in CMake since the introduction of targets.

Presets reduce errors, improve consistency across teams, and help CI pipelines stay clean. By providing a standardized way to configure good build configurations for development, testing, and release builds, CMakePresets can significantly reduce the entry barrier for new contributors and help ensure that everyone is using the same build settings. Always include and maintain presets for your project, and encourage contributors to use them or create their own CMakeUserPresets.json files for their specific needs.

3. Still using global variables instead of targets

Modern CMake is all about targets - and has been for over a decade. Yet many projects still rely on global variables and configuration, making things messy and error-prone.

Targets make everything modular, scoped, and easier to maintain. Use commands like target_link_libraries, target_include_directories, and target_compile_options to apply settings only where needed.

And don’t just slap PUBLIC everywhere. Use PRIVATE by default, and only expose what’s absolutely necessary. You’ll reduce build times and avoid unnecessary rebuilds.

And if you still need global project settings, use options() with good descriptions and defaults, but keep them minimal. The goal is to avoid global state as much as possible.

4. Writing non-portable build instructions

It’s easy to hack together something that works on your machine and CMake’s possibility to call external commands and scripts is very powerful. But that doesn’t mean it’s portable. Using platform-specific paths, scripts, or flags ties your build to a single setup - and breaks everything elsewhere.

Stick to CMake’s built-in features:

• Stick global compiler flags into CMake presets.
• Use target_compile_options, target_compile_definitions, etc., instead of global flags.
• Use cmake -P and cmake -E for scripting, not Bash or PowerShell.
• Use CMake’s built-in function to find things like libraries and include directories - No hardcoded paths in CMakeLists.txt!
• Set up toolchain files for cross-compilation.

Portability isn’t just for open-source projects. It’ll save you headaches even in single-platform codebases - especially when onboarding new devs or upgrading tools.

5. Half-baked library setup

Ever try using a library only to discover it’s missing version info, has broken exports, or forces a pile of transitive includes on you?

That’s what happens when the setup of the library target is incomplete or sloppy.

Don’t just make everything PUBLIC. Be precise with PRIVATE, PUBLIC, and INTERFACE. Set symbol visibility correctly, especially for shared libs. And always include version info (VERSION, SOVERSION) for libraries - it helps tools, consumers, and future-you.

For a solid setup, here’s a line-by-line guide.

6. No install instructions

You might think install rules don’t matter if you’re not publishing a library. But even in internal projects, a clean install() setup can make packaging, CI pipelines, or dev workflows way smoother.

It’s not hard: a few lines in your CMakeLists.txt go a long way. Think of it as a foundation for future packaging, or just making builds easier to deploy locally.

7. Neglecting maintenance of your CMakeLists.txt

CMake scripts aren’t “write-once” files. They need love too.

Old, bloated, or inconsistent CMakeLists.txt files are painful to work with and a nightmare for newcomers. Regularly refactor and clean up your build scripts - just like you do with code.

Update to modern CMake practices. Drop old workarounds. Add comments. Keep dependencies clean. You’ll thank yourself later.

How to fix it

If your team is grumbling about CMake, chances are you’re stuck in one - or several - of these anti-patterns. But you don’t have to fix everything overnight.

Which one hurts the most depends on the project and the team, but in general, the anti-patterns often go hand in hand. Modernize your CMake scripts and tackle one thing at the time and each step brings back a little bit of joy to the build process.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Even if you can’t make things perfect, starting with small steps can lead quickly to improvements. Just applying PRIVATE and PUBLIC correctly can go a long way, replacing hardcoded paths with find_package and using CMake’s built-in functions to find libraries and include directories can make the build process more robust and portable. And usually with every step you take, you will find that the build process becomes easier to understand and maintain, leading to a more enjoyable development experience for everyone involved.

What drives the cost of change in software?

When talking about the cost of software products, companies often talk about total cost of ownership (TCO), which in simple terms means, “How much does it cost to keep the software running and up to date?” While the cost of operating software can be a significant cost factor contributing to the TCO, often the far larger cost factor is the amount of time software developers spend changing the code, improving or adding features, and fixing defects.

Changing and adapting software can be the major cost driver in software development, even before operational costs.

When maintaining software over time, there is always a base layer of effort needed just to keep the software running and up to date. These are things like changes in underlying hardware, operating systems, evolution of libraries, etc. This is what I refer to as the fixed cost of change. This kind of housekeeping is an important investment in keeping the cost of change low, but it is often hard to predict when this needs to be done. However, usually the far larger share of the cost is the controllable cost of change. This is driven by the number of feature requests, defects reported, volatility of the user base, and evolving user needs. These changes can often be planned based on observing the running system and feedback received. Thus, a development team can usually exert some control over these factors through deciding when, how, or if to implement a change, or by accepting certain limitations in functionality or stability instead of fixing a defect.

Looking at the typical software development process, any change made on an existing codebase usually involves the following steps:

1. Identifying the requirements of the change - Slicing requirements into small, incremental changes helps to decide whether the change is worth implementing or not.
2. Looking for the parts of the code that need to be changed - A modularized, consistently designed codebase makes finding the parts that need to be changed easier and reduces the impact of the change.
3. Changing the code and implementing the change - Good software craft and clean implementation reduce testing effort.
4. Testing and validating the changes - Good test coverage reduces the risk of regression.
5. Deploying the changes to the production environment - Frequent and easy deployment enables fast feedback and new requirements.

Each of these steps can be a source of cost, and each of them can be optimized to reduce the cost of change, which usually impacts the next step again. Generally, the faster a change can be implemented, tested, and deployed, the faster the loop can be closed and the faster value is generated. This includes also the waiting time between the steps. Nothing costs more than a finished change that is lying around for weeks or months before it is deployed and validated.

Controlling the fixed cost of change

One of the most frequent shortcomings regarding high cost of change that I see is not investing in the fixed cost of change frequently. Keeping your software reasonably up to date and doing so in frequent, but small increments is one of the major investments in keeping the cost of change small. A frequent mistake of software teams is not keeping the software stack current and up to date, especially once the speed of feature development slows down and once the software is considered mature. The effect is then that even if a small change is needed, the cost of change is high because the first thing a developer has to do is update the software stack to a version that is still supported and maintained. While this is not only a time-consuming factor, it is often a high risk to introduce hidden regression bugs into the software, leading to a much higher validation and testing cost than what would be needed just to test the new feature.

Keeping your software stack up to date is not glamorous work, but it is one of the most important investments in keeping the cost of change low.

When it comes to the controllable cost of change, there are a number of strategies that can be applied to reduce the cost of change in your code. Of the steps identifying the requirements, looking for the parts of the code that need to be changed, changing the code, testing the changes, validating the changes, and deploying the changes, all have potential for optimization. Here are some strategies that can be applied to reduce the cost of change in your code:

Identify requirements - Say “No” to a change

Let’s start at the very beginning with identifying requirements. The biggest cost saver here is saying “No” to a change. One of the twelve principles of the agile manifesto is “Simplicity–the art of maximizing the amount of work not done–is essential.” The equation is simple: the fewer features and code there are in a product, the less there is to maintain. To do so, a clear discussion about the value of the change is necessary. Ask “Do we (or one of our stakeholders) benefit if we do this?” If the answer is not a clear “yes,” then don’t do it. To help with this decision, rigorous slicing of stories into small, incrementally applicable changes is a must. This is done to reduce development effort while maximizing the value of the change. A frequent mistake here is that teams at this point are not talking about the value, but about the predicted cost only. A typical pattern I observe is that developers are asked to estimate the change, and if they give a low enough number, the change is applied.

A common mistake when deciding whether to fix a defect or add a feature is to talk about cost only instead of value first.

Another frequent omission at that stage is to not talk about what to remove from the product and talk about deprecation. If a product is running over decades, it is natural that some features at one point will become obsolete or too cumbersome to use and maintain. Having a clear strategy on how to remove or replace features from a product is as important as adding new ones. That usually involves a clear communication strategy to the users and stakeholders, a clear way to mark features as being phased out, and a clear plan on how to remove the feature from the codebase. Handling deprecation well can save a lot of cost in the long run. Removing bad or dead code reduces the noise developers have to dig through when looking for the parts of the code that need to be changed.

Finding the parts of the code that need to be changed

Once the decision is made to implement a change, the next step is to find the parts of the code that need to be changed. This is where good software design and architecture come into play. The more modularized and decoupled your code is, the easier it is to find the parts of the code that need to be changed. There is a high chance that after a few years of development, the people maintaining the code are no longer the same people who wrote the code. And even if they are, there is a high chance that a lot of detail just got forgotten. If the code is well structured and modularized, it is much easier for new developers to find their way around the codebase and to understand what needs to be changed. Build your software around a consistent architecture and design patterns, so it is easier to find the parts of the code that need to be changed. This can start at the smallest level, for instance, on how you use functions. Are you using non-const output parameters in functions, or are you strictly using structs as return values? Do you use exceptions or error codes to signal errors? You may like or not like some of these patterns, whatever you choose, be consistent.

Consistency in software design and architecture is more important than the actual choice of the pattern, to make locating the parts of the code that need to be changed easier.

Consistency extends all the way up to the architecture patterns like MVC, MVVM, Clean Architecture, or Hexagonal Architecture and whatever else is out there. Again, consistency is more important than the actual choice of the pattern, although choosing the completely wrong architecture can be a major cost driver. Whatever architecture you pick, favor strong modularization, low coupling, and high internal cohesion for your modules to reduce the cost of change.

Stick to your chosen design within a module, but decouple modules as much as possible. If your code is tightly coupled, you will have to change many other parts of the code when you change one part. This is a major cost driver in software development. The more code changed, the more likely you are to introduce bugs and the higher the testing effort will be. This, of course, has a backward effect on the decision of whether to implement the changed requirement or not. If you are sure that the change will be localized and not affect other parts of the code, it’s way easier to say “yes” to a change. Once you found the code, the fun part starts - changing the code and implementing the change.

Implementing the change

When implementing a change, having invested in good software practices upfront pays off a lot. But even if that was not done in the past, then start applying good practices with the change on hand. It might as well be that when the original code was written, there were no proper unit tests around, TDD was not practiced, and no code reviews were done. So what? Start doing that now! One of the most important things you can do at that stage is to further reduce the cost of change. I tend to say that every implementation should start with a refactoring first. Maybe the existing code is not following the current coding standards, has the wrong level of abstraction, or is not using any of the patterns you have chosen for that particular module. Don’t build on broken stilts; refactor the code first, so it is easier to understand and change.

Every implementation should strive to further reduce the cost of change (or at least not increase it significantly).

There is, of course, a trade-off on how much can be done. At one point, we might have to accept that some parts of the code are just too costly to repair and that we hit a flat spot on how much we can reduce the cost of change. The trade-off here is usually whether to sacrifice internal coherence of a module but improve decoupling and isolation of “bad code” more. On a whim, I usually try to isolate first rather than to keep coherence, but that is a personal preference and depends on a lot of factors.

A very good practice to make sure the cost of change stays manageable when implementing new features is to use a TDD approach and relentlessly apply the full cycle, which includes refactoring of the original code. To skip the last step in the TDD cycle is a direct invitation to increase the cost of change. The other benefit of a TDD approach is that test coverage of new code stays high, which helps with verifying that the change is correct and that no regression bugs are introduced.

Testing and validating the changes

When thinking about the cost of a new feature, the testing and validation cost is often forgotten or underestimated. This often goes back to the segmentation of the system and how localized the change is. The splash radius of a change is often a very good indicator of how high the testing and validation cost will be. If the change is localized, the testing effort is usually low; the wider the effects, the more expensive testing goes. For instance, optimizing the performance of a single algorithm is usually a very localized change, and testing through a benchmark of before/after can be sufficient. Optimizing or changing a full workflow in the business logic could have a much wider effect and require a lot more testing, often involving manual testing - which is expensive.

While the verification of a feature - aka it works as defined - can often be automated, the validation - aka it works as expected - is often a manual process. The more manual the validation process is, usually the higher the contribution to the cost of change. It often pays off to think of what kind of validation is needed for a feature before actually implementing it. Sometimes validation of a change can only be done by the end user, so an easy way for (selectively) deploying changes to a subset of users can be a good strategy to reduce the cost of change.

Deploying the changes

So the change is now implemented and tested! Very good, now let’s ask the developers to roll it out to the production environment. This is where the cost of change can skyrocket. If the deployment process is manual, error-prone, and time-consuming, the cost of change is high even for the tiniest change. This can go up to the point where developers avoid even the tiniest fixes for fear of the cost of deployment.

Invest in an easy-to-operate and automated deployment process to reduce the cost of change and deploy changes one by one instead of bundled together into large releases.

The deployment process is often a neglected part of the software development process, but it is a very important part of the cost of change. If deployment to production is hard and involves jumping through seven hoops to get it done, a common pattern is that multiple changes are bundled together to reduce the cost of deployment. The problem here arises that at one point the feedback loop gets a lot longer and that big releases have a much higher chance to introduce hidden regression. Another pattern frequently observed is that if deployment is expensive, only big and heavy changes are even passing the decision threshold at the beginning of the cost-of-change cycle. Because, why spend days deploying a small change that might not even be needed? The result is software that might just do what it is supposed to do, but the many small kinks and annoyances that are never fixed make it a pain to use.

As long as a feature is not deployed, it only generates cost, but no value, so early deployment of any change will bring more value over time. This leads to one of the most destructive and behavioral patterns in software engineering. Only discussing software development from the cost perspective when evaluating whether to apply a change or not. While it might be true that software development is not cheap, holding back the work done and not letting it generate value is even more expensive. This closes the cycle of the cost of change because if features are never deployed, the cost of change for the next feature might be even higher.

Optimizing for reduced cost of change is a systemic problem

Being aware of the cycle that drives the cost of change is the first step to reduce it. The cost of change in software engineering is not just a technical problem, but a systemic problem that involves the whole team or even larger parts of an organization.

Optimizing the cycle can be a challenging task; however, if the cycle is observed, then one can usually spot the major bottlenecks quickly. Start where it hurts the most and where work piles up because it’s waiting for action. Whether it is improving the deployment infrastructure and processes, investing in test automation, improving the architecture or design of the software, or just training to say “No” to a change, there are many ways to reduce the cost of change in software.

CMake Dependency Providers in a nutshell

Dependency providers are a new feature introduced in CMake 3.24 that allows package managers to provide dependency information directly to CMake. This information includes the location of the libraries, their include directories, and other necessary information for building the project. When a dependency provider is registered in CMake each call to find_package or FetchContent_MakeAvailable will trigger the corresponding function in the dependency provider script.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Behind the curtains, a dependency provider is a CMake script that provides functions to locate dependencies through whatever means it can. In the case of Conan, this means calling calling conan install with the appropriate toolchain configuration and then extracting the necessary information from the files generated by Conan. To install a depenndency provider the script containing the provider definition is passed using the CMAKE_PROJECT_TOP_LEVEL_INCLUDES variable either from the command line or through a CMake preset The official CMake documentation provides an in-depth explanation of how dependency providers work and how to create your own.

Setting up Conan as a CMake Dependency Provider

To set up Conan as a dependency provider we need three things:

• CMake 3.24 or later
• Conan 2.0.2 or later
• The Conan CMake helper script to install it as a dependency provider

Installing CMake and Conan is straightforward and well-documented, so we won’t cover it here. The Conan CMake helper script is a CMake script that sets up Conan as a dependency provider in your project can be found in the cmake-conan repository. As of May 2024 the support is still experimental, but it should be stable enough for most use cases. To use the Conan CMake helper script, you can either download it manually or use it as a git submodule in your project. Since dependency providers have to be configured at the very beginning of the CMake configuration process, getting it over FetchContent or similar is not an option.

I usually end up with a directory structure like this:

├── cmake-conan # git submodule
│   ├── conan_provider.cmake
│   └── ...
├── CMakeLists.txt
├── conanfile.txt
└── src
    └── ...

The conan_provider.cmake script is the Conan CMake helper script that we will use to set up Conan as a dependency provider. The conanfile.txt file is the Conan configuration file that lists the dependencies of the project. The src directory contains the source code of the project and the CMakeLists.txt file contains the main CMake configuration.

Preparing the Conan and CMake configuration

Let’s assume we are building a simple project that depends on the fmt library for string formatting, which is available from the conan center repository. One of the nice things about dependency providers is that the CMakeLists.txt does not need any special configuration and can just use find_package and target_link_libraries as usual. The Conan CMake helper script will take care of the rest. The CMakeLists.txt file for the project looks like this:

cmake_minimum_required(VERSION 3.24)

project(
    hello_world
    LANGUAGES CXX
)

find_package(fmt 10.2.1 REQUIRED)
add_executable(hello src/main.cpp)
target_link_libraries(
    hello
    PRIVATE fmt::fmt
)

Next, The conanfile.txt file lists the dependencies of the project:

[requires]
fmt/10.2.1

[generators]
CMakeDeps

Using the CMakeDeps generator in the conanfile.txt file tells Conan to generate the necessary information for CMake’s find_package function. This is necessary for the Conan CMake helper script to work correctly. There is also the CMakeToolchain generator that generates a toolchain file for CMake, but this is not recommended when using Conan as a dependency provider.

With that in place, we can now set up Conan as a dependency provider in the CMakeLists.txt file:

Using Conan as a CMake Dependency Provider

To set up Conan as a dependency provider, we need to pass the conan_provider.cmake script to CMake using the CMAKE_PROJECT_TOP_LEVEL_INCLUDES variable. This can be done either from the command line or through a CMake preset. The conan_provider.cmake script is the Conan CMake helper script that sets up Conan as a dependency provider.

cmake -S . -B build -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES=./cmake-conan/conan_provider.cmake -DCMAKE_BUILD_TYPE=Debug

Note that Conan requires the CMAKE_BUILD_TYPE to be set in order to download or build the correct version of the dependencies. The conan_provider.cmake script will take care of setting up the necessary Conan profiles and installing the dependencies in the local cache. The binaries and include files for the dependencies will be placed in the local Conan cache, which is usually located in the user’s home directory. Only the configuration files generated by Conan will be placed in the build directory. By default the Conan helper script is configured to use the default Conan profile and it tries to build any missing dependencies from source. These settings can be changed by setting the CONAN_BUILD_PROFILE and CONAN_INSTALL_ARGS variables respectively.

After running CMake, the project can be built as usual:

cmake --build build

And with that the project the project is set up to use Conan as a dependency provider. I recommend to use CMake presets to make the configuration more reproducible and to avoid having to remember the command line arguments, especially when working with multiple configurations or platforms. For public projects I generally include the Conan helper script as a git submodule but depending on the project it might be easier to just download it manually.

Is it really that easy?

The introduction of CMakes dependency providers closes a long-existing gap in CMake’s dependency management and like CMake Presets I consider them a vast improvement regarding the tooling situation around C++ projects. So far I have only used Conan as a dependency provider but I am looking forward to seeing how other package managers will integrate with CMake. What I particularly like is that the CMake configuration can remain simple and platform agnostic while the complex dependency management is handled by Conan. This makes it easier to switch between different build systems or to integrate the project into a larger build system and in my opinion is a great incentive to use Conan for C++ projects.