This article is based on the QRLite-Project. QRLite is a small, local-only and ad-free app for reading QR-Codes that runs on Desktop and Android. It is a C++/Qt6 Android app, using CMake presets, Conan 2 as a CMake dependency provider, and Qt6’s own qt-cmake tooling to go from source to a single installable .aab (Android App Bundle) covering arm and x86 architectures.¹

Let’s see how it all comes together.

The high level build process for multi-ABI Android App Bundles

The basic setup for a Qt/CMake Android build is covered in a previous post. That post covers NDK/SDK installation, environment variables, basic CMakePresets.json anatomy, and adb install. All are still valid, and we won’t repeat it here. However this time, we have some dependencies to manage, and this post picks up from there and adds Conan 2 + Qt6, then we go all the way to creating a multi-ABI bundle.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Before we dive into the details of Conan and Qt integration, let’s outline the high-level steps involved in building a multi-ABI Android App Bundle with CMake. At the baseline, an aab file is essentially a collection of packages for different plattforms, so the process of building one with C++ involves:

• Configuring CMake for each target ABI.
• Fetching and resolving dependencies for each ABI.
• Building the project for each ABI.
• Collecting the resulting binaries and resources.
• Packaging them into a single .aab file.

While there exist the qt-cmake wrapper to simplify Qt-specific configuration for Android, the underlying challenge of building for multiple ABIs remains. Each ABI still requires a separate CMake configuration and build step. Also qt-cmake has a few quirks and additional steps compared to plain CMake and if your stack is not Qt-centric, plain CMake is often easier to work with. So the approach here is to use plain CMake for building and testing and only use qt-cmake when necessary for packaging the final .aab. Let’s walk through each of these steps in more detail.

The dependency problem: Conan as a CMake dependency provider

QRLite depends on zxing-cpp for QR decoding and Catch2 for its test suite, which will not be bundled into the final Android App Bundle.

To manage these dependencies across multiple ABIs, we need a tool that understands both CMake and cross-compilation. Conan solves the “build the right binary for the right ABI” half of that problem. CMake’s dependency provider mechanism is what lets Conan do it transparently, from inside find_package(), instead of requiring every consumer of the project to run a separate conan install step by hand and remember to keep it in sync with the CMake configure. Since CMake 3.24, setting

set(CMAKE_PROJECT_TOP_LEVEL_INCLUDES "path/to/conan_provider.cmake")

before the first project() call registers a script that CMake consults whenever find_package() can’t resolve a package the normal way. The script is cmake-conan, which QRLite vendors as a git submodule at cmake-conan/. In CMakePresets.json this becomes a small hidden preset that every other configuration inherits from:

{
    "name": "conan-dependency-provider",
    "hidden": true,
    "cacheVariables": {
        "CMAKE_PROJECT_TOP_LEVEL_INCLUDES": "${sourceDir}/cmake-conan/conan_provider.cmake"
    }
}

With that in place, the provider intercepts find_package(ZXing REQUIRED) and find_package(Catch2) and Conan runs automatically: it reads conanfile.txt, resolves and (if needed) builds the dependency graph for the active profile, and generates the CMakeDeps config files that find_package() then picks up as if they’d always been there. QRLite’s conanfile.txt is intentionally minimal:

[requires]
catch2/3.10.0
zxing-cpp/2.3.0

[generators]
CMakeDeps

CMakeDeps is the generator that matters here — it’s what produces the Find<Package>.cmake/<package>-config.cmake files that make Conan-provided dependencies look like any other CMake package to find_package(). For a more in-depth explanation, see the Article @Conan as CMake Dependency Provider”.

Host and build profiles

Once you’re cross-compiling, “what compiler and settings should this dependency be built with” stops having one answer, Conan needs to know about two toolchains at once:

• the build profile: the machine actually running the compiler (your desktop/CI container, x86_64 Linux, system GCC/Clang)
• the host profile: the machine the binary will run on (an Android device — a given ABI, the NDK’s Clang, libc++, a minimum API level)

For a desktop build these happen to be the same, so Conan defaults quietly work. For Android they diverge on almost every axis: arch, os, compiler, compiler.libcxx, even compiler.version (the NDK ships its own Clang, not whatever clang). QRLite’s devcontainer and CI image (bernedom/qtandroidbuilder) ship pre-built Conan profiles for exactly this split, one host profile per Android ABI/build type (android-debug, android-release) and a build profile for the container itself (android). A host profile looks roughly like:

[settings]
arch=armv8
os=Android
os.api_level=31
compiler=clang
compiler.version=18
compiler.libcxx=c++_shared
compiler.cppstd=gnu17
build_type=Release

[conf]
tools.android:ndk_path=$ANDROID_NDK_HOME

with a separate one per ABI (armv7, armv8, x86_64 map to armeabi-v7a/arm64-v8a/x86_64). The desktop build profile is the “normal” one, plain os=Linux, system C++-Compiler. CMakePresets.json wires the pair in as cache variables that cmake-conan reads directly:

{
    "name": "conan-android-debug",
    "hidden": true,
    "cacheVariables": {
        "CONAN_HOST_PROFILE": "android-debug",
        "CONAN_BUILD_PROFILE": "android"
    }
}

Making find_package work for Conan + Qt6 while cross-compiling

Getting Conan to produce the right binaries is half the battle; getting CMake’s find_package() to actually find them while an Android toolchain file is active is the other half. The NDK’s CMake toolchain file is deliberately strict about this — by default it sets CMAKE_FIND_ROOT_PATH_MODE_PACKAGE/LIBRARY/INCLUDE to ONLY, meaning find_package() will only look inside CMAKE_FIND_ROOT_PATH (the sysroot). That’s correct behaviour for system libraries, but it also means CMake will silently refuse to see Conan’s host-profile packages, which live outside the NDK sysroot entirely. QRLite’s android-ndk hidden preset relaxes exactly this:

"cacheVariables": {
    "ANDROID_PLATFORM": "31",
    "ANDROID_SDK_ROOT": "/opt/android",
    "CMAKE_FIND_ROOT_PATH_MODE_PACKAGE": "BOTH",
    "CMAKE_FIND_ROOT_PATH_MODE_LIBRARY": "BOTH",
    "CMAKE_FIND_ROOT_PATH_MODE_INCLUDE": "BOTH"
}

BOTH tells CMake to search the sysroot and the ordinary CMAKE_PREFIX_PATH/CMAKE_MODULE_PATH, which is where both Conan’s generated package files and Qt6’s Android package live.

Qt6 itself adds a second wrinkle: unlike Qt5’s single CMAKE_PREFIX_PATH from the prior article, Qt6 for Android ships one full install per ABI (android_armv7/, android_arm64_v8a/, android_x86_64/), each exposed through a QT_PATH_ANDROID_ABI_<abi> variable when you’re building for multiple ABIs at once. QRLite’s top-level CMakeLists.txt combines Conan’s per-ABI output directory and Qt6’s per-ABI install into one manual prefix-path prepend, keyed on CMAKE_ANDROID_ARCH_ABI:

if(ANDROID)
  set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE BOTH)
  set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY BOTH)

  if(CMAKE_ANDROID_ARCH_ABI STREQUAL "armeabi-v7a")
    list(PREPEND CMAKE_PREFIX_PATH "${QT_PATH_ANDROID_ABI_armeabi-v7a}")
    list(PREPEND CMAKE_PREFIX_PATH "${CONAN_ARMEABI_V7A_DIR}")
    list(PREPEND CMAKE_MODULE_PATH "${CONAN_ARMEABI_V7A_DIR}/cmake")
  elseif(CMAKE_ANDROID_ARCH_ABI STREQUAL "arm64-v8a")
    list(PREPEND CMAKE_PREFIX_PATH "${QT_PATH_ANDROID_ABI_arm64-v8a}")
    list(PREPEND CMAKE_PREFIX_PATH "${CONAN_ARM64_DIR}")
    list(PREPEND CMAKE_MODULE_PATH "${CONAN_ARM64_DIR}/cmake")
  elseif(CMAKE_ANDROID_ARCH_ABI STREQUAL "x86_64")
    list(PREPEND CMAKE_PREFIX_PATH "${QT_PATH_ANDROID_ABI_x86_64}")
    list(PREPEND CMAKE_PREFIX_PATH "${CONAN_X86_64_DIR}")
    list(PREPEND CMAKE_MODULE_PATH "${CONAN_X86_64_DIR}/cmake")
  endif()
endif()

find_package(Qt6 COMPONENTS Core Gui Multimedia Qml Quick REQUIRED)
find_package(ZXing REQUIRED)

This is a bit of a workaround to make multi-ABI builds work with Conan and Qt and it would be nicer, if this would be covered by one of the presets, but so far this did not work out as we see later. This hack exists because a single CMake configure only ever targets one ABI, but when creating the bundle, qt-cmake re-invokes CMake three times against the same source tree from a fourth, top-level configure, and that fourth configure needs to know where each of the three per-ABI Conan installs ended up. CONAN_ARM64_DIR and friends are just plain cache that are passed to the dependency provider and ultimately used by the bundle preset to locate each ABI’s Conan install.

From one APK to one bundle: why three ABIs means three builds

If you’re building an .apk you adb install yourself can happily target one ABI. What the Play Store wants is different: a single .aab that contains the app’s shared libraries for every ABI you support, with Google Play generating per-device APKs from it at install time. While CMake is great for cross-compiling it cannot build for more than one ABI at a time. A shortcoming that regularly sparks dicussions in the community.

And unfortunately, the android configuration such as ANDROID_ABI are configure-time settings. So the only way to get three ABIs’ worth of native libraries is to run three independent configure+build cycles — which is exactly what QRLite’s per-ABI presets (android-armeabi-v7a, android-arm64-v8a, android-x86_64) are for, each with its own binaryDir:

cmake --preset ci-ninja-android-armeabi-v7a-debug
cmake --preset ci-ninja-android-arm64-v8a-debug
cmake --preset ci-ninja-android-x86_64-debug

cmake --build build_android_armeabi-v7a
cmake --build build_android_arm64-v8a
cmake --build build_android_x86_64

At this point you have three separate build_android_<abi>/ trees, each with its own .so, and each with its own Conan install directory (build_android_<abi>/conan/) built against that ABI’s host profile. Three APKs’ worth of native code, but Play Store bundles aren’t “three APKs concatenated,” they’re one Gradle-shaped Android project with all three .so sets dropped into the right per-ABI jniLibs folders. Producing that is a fourth, different kind of build.

Tying it together: the bundle build

Qt6 has first-class support for aab of build, but it’s driven through qt-cmake rather than plain cmake, and it needs to know where all three of the per-ABI pieces (Qt installs and Conan installs) actually live, since this configure isn’t cross-compiling anything itself; it’s orchestrating the three that already ran. That’s the android-all-archs preset:

{
    "name": "android-all-archs",
    "hidden": true,
    "inherits": ["android-ndk"],
    "cacheVariables": {
        "QT_ANDROID_ABIS": "x86_64;arm64-v8a;armeabi-v7a",
        "QT_ANDROID_BUILD_ALL_ABIS": "OFF"
    },
    "binaryDir": "build_android_bundle"
},
{
    "name": "Qt-android-all-archs",
    "hidden": true,
    "cacheVariables": {
        "QT_CHAINLOAD_TOOLCHAIN_FILE": "$env{ANDROID_NDK_HOME}/build/cmake/android.toolchain.cmake",
        "CONAN_ARM64_DIR": "${sourceDir}/build_android_arm64-v8a/conan",
        "CONAN_ARMEABI_V7A_DIR": "${sourceDir}/build_android_armeabi-v7a/conan",
        "CONAN_X86_64_DIR": "${sourceDir}/build_android_x86_64/conan",
        "QT_PATH_ANDROID_ABI_armeabi-v7a": "/opt/Qt/6.11.1/android_armv7/",
        "QT_PATH_ANDROID_ABI_arm64-v8a": "/opt/Qt/6.11.1/android_arm64_v8a/",
        "QT_PATH_ANDROID_ABI_x86_64": "/opt/Qt/6.11.1/android_x86_64/",
        "QT_HOST_PATH": "/opt/Qt/6.11.1/gcc_64/"
    },
    "environment": {
        "QT_HOST_PATH": "/opt/Qt/6.11.1/gcc_64/"
    }
}

This is the missing piece from the “hack” in the top-level CMakeLists.txt earlier: CONAN_ARM64_DIR, CONAN_ARMEABI_V7A_DIR, and CONAN_X86_64_DIR are set here explicitly, pointing back at the three builds that already ran. QT_ANDROID_BUILD_ALL_ABIS=OFF combined with an explicit QT_ANDROID_ABIS list tells Qt6 “don’t try to build these yourself, I’ve already built them, just assemble the bundle from what’s on disk.” Two details matter for actually running this configure step:

• It has to be invoked with qt-cmake from one of the Android Qt installs, not plain cmake. qt-cmake is what wires in Qt6’s Android packaging machinery (the aab target itself doesn’t exist otherwise).
• QT_ANDROID_MULTI_ABI_FORWARD_VARS needs to include ANDROID_PLATFORM so the per-ABI Android platform level set in the android-ndk preset actually propagates into the bundle project.

Put together, the full sequence looks like this:

cmake --preset ci-ninja-android-armeabi-v7a-debug
cmake --preset ci-ninja-android-arm64-v8a-debug
cmake --preset ci-ninja-android-x86_64-debug

cmake --build ./build_android_armeabi-v7a
cmake --build ./build_android_arm64-v8a
cmake --build ./build_android_x86_64

/opt/Qt/6.11.1/android_x86_64/bin/qt-cmake \
  --preset ci-ninja-android-all-archs-debug \
  -DQT_ANDROID_MULTI_ABI_FORWARD_VARS="ANDROID_PLATFORM"

cmake --build ./build_android_bundle/ --target aab

That last build produces build_android_bundle/QRLiteApp/android-build/build/outputs/bundle/debug/android-build-debug.aab, a single file containing native code for all three ABIs, ready for signing.

One more thing worth flagging if you’re following along with your own project: QRLiteApp’s CMakeLists.txt sets QT_ANDROID_TARGET_SDK_VERSION 36. Most Qt/Android tutorials and Stack Overflow answers you’ll find still assume an SDK version several years old, but Google’s Play Store minimum target API requirement moves roughly once a year, and Qt’s own defaults lag behind it. Worth checking you’re not building against a target SDK version Play Store will reject before you get to the upload step.

What we have, and what’s next

So at this point, we have Conan 2 resolving zxing-cpp and Catch2 per-ABI through a CMake dependency provider, find_package() correctly seeing both Conan and Qt6 across an Android cross-compile, and a qt-cmake-driven bundle step that turns three independent ABI builds into one .aab.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

At this point you can sign the .aab and upload it to the Google Play Store (or any derivative app store) for distribution. While creating an .aab is a step up in complexity compared to a single-ABI APK, it simplifies distribution and ensures that users get the best possible version of your app for their device, so it is generally worth the extra effort.

Good software design is a survival trait

Before the massive adoption of AI-assisted coding, producing running software was expensive and time-consuming, not just in money, but in focus, time and cognitive energy. This scarcity forced software engineers to think about the design of their software. A badly written system meant weeks of lost work and frustration. A wrongly chosen data structure could jeopardize the performance of an entire system and a rewrite of it could take months.

This is why practices such as domain driven design, test-driven development, and code reviews were invented and popularized. They were not academic exercises, but survival traits.

If implementing is expensive, you have to think before you implement.

An architect spending days thinking about how data is modeled doesn’t do this out of nostalgia. She does it because mistakes there could cost months of work later. This economic pressure is what formed modern software craft.

What AI-Agents really change

The adoption of AI-assisted coding relieves this economic pressure. An AI-Agent doesn’t iterate two or three times on a feature, it iterates twenty or thirty times, and can implement it in a few hours instead of days. This is technically impressive. But it is a brute-force approach to the problem. Instead of reducing the problem scope through careful design, the solution space is explored through sheer computational power.

The principle is not new. Brute-Force approaches are a legitimate strategy, as long as the problem space is small enough. But software design does not have a small - or even bounded - problem space. The combination of requirements, architectures, use cases and systematic constraints are exponential. And so is the needed power to cover this with a brute-force approach.

One could argue that reflection hasn’t disappeared, it has just moved into the prompt: writing a good request requires thinking too. But specifying a request is not the same act as designing a system.

The old reflection didn’t come from developers being virtuous, it came from hitting resistance while building.

You’d start modeling a piece of data, run into an edge case mid-implementation, and that friction would force a redesign. Agentic coding absorbs that friction on the agent’s side: the human never hits the wall that used to trigger the rethink, because the wall gets hit and silently worked around inside the AI’s iteration loop. And design thinking has always depended on stepping away from the keyboard: sleeping on a decision, sketching it out, arguing about it with a colleague. The faster and tighter the prompt-to-code loop gets, the less room there is for that incubation. Prompting is optimized for continuous momentum, which is structurally opposed to the pause that produces the best design decisions.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

Trading thinking time for production time

That lost friction doesn’t just remove a delay, it skips the specific questions it used to force: is this really needed? Is it the right approach, and is it the right time? Which edge-cases are relevant, which trade-offs are acceptable, and which requirements actually matter if they change? Agentic coding jumps straight over these questions. Humans front-feed requirements to the AI-Agent, and the AI-Agent produces code: the code is written and in production before the questions even get asked. And why not - if it proves to be a wrong decision, the AI-Agent can just iterate again and produce a new version.

The problem here is that the most expensive part of developing and running software are often not bugs, but incorrect requirements, dead code that needs maintenance and accidental complexity in products.

The cost of a wrong decision is not the cost of fixing a bug, but the cost of maintaining a wrong decision for years.

And this is where agentic coding fails: it does not help to make better decisions, it only helps to implement them faster. Implementing a feature, that nobody needs in lightning fast time is not efficiency it is fully automated waste of resources.

Other symptoms to watch out for

We all cheer the speed of agentic coding, but we should also watch out for the symptoms of a growing problem. There are several other symptoms that indicate that we are heading for a smelly place in software engineering.

Teams are losing collective understanding of the system they are trying to build and systems are losing their architectural cohesion. For a long time system understanding was gained by building the system itself. The act of implementing, testing and refactoring a system was the way to gain understanding of it. If implementation is treated as an afterthought, systems become black boxes within their own product. The team is degraded to archivists of code they no longer understand.

With this brute force approach where AI-Agents optimize for local goals, the overall system architecture is at risk and software becomes bloated. As a result operating costs will rise and the system will become more fragile. The cost of maintaining a system is not just the cost of fixing bugs, but also the cost of understanding it.

If the system becomes too complex to understand, it becomes too expensive to maintain.

Technical debts rise faster and so does the interest paid into it through refactoring. Even with assisted refactorings if they are large enough they carry risks in destabilizing a running system. Watching out for code churn and the growth of the code base is a good indicator for the health of a software project.

A common pitch from AI-tooling vendors is deceptively simple: just wrap the whole process into another AI-Agent that is responsible for the overall system architecture. But “just add another layer of abstraction” is a long known anti-pattern in software engineering. It is again a brute-force approach to a problem that is not bounded.

Conclusion

So what is the solution? Should we stop using AI-assisted coding? No, it is a productivity boost and it can help to implement features faster. But it does not eliminate the need for good design, reflection and above all it does not invalidate many of the practices around good software craft.

The answer is not more AI, but more discipline in software engineering.

The practices that were adopted because implementation was expensive are still valid, even if implementation becomes cheap. The need for good design, reflection and discipline in software engineering is not going away. It is just that the economic pressure that forced us to adopt these practices has shifted. If implementation becomes cheap and we don’t stop to think, we will end up in a world of bloated, fragile and unmaintainable software.

The term “software craft” has two dimensions: The craft of producing running software and the intellectual judgment on what to produce in the first place. The first dimension is what AI-assisted coding is good at, the second dimension is what it is bad at. The question whether a system is still suited to its purpose in a few years from now is not a question of implementation speed and will remain a question of human judgment in many cases. If you think agentic coding is a silver bullet for software engineering, you are mistaking speed for direction and you might be in for a rude awakening.

Big releases are a big risk

In many embedded software projects, releases are rare and stressful events. They often involve a lot of manual work, coordination between teams, and a high risk of things going wrong. The alternative is to make releases small, frequent and boring. Instead of bundling months of development into a single firmware version, deliver incremental improvements continuously. Each change should be small enough that its impact is well understood and reversible.

When releases are small and routine, the feedback loop with customers becomes much shorter and product development becomes more agile. While embedded software probably never will be able to ship as fast as pure software products, there are still many ways to speed up the delivery process and make it more reliable. However, moving from multiple months between releases to a few weeks or even days can have a huge impact on your product and your customers. It allows you to respond to customer feedback more quickly, fix bugs faster, and deliver new features sooner.

Making Releases Routine

To enable faster releases for embedded devices, you need to invest in a few key capabilities that reinforce each other. Some of these are technical foundations, others are organizational habits or cultural shifts, but most can be introduced incrementally without turning the whole company upside down. The goal is to make releasing feel like just another part of everyday development work rather than a nerve‑wracking special event.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

The first step is to treat releases as a routine, not a ceremony. If you ship only once or twice a year, every release feels exceptional, with custom scripts, ad‑hoc decisions and lots of manual coordination. This naturally increases risk, because nobody gets enough practice to truly master the process. By contrast, when releases happen regularly, the team gains muscle memory. You can get there by deliberately increasing the frequency of internal releases first: ship every sprint to a staging environment, then to a small set of internal devices, and only then to customers. Over time, the release pipeline becomes so familiar that pushing a new version out is as unremarkable as merging a pull request.

Shifting the attitude inside the team towards releases is one thing, but to go to full power you also need the technical capabilities to deliver the releases. Over‑The‑Air (OTA) updates are the technical backbone that makes this routine possible at scale. Without a solid OTA mechanism, delivering embedded software means physical access to devices, manual interventions, or complex field procedures. This makes releases cumbersome and risky. A robust OTA solution lets you update devices in the field reliably, with support for progressive rollouts, version tracking and rollbacks. There are many mature OTA-solutions out there, and most can be conveniently integrated with your build system and CI/CD pipeline. With OTA in place, you can decouple the release process from physical logistics and manual steps. The outcome is that “shipping” becomes an automated action triggered from your pipeline rather than a logistical project involving manual coordination.

The ability to update devices in the field is a big enabler, but to profit the most from it automating the delivery pipeline is a must. Automated testing via a powerful CI/CD pipeline is what turns frequent updates from reckless to safe. If every change can be built, tested and validated automatically, then the risk of each individual release drops dramatically. The testing possibility and complexity for embedded software is often huge, but even automating a subset of tests can have a big impact on confidence. Build up an efficient and automated testing pipeline that covers the critical paths of your software. Start by putting the entire build under CI so that every commit produces a reproducible artifact. Then layer on unit tests, static analysis, and hardware‑in‑the‑loop or end‑to‑end tests where feasible. Over time, your pipeline becomes a safety net: if something slips through, the pipeline catches it before customers ever see it. This confidence is what allows you to ship more often without losing sleep.

Finally, shipping faster requires shifting product management away from chasing big, “heroic” releases and towards delivering features continuously. Large, monolithic releases encourage long planning cycles, heavy upfront design and a tendency to cram in “just one more feature” which delays getting real feedback. Shift from delivering major versions that “make everying better” to delivering single features or small improvements that can be released independently. Product managers play a crucial role here by setting expectations with stakeholders about incremental releases, and measuring success by customer impact rather than marketing the next big release “that makes everything better”. You can support this shift by aligning roadmaps with short delivery cadences, encouraging experimentation, and using metrics such as lead time and value delivered instead of just counting features.

When you combine these four elements - releases as routine, OTA as plumbing, automated testing as your safety net and product management focused on incremental value - you create a delivery system that is both faster and more resilient. Releases stop being terrifying, once‑in‑a‑while events and become a steady, predictable flow of improvements. That, in turn, lets your teams spend less time orchestrating risky upgrades and more time building products your customers actually care about.

Encryption at rest in embedded Linux in a nutshell

On a high level, encryption at rest means that all data stored on the device is encrypted when the device is powered off. This ensures that if an attacker gains physical access to the device, they cannot read the data without the encryption key. In practice, this is typically achieved by encrypting the root filesystem and any other sensitive partitions using a strong encryption algorithm such as AES. When the device boots up, the encryption key is retrieved from a secure location (e.g. One-Time Programmable memory (OTP)) and used to unlock the encrypted partitions, allowing the operating system to access the data as needed. While this adds some complexity to the boot process and comes with a slight performance overhead for I/O operations, the security benefits far outweigh these drawbacks for most applications.

In practice for embedded Linux devices, encryption at rest can be implemented using LUKS (Linux Unified Key Setup), which is a widely used disk encryption specification for Linux. The tool to manage LUKS encrypted partitions is cryptsetup and is well supported in the Linux kernel through the dm-crypt kernel module.

Let’s look at how these steps can be implemented in detail with yocto.

Creating a LUKS-encrypted root filesystem with yocto

To implement encryption at rest using LUKS on embedded Linux devices based on the Raspberry Pi Compute Module 4 with the yocto Project, the following steps are necessary:

• Create an initramfs image that includes the cryptsetup tool and any necessary dependencies.
• Modify the bootloader configuration to load the initramfs image during the boot process.
• Create an init script within the initramfs that handles the unlocking of the LUKS-encrypted root filesystem using the encryption key stored in OTP memory.
• Implement a provisioning process during the first boot to create the LUKS-encrypted root filesystem and store the encryption key in OTP memory.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

Since the encryption of the partitions is device specific and needs to be done only once, a provisioning process is required during the first boot of the device. As a consequence, the device image created from yocto is unencrypted initially and will only be encrypted during the provisioning process. A convenient way to achieve this is to use an A/B partition scheme, where one partition is used for provisioning and the other for normal operation. During the first boot, the init script generates a unique encryption key for the device, creates the LUKS-encrypted root filesystem, copies the unencrypted root filesystem into the encrypted container, and stores the encryption key in OTP memory. On subsequent boots, the init script retrieves the encryption key from OTP memory and unlocks the LUKS-encrypted root filesystem for normal operation. So the init script needs to handle two scenarios: the first boot (provisioning) and subsequent boots (normal operation).

But first of all, the initramfs image needs to be created and the bootloader configuration modified.

Create initramfs with cryptsetup

Since the initramfs is just a minimal Linux image, we can create it using yocto by defining a custom image recipe. The image recipe should include the cryptsetup package and any other necessary dependencies. Here is an example of how to create an initramfs image with cryptsetup. Let’s call this recipe initramfs-cryptsetup.bb:

inherit core-image

PACKAGE_INSTALL = "provisioning-initramfs-init cryptsetup e2fsprogs rsync userland openssl rpi-eeprom"

IMAGE_FEATURES = ""

PACKAGE_EXCLUDE = "kernel-image-*"

IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"

INITRAMFS_MAXSIZE = "262144"

do_install_bootfiles() {
    
    install -m 644 ${IMGDEPLOYDIR}/${IMAGE_BASENAME}-${MACHINE}.cpio.gz ${DEPLOY_DIR_IMAGE}/bootfiles/initramfs.cpio.gz
    
}

addtask do_install_bootfiles after do_image_cpio before do_image_complete

If we look at this a bit closer, we can see that the image recipe inherits from core-image, which provides a basic image structure. The PACKAGE_INSTALL variable specifies the packages to be included in the initramfs image, including cryptsetup for LUKS support provisioning-initramfs-init, which is our custom package that contains the init script for handling the unlocking of the LUKS-encrypted root filesystem. Additionally, other necessary packages such as e2fsprogs, rsync, userland, openssl, and rpi-eeprom are included to support various functionalities required for the provisioning process, create the encryption keys and store them in the OTP memory of the Raspberry Pi CM4. Since the initramfs is intended to only run the provisioning and unlocking process, we exclude the kernel image packages by setting the PACKAGE_EXCLUDE variable to kernel-image-*. The IMAGE_FSTYPES variable is set to ${INITRAMFS_FSTYPES} to specify the desired initramfs file types (e.g., cpio.gz). The INITRAMFS_MAXSIZE variable is set to 262144 (256MB) to define the maximum size of the initramfs image. IMAGE_FEATURES is deliberately left empty to avoid including unnecessary features in the initramfs image. The do_install_bootfiles function installs the generated initramfs image to the appropriate location for the bootloader to access it during the boot process.

Next we need to make sure that this initramfs image is loaded by the bootloader during the boot process.

Modify bootloader configuration to load initramfs

To modify the bootloader configuration to load the initramfs image during the boot process, we need to update the bootloader settings to include the initramfs image. For Raspberry Pi devices, this is typically done by modifying the config.txt file located in the boot partition. In the case of the Raspberry Pi recipes this can be achieved by modifying the RPI_EXTRA_CONFIG variable in the machine configuration file (e.g., raspberrypi4.conf or raspberrypi-cm4.conf). Here is an example of how to modify the bootloader configuration to load the initramfs image:

RPI_EXTRA_CONFIG += "
...
initramfs initramfs.cpio.gz\n\
..."

This tells the bootloader to load the specified initramfs image (initramfs.cpio.gz) during the boot process, but it is only half the story. To make this accessible, we also need to make sure that the initramfs image is copied to the boot.img container during the image creation process. If your distribution is not yet using secure boot, I suggest to implement this first. Here is a step by step guide for secure boot using yocto.

In the recipe creating the boot.img we need to add the dependency to our initramfs image and copy it into the boot image. Here is an example of how to do this:

inherit deploy nopackages

DEPENDS += "rpi-bootfiles rpi-cmdline rpi-config linux-raspberrypi dosfstools-native mtools-native initramfs-cryptsetup"

# The raspberry pi bootloader files are located in the following directories
BOOTFILES_DIR = "${WORKDIR}/bootfiles" 
DTBO_DIR = "${DEPLOY_DIR_IMAGE}"

S = "${WORKDIR}/boot-img-container"

do_compile() {
    ...
   # Create boot.img file code, omitted for brevity here
   ...
}

do_compile[depends] += "initramfs-cryptsetup:do_install_bootfiles"

By adding the initramfs-cryptsetup dependency to the DEPENDS variable, we ensure that the initramfs image is built before the boot.img is created. The do_compile[depends] line ensures that the initramfs image is installed to the bootfiles directory before creating the boot.img. With these modifications and together with the modifications of the RPI_EXTRA_CONFIG variable, the initramfs will be included in the boot.img and loaded by the bootloader during the boot process. At this point we have everything in place to start looking at how the unlocking of the LUKS-encrypted root filesystem during boot and the provisioning process during the first boot can be implemented.

Init script for unlocking LUKS-encrypted root filesystem

Inside the initramfs there is an init script, which is a shell script that runs during the initramfs stage of the boot process. In our case, the init script is responsible for handling both the provisioning process during the first boot and the normal unlocking process during subsequent boots. First we create a new recipe for placing the init script inside the initramfs-image, let’s call it provisioning-initramfs-init.bb:

SRC_URI = "file://provisioning-initramfs-init"

# Installs shell and binaries required for the initscript
RDEPENDS:${PN} += "${VIRTUAL-RUNTIME_base-utils}"

do_install() {
    # Install the init script itself
    install -m 0755 ${WORKDIR}/provisioning-initramfs-init ${D}/init
    # Kernel will panic if /dev doesn't exist
    install -d ${D}/dev
    # Required for logging to work
    mknod -m 622 ${D}/dev/console c 5 1

    # Add the mount points
    install -d ${D}/mnt
}

FILES:${PN} = "/init /dev/console /proc /sys /mnt"

This recipe is included in the initramfs image by adding it to the PACKAGE_INSTALL variable in the initramfs-cryptsetup.bb recipe as shown earlier. The actual init script (provisioning-initramfs-init) contains the logic for both the provisioning and normal unlocking processes. Additionally, it creates the /dev directory and the /dev/console device to allow for logging over uart, but this is mainly for debug purpose.

So far we have set up the initramfs image with the necessary tools and created a recipe to include the init script. But this does not yet cover the actual logic for unlocking the LUKS-encrypted root filesystem and the provisioning process. Let’s look at how this can be implemented in the init script.

The init script

The init script is a shell script that runs during the initramfs stage of the boot process. In our case it it checks whether the device has been provisioned (i.e., whether the LUKS-encrypted root filesystem has been created and the encryption key stored in OTP memory). If not, it performs the provisioning process; otherwise, it retrieves the encryption key from OTP memory and unlocks the LUKS-encrypted root filesystem for normal operation.

The following flowchart illustrates the provisioning and boot process:

flowchart TD
    
    %% Boot chain
    INITRAM[Load initramfs from boot.img] --> KEYCHECK{"does a private key in the OTP exist?"}
    KEYCHECK -- No (do provisioning) --> GENKEY[Generate key and store in OTP]
    GENKEY --> ENCRYPTROOTFSB[create a LUKS container for <span style='color:orange'>rootfsB</span> using the key from OTP]
    ENCRYPTROOTFSB --> POPULATEB[Populate <span style='color:orange'>rootfsB</span> with a copy of <span style='color:green'>rootfsA</span>]
    POPULATEB --> ENCRYPTROOTFSA[create a LUKS container for <span style='color:green'>rootfsA</span> using key from OTP]
    ENCRYPTROOTFSA --> POPULATEA[Populate <span style='color:green'>rootfsA</span> with a copy from <span style='color:orange'>rootfsB</span>]
    POPULATEA --> MAP[Map all LUKS containers using key from OTP]
    MAP --> MOUNT[Mount all LUKS containers]
    MOUNT --> BOOT[Boot into selected rootfs]
    KEYCHECK -- Yes --> MAP
    

The path for already provisioned devices is quite straight forward: Retrieve the encryption key from OTP memory, map the LUKS containers using cryptsetup, mount the root filesystem, and call switch_root to boot into the LUKS-encrypted root filesystem.

The provisioning process is more complex and requires several steps to create the LUKS-encrypted root filesystem and store the encryption key in OTP memory. Our script will perform the following steps during the provisioning process:

1. Detect if the device is already provisioned by checking if the encryption key is present in OTP memory and if the root filesystem partitions are already LUKS encrypted.
2. Generate a unique encryption key for the device using a secure random number generator.
3. Store the generated encryption key in the One-Time Programmable (OTP) memory of the Raspberry Pi CM4 using the appropriate tools (e.g., rpi-eeprom).
4. Create a LUKS-encrypted container for the root filesystem using the generated encryption key.
5. Copy the unencrypted root filesystem into the LUKS-encrypted container.
6. Boot into the newly created LUKS-encrypted root filesystem.

One question to clarify is where to store the unencrypted root filesystem before the provisioning process. Since LUKS does not support in-place encryption of existing filesystems, we need to have the unencrypted root filesystem available somewhere during the first boot. If you are using an A/B partition scheme, you can store the unencrypted root filesystem in one of the partitions (e.g., partition A) and use the other partition (e.g., partition B) for the LUKS-encrypted root filesystem. During the provisioning process, the init script can copy the unencrypted root filesystem from partition A into the LUKS-encrypted container on partition B and then do the same vice versa from the now encrypted B partition to A. While this complicates the init script somewhat, it streamlines the provisioning process significantly, as no external media is required to provide the unencrypted root filesystem during the first boot and the entire image can be flashed at once using .wic images or similar.

So let’s look at the key commands that need to be executed in the init script for both the provisioning and normal unlocking processes.

Provisioning and boot process

First we need to check if the device is already provisioned. The simplest way to do this is to check if all partitions are already LUKS encrypted and if there is a non-zero key stored in OTP memory. If the OTP key is not set, we need to generate a new encryption key and store it in OTP memory. The RPI eeprom tools provide a convenient way to do this:

rpi-otp-private-key -w $(openssl rand -hex 32) -y

If the key is either generated or already present in OTP memory, we read it out and store it in a temporary key file for use with cryptsetup:

rpi-otp-private-key -r > /keyfile.bin
# in case the private key is only zeroes, the file is not created
if [ ! -f /keyfile.bin ] || [ ! -s /keyfile.bin ]; then
    echo "[initramfs] FATAL ERROR: OTP private key file /keyfile.bin missing or empty. This is fatal!"
    exec /bin/sh
fi

Note, that on error we drop into a shell for debugging purposes, but in a production system you might want to reboot -f or take halt the system entirely to avoid leaving the device in an inconsistent or vulnerable state.

⚠️ Dropping into a shell on error is dangerous for production systems. Use with caution and only for debugging purposes. ⚠️

Next we check if the root filesystem partitions are already LUKS encrypted or need to be provisioned. Here is an example of how to do this:

ROOTFS_A="/dev/mmcblk0p5"   # rootfs_A
ROOTFS_B="/dev/mmcblk0p6"   # rootfs_B
OTP_KEY_SET=false
rpi-otp-private-key -c && OTP_KEY_SET=true
cryptsetup isLuks "${ROOTFS_B}" || { [ "${OTP_KEY_SET}" = false ] && ROOTFS_B_NEEDS_SETUP=true; }
cryptsetup isLuks "${ROOTFS_A}" || { [ "${OTP_KEY_SET}" = false ] && ROOTFS_A_NEEDS_SETUP=true; }

The rpi-otp-private-key -c command stems from the rpi-eeprom yocto recipe and checks if a private key is stored in OTP memory. If the OTP is non-zero, it returns 0 else 1. We use this to set the OTP_KEY_SET variable accordingly. Next we check if the rootfs partitions are LUKS containers with the cryptsetup isLuks command. If either of the rootfs partitions is not a LUKS container and the OTP key is not set, we mark that partition as needing setup.

For this we create two helper functions: set_up_luks_partition and provision_luks_partition. The first function creates the LUKS container on the specified partition, while the second function copies the unencrypted root filesystem into the LUKS container and stores the encryption key in OTP memory. Here is an example implementation of these functions:

set_up_luks_partition(){
    target_partition="$1"
    mapping_name="$2"

    echo "[initramfs] Setting up LUKS on ${target_partition}..."
    cryptsetup luksFormat "${target_partition}" --batch-mode --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to format LUKS partition ${target_partition}."
        exec /bin/sh
    }

    echo "[initramfs] formatting ${target_partition} to ext4."

    cryptsetup luksOpen ${target_partition} ${mapping_name} --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to open LUKS partition ${target_partition}."
        exec /bin/sh
    }   

    mkfs.ext4 /dev/mapper/${mapping_name} || {
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to format LUKS partition ${target_partition}."
        exec /bin/sh
    }

    
    cryptsetup luksClose ${mapping_name}
    echo "[initramfs] LUKS partition ${target_partition} is set up and ready."
}

First, the set_up_luks_partition function takes two arguments: the target partition to be encrypted and the mapping name for the LUKS container. It uses cryptsetup luksFormat to create the LUKS container on the specified partition, using a predefined key file (/keyfile.bin) for simplicity. After creating the LUKS container, it opens it with cryptsetup luksOpen, formats it to ext4 using mkfs.ext4, and then closes the LUKS container. If any of these steps fail, an error message is printed, and the system reboots. However depending on where things go wrong this might leave you with a bricked device, so be careful when testing this!

Once the LUKS container is set up, we can proceed to the provisioning process with the provision_luks_partition function:

provision_luks_partition() {
    target_partition="$1"
    mapping_name="$2"
    source_partition="$3"

    echo "[initramfs] Opening LUKS partition ${target_partition}..."
    cryptsetup luksOpen ${target_partition} ${mapping_name} --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to open LUKS partition ${target_partition}."
        exec /bin/sh
    }   

    echo "[initramfs] mounting ${mapping_name} to /mnt/target."
    mkdir -p /mnt/target
    mount /dev/mapper/${mapping_name} /mnt/target || {
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to mount LUKS partition ${target_partition}."
        exec /bin/sh
    }
    mkdir -p /mnt/src
    mount ${source_partition} /mnt/src || {
        
        umount /mnt/target
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to mount source partition ${source_partition}."
        exec /bin/sh
    }

    echo "[initramfs] Copy everything from ${source_partition} (/mnt/src) to /mnt/target using rsync."
    rsync -aHAXx /mnt/src/ /mnt/target/ || {
        umount /mnt/target
        cryptsetup luksClose ${mapping_name}
        echo "[initramfs] ERROR: Failed to copy data from /mnt/src to /mnt/target."
        exec /bin/sh
    }

    umount /mnt/target
    umount /mnt/src
    cryptsetup luksClose ${mapping_name}

}

The provision_luks_partition function takes three arguments: the target partition to be provisioned, the mapping name for the LUKS container, and the source partition containing the unencrypted root filesystem. It opens the LUKS container on the target partition, mounts it to /mnt/target, and mounts the source partition to /mnt/src. It then uses rsync to copy all data from the source partition to the target LUKS-encrypted partition. After the data transfer is complete, it unmounts both partitions and closes the LUKS container. If any step fails, an error message is printed, and the system reboots.

We then can use these functions to handle the provisioning process during the first boot. Here is an example of how to use these functions in the init script:

if [ "${ROOTFS_B_NEEDS_SETUP}" = true ]; then
    set_up_luks_partition ${ROOTFS_B} "rootfs_b" 
    provision_luks_partition ${ROOTFS_B} "rootfs_b" ${ROOTFS_A}
    
else
    echo "[initramfs] Rootfs_B partition ${ROOTFS_B} is already LUKS formatted. Skipping setup and provisioning."
fi

# Create a LUKS container for mmcblk0p5 (rootfs_A) if needed
if [ "${ROOTFS_A_NEEDS_SETUP}" = true ]; then
    echo "[initramfs] Rootfs_A partition ${ROOTFS_A} is not LUKS formatted. Setting up LUKS."
    set_up_luks_partition ${ROOTFS_A} "rootfs_a"
    # map rootfsb to copy data from it
    cryptsetup luksOpen ${ROOTFS_B} "rootfs_b" --key-file /keyfile.bin || {
        echo "[initramfs] ERROR: Failed to open LUKS partition ${ROOTFS_B}."
        exec /bin/sh
    }
    provision_luks_partition ${ROOTFS_A} "rootfs_a" "/dev/mapper/rootfs_b"
    cryptsetup luksClose "rootfs_b"
else
    echo "[initramfs] Rootfs_A partition ${ROOTFS_A} is already LUKS formatted. Skipping setup and provisioning."
fi

Note that while the first provisioning is done from an unencrypted partition to an encrypted one, the second provisioning is done from an already encrypted partition to another encrypted partition. For this we need to open the source LUKS container first and use the mapped device as source for the provision_luks_partition function.

Once the provisioning process is complete, we can proceed to unlock the LUKS-encrypted root filesystem for normal operation during subsequent boots. Here is an example of how to do this in the init script, for simplicity let’s say always into rootfs_A:

cryptsetup luksOpen ${ROOTFS_A} rootfs_a --key-file /keyfile.bin || {
    echo "[initramfs] ERROR: Failed to open LUKS partition ${ROOTFS_A}."
    exec /bin/sh
}
mount /dev/mapper/rootfs_a /mnt/rootfs || {
    cryptsetup luksClose rootfs_a
    echo "[initramfs] ERROR: Failed to mount LUKS partition ${ROOTFS_A}."
    exec /bin/sh
}

At this point, the LUKS-encrypted root filesystem is unlocked and mounted to /mnt/rootfs, allowing the normal boot process to continue.

Finally, we need to switch to the unlocked root filesystem and continue the normal boot process. This can be done using the switch_root command:

exec switch_root /mnt/rootfs /sbin/init

At this point, the init script has completed its tasks, and the normal boot process continues from the unlocked LUKS-encrypted root filesystem.

Conclusion

Getting an embedded device to support encryption at rest may look daunting at first, but as illustrated yocto makes it relatively straightforward to create the necessary initramfs image and modify the bootloader configuration. The more complex part is the provisioning process during the first boot, which requires careful handling of encryption keys and data transfer. While such a self-provisioning process may be comfortable to use, it might come with some security implications. First of all, until the provisioning is done, the unencrypted root filesystem is present on the device, which means that an attacker with physical access to the device could potentially read and modify the unencrypted data before the provisioning process is complete. Additionally, the provisioning process itself involves copying data from an unencrypted partition to an encrypted one, which could be vulnerable to interception or tampering if not properly secured. Therefore, it is crucial to ensure that the provisioning process is executed in a secure environment and that appropriate measures are taken to protect the unencrypted data during this phase. Using an external source to pull the unencrypted root filesystem during provisioning could mitigate some of these risks, but it also adds complexity to the provisioning process.

Another concern is that the provisioning code is still present in the initramfs even after the device has been provisioned. An attacker who gains access to the device could potentially exploit vulnerabilities in the provisioning code to compromise the security of the encrypted root filesystem. To mitigate this risk, it is advisable to implement a mechanism to disable or remove the provisioning code from the initramfs after the first successful boot and provisioning process. If your device supports update mechanisms, you could push an updated initramfs without the provisioning code after the first boot.

Despite these considerations, implementing encryption at rest using LUKS on embedded Linux devices based on the Raspberry Pi Compute Module 4 with the yocto Project provides a robust solution for protecting sensitive data and will make exploiting the device significantly harder, even if an attacker has physical access.

Like what you read?

We help teams design, build, and rescue complex software projects. From tricky architectures to hands-on delivery, we bring senior expertise where it matters most. Curious how we can help?

Learn more or or contact us

What is secure boot?

There are many aspects to consider when building a secure embedded device. One of the first steps towards security is making sure that a device runs only trusted software. This is generally called Secure boot and it is preventing attackers from loading malicious code on the device during the boot process. Secure boot is typically implemented using a chain of trust, where each stage of the boot process verifies the integrity and authenticity of the next stage before executing it.

In practice, this means that the device’s bootloader is cryptographically signed, and the hardware verifies this signature before executing the bootloader. If the signature is invalid, the boot process is halted, preventing the execution of potentially malicious code. For a more in-depth documentation on secure boot for raspberry pi devices, check out the Raspberry Pi Secure Boot documentation.

Implementing secure boot with Yocto

On a high level, implementing secure boot for an embedded Linux device using Yocto involves the following steps:

• Enable trusted boot support on the device itself - In the case of the Raspberry Pi CM4, this involves configuring the EEPROM firmware to enable secure boot features.
• Provide the public key used for verifying the signature of the bootloader to the device and store it on the EEPROM.
• Create a boot.img container that includes the bootloader, kernel, and device tree blobs (DTBs) - This container needs to be signed using a public key.
• Configure Yocto to build and sign the boot.img - This involves creating custom Yocto layers and recipes to handle the signing process and ensure that the boot.img is correctly formatted.
• Replace existing boot files with the signed boot.img and pack it into the boot partition of the device’s storage medium.

Before we dive into the details, we enable trusted boot on the Raspberry Pi CM4.

Enabling trusted boot on the Raspberry Pi CM4

Trusted boot on the Raspberry Pi CM4 needs to be enabled in the boot EEPROM by setting the SIGNED_BOOT flag in the boot.conf and by providing the public key to verify the signature. This is done using the rpiboot tool, which allows us to write the necessary configuration to the EEPROM. The usbboot repository provides the necessary tools and documentations to update the EEPROM for Raspberry Pi.

For illustration purposes, this article shows how to enable secure boot using a self-signed key pair. In a production environment, you would typically use a key pair issued by a trusted certificate authority (CA).

First, we generate a public/private key pair using OpenSSL:

openssl genrsa 2048 > keypair.pem

Next, we set up the boot.conf file to enable signed boot by setting SIGNED_BOOT=1. Then we use the update-pieeprom tool from rpiboot to update the EEPROM with the public key and flash it to the device:

update-pieeprom -k  keypair.pem
rpiboot -d /path/to/bootconf/

From this point on, the device will expect a signed boot.img during the boot process, the signature of it is expected to be present in a boot.sig file. If we try to boot an unsigned image, the device will halt the boot process with a Bad signature boot.sig error. To disable secure boot, we can overwrite the EEPROM again with SIGNED_BOOT=0.

Now that secure boot is enabled on the device, we can move on to configuring Yocto to build and sign the boot.img.

Configuring Yocto to build and sign the boot.img

Now that secure boot is enabled on the device and the key pair is generated, we can configure Yocto to build and sign the boot.img. This involves creating a custom Yocto recipe to create the boot container and to handle the signing process.

By default the raspberrypi layers in yocto store the boot files directly in the boot partition, so once we have a signed boot.img, we need to override the default behavior to use our signed image instead.

Let’s start with creating the boot.img container

Creating the boot.img container

The boot.img container is a fat16 formatted file that contains the bootloader, kernel, and device tree blobs (DTBs). We can create a custom Yocto recipe to build this container, let’s call this boot-img-container.bb:

The recipe will do the following things:

• Create a staging directory to hold the boot files
• Copy the bootloader, kernel, and DTBs to the staging directory
• Create a fat16 formatted boot.img
• Copy the boot files from the staging directory to the boot.img

Let’s look at the recipe code:

inherit deploy nopackages

DEPENDS += "rpi-bootfiles rpi-cmdline rpi-config linux-raspberrypi dosfstools-native mtools-native"

# The raspberry pi bootloader files are located in the following directories
BOOTFILES_DIR = "${WORKDIR}/bootfiles" 
DTBO_DIR = "${DEPLOY_DIR_IMAGE}"

S = "${WORKDIR}/boot-img-container"

do_compile() {
    # Create staging and target directory
    mkdir -p ${B}/boot-img-container-staging/overlays
    mkdir -p ${S}

    #collect all bootfiles from BOOTFILES_DIR and DTBO_DIR and and put it into the staging dir
    find ${BOOTFILES_DIR} -maxdepth 1 -type f ! -name 'cmdline.txt' -exec install -m 0644 -t ${B}/boot-img-container-staging/ {} +

    # copy all resolved .dtbo symlinks (the image is vfat formatted and symlinks are not supported)
    find ${DTBO_DIR} -type f -name '*.dtbo' -exec install -m 0644 -t ${B}/boot-img-container-staging/overlays {} +

    # copy all dtb files except overlay_map.dtb
    find ${DEPLOY_DIR_IMAGE} -type f -name '*.dtb' ! -name 'overlay_map.dtb' -exec install -m 0644 -t ${B}/boot-img-container-staging/ {} +

    # the overlay map file needs to be placed into the overlays folder too
    install -m 0644 ${DEPLOY_DIR_IMAGE}/overlay_map.dtb ${B}/boot-img-container-staging/overlays/overlay_map.dtb

    # copy the kernel image
    install -m 0644 ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE_DIRECT} ${B}/boot-img-container-staging/${SDIMG_KERNELIMAGE}

    BOOT_STAGING_DIR="${B}/boot-img-container-staging"
    BOOT_STAGING_SIZE_BYTES=$(find "$BOOT_STAGING_DIR" -type f -printf '%s\n' 2>/dev/null | awk '{s+=$1} END{print s+0}')
    
    # Create a FAT image sized to fit the staged files (+ padding), format it, copy files.
    BOOT_IMG="${S}/boot.img"

    # Add padding and round up to nearest MiB for filesystem overhead use awk because shell arithmetic is poorly supported in bitbake
    PADDING=$(awk 'BEGIN{printf "%d\n", 8*1024*1024}')
    PADDED_SIZE=$(awk -v a="$BOOT_STAGING_SIZE_BYTES" -v b="$PADDING" 'BEGIN{printf "%d\n", a+b}')
    MIN_SIZE=$(awk 'BEGIN{printf "%d\n", 1*1024*1024}')
    if [ "$PADDED_SIZE" -lt "$MIN_SIZE" ]; then
        PADDED_SIZE="$MIN_SIZE"
    fi
    PADDED_SIZE=$(awk -v n="$PADDED_SIZE" 'BEGIN{s=1024*1024; printf "%d\n", int((n + s - 1) / s) * s }')

    # Create an empty file of the desired size
    rm -f "$BOOT_IMG"
    truncate -s "$PADDED_SIZE" "$BOOT_IMG"

    # Create FAT16 filesystem
    mkfs.vfat -F 16 -n BOOT "$BOOT_IMG"

    # Copy files to the FAT image
    mcopy -i ${BOOT_IMG} -s ${BOOT_STAGING_DIR}/* ::/

}
do_compile[depends] += "rpi-bootfiles:do_deploy"
do_compile[depends] += "rpi-cmdline:do_deploy"
do_compile[depends] += "rpi-config:do_deploy"
# make do_compile depend on the boot files and kernel image
do_compile[filedeps] += " \
    ${BOOTFILES_DIR}/* \
    ${DEPLOY_DIR_IMAGE}/*.dtb \
    ${DEPLOY_DIR_IMAGE}/*.dtbo \
    ${DEPLOY_DIR_IMAGE}/overlay_map.dtb \
    ${DEPLOY_DIR_IMAGE}/${KERNEL_IMAGETYPE_DIRECT} \

First of all, we define the recipe as a deployment recipe by inheriting the deploy class. This allows us to place the generated boot.img in the deploy directory for later use. Since the recipe does not produce any packages to be put in the root filesystem, we also inherit the nopackages class. As for dependencies, we need the rpi-bootfiles and linux-raspberrypi recipes to provide the necessary boot files and kernel image. Additionally, we need dosfstools-native and mtools-native to create and manipulate the FAT filesystem.

Next we define the directories where the boot files are located. The BOOTFILES_DIR variable points to the directory where the Raspberry Pi bootloader files are located, while the DTBO_DIR variable points to the deploy directory where the DTBs are located.

And now begins the actual work. While we do not generate packages, we still compile an artifact, so we implement the do_compile function.

After creation of the staging and the target directory, we collect all necessary boot files from the defined directories and copy them to the staging directory. We make sure to copy the resolved .dtbo files instead of the symlinks, since the FAT filesystem does not support symlinks. These are all the find ... -exec install ... commands in the recipe. We also copy the kernel image to the staging directory.

Next comes the actual creation of the boot.img. We first calculate the size needed for the image by summing up the sizes of all files in the staging directory. We then add some padding and round up to the nearest MiB to account for filesystem overhead. This is a bit of an optimization to avoid wasting space, for a first implementation you could also just pick a fixed size that is large enough to hold all files. Since shell arithmetic is poorly supported in bitbake, I used awk for the calculations.

With the calculated size, we create an empty file of that size using truncate, format it as a FAT16 filesystem using mkfs.vfat, and finally copy all files from the staging directory to the boot.img using mcopy.

And with that, we have our boot.img container created!

Lastly, we make sure that the do_compile task depends on the deployment of the boot files and kernel image, so that it is executed after those are available. We also add the boot files and kernel image as file dependencies to ensure that any changes to these files will trigger a rebuild of the boot.img.

Next up is signing the boot.img.

Signing the boot.img

After creating the boot.img, we need to sign it using our private key. For debugging purposes, it is convenient to also be able to create unsigned images, so let’s add the signing as a separate recipe that depends on the boot-img-container recipe. This way we can choose whether to build a signed or unsigned image by including the appropriate recipe in our build.

Signing is relatively straightforward:

• Find the generated boot.img from the boot-img-container recipe
• Generate a sha256 hash of the boot.img to create a digest
• Sign the digest using the private key to create the signature
• Copy the boot.sig file to the deploy directory

The resulting recipe could look like this:

inherit nopackages

DEPENDS += "boot-img-container openssl-native xxd-native"

BOOT_IMG_PATH = "${DEPLOY_DIR_IMAGE}/boot.img"
SIG_FILE_PATH = "${DEPLOY_DIR_IMAGE}/boot.sig"
# Path to the .pem file containing the private key for signing 
BOOT_IMG_CERTIFICATE_PEM = "${THISDIR}/files/development-1.key.pem"

do_compile() {
    if [ -z "${BOOT_IMG_CERTIFICATE_PEM}" ]; then
        bbfatal "BOOT_IMG_CERTIFICATE_PEM is not set. Please specify the path to the .pem file."
    fi

    if [ -f "${BOOT_IMG_PATH}" ]; then
        # Generate a signature file containing the sha256 hash of boot.img
        sha256sum "${BOOT_IMG_PATH}" | awk '{print $1}' > "${SIG_FILE_PATH}"
        # next is the time of signing 
        printf 'ts: %s\n' "$(date -u +%s)" >> "${SIG_FILE_PATH}"

        # Generate RSA signature of the boot.img using openssl and hash it using sha256
        # append at the end of the sig file

        sig=$(openssl dgst -sha256 -sign "${BOOT_IMG_CERTIFICATE_PEM}" "${BOOT_IMG_PATH}" | xxd -c 4096 -p )
        printf 'rsa2048: %s\n' "${sig}" >> "${SIG_FILE_PATH}"
    else
        bbwarn "boot.img not found, skipping signature generation."
    fi
}

do_deploy() {
    if [ -f "${SIG_FILE_PATH}" ]; then
        install -D -m 0644 "${SIG_FILE_PATH}" "${D}${DEPLOYDIR}/boot.sig"
    else
        bbwarn "Signature file not found, skipping installation."
    fi
}

addtask do_compile after do_fetch before do_install
addtask do_deploy after do_compile before do_build
do_compile[depends] += "boot-img-container:do_deploy"

This recipe inherits the nopackages class since it does not produce any packages. It depends on the boot-img-container recipe to ensure that the boot.img is available for signing. Additionally, it depends on openssl-native and xxd-native to perform the signing operation.

The BOOT_IMG_PATH variable points to the location of the generated boot.img, while the SIG_FILE_PATH variable defines where the signature file will be created. The BOOT_IMG_CERTIFICATE_PEM variable should point to the private key file used for signing, this should be overridden with the actual path to the private key in a secure manner (i.e. by pulling it from an AWS Secrets Manager, Azure Key Vault, or similar secure storage).

The do_compile function first checks if the BOOT_IMG_CERTIFICATE_PEM variable is set and if the boot.img file exists. If both checks pass, it generates a SHA256 hash of the boot.img and writes it to the signature file. It also appends a timestamp to the signature file for reference. Then, it generates an RSA signature of the boot.img using the private key and appends it to the signature file.

And with that, we have our signed boot.img and boot.sig files ready for deployment! The do_deploy function installs the boot.sig file to the deploy directory, so the image creation command automatically packs it.

Lastly, we need to tell bitbake to execute the do_compile and do_deploy tasks at the appropriate times in the build process by adding them to the task graph. We do this by placing the do_compile task before the do_install task and the do_deploy task before the do_build task. We also make sure that the do_compile task depends on the deployment of the boot-img-container recipe to ensure that the boot.img is available for signing.

With that we have the recipes ready to create and sign the boot.img. The last step is to make sure that the signed image is used during the build process instead of the default boot files.

Using the signed boot.img in the build

Since the recipes are not packages, we add them to the machine part of the image configuration file (i.e. raspberrypi-cm4.conf):

IMAGE_BOOT_FILES = "boot.img config.txt boot.sig"
EXTRA_IMAGEDEPENDS += "boot-img-container sign-boot-img"
do_image_wic[depends] += "sign-boot-img:do_compile"

First the existing IMAGE_BOOT_FILES variable is overridden to include the boot.img and boot.sig files instead of the default boot files produced by the raspberrypi layer. Next, we add the boot-img-container and sign-boot-img recipes to the EXTRA_IMAGEDEPENDS variable to ensure that they are built before the image is created.

The recipes to create and sign the boot image need to be executed before the image creation process, so we add a dependency from the do_image_wic task to the do_compile task of the sign-boot-img recipe. If you are using a different image creation method (i.e. do_image_sdcard), you need to adjust this accordingly.

And with that, we have successfully integrated secure boot into our Yocto build process for the Raspberry Pi CM4. You can now build your image as usual using bitbake, and the resulting image will include the signed boot.img and boot.sig files, ensuring that only trusted software is executed during the boot process.

What about further improvements?

The recipes and approach described in this post provide a basic implementation of secure boot for an embedded Linux device using Yocto. For production use one improvement that needs to be made is to securely manage the private key used for signing the boot.img. Storing the private key directly in the source code or build environment is not secure and can lead to compromise of the key. Instead, consider using a secure key management solution, such as a hardware security module (HSM) or a cloud-based key management service (KMS), to store and manage the private key.

The Cyber Resilience Act at a glance

At the highest level, the Cyber Resilience Act (CRA) is a regulation by the European Union that aims to enhance the cybersecurity of products with digital elements, including software and connected hardware devices. The CRA establishes baseline security requirements for these products throughout their lifecycle, from design and development to deployment and maintenance. The term “products with digital elements” is defined broadly and includes any product that relies on software to function, such as IoT devices, medical devices, industrial control systems, and general-purpose software applications.

At the core, the CRA focuses on three main areas:

• Security-by-design: Integrate security measures into the product design and development process from the outset.
• Vulnerability management and incident response: Establish processes for identifying, reporting, and remediating vulnerabilities throughout the product lifecycle. Develop and maintain an incident response plan to address security incidents effectively.
• Compliance documentation: Maintain comprehensive documentation to demonstrate compliance with CRA requirements.

Let’s look at some of the key requirements of the CRA in more detail.

Security-by-design

Security by design is a fundamental principle of the CRA and is often considered the most technical part. This means that security considerations must be part of the design and development process of software products. Companies need to implement secure coding practices, conduct regular security testing (e.g., static and dynamic analysis, fuzzing, penetration testing), and ensure that products are resilient against common cyber threats. Additionally, products must be designed to minimize attack surfaces and protect sensitive data. But what about existing products? Do they need to be redesigned from scratch? Not necessarily. The CRA recognizes that many products are already in the market and allows for a risk-based approach to security improvements.

The CRA recognizes that many products are already in the market and allows for a risk-based approach to security improvements.

This means companies need to assess the security and perform a threat analysis of their existing products first. After this assessment, they can prioritize security enhancements based on the identified risks and start implementing necessary measures to improve security over time. An important point here is that many security issues might be mitigated by means other than code changes, e.g., by limiting exposure to threat vectors or improving user guidance.

Vulnerability management and incident response

The authors of the CRA understand that no software is perfect and vulnerabilities will inevitably be discovered over time. Therefore, the CRA mandates that companies establish vulnerability management and incident response processes. At the core, vulnerability management means setting up a way to report and triage vulnerabilities, a way to inform customers about the existence of a vulnerability, and ensuring timely remediation of identified vulnerabilities. Again, this sounds like a lot of work, but in many cases existing processes for receiving user feedback or bug reports can be extended to cover security vulnerabilities.

While proactively addressing vulnerabilities is important, companies must also be prepared to respond effectively when security incidents occur. This is closely tied to vulnerability management and may even use some of the same facilities.

Incident response means to have a clear plan in place that outlines roles, responsibilities, and communication channels during a security incident.

The most important part of incident response is to have a clear plan in place that outlines roles, responsibilities, and communication channels during a security incident. Make it explicit who is taking the lead, who needs to sit in the crisis room, and what response times are expected. Who can make decisions to enable containing the incident, mitigating its impact, and recovering normal operations. The third part to be ready for the CRA is comprehensive compliance documentation.

Compliance documentation

Chances are that if you are affected by the CRA, you already have some level of compliance documentation in place, because many other regulations (e.g., GDPR, industry-specific regulations) also require documentation of security practices. If so, good news: you can build upon your existing documentation efforts. If you have not yet started, don’t panic; creating the necessary documentation is not an unmanageable task. So what kind of documentation is required? First, the efforts described in the previous sections need to be documented. This includes documenting the security-by-design practices, vulnerability management processes, and incident response plans discussed earlier.

Don’t panic, creating the necessary documentation for the CRA is not an unmanageable task.

Additionally, companies must maintain records of security testing results, risk assessments, and any security-related decisions made during the product lifecycle. In practice this means nothing more than keeping the CI/CD and testing results of your releases ready and creating a comprehensive changelog. Additionally, the CRA requires companies to keep track of third-party components used in their products, including open-source libraries. This means maintaining a Software Bill of Materials (SBOM) that lists all components, their versions, and any known vulnerabilities associated with them. This can be automated to a large extent using existing tools that generate SBOMs as part of the build process.

The compliance documentation is the final piece of the puzzle to be ready for the CRA. While security-by-design and vulnerability management are the more practical parts, compliance documentation ensures that companies can demonstrate their adherence to CRA requirements. So where to start?

Get started with the Cyber Resilience Act

Preparing for the CRA may seem overwhelming at first, but breaking it down into manageable steps can make the process more approachable. An incremental approach is often the best way to build up the necessary capabilities to pass an audit and get your products certified.

Start with a high-level gap assessment to identify where your current practices stand in relation to the CRA requirements. Then implement a “CRA-Light” process that covers the most critical aspects of the CRA without overwhelming your teams. Start with the “who” regarding vulnerability management and incident response. Do a high-level STRIDE threat model for your main products to identify the biggest risks. Automate SBOM generation in your build process and start collecting compliance documentation as you go along.

While this might not cover all aspects of the CRA right away, it establishes a foundation that can be built upon over time. Building up CRA compliance incrementally allows teams to adapt and improve their processes without feeling overwhelmed by the full scope of the regulation from the start. And remember, passing audits on the first try is nice, but often not the case—so build a process that allows you to continuously improve over time.

After all, the CRA is not just about compliance; it’s about enhancing the overall security posture of products and protecting users from cyber threats. By taking proactive steps to meet CRA requirements, companies can not only ensure market access in the EU but also contribute to a safer digital ecosystem.

std::expected in a nutshell

At the core, std::expected<T, E> is a returnable type that can either hold a value of type T (indicating success) or an error of type E (indicating failure). This makes it clear to the caller that a function can fail and provides a structured way to handle that failure.

Let’s look at a simple example of a function that computes the square root of a number, returning an error if the input is negative:

#include <expected>
#include <cmath>

std::expected<double, std::string> safe_sqrt(double x) {
    if (x < 0) {
        return std::unexpected("Negative input");
    }
    return std::sqrt(x);
}

...

// Usage
const auto result = safe_sqrt(-1);
if (result) {
    std::cout << "Square root: " << *result << '\n';
} else {
    std::cout << "Error: " << result.error() << '\n';
}

In this example, safe_sqrt returns an std::expected<double, std::string>. If the input is valid, it returns the square root; otherwise, it returns an error message. The caller can then check if the result is valid and handle the error accordingly. So how does this compare to traditional error handling methods?

Comparison to Traditional Error Handling

Before std::expected, there were typically two main approaches to error handling in C++: exceptions and error codes. While exceptions can be powerful, they typically bring with them more complexity in control flow and then there is the discussion which errors should cause an exception to be thrown and which should not. The benefit of exceptions is that they allow for clean separation of error handling code and for propagation of errors up the call stack. Error codes on the other hand tend to either clutter the code by requiring out-parameters or have the problem of being either ignored or misunderstood by the caller. While nodiscard can help with ignored return values, it still does not solve the problem that the caller has to semantically understand the meaning of the return value.

std::expected provides a middle ground. It makes error handling explicit in the type system, allowing to pass semantic information about the error back to the caller. The beauty of std::expectedis also, that it can help to discern between expected or recoverable errors (e.g. file not found, invalid input) and unexpected or unrecoverable errors (e.g. out of memory, logic errors) which should still be handled via exceptions.

Tip: Use std::expected for recoverable errors where the caller can take action based on the error, and reserve exceptions for truly exceptional situations.

Let’s look at a more complex example that demonstrates how std::expected can be used in a real-world scenario.

Real world example: Reading a QR code from an image

Let’s suppose we want to write a function that reads a QR code from binary image data. The function generally has three paths:

1. The image contains a valid QR code and we can return the decoded string.
2. The image does not contain a QR code and we want to return an error indicating that.
3. The image data is unreadable (e.g. corrupted or unrecognizable format) and we want to throw an exception.

While the first two paths are expected and recoverable errors, the third path is an unexpected error that should be handled via exceptions. So the implementation could look like this:

#include <expected>
#include <string>
#include <stdexcept>
#include <vector>   

std::expected<std::string, std::string> read_qr_code(std::vector<uint8_t> const& image_data) {
    
    if (image_data.empty() || check_if_corrupted(image_data)) {
        throw std::invalid_argument("Invalid image data");
    }

    // Assume parse_image_data is a function that parses the image data and returns the QR code string or throws on failure
    std::string parsed_data = parse_image_data(image_data); // May throw exceptions on failure

    if (parsed_data.empty()) {
        return std::unexpected("No QR code found");
    }

    return parsed_data;
}

Note that in this example, both the success and error type are strings, but they could be any type. In a lot of cases, it might still make sense to use an enum or a custom error type for the error case to make it more structured. However, by using std::expected, we already are able to add a lot more context to the function without cluttering the code. This already is a big improvement over returning, but there is more. std::expected also provides a set of powerful combinators for composing operations that may fail, allowing for more elegant error handling.

Monadic chaining with and_then

One of the benefits of std::expected is that it allows for monadic chaining, A technique widely used in functional programming. Monadic chaining lets you compose a sequence of operations that may fail, without deeply nested if/else. With std::expected, you chain:

• and_then when the next step itself may fail and returns another expected.
• transform when the next step cannot fail and just maps the value.
• or_else to act on or recover from an error.

Below is a continuation of the QR example, showing a pipeline that: 1) reads the QR payload, 2) validates it, 3) parses it as a URI, 4) extracts the host (pure mapping), 5) adds context to the error if anything failed.

// Reuse the earlier read_qr_code signature:
// std::expected<std::string, std::string> read_qr_code(const std::vector<uint8_t>&);

#include <expected>
#include <string>
#include <vector>

struct Uri {
    std::string scheme;
    std::string host;
    std::string path;
};

std::expected<std::string, std::string>
validate_payload(std::string const& s) {
    if (s.empty()) {
        return std::unexpected("Empty QR payload");
    }
    if (s.size() > 4096) { // arbitrary sanity limit
        return std::unexpected("QR payload too large");
    }
    return s; // valid as-is
}

std::expected<Uri, std::string>
parse_uri(std::string const& s) {
    auto starts_with = [&](std::string const& p) {
        return s.rfind(p, 0) == 0;
    };

    Uri u;
    if (starts_with("https://")) {
        u.scheme = "https";
    } else if (starts_with("http://")) {
        u.scheme = "http";
    } else {
        return std::unexpected("Not an http(s) URI");
    }

    // very naive split: scheme://host/path
    auto pos = s.find("://");
    auto rest = s.substr(pos + 3);
    auto slash = rest.find('/');
    if (slash == std::string::npos) {
        u.host = rest;
        u.path = "/";
    } else {
        u.host = rest.substr(0, slash);
        u.path = rest.substr(slash);
    }
    if (u.host.empty()) {
        return std::unexpected("Missing host");
    }
    return u;
}

std::string host_from(Uri const& u) {
    return u.host; // pure mapping, cannot fail
}

std::expected<std::string, std::string>
annotate_error(std::string const& err) {
    return std::unexpected(std::string{"QR processing failed: "} + err);
}

// Usage: linear, early-exiting pipeline
std::expected<std::string, std::string>
extract_qr_host(std::vector<uint8_t> const& image) {
    return read_qr_code(image)
        .and_then(validate_payload)        // may fail -> expected<Payload, E>
        .and_then(parse_uri)               // may fail -> expected<Uri, E>
        .transform(host_from)              // cannot fail -> expected<std::string, E>
        .or_else(annotate_error);          // act on error path, keep E the same
}

As we see, the resulting code in extract_qr_host is linear and easy to read. Each step is clearly defined, and error handling is centralized without deeply nested conditionals. The use of and_then, transform, and or_else makes the intent of each operation explicit.

There are some pitfalls and good practices to keep in mind when using monadic chaining with std::expected:

• Keep the error type E consistent across and_then/or_else steps. If you must change it, use transform_error.
• Use and_then only with functions that return expected<..., E>. Use transform for pure mappings returning plain values.
• The chain short-circuits on the first error, returning that error downstream. So having a catch-all or_else at the end is a good practice.
• If a step can throw, those exceptions still propagate unless caught and converted to std::unexpected.
• Prefer passing by const& in chain steps to avoid copies or use move semantics. However the guaranteed copy elision in C++17 and later often makes this less of a concern.

With these practices in mind, std::expected and its combinators can greatly enhance the clarity and robustness of error handling in your C++ code.

final thoughts

With the arrival of std::expected in C++23 there is another powerful tool in C++ to allow more expressive code in a functional programming style. This can make applications that do a lot of data processing and have many recoverable failure paths much cleaner and easier to maintain. While it does not replace exceptions for unrecoverable errors, it nicely complements them by providing a structured way to handle expected errors. And the beauty of it is, that it still works seamlessly with existing C++ code and libraries - So no need to go all in and change it everywhere. So give it a try in your next C++ project and see how it can improve your error handling!

While std::expected became part of the C++ standard with C++23, it has been around for quite some time as an open-source library by Sy Brand before that.

Why build your own AI agent?

While tapping into an existing online service to build your custom AI agent might be convenient, there are several reasons why you might want to build your own. There might be privacy concerns, or you might need offline access to the agent or maybe you want to customize the agent to your specific needs. Building your own AI agent gives you full control over the model, the data and the functionality. You can tailor it to your specific use case and integrate it seamlessly into your existing systems. A great place to start building your own AI agent is the llama.cpp project.

Llama.cpp is a C/C++ implementation of Facebook’s LLaMA model that allows you to run LLMs on your own hardware. It is designed to be efficient and lightweight, making it suitable for running on a wide range of devices, from high-end servers to low-power edge devices. Llama.cpp supports various LLaMA models, including the smaller ones that can run on consumer-grade hardware. It offers a wide range of cross-platform support including various GPU backends for acceleration, but also runs on CPU only systems. So let’s see how to get started with llama.cpp.

Building a simple AI agent with llama.cpp

So let’s build a simple executable that uses llama.cpp to answer simple prompts by using a pre-trained model. You can find the complete code on on GitHub it the bernedom/LlamaPlayground repository which is inspired by the simple example provided in the llama.cpp repository. The executable will be called LLamaPlayground and can be used like this:

> LLamaPlayground -m ./models/mymodel.gguf What is the capital of Switzerland?

The capital of Switzerland is Bern.

To get started we need a few things:

• A C++17 compatible compiler and CMake
• llama.cpp as library, which we can get by using CMakes FetchContent module. (Unfortunately the conan module for llama.cpp is outdated at the time of writing)
• A LLM-Model as .gguf file which can be obtained from various sources, e.g. Huggingface

Setting up the project

Let’s start by setting up a simple CMake project, that creates our executable and pulls in llama.cpp as a dependency. To kickstart your project with a basic CMake and dependency handling already set up, consider this CMake/Conan/Catch2 Template.

In the CMakeLists.txt we need to pull in llama.cpp using FetchContent and link it to our executable like this

...
include(FetchContent)
FetchContent_Declare(
        llama
        GIT_REPOSITORY https://github.com/ggml-org/llama.cpp.git
        GIT_TAG master
)
FetchContent_MakeAvailable(llama)
...

This gives us access to the llama target that we can link to our executable like this target_link_libraries(LLamaPlayground PRIVATE llama).

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Next we need to obtain a model. You can find various models on Huggingface. For this example I downloaded the ggml-org/gemma-3-1b-it-GGUF model and placed it in a models folder next to the executable. This is a fairly lightweight model that accepts text or images as input and generates text as output.

The easiest way to get the model is to use the huggingface CLI. You can install it using pip:

pip install huggingface_hub[cli]

To download the model, you can use the following command:

hf download ggml-org/gemma-3-1b-it-GGUF --local-dir ./models

And with that we are ready to start coding.

Coding the AI agent

Our simple AI agent will have to do three basig things:

1. Model Loading: Loading the pre-trained model file.
2. Prompt Tokenization: Converting the input prompt into tokens.
3. Text Generation: Using the model to predict and generate tokens.

I’ll skip the boilerplate code to parse the arguments and focus on the llama.cpp specific parts, for the full code head over to the Github repository containing the example code

Model Loading

First let’s load the model and initialize it.

llama_model_params model_params = llama_model_default_params();
model_params.n_gpu_layers = ngl;

llama_model *model = llama_model_load_from_file(model_path.c_str(), model_params);

For simplicity we use the default model parameters, but you can customize them as needed. To illustrate how this works, the n_gpu_layers parameter specifies how many layers of the model should be offloaded to the GPU for acceleration. If you don’t have a GPU or want to run on CPU only, you can set this to 0. That is already all the magic needed to load the model, lets go to the next step Prompt Tokenization.

Prompt Tokenization

To convert the user-supplied prompt into tokens we need to convert it from a string into tokens that the model can understand. The prompt is tokenized using the model’s vocabulary. Tokenization involves splitting the input string into smaller units (tokens) that the model can process, these are no longer human readable strings but rather integer values that represent words or subwords.

const llama_vocab *vocab = llama_model_get_vocab(model);
const int n_prompt = -llama_tokenize(vocab, prompt.c_str(), prompt.size(), nullptr, 0, true, true);

std::vector<llama_token> prompt_tokens(n_prompt);
llama_tokenize(vocab, prompt.c_str(), prompt.size(), prompt_tokens.data(), prompt_tokens.size(), true, true);

The first call to llama_tokenize with a nullptr for the output tokens is used to determine how many tokens are needed for the prompt. It’s a particularity of llama.cpp that the first call returns a negative value, because it failed to write the tokens to a nullptr. The absolute value of the returned integer indicates the number of tokens required.

The second call actually performs the tokenization and fills the prompt_tokens vector with the resulting tokens. With this we can initialize the context for the model and prepare it for text generation.

llama_context_params ctx_params = llama_context_default_params();
ctx_params.n_ctx = n_prompt + n_predict - 1;
ctx_params.n_batch = n_prompt;

llama_context *ctx = llama_init_from_model(model, ctx_params);

The context defines the model’s working environment, including the number of tokens to process and the batch size. For our small example we set the context size to the sum of the prompt tokens and the number of tokens we want to predict. The batch size is set to the number of prompt tokens, which means that the model will process all prompt tokens in one go.

Now that we have the prompt tokenized, we can move on to the final step: Text Generation.

Text Generation

The main loop evaluates the model and generates tokens iteratively. Each token is sampled, converted back to text, and appended to the output. for this we loop until we reach the desired number of predicted tokens or encounter an end-of-generation token.

llama_batch batch = llama_batch_get_one(prompt_tokens.data(), prompt_tokens.size());

for (int n_pos = 0; n_pos + batch.n_tokens < n_prompt + n_predict;) {
    llama_decode(ctx, batch);

    llama_token new_token_id = llama_sampler_sample(smpl, ctx, -1);
    if (llama_vocab_is_eog(vocab, new_token_id)) break;

    char buf[128];
    int n = llama_token_to_piece(vocab, new_token_id, buf, sizeof(buf), 0, true);
    std::cout << std::string(buf, n);

    batch = llama_batch_get_one(&new_token_id, 1);
}

First we fetch a batch of tokens to process, in our case this might be the entire prompt, but since we might want to extend to larger prompts, we enter a loop.

Inside the loop the current batch is decoded using llama_decode, which processes the tokens and updates the model’s internal state. Then we sample a new token using llama_sampler_sample, which selects the next token based on the model’s predictions. If the sampled token is an end-of-generation token, we break out of the loop.

The sampled token is then converted back to a human-readable string using llama_token_to_piece and printed to the console. Finally, we prepare the next batch containing just the newly generated token for the next iteration of the loop. And that is all the code we need to build a simple AI agent using llama.cpp. You can find the complete code on on GitHub it the bernedom/LlamaPlayground repository.

To run the AI agent, you need to compile the code using CMake and your C++ compiler. Make sure you have the model file in the specified path. Beware that even the smaller models can be quite memory/hungry, so make sure you have enough RAM available.

Whats next?

This is a very simple example to get you started with llama.cpp and building your own AI agent. As the talk of edge AI and on-device AI is getting more and more popular, llama.cpp might be a great starting point to explore this field. There are of course many more things to explore be it towards more efficiency or hardware acceleration through CUDA or Vulkan or more advanced prompt handling and context management. Whether you find any practical use for it or just play around with it, I hope this post helped you to get started.

What is artifact based CI?

Let’s look at a typical CI/CD pipeline: When a developer pushes code changes, the CI system triggers a build process that compiles the code, runs tests, and produces artifacts (e.g., binaries, libraries, documentation). In a traditional setup, every time a change is pushed, the entire build process is executed from scratch, which can be time-consuming and resource-intensive.

For a simple and small project such a straight forward pipeline is all what it takes. However if projects grow larger and contain multiple components, the build process can become complex and time-consuming. This is where artifact based CI comes into play. A slightly more complicated example might be an application that consists of several modules, such as an executable and several libraries, where the executable depends on the libraries. A trivial approach to CI/CD would be to build all the libraries from scratch on every change, then deploy all together to a test-environment, run the tests and if successfully, package them and publish them somewhere or deploy them to the target systems.

Since the libraries and the executable should be buildable individually, building everything from scratch, even if only a part of the code has changed is wasteful. Also, if for example only code in the executable changes, running the full library build and test process is not necessary. So to optimize this, we can introduce artifact based CI.

Artifact based CI is a practice where the output of your build process (the artifacts) are stored and reused in subsequent builds. This means that instead of rebuilding everything from scratch every time, we can leverage previously built artifacts, which can save a lot of time and resources.

If done right, artifact based CI can not just speed up your builds and increase the reliability of your CI/CD pipelines, but they are also a great way to test deployment processes on the go. By deploying the same artifacts that will be used in production, you can catch deployment issues early in the development process.

The same pipeline as before, but now with artifact based CI could look like this:

Wow, this looks a lot more complicated, but let’s break it down a bit. The first thing introduced here is some form of decision making to determine whether a component has changed or not. This can often be done by checking the version control systems for changes in the relevant files or directories. What we also need is a way to store the artifacts produced by the build process. Many CI/CD systems such as github actions or gitlab pipelines offer a simple way to store and retrieve build outputs, but for more complex applications a dedicated artifact repository, such as JFrog Artifactory, Nexus Repository might be better suited. The key is to ensure that the artifacts are versioned and easily retrievable. The last thing to notice is that this already gives us a way to speed up the build process, even if we need to build everything from scratch, because we can parallelize the jobs for building the libraries.

When moving to artifact based CI, the central problem usually revolves around how to efficiently store and retrieve the artifacts, how to version them properly, and how to ensure that the right artifacts are used in the right places.

Finding the right artifact

The core concept of artifact based CI rotates around storing and retrieving build artifacts efficiently. Looking a bit closer this the main problem to be solved is that we need to find the right artifact for the current build. So putting up a good versioning and naming scheme is crucial. In a typical workflow we have three kind of versions of artifacts that we need to consider:

• Development builds: These are the artifacts produced during the development process, often for branches or pull requests. Typically, these builds are only used by a small group of developers (or even a single one) working on a specific feature or bug fix. They are usually stored temporarily and can be discarded after a certain period of time or when the feature is merged into the main branch.

• Released builds: These are the artifacts that are deployed to production environments. They should be stored permanently and should be easily retrievable.

• Staged builds: These are the artifacts used for testing purposes, such as staging or QA environments. They can be a mix of development and production builds, depending on the testing requirements. If all goes well, these artifacts might be promoted to production builds or discarded if newer artifacts are available.

The naming and versioning of the artifacts should reflect these different stages. A common approach is to use semantic versioning, where the version number consists of three parts: major, minor, and patch. For example, a development build might be named myapp-1.0.0-dev-123, where 123 is a build number, a commit hash, or a branch name (or a combination of all of them). A released build might be named myapp-1.0.0, and a staged build might be named myapp-1.0.0-staging.

This way it is easy to identify where an artifact comes from and what it is used for. If building for multiple platforms or configurations, the naming scheme can be extended to include the platform and configuration, such as myapp-1.0.0-linux-x86_64, myapp-1.0.0-windows-x86_64, etc.

For a start, a simple storage solution for these artifacts might be enough, but as the project grows, it is worth considering a dedicated artifact repository and a package manager to handle the artifacts. If you have the versioning and naming scheme covered, you are already a huge step further towards efficient artifact-based CI, and further improvements can usually be made incrementally.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Further improvements

Moving from a build-everything-all-the-time approach to an artifact-based CI approach can be a bit of a paradigm shift, but it can lead to significant improvements in build times and reliability. Once the CI pipeline is set up to handle artifacts, the door is open to further optimizations, such as caching dependencies, parallelizing builds, and more. Depending on the team’s setup, it can even be beneficial to make the artifacts available to developers locally, so they can test their changes against the same artifacts that will be used in production.

As always, the key is to start somewhere and iterate and improve over time. Shifting to artifact-based CI is a great step towards more efficient and reliable CI/CD pipelines, and it can help teams to focus on delivering value rather than dealing with long build times and waiting for feedback.

Let’s fix that.

1. Vendoring everything instead of using find_package

One of the worst time-wasters: bundling dependencies directly into your project (aka “vendoring”), either as source code or worse, as precompiled binaries.

It bloats your repo, makes updates a hassle, and creates fragile builds. Modern CMake projects should use find_package to locate and integrate dependencies cleanly. Many libraries now provide CMake config files out of the box. If they don’t, writing your own Find<Package>.cmake is a good step toward future-proofing.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Even better, tools like Conan or vcpkg can take care of dependency management for you. Stop hardcoding paths - let CMake and your package manager handle it.

2. Not using CMakePresets

CMake Presets are a game-changer - use them! Since their introduction, they’ve made managing multiple build configurations so much easier. Instead of fiddling with command-line flags or custom scripts, you can define builds cleanly in CMakePresets.json.

CMake presets are one of the most impactful features introduced in CMake since the introduction of targets.

Presets reduce errors, improve consistency across teams, and help CI pipelines stay clean. By providing a standardized way to configure good build configurations for development, testing, and release builds, CMakePresets can significantly reduce the entry barrier for new contributors and help ensure that everyone is using the same build settings. Always include and maintain presets for your project, and encourage contributors to use them or create their own CMakeUserPresets.json files for their specific needs.

3. Still using global variables instead of targets

Modern CMake is all about targets - and has been for over a decade. Yet many projects still rely on global variables and configuration, making things messy and error-prone.

Targets make everything modular, scoped, and easier to maintain. Use commands like target_link_libraries, target_include_directories, and target_compile_options to apply settings only where needed.

And don’t just slap PUBLIC everywhere. Use PRIVATE by default, and only expose what’s absolutely necessary. You’ll reduce build times and avoid unnecessary rebuilds.

And if you still need global project settings, use options() with good descriptions and defaults, but keep them minimal. The goal is to avoid global state as much as possible.

4. Writing non-portable build instructions

It’s easy to hack together something that works on your machine and CMake’s possibility to call external commands and scripts is very powerful. But that doesn’t mean it’s portable. Using platform-specific paths, scripts, or flags ties your build to a single setup - and breaks everything elsewhere.

Stick to CMake’s built-in features:

• Stick global compiler flags into CMake presets.
• Use target_compile_options, target_compile_definitions, etc., instead of global flags.
• Use cmake -P and cmake -E for scripting, not Bash or PowerShell.
• Use CMake’s built-in function to find things like libraries and include directories - No hardcoded paths in CMakeLists.txt!
• Set up toolchain files for cross-compilation.

Portability isn’t just for open-source projects. It’ll save you headaches even in single-platform codebases - especially when onboarding new devs or upgrading tools.

5. Half-baked library setup

Ever try using a library only to discover it’s missing version info, has broken exports, or forces a pile of transitive includes on you?

That’s what happens when the setup of the library target is incomplete or sloppy.

Don’t just make everything PUBLIC. Be precise with PRIVATE, PUBLIC, and INTERFACE. Set symbol visibility correctly, especially for shared libs. And always include version info (VERSION, SOVERSION) for libraries - it helps tools, consumers, and future-you.

For a solid setup, here’s a line-by-line guide.

6. No install instructions

You might think install rules don’t matter if you’re not publishing a library. But even in internal projects, a clean install() setup can make packaging, CI pipelines, or dev workflows way smoother.

It’s not hard: a few lines in your CMakeLists.txt go a long way. Think of it as a foundation for future packaging, or just making builds easier to deploy locally.

7. Neglecting maintenance of your CMakeLists.txt

CMake scripts aren’t “write-once” files. They need love too.

Old, bloated, or inconsistent CMakeLists.txt files are painful to work with and a nightmare for newcomers. Regularly refactor and clean up your build scripts - just like you do with code.

Update to modern CMake practices. Drop old workarounds. Add comments. Keep dependencies clean. You’ll thank yourself later.

How to fix it

If your team is grumbling about CMake, chances are you’re stuck in one - or several - of these anti-patterns. But you don’t have to fix everything overnight.

Which one hurts the most depends on the project and the team, but in general, the anti-patterns often go hand in hand. Modernize your CMake scripts and tackle one thing at the time and each step brings back a little bit of joy to the build process.

CMake Best Practices - The book

CMake Best Practices: Discover proven techniques for creating and maintaining programming projects with CMake. Learn how to use CMake to maximum efficiency with this compendium of best practices for a lot of common tasks when building C++ software.

Get it from Packt

Even if you can’t make things perfect, starting with small steps can lead quickly to improvements. Just applying PRIVATE and PUBLIC correctly can go a long way, replacing hardcoded paths with find_package and using CMake’s built-in functions to find libraries and include directories can make the build process more robust and portable. And usually with every step you take, you will find that the build process becomes easier to understand and maintain, leading to a more enjoyable development experience for everyone involved.