Designing Firmware for Testability: Unit Testing, Mocks, HIL and Fault Injection

Part 9 of 12 in the Embedded Firmware Architecture series · Phase 4: Verify · Level: Intermediate

Article 08 ended on questions you can’t answer by reading the code, or by testing on hardware alone. Does your bootloader actually reject an image with one flipped bit in the signature? Does anti-rollback actually block a downgrade, or is the comparison off by one? Does the parser fault instead of writing past its buffer?

You could flash a board and try it. Once. Manually. Then someone changes the linker script and you’d have to do it again, and you won’t, because that loop takes eleven minutes and there are forty other things to check.

The alternative isn’t more testing effort. It’s firmware testability designed into the architecture, so the answers are cheap to obtain, and the good news is that this series has been building it for eight articles without saying so. The interface in article 01, the layer rules in article 04, run-to-completion in article 06, the result_t boundary in article 02: every one of those is a place where a test can substitute its own implementation.

This article is about collecting on that. What the seams buy for unit testing on a host, what host testing genuinely cannot tell you, and the handful of exhaustive tests that are affordable in embedded and nowhere else.

---

The Dual-Target Build

Everything here depends on one thing: your code compiling for two targets from one source tree. Article 03’s sharpest tell was gcc -c src/application/*.c: if it doesn’t build for the host, nothing below matters.

Structurally this is less work than people expect. The rule is that a module’s target depends on which layer it’s in:

# Host-buildable: no vendor headers, no target assumptions
add_library(app_core STATIC
    src/application/thermal_policy.c
    src/application/control_law.c
    src/services/config_service.c
    src/services/sensor_service.c
    src/middleware/modbus_frame.c
    src/middleware/fsm.c)

target_include_directories(app_core PUBLIC src/include)
# Deliberately NOT linked against anything vendor-specific

if(CMAKE_CROSSCOMPILING)                 # arm-none-eabi build tree
    # Target-only: drivers, BSP, startup
    add_library(platform STATIC
        src/drivers/spi_bus.c
        src/drivers/uart_rs485.c
        src/bsp/board_h743.c)
    target_link_libraries(platform PRIVATE stm32h7_hal)

    add_executable(firmware.elf src/main.c)
    target_link_libraries(firmware.elf app_core platform)
else()                                   # host build tree
    add_executable(tests test/run_all.c ${TEST_SOURCES})
    target_link_libraries(tests app_core test_doubles)
endif()

app_core is the interesting line. Everything in it compiles with a host compiler and links against either real drivers or test doubles, and the set of files in that list is your testability metric. Track it. It should grow.

Two rules keep it honest. app_core never links a vendor library; if it needs to, something above the driver layer has vendor knowledge and article 04’s boundary was crossed. And the test build uses the same warning and undefined-behaviour flags as the target build, so you’re not testing subtly different semantics.

The if(CMAKE_CROSSCOMPILING) split is there because CMake configures one toolchain per build tree, so firmware.elf and the host tests executable cannot come out of the same configure. Use two build trees, one with the Arm toolchain file and one with the host compiler.

CMake presets make the two trees one command each (cmake --preset target, cmake --preset host), and app_core is built in both.

Add both builds to CI. The host build failing is a layering violation report, not just a broken test.

---

What Host Tests Cannot Tell You

This is the part most writing on this topic skips, and skipping it is how teams end up with 4,000 green tests and a device that hangs in the field.

Host testing verifies logic. It cannot verify anything that depends on the machine actually executing it. Specifically:

Cache coherency. Your fake DMA writes to a buffer and your test reads it back correctly, because on a host there’s one coherent memory. On an H7, the real DMA writes to AXI SRAM while the CPU reads a stale cache line. Every article-02 MPU decision is invisible to host tests.

Real interrupt latency and preemption. A host test calls process_byte() directly. It doesn’t reproduce an ISR arriving between two instructions of a non-atomic update. Article 05’s priority inversion and article 03’s shared-state races are invisible here.

Actual timing. Article 05’s entire schedulability analysis rests on measured worst-case execution times. A host runs your control law in 200 ns on a machine with a branch predictor and megabytes of cache. That number tells you nothing about 480 MHz with a 16 KB D-cache.

Peripheral behaviour and errata. Your fake SPI always completes. The real one has a silicon erratum about a specific clock-phase and DMA combination, and no model of it exists in your test.

Alignment and access faults. Unaligned access that works on x86 faults on some Cortex-M configurations. Same source, different outcome.

Anything about power. Brownout, rail sequencing, the recovery storm from article 07.

So the honest framing is that host tests prove your logic is right, which is most of your codebase and most of your behavioural bugs. They prove nothing about your system, and pretending otherwise is worse than not having them. A green suite creates confidence proportional to its coverage, and that confidence is misplaced for exactly the class of bug that reaches customers.

Which is what HIL is for, and why the answer is both rather than either. An emulator tier fits between the two: QEMU or Renode runs the real Arm binary, startup code and RTOS included, on every commit without a board, which catches linker, startup and toolchain problems early, though its peripheral models are no substitute for silicon.

---

The Seams You Already Have

Four substitution points, all built in earlier articles for other reasons.

Time. The single most valuable seam, and the one people skip. Any module that asks the system what time it is has an untestable dependency buried inside it.

The pattern that works without turning every function signature into noise: one clock per module, set at init.

typedef uint32_t (*clock_fn_t)(void);

void sensor_service_init(sensor_service_t *s, clock_fn_t clock) {
    s->now = clock;
}

Production passes HAL_GetTick. Tests pass a function reading a variable they control, and a five-minute timeout sequence executes in microseconds:

static uint32_t fake_now = 0;
static uint32_t fake_clock(void) { return fake_now; }

fake_now = 0;      service_tick(&s);   assert(s.state == ST_WAITING);
fake_now = 1999;   service_tick(&s);   assert(s.state == ST_WAITING);
fake_now = 2000; service_tick(&s); assert(s.state == ST_WAITING);   /* boundary: times out after more than 2000 ms */
fake_now = 2001;   service_tick(&s);   assert(s.state == ST_TIMEOUT);

The trap is going too far the other way and injecting a clock into every function. One per module, held in the module’s context, is the right granularity.

Storage. A RAM-backed fake flash with real sector semantics: erase sets to 0xFF, writes only clear bits, writes must be aligned. On STM32H7 internal flash there is one more rule: a 256-bit flash word is programmed once, with ECC, and programming a written word again is an error even if it only clears bits, so the fake must reject a second program of any written word or it will pass the flash-word counters in articles 07 and 08 that fail on silicon. Getting those semantics right in the fake is what makes it useful; a fake that lets you program a location that was never erased won’t catch the bug where you did. These are internal and NOR flash semantics; the KIC-400’s config sits in I2C EEPROM, which has no erase step, so a fake for that store models page boundaries and write-cycle time instead.

Transport. Article 01’s telemetry_sink_t and article 04’s sensor_if_t. Bind a recorder, assert on the bytes.

The fault sink. Article 02’s fault_emit() is already a seam. In tests, bind it to a recorder and assert the right fault was emitted with the right source and code, which is how you verify that error paths reported correctly, not just that they returned an error.

---

Fault Injection at the Boundary, Not With #ifdef

Here’s a rule I’d defend hard:

If you need #ifdef TEST inside a module to inject a fault, your seam is in the wrong place.

The version everyone writes first:

result_t spi_transfer(...) {
#ifdef TEST_INJECT_SPI_TIMEOUT
    if (g_inject_timeout) return RES_TIMEOUT;
#endif
    /* ...real implementation... */
}

Three problems. The production build and the test build now compile different code, so you aren’t testing what ships. The injection points multiply until the module is half scaffolding. And you can only inject faults where someone thought to add a hook.

Inject by substitution instead. The sensor service takes a sensor_if_t; the test binds an implementation that fails:

static result_t always_timeout(void *ctx, sensor_sample_t *out) {
    (void)ctx; (void)out;
    return RES_TIMEOUT;
}

static result_t fail_after_n(void *ctx, sensor_sample_t *out) {
    fail_ctx_t *f = ctx;
    if (f->calls++ >= f->fail_at) return RES_NO_DEVICE;
    out->value = f->good_value;
    return RES_OK;
}

Now you can test article 07’s entire escalation ladder (retry once, reset the peripheral, mark degraded, isolate) with no conditional compilation anywhere and the production module compiled exactly as it ships.

The one legitimate exception is fault injection below the seam, on target, for HIL: making the real SPI driver fail so you can test recovery against real hardware. That’s a separate build variant with a stated purpose, not hooks scattered through production modules.

---

The Exhaustive Tests You Can Actually Afford

Embedded has an advantage general software doesn’t: the state spaces are small enough to enumerate. Four cases where you can test everything rather than sampling.

The state machine

Article 06 covered this: N states × M events is enumerable, runs on a host in milliseconds, and finds the transitions you never specified. Not repeating it here, but it’s the template for the other three.

The fault policy table

Article 07’s severity mapping is data:

{ SRC_SPI, RES_TIMEOUT, CLASS_RECOVERABLE, DOMAIN_SENSING },

So iterate every (source, code) pair in your enums and assert the lookup returns a defined class. Missing entries are exactly the failures that will hit an undefined default at 3 a.m. in a substation.

for (src_t s = 0; s < SRC_COUNT; s++)
    for (result_t r = 1; r < RES_COUNT; r++)
        assert(fault_classify(s, r) != CLASS_UNDEFINED);

Three lines, and it fails the build the day someone adds a result_t value without deciding what it means.

Anti-rollback boundaries

Article 08’s security version check is one comparison, which is exactly the kind of code that’s off by one and never noticed:

struct { uint32_t stored, incoming; bool expect_accept; } cases[] = {
    { 5, 4, false },                 /* older: reject */
    { 5, 5, true },                  /* equal: accept (reinstall) */
    { 5, 6, true },                  /* newer: accept */
    { 0, 0, true },                  /* first boot */
    { 0xFFFFFFFF, 0, false },        /* max stored */
    { 0, 0xFFFFFFFF, true },         /* max incoming */
};

/* The store, not just the comparison: an erased log must read as the floor */
fake_flash_erase_all(&flash);
assert(secver_read_min() == 0);      /* never 0xFFFFFFFF, or every image is rejected */

Six cases, and the last two catch a signed comparison someone wrote by accident. The 0xFFFFFFFF row deserves a second look, because on flash that is exactly what an erased word reads as. If secver_read_min() ever returns raw erased contents, after a log that was never initialised or was erased by mistake, the device rejects every image including the one it is running, and that is a brick. So the final assertion tests the store as well as the comparison. Note that an append-only flash-word log, whose value is the count of written words, cannot represent 0xFFFFFFFF as a value at all (a 128 KB sector holds about 4,096 words), so that reading from it is always a fault to handle, never a real version.

Power loss at every single byte

This is the one I’d put in front of anyone who thinks host testing is second-class.

Article 02’s config service must survive power loss mid-write (REQ-08). On hardware you test that by cutting power during a write, which gives you one data point per attempt at roughly thirty seconds each. With a fake flash that fails at an injectable offset, you test every offset:

for (size_t fail_at = 0; fail_at < TOTAL_WRITE_BYTES; fail_at++) {
    fake_flash_load(&flash, known_good_image);
    fake_flash_fail_after(&flash, fail_at);      /* writes stop here */

    result_t r = config_commit(&cfg, &new_config); /* will fail somewhere */
    (void)r; /* judged by config_load() below; GCC ignores a (void) cast on the call itself */

    fake_flash_clear_failure(&flash);            /* "power comes back" */

    config_t loaded;
    assert(config_load(&cfg, &loaded) == RES_OK);       /* always loads  */
    assert(config_equal(&loaded, &old_config)           /* old ...       */
        || config_equal(&loaded, &new_config));         /* ...or new     */
}

The assertion is the atomicity requirement stated as code: after any interruption, the device loads either the old configuration or the new one, never a mixture and never nothing. A few hundred iterations, a few milliseconds, every possible interruption point covered.

Make the fake honest about how power loss really lands, though. Stopping writes cleanly at a byte boundary is kinder than hardware: an EEPROM interrupted mid page write, or an H7 flash word interrupted mid program, can hold indeterminate data across the whole page or word. A fake that only truncates will pass designs that fail in the field, so model the in-flight unit as garbage. On H7 internal flash it can be worse than garbage: a 256-bit flash word interrupted mid program can carry an ECC pattern that fails as a double-bit error, which sets DBECCERR and raises a bus fault on the read instead of returning bad data. config_load() has to survive that, by checking the ECC flags or handling the bus fault and treating the copy as corrupt, or the first read of one torn word takes the device down on every boot. The fake has to model it as well:

/* Torn-write model: the unit being written when power fails is indeterminate */
static void fake_flash_power_fail(fake_flash_t *f, size_t offset)
{
    size_t page = offset / f->page_size;
    for (size_t i = 0; i < f->page_size; i++) {
        f->mem[page * f->page_size + i] = (uint8_t)rand();
    }
    /* H7 internal flash: an interrupted 256-bit word may fail ECC on read */
    if (f->models_ecc && (rand() & 1)) {
        f->ecc_bad[page] = true;   /* fake_flash_read() then returns RES_ECC_DOUBLE */
    }
}

For the H7 store, page_size is the 32-byte flash word, and RES_ECC_DOUBLE stands in for the bus fault: the loop above still asserts that config_load() returns the old or the new configuration. Seed rand() per iteration and log the seed, so any failure reproduces exactly.

Try that on hardware and you’re there for two hours to cover a fraction of the offsets. This is the strongest argument for the architecture the whole series has been building, and the same technique transfers directly to article 12’s update staging.

---

Test Doubles, Briefly

Five words that get used interchangeably. The distinction matters only because picking the heaviest one by default wastes time.

Stub: returns a canned answer. always_timeout above. Use it to force one path.

Fake: a working simplified implementation. The RAM-backed flash. Use it when the dependency has real behaviour you depend on.

Spy: a stub that also records how it was called, for the test to inspect afterwards. The fault_emit recorder from the seams section is a spy: the test runs the code, then asserts on what it captured. Use it when the outcome you care about is a call, not a return value.

Mock: programmed with expectations before the code runs, and fails the test itself when a call doesn’t match. “fault_emit is called once, with SRC_SPI and RES_TIMEOUT, and nothing else is.” Use it when how you called something, including order and count, is the requirement.

Simulator: models a system. A Modbus master sending real frames with real timing. Use it for protocol conformance, rarely otherwise.

Pick the simplest one that proves the requirement. Most embedded tests need a stub, sometimes a spy, and occasionally a fake.

Link-time substitution. Everything above substitutes through an interface the module already takes, a function pointer or a sensor_if_t. The other way in is the linker. CMock (part of the Unity and Ceedling toolset) and fff (the Fake Function Framework) read a module’s header and generate a replacement for every function in it, with call counts, captured arguments and programmable return values. The test build links the generated file and simply leaves the real .c out of the test executable, while the code under test calls spi_transfer() by name exactly as it does on target. Production code doesn’t change, there is no indirection on the hot path, and nobody has to invent an interface for a dependency that will only ever have one real implementation.

Prefer it for leaf dependencies called directly by name: drivers below the seam, thin vendor HAL wrappers, fault_emit() when you only need to check that it was called. Prefer function-pointer interfaces when production itself has more than one implementation (several sensor types behind sensor_if_t), when one test binary needs both the real and the fake version of a function, or when the binding changes at runtime. Link-time substitution is all or nothing per executable: once the real spi_bus.c is out of the link, no test in that binary can use it, which is why CMock-based projects usually build one test executable per module.

---

Testing RTOS Code Without the Scheduler

Don’t start FreeRTOS to test a task’s logic. Split the task into a loop and a step:

/* target */
void modbus_rtu_task(void *arg) {
    for (;;) {
        event_t ev;
        if (xQueueReceive(q, &ev, ticks_to_deadline(&ctx)) == pdTRUE)   /* absolute deadline, as in article 06 */
            modbus_step(&ctx, &ev);
        else                                modbus_step(&ctx, &TICK_EVENT);
    }
}

/* host test: no scheduler, no queue, no target */
modbus_step(&ctx, &(event_t){ .id = EV_BYTE, .param = 0x01 });

Article 06’s run-to-completion property is what makes this legal. A step function that never blocks is a pure function of state and event, so calling it directly from a test is equivalent to the scheduler calling it.

What still needs the real RTOS, on target: priority assignment behaving as analysed, queue depths under real burst load, stack high-water marks, and startup ordering. Those are integration tests, and there should be a few dozen of them, not thousands.

And don’t test the RTOS itself. FreeRTOS’s queue implementation is not your responsibility.

---

HIL: The Minimum Viable Rig

Hardware-in-the-loop gets presented as a rack of instrumentation, which makes teams conclude it’s out of reach and skip it entirely. It isn’t.

A rig that answers most of the questions host tests can’t:

Part	Purpose
USB relay board	Cut and restore power on command
A spare dev board	Stimulus and monitoring: drive sensor inputs, timestamp GPIO
USB-serial adapter	Console and log capture
Programmer/debugger	Flash, and read backup SRAM after a reset on RDP Level 0 builds (on Level 1 units, read it through a firmware diagnostic command)
USB-RS485 adapter	Act as a Modbus master

Around a hundred dollars of parts and a Python script. What it buys, none of which is available on a host:

Power interruption at speed. Cut power during a config write or an OTA, restore it, assert the device boots and the config is intact. The host test proved the algorithm; this proves the flash actually behaves as your fake assumed.

Boot timing. Article 08’s secure boot budget, measured on the real part with the real image size.

Real peripheral bring-up. Does the ADC actually produce the values the fake claimed?

Reset-reason capture. Force a watchdog reset, read backup SRAM, verify article 07’s crash record is populated and correct. That test is nearly impossible to trust any other way.

Long-running behaviour under real interrupt load.

Run it nightly, not per commit. Twenty to fifty HIL tests is a healthy number. If you have five hundred, most of them belong on the host.

---

Soak Tests and What to Log

Some failures need hours. Memory drift, counter overflow, queue accumulation, reconnect logic that leaks a socket every thousand attempts.

A soak test that only asserts “it was still running at the end” wastes the hours. Log the four numbers from article 05 at a fixed interval and look at the trend:

• Stack high-water mark per task: trending down means something’s growing
• Heap or arena headroom: minimum-ever free bytes catches leaks, and the largest free block catches fragmentation, since total free can stay flat while the largest block shrinks
• Peak queue depth per queue: validates your burst sizing against reality
• Drop and error counters: any non-zero value nobody noticed

A 72-hour soak producing a flat line on all four is real evidence. The same soak producing “no crash” is an anecdote.

---

Branch Coverage, Honestly

Coverage doesn’t measure correctness. It measures which lines never executed, which is genuinely useful as long as you don’t turn the number into a target.

The number worth looking at isn’t overall coverage. It’s coverage of error paths, and in most firmware it’s close to zero, which is precisely backwards, because that’s where the field failures live. Instrument your result_t handling branches specifically and you’ll get a far more informative number than 78% overall.

Branch coverage over line coverage, always. This is fully line-covered and half-tested:

if (temperature > OVER_TEMP_LIMIT) { thermal_protect(); }

One test at 90 °C hits every line. It never checks that 70 °C doesn’t trip protection, which is the failure a customer notices.

Branch coverage isn’t the ceiling, either. DO-178C Level A requires, and ISO 26262 highly recommends at ASIL D, MC/DC (modified condition/decision coverage), which shows that every condition in a decision independently changes its outcome; in those sectors, choose a coverage tool that measures it from day one.

---

CI Pipeline for Embedded Tests

Not every stage on every commit. Match the cost of the check to the cost of the feedback:

Pull request     Build (both targets) + static analysis + host tests
                 Target: under 3 minutes

Merge to main    The above + integration tests on target + coverage

Nightly          HIL suite + 8-hour soak

Release          Full HIL + 72-hour soak + power-cycle campaign
                 + security test vectors from article 08

Concrete choices so the table is actionable: Unity with CMock, or CppUTest, for host tests; gcov with lcov for coverage; cppcheck or clang-tidy for static analysis, plus a MISRA C checker if your sector expects one. Pick one of each and keep them pinned in the build container.

Two additions earn their place in the host tree. Build it with -fsanitize=address,undefined, which turns silent out-of-bounds writes and undefined shifts into test failures. And give every parser that touches untrusted bytes a fuzz target: a libFuzzer harness that feeds arbitrary frames to the Modbus frame decoder looks for exactly the length and index bugs that article 08 calls the likeliest shipped vulnerability, and it reuses the same host build.

/* test/fuzz_modbus.c
   clang -g -O1 -fsanitize=fuzzer,address,undefined -Isrc/include \
         test/fuzz_modbus.c src/middleware/modbus_frame.c -o fuzz_modbus */
#include <stddef.h>
#include <stdint.h>
#include "modbus_frame.h"

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
    modbus_frame_t f;
    result_t r = modbus_frame_decode(data, size, &f);  /* must never read past size */
    (void)r;
    return 0;
}

Speed on the first line is an architectural constraint, not a nice-to-have. A host suite that runs in five seconds gets run on every save. One that takes forty-five minutes gets run by CI and ignored by humans, which means feedback arrives after the developer has moved on, when fixing it costs ten times more.

If your host suite is slow, something in it is touching real time, real I/O, or real hardware. Find it.

---

Next

You can now verify the firmware behaves correctly. Today, on today’s toolchain, against today’s hardware.

The KIC-400 ships for ten years. Over that span the compiler gets three major versions, the MCU goes end-of-life and is replaced by a pin-compatible part with different errata, your TLS library has a CVE and the fix needs a newer API, two of the four sensors are discontinued, and every engineer who built it has moved on. The test suite still passes, on a machine nobody can rebuild.

Article 10: Designing Firmware for 10-Year Products: building for a decade of change you can’t predict.

---

References

• Unity test framework
• CppUTest
• CMake presets
• libFuzzer

---

Series Index

Overview, reading paths and related articles: Embedded Firmware Architecture series guide.

Phase 1: Decide

01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture
03. Firmware Architecture Anti-Patterns

Phase 2: Build

04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture

Phase 3: Fail safely

07. Error Handling & Recovery
08. Firmware Security Architecture

Phase 4: Verify

09. Designing Firmware for Testability (you are here)

Phase 5: Operate

10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates