Article 08 ended on a question you can’t answer by reading code, only by fault injection. Does your bootloader actually reject an image with one flipped bit in the signature? Does anti-rollback actually block a downgrade, or is the comparison off by one? Does the parser fault instead of writing past its buffer?
You could flash a board and try it. Once. Manually. Then someone changes the linker script and you’d have to do it again, and you won’t, because that loop takes eleven minutes and there are forty other things to check.
The alternative isn’t more testing effort. It’s an architecture where the answers are cheap to obtain, and the good news is that this series has been building it for eight articles without saying so. The interface in article 01, the layer rules in article 04, run-to-completion in article 06, the result_t boundary in article 02: every one of those is a place where a test can substitute its own implementation.
This article is about collecting on that. What the seams buy, what host testing genuinely cannot tell you, and the handful of exhaustive tests that are affordable in embedded and nowhere else.
The Dual-Target Build
Everything here depends on one thing: your code compiling for two targets from one source tree. Article 03’s sharpest tell was gcc -c src/application/*.c — if it doesn’t build for the host, nothing below matters.
Structurally this is less work than people expect. The rule is that a module’s target depends on which layer it’s in:
# Host-buildable: no vendor headers, no target assumptions
add_library(app_core STATIC
src/application/thermal_policy.c
src/application/control_law.c
src/services/config_service.c
src/services/sensor_service.c
src/middleware/modbus_frame.c
src/middleware/fsm.c)
target_include_directories(app_core PUBLIC src/include)
# Deliberately NOT linked against anything vendor-specific
# Target-only: drivers, BSP, startup
add_library(platform STATIC
src/drivers/spi_bus.c
src/drivers/uart_rs485.c
src/bsp/board_h743.c)
target_link_libraries(platform PRIVATE stm32h7_hal)
# Two consumers of app_core
add_executable(firmware.elf src/main.c)
target_link_libraries(firmware.elf app_core platform)
add_executable(tests test/run_all.c ${TEST_SOURCES})
target_link_libraries(tests app_core test_doubles)app_core is the interesting line. Everything in it compiles with a host compiler and links against either real drivers or test doubles, and the set of files in that list is your testability metric. Track it. It should grow.
Two rules keep it honest. app_core never links a vendor library — if it needs to, something above the driver layer has vendor knowledge and article 04’s boundary was crossed. And the test build uses the same warning and undefined-behaviour flags as the target build, so you’re not testing subtly different semantics.
Add both builds to CI. The host build failing is a layering violation report, not just a broken test.
What Host Tests Cannot Tell You
This is the part most writing on this topic skips, and skipping it is how teams end up with 4,000 green tests and a device that hangs in the field.
Host testing verifies logic. It cannot verify anything that depends on the machine actually executing it. Specifically:
Cache coherency. Your fake DMA writes to a buffer and your test reads it back correctly, because on a host there’s one coherent memory. On an H7, the real DMA writes to AXI SRAM while the CPU reads a stale cache line. Every article-02 MPU decision is invisible to host tests.
Real interrupt latency and preemption. A host test calls process_byte() directly. It doesn’t reproduce an ISR arriving between two instructions of a non-atomic update. Article 05’s priority inversion and article 03’s shared-state races are invisible here.
Actual timing. Article 05’s entire schedulability analysis rests on measured worst-case execution times. A host runs your control law in 200 ns on a machine with a branch predictor and megabytes of cache. That number tells you nothing about 480 MHz with a 16 KB D-cache.
Peripheral behaviour and errata. Your fake SPI always completes. The real one has a silicon erratum about a specific clock-phase and DMA combination, and no model of it exists in your test.
Alignment and access faults. Unaligned access that works on x86 faults on some Cortex-M configurations. Same source, different outcome.
Anything about power. Brownout, rail sequencing, the recovery storm from article 07.
So the honest framing is that host tests prove your logic is right, which is most of your codebase and most of your behavioural bugs. They prove nothing about your system, and pretending otherwise is worse than not having them. A green suite creates confidence proportional to its coverage, and that confidence is misplaced for exactly the class of bug that reaches customers.
Which is what HIL is for, and why the answer is both rather than either.
The Seams You Already Have
Four substitution points, all built in earlier articles for other reasons.
Time. The single most valuable seam, and the one people skip. Any module that asks the system what time it is has an untestable dependency buried inside it.
The pattern that works without turning every function signature into noise: one clock per module, set at init.
typedef uint32_t (*clock_fn_t)(void);
void sensor_service_init(sensor_service_t *s, clock_fn_t clock) {
s->now = clock;
}Production passes HAL_GetTick. Tests pass a function reading a variable they control, and a five-minute timeout sequence executes in microseconds:
static uint32_t fake_now = 0;
static uint32_t fake_clock(void) { return fake_now; }
fake_now = 0; service_tick(&s); assert(s.state == ST_WAITING);
fake_now = 1999; service_tick(&s); assert(s.state == ST_WAITING);
fake_now = 2001; service_tick(&s); assert(s.state == ST_TIMEOUT);The trap is going too far the other way and injecting a clock into every function. One per module, held in the module’s context, is the right granularity.
Storage. A RAM-backed fake flash with real sector semantics — erase sets to 0xFF, writes only clear bits, writes must be aligned. Getting those semantics right in the fake is what makes it useful; a fake that lets you program a location that was never erased won’t catch the bug where you did. These are internal and NOR flash semantics; the KIC-400’s config sits in I2C EEPROM, which has no erase step, so a fake for that store models page boundaries and write-cycle time instead.
Transport. Article 01’s telemetry_sink_t and article 04’s sensor_if_t. Bind a recorder, assert on the bytes.
The fault sink. Article 02’s fault_emit() is already a seam. In tests, bind it to a recorder and assert the right fault was emitted with the right source and code — which is how you verify that error paths reported correctly, not just that they returned an error.
Fault Injection at the Boundary, Not With #ifdef
Here’s a rule I’d defend hard:
**If you need
#ifdef TESTinside a module to inject a fault, your seam is in the wrong place.**
The version everyone writes first:
result_t spi_transfer(...) {
#ifdef TEST_INJECT_SPI_TIMEOUT
if (g_inject_timeout) return RES_TIMEOUT;
#endif
/* ...real implementation... */
}Three problems. The production build and the test build now compile different code, so you aren’t testing what ships. The injection points multiply until the module is half scaffolding. And you can only inject faults where someone thought to add a hook.
Inject by substitution instead. The sensor service takes a sensor_if_t; the test binds an implementation that fails:
static result_t always_timeout(void *ctx, sensor_sample_t *out) {
(void)ctx; (void)out;
return RES_TIMEOUT;
}
static result_t fail_after_n(void *ctx, sensor_sample_t *out) {
fail_ctx_t *f = ctx;
if (f->calls++ >= f->fail_at) return RES_NO_DEVICE;
out->value = f->good_value;
return RES_OK;
}Now you can test article 07’s entire escalation ladder — retry once, reset the peripheral, mark degraded, isolate — with no conditional compilation anywhere and the production module compiled exactly as it ships.
The one legitimate exception is fault injection below the seam, on target, for HIL: making the real SPI driver fail so you can test recovery against real hardware. That’s a separate build variant with a stated purpose, not hooks scattered through production modules.
The Exhaustive Tests You Can Actually Afford
Embedded has an advantage general software doesn’t: the state spaces are small enough to enumerate. Four cases where you can test everything rather than sampling.
The state machine
Article 06 covered this: N states × M events is enumerable, runs on a host in milliseconds, and finds the transitions you never specified. Not repeating it here, but it’s the template for the other three.
The fault policy table
Article 07’s severity mapping is data:
{ SRC_SPI, RES_TIMEOUT, CLASS_RECOVERABLE, DOMAIN_SENSING },So iterate every (source, code) pair in your enums and assert the lookup returns a defined class. Missing entries are exactly the failures that will hit an undefined default at 3 a.m. in a substation.
for (src_t s = 0; s < SRC_COUNT; s++)
for (result_t r = 1; r < RES_COUNT; r++)
assert(fault_classify(s, r) != CLASS_UNDEFINED);Six lines, and it fails the build the day someone adds a result_t value without deciding what it means.
Anti-rollback boundaries
Article 08’s security version check is one comparison, which is exactly the kind of code that’s off by one and never noticed:
struct { uint32_t stored, incoming; bool expect_accept; } cases[] = {
{ 5, 4, false }, /* older — reject */
{ 5, 5, true }, /* equal — accept (reinstall) */
{ 5, 6, true }, /* newer — accept */
{ 0, 0, true }, /* first boot */
{ 0xFFFFFFFF, 0, false }, /* max stored */
{ 0, 0xFFFFFFFF, true }, /* max incoming */
};Six cases, and the last two catch a signed comparison someone wrote by accident.
Power loss at every single byte
This is the one I’d put in front of anyone who thinks host testing is second-class.
Article 02’s config service must survive power loss mid-write — REQ-08. On hardware you test that by cutting power during a write, which gives you one data point per attempt at roughly thirty seconds each. With a fake flash that fails at an injectable offset, you test every offset:
for (size_t fail_at = 0; fail_at < TOTAL_WRITE_BYTES; fail_at++) {
fake_flash_load(&flash, known_good_image);
fake_flash_fail_after(&flash, fail_at); /* writes stop here */
(void)config_commit(&cfg, &new_config); /* will fail somewhere */
fake_flash_clear_failure(&flash); /* "power comes back" */
config_t loaded;
assert(config_load(&cfg, &loaded) == RES_OK); /* always loads */
assert(config_equal(&loaded, &old_config) /* old ... */
|| config_equal(&loaded, &new_config)); /* ...or new */
}The assertion is the atomicity requirement stated as code: after any interruption, the device loads either the old configuration or the new one, never a mixture and never nothing. A few hundred iterations, a few milliseconds, every possible interruption point covered.
Try that on hardware and you’re there for two hours to cover a fraction of the offsets. This is the strongest argument for the architecture the whole series has been building, and the same technique transfers directly to article 12’s update staging.
Test Doubles, Briefly
Four words that get used interchangeably. The distinction matters only because picking the heaviest one by default wastes time.
Stub — returns a canned answer. always_timeout above. Use it to force one path.
Fake — a working simplified implementation. The RAM-backed flash. Use it when the dependency has real behaviour you depend on.
Mock — records and verifies interactions. “fault_emit was called once, with SRC_SPI and RES_TIMEOUT.” Use it when how you called something is the requirement.
Simulator — models a system. A Modbus master sending real frames with real timing. Use it for protocol conformance, rarely otherwise.
Pick the simplest one that proves the requirement. Most embedded tests need a stub and occasionally a fake.
Testing RTOS Code Without the Scheduler
Don’t start FreeRTOS to test a task’s logic. Split the task into a loop and a step:
/* target */
void modbus_rtu_task(void *arg) {
for (;;) {
event_t ev;
if (xQueueReceive(q, &ev, timeout)) modbus_step(&ctx, &ev);
else modbus_step(&ctx, &TICK_EVENT);
}
}
/* host test — no scheduler, no queue, no target */
modbus_step(&ctx, &(event_t){ .id = EV_BYTE, .param = 0x01 });Article 06’s run-to-completion property is what makes this legal. A step function that never blocks is a pure function of state and event, so calling it directly from a test is equivalent to the scheduler calling it.
What still needs the real RTOS, on target: priority assignment behaving as analysed, queue depths under real burst load, stack high-water marks, and startup ordering. Those are integration tests, and there should be a few dozen of them, not thousands.
And don’t test the RTOS itself. FreeRTOS’s queue implementation is not your responsibility.
HIL: The Minimum Viable Rig
Hardware-in-the-loop gets presented as a rack of instrumentation, which makes teams conclude it’s out of reach and skip it entirely. It isn’t.
A rig that answers most of the questions host tests can’t:
| Part | Purpose |
|---|---|
| USB relay board | Cut and restore power on command |
| A spare dev board | Stimulus and monitoring: drive sensor inputs, timestamp GPIO |
| USB-serial adapter | Console and log capture |
| Programmer/debugger | Flash, and read backup SRAM after a reset |
| USB-RS485 adapter | Act as a Modbus master |
Around a hundred dollars of parts and a Python script. What it buys, none of which is available on a host:
Power interruption at speed. Cut power during a config write or an OTA, restore it, assert the device boots and the config is intact. The host test proved the algorithm; this proves the flash actually behaves as your fake assumed.
Boot timing. Article 08’s secure boot budget, measured on the real part with the real image size.
Real peripheral bring-up. Does the ADC actually produce the values the fake claimed?
Reset-reason capture. Force a watchdog reset, read backup SRAM, verify article 07’s crash record is populated and correct. That test is nearly impossible to trust any other way.
Long-running behaviour under real interrupt load.
Run it nightly, not per commit. Twenty to fifty HIL tests is a healthy number. If you have five hundred, most of them belong on the host.
Soak Tests and What to Log
Some failures need hours. Memory drift, counter overflow, queue accumulation, reconnect logic that leaks a socket every thousand attempts.
A soak test that only asserts “it was still running at the end” wastes the hours. Log the four numbers from article 05 at a fixed interval and look at the trend:
- Stack high-water mark per task — trending down means something’s growing
- Heap or arena high-water mark — flat is fine, drifting up is fragmentation
- Peak queue depth per queue — validates your burst sizing against reality
- Drop and error counters — any non-zero value nobody noticed
A 72-hour soak producing a flat line on all four is real evidence. The same soak producing “no crash” is an anecdote.
Coverage, Honestly
Coverage doesn’t measure correctness. It measures which lines never executed, which is genuinely useful as long as you don’t turn the number into a target.
The number worth looking at isn’t overall coverage. It’s coverage of error paths, and in most firmware it’s close to zero — precisely backwards, because that’s where the field failures live. Instrument your result_t handling branches specifically and you’ll get a far more informative number than 78% overall.
Branch coverage over line coverage, always. This is fully line-covered and half-tested:
if (temperature > OVER_TEMP_LIMIT) { thermal_protect(); }One test at 90 °C hits every line. It never checks that 70 °C doesn’t trip protection, which is the failure a customer notices.
The Pipeline
Not every stage on every commit. Match the cost of the check to the cost of the feedback:
Pull request Build (both targets) + static analysis + host tests
Target: under 3 minutes
Merge to main The above + integration tests on target + coverage
Nightly HIL suite + 8-hour soak
Release Full HIL + 72-hour soak + power-cycle campaign
+ security test vectors from article 08Speed on the first line is an architectural constraint, not a nice-to-have. A host suite that runs in five seconds gets run on every save. One that takes forty-five minutes gets run by CI and ignored by humans, which means feedback arrives after the developer has moved on, when fixing it costs ten times more.
If your host suite is slow, something in it is touching real time, real I/O, or real hardware. Find it.
Next
You can now verify the firmware behaves correctly. Today, on today’s toolchain, against today’s hardware.
The KIC-400 ships for ten years. Over that span the compiler gets three major versions, the MCU goes end-of-life and is replaced by a pin-compatible part with different errata, your TLS library has a CVE and the fix needs a newer API, two of the four sensors are discontinued, and every engineer who built it has moved on. The test suite still passes — on a machine nobody can rebuild.
Article 10: Designing Firmware for 10-Year Products — building for a decade of change you can’t predict.
Series Index
Phase 1 — Architecture
01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture
03. Firmware Architecture Anti-Patterns
Phase 2 — Implementation
04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture
Phase 3 — Failure & Resilience
07. Error Handling & Recovery
08. Firmware Security Architecture
Phase 4 — Verification
09. Designing Firmware for Testability (you are here)
Phase 5 — Product Lifecycle
10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates
