Part 7 of 12 in the Embedded Firmware Architecture series · Phase 3: Fail safely · Level: Advanced
Every state machine in article 06 had a FAULT state. Every one of them was a box with nothing in it, and no firmware error handling behind it.
That’s not an accident of the writing. It’s how most firmware is actually built. The fault path is the least-exercised code in the product, written last, reviewed least, and executed only when conditions are already bad, which is precisely when you need it to be the most dependable code you have.
This article is about what goes in that box. Not error propagation: article 02 built result_t and the layered model, and article 03 covered what happens when errors get swallowed. This is the part after detection: classification that actually drives behaviour, recovery that’s budgeted rather than hoped for, and preserving enough evidence that the failure is diagnosable from a log file in another country.
Errors and Faults Are Different Things
Article 02 drew this line and deferred it here. It’s the most consequential distinction in failure handling, and conflating the two is why devices limp along in corrupted states.
An error is an expected outcome you have a plan for. The sensor timed out. A Modbus frame had a bad CRC. The socket closed. These are things the physical world does. Count them, handle them, carry on.
A fault is a broken invariant. The config CRC is valid but the sequence number moved backwards. A stack canary is gone. An MPU violation fired. A queue that the design says can never fill is full.
The test that separates them, and it’s a single question:
Can the system correctly continue?
If yes, it’s an error, and recovery is appropriate. If no, because continuing means operating on state you can’t trust, it’s a fault, and running recovery is the wrong move. Recovery assumes the system is sound and something external failed. When the system itself is unsound, recovery executes on a corrupted foundation and produces worse outcomes than stopping.
So faults get a different path entirely: record what you can, reach the safe state, reset. No retry, no degrade, no attempt to be clever. Article 02’s KIC-400 drives two analog outputs; a fault means those outputs go to their defined safe value before anything else happens, including logging.
Firmware Error Handling: Classification That Drives Behaviour
Five severity levels, and the reason for five rather than three is that each maps to a different first action. A classification that doesn’t change what happens next isn’t worth having.
| Class | Meaning | First action | Example |
|---|---|---|---|
| Transient | May clear by itself | Count, continue | One bad CRC on a Modbus frame |
| Recoverable | Needs an action, system is sound | Retry once, then reset the peripheral | I2C timeout, sensor NAK |
| Degrading | Capability lost, product still useful | Isolate, mark capability unavailable | One of four sensors dead |
| Critical | Product can’t do its primary job | Safe state, keep diagnostics alive | Control feedback lost |
| Fatal | Invariant broken, state untrustworthy | Record, safe outputs, reset | Memory corruption, bad config sequence |
Two things to get right here.
Classification belongs to the code that detects, but the severity isn’t fixed by the detector. The SPI bus manager knows a transfer timed out. That’s the detection, and it’s transient or recoverable. Whether one dead sensor is degrading or critical depends on what that sensor does, and only the application knows. So drivers report what happened; the fault policy maps it to severity.
That mapping should live in one table, at the application layer, where it can be read in a review:
static const fault_policy_t policy[] = {
{ SRC_SPI, RES_TIMEOUT, CLASS_RECOVERABLE, DOMAIN_SENSING },
{ SRC_SPI, RES_NO_DEVICE, CLASS_DEGRADING, DOMAIN_SENSING },
{ SRC_ADC, RES_TIMEOUT, CLASS_CRITICAL, DOMAIN_CONTROL },
{ SRC_CONFIG, RES_SEQUENCE, CLASS_FATAL, DOMAIN_SYSTEM },
};Notice SRC_ADC, RES_TIMEOUT is critical while the same error from SPI is recoverable. Same failure, different meaning, because the ADC feeds the control loop and the SPI sensors don’t. That judgement is a product decision and it belongs in a product-layer table, not scattered across drivers.
Degraded is per capability, not global. A single DEGRADED flag tells an operator nothing and tells your telemetry less. What you need is which capability is gone:
uint32_t degraded_mask; /* bit per capability */
#define CAP_SENSING_FULL (1u << 0)
#define CAP_CLOUD_TELEM (1u << 1)
#define CAP_MODBUS_TCP (1u << 2)One word, and a field engineer can tell the difference between “lost cloud connectivity” and “running on three of four sensors” without a site visit.
Recovery Has a Worst-Case Execution Time
This is the part most treatments of this topic leave out, and it’s the one that turns a recovery mechanism into an outage.
Article 05 built a schedulability analysis from task periods and execution budgets. Every number in that table was a normal-path number. Then a peripheral fails, recovery runs, and recovery’s execution time is nowhere in the analysis.
Work it out for I2C bus recovery on the KIC-400’s EEPROM:
Detect timeout 25 ms (the configured bus timeout)
Retry once 25 ms
Nine clock pulses at 100 kHz 90 µs
Generate STOP ~20 µs
Peripheral reset and reinit ~200 µs
Re-read the page ~5 ms
--------
Worst case ~55 msFifty-five milliseconds of latency for the config service. What it costs everyone else depends on how the driver waits, and the two cases give very different answers.
Polling driver. If the I2C driver busy-polls its timeouts, as ST’s HAL does in polling mode, the System task, which hosts the config service at priority 2, holds the CPU for the full 55 ms. Everything below it waits, which here is Logging. Firmware Update shares priority 2 and gets time only if configUSE_TIME_SLICING is on. Call the same polling driver from a task at priority 5 and every task below it stalls for 55 ms, Modbus TCP included, whose 100 ms deadline then survives only just.
Blocking driver. If the driver is interrupt or DMA driven and waits on an RTOS primitive, the two 25 ms timeouts are spent blocked and lower tasks run normally. The CPU cost is then roughly 110 µs of bit-banging plus about 200 µs of reinitialisation.
Same recovery, two very different numbers. Know which one your driver gives you before you put it in the analysis.
Three rules fall out, and they’re the practical content of this section:
Recovery runs at the priority of the thing being recovered, never higher. Config EEPROM recovery is a config-service concern and belongs at the config service’s priority. The temptation to raise it (“this is urgent, we need the bus back”) is exactly backwards. Recovery is never more urgent than the deadlines it would displace.
Recovery’s WCET goes in the task table. Add a column. If a task’s recovery path is longer than its period, that task cannot recover in-line and needs to hand off to a lower-priority recovery context.
The control task never recovers anything. Article 02 put it at the top with a hard 1 ms deadline and a never-blocks policy. That policy has no exception for recovery. If the control task’s own inputs fail, it uses last-known-good with an age stamp and posts a request. Someone else does the work.
Getting this wrong produces the most frustrating class of field failure: a device that was handling a minor peripheral glitch fine until the recovery logic caused a deadline miss, which triggered a watchdog, which reset a device that was never actually broken.
Progressive Recovery, Worked Properly
Article 02 gave the escalation ladder: count, log, retry once, reset peripheral, degrade, isolate, controlled reset. Here’s what one rung actually looks like when you build it, because “reset the peripheral” hides real work.
I2C bus recovery
I2C deserves the example because its failure mode is genuinely nasty: a slave interrupted mid-byte can hold SDA low indefinitely, and no amount of resetting the master peripheral fixes it, because the bus is stuck outside the master.
The sequence that actually works:
result_t i2c_bus_recover(i2c_bus_t *bus)
{
/* 1. Release the peripheral's hold on the pins */
i2c_peripheral_disable(bus);
gpio_set_mode(bus->scl, GPIO_OUTPUT_OD);
gpio_set_mode(bus->sda, GPIO_INPUT);
/* 2. Nine clocks lets a stuck slave finish its byte and see a NACK */
for (int i = 0; i < 9; i++) {
gpio_write(bus->scl, 0); delay_us(5);
gpio_write(bus->scl, 1); delay_us(5);
if (gpio_read(bus->sda)) break; /* released early */
}
if (!gpio_read(bus->scl)) {
return RES_NO_DEVICE; /* a slave holds SCL low: clocking cannot fix it */
}
/* 3. Manual STOP: SCL low, SDA low, then SDA low→high while SCL is high */
gpio_write(bus->scl, 0); delay_us(5);
gpio_set_mode(bus->sda, GPIO_OUTPUT_OD);
gpio_write(bus->sda, 0); delay_us(5);
gpio_write(bus->scl, 1); delay_us(5);
gpio_write(bus->sda, 1); delay_us(5);
/* 4. Hand the pins back and reinitialise */
gpio_set_mode(bus->scl, GPIO_AF_I2C);
gpio_set_mode(bus->sda, GPIO_AF_I2C);
i2c_peripheral_init(bus);
return gpio_read(bus->sda) ? RES_OK : RES_NO_DEVICE;
}Nine clocks because the slave may be anywhere in a byte plus its ACK, so nine guarantees it completes the transfer and sees a NACK. The manual STOP because the peripheral can’t generate one while the bus is held. One detail the listing leaves to gpio_set_mode(): write the output data bits high before switching a pin from alternate function to open-drain output, or SCL glitches low on the way in and can confuse the slave you are trying to rescue.
Two things worth saying plainly about this, because most write-ups of the nine-clock trick stop before them.
It doesn’t always work. If the slave has genuinely latched up (brownout mid-transaction, ESD event), clocking won’t recover it. Which means your architecture needs a rung below peripheral reset: power-cycling the device. If the EEPROM’s supply isn’t switchable, that rung doesn’t exist and the honest escalation is straight to marking the device offline. Find that out at schematic review, not in the field. A load switch on a bus-critical device is one BOM line and it’s the difference between a recoverable fault and a truck roll.
The recovery itself must be bounded. Those delay_us(5) calls total at most 110 µs of busy-wait: 90 µs for nine clocks and 20 µs for the STOP. Acceptable once. Called in a loop by a retry policy that doesn’t count, it’s a task that stops meeting its deadline while nominally working.
Recovery Storms
Correlated failure is underrated, and it produces outages that look inexplicable in the logs.
A brownout dips the rail for 40 ms. Three sensors NAK, the EEPROM times out, the Ethernet PHY drops link. Five subsystems detect failure within the same millisecond. Each starts its retry policy. Each logs. Each requests a degrade evaluation. The CPU, which was at 27% utilisation, is now saturated with recovery, and the control task, which was fine because its ADC path is in DTCM and never noticed, misses a deadline because everything below it is thrashing.
The device resets. On reboot, the rail is fine, everything works, and the log shows five simultaneous failures and a watchdog reset with no obvious cause.
Three mitigations:
Jitter your backoff. Exponential backoff with a fixed constant makes correlated failures retry in lockstep forever, every subsystem retrying at 10 ms, then 50 ms, then 250 ms, together. Add randomness:
uint32_t backoff_ms(uint8_t attempt)
{
if (attempt > 8u) attempt = 8u; /* shifting by 32 or more is undefined */
uint32_t base = 10u << attempt; /* 10, 20, 40, 80... */
if (base > 2000u) base = 2000u;
return base + (rand_u32() % (base / 2u + 1u)); /* +0 to 50% jitter */
}This is the thundering-herd fix from distributed systems, and it applies directly to a device with several independent recovery loops.
Rate-limit recovery globally. One recovery action at a time, system-wide, arbitrated by whoever owns fault policy. Five subsystems needing recovery isn’t five parallel recoveries. It’s a queue.
Correlate before you act. Five failures in the same millisecond across unrelated domains is not five independent faults. It’s one cause: usually power, sometimes clock, occasionally a shared reset line. The right response is to record the correlation and treat it as a single event, not to run five recovery procedures for a problem none of them addresses.
That last one needs almost no code: a timestamp on each fault record and a check for how many arrived inside the same few milliseconds. It turns an incomprehensible log into a diagnosis.
The Watchdog, Supervised
A watchdog kicked from the main loop tells you the main loop is running. That’s all it tells you, and it’s rarely the thing you need to know.
for (;;) {
do_work();
watchdog_kick(); /* Task A died twenty minutes ago. Nobody noticed. */
}Per-task deadlines, one gate
Each supervised task checks in. The health monitor kicks the hardware watchdog only if every supervised task has checked in within its own deadline:
typedef struct {
volatile uint32_t last_checkin_ms; /* written by the task, read by the monitor */
uint32_t deadline_ms; /* generous multiple of the task's period */
uint32_t missed_count;
} task_health_t;
static bool all_healthy(uint32_t now)
{
for (int i = 0; i < SUPERVISED_COUNT; i++) {
if ((now - health[i].last_checkin_ms) > health[i].deadline_ms) {
fault_emit(SRC_HEALTH, RES_FAULT, i); /* record WHICH task */
return false;
}
}
return true;
}Recording which task failed before the reset is the entire value. Without it, a watchdog reset tells you the device died. With it, you know the sensor task stopped checking in, which is a diagnosis.
Two details make this production-ready. Emit the fault once, on the transition from healthy to unhealthy, not on every 100 ms tick while it stays that way, or the fault ring fills with copies of one event. And arm supervision per task only after its first check-in, so a task that legitimately starts late in the boot sequence does not trip the monitor before it has run.
Set deadlines generously, several times the task’s period, because a health monitor that trips on normal jitter is worse than no health monitor, and it will be disabled by the second engineer who gets paged about it.
Two watchdogs, two jobs
On an STM32H7 you have both, and they catch different failures.
IWDG runs from the LSI, an independent RC oscillator. It survives main clock failure, it can’t be stopped once started, and it’s your backstop. This is the one the health monitor feeds.
WWDG runs from the APB clock and has a window: kicking too early also resets you. That catches a different bug, a loop spinning fast because a state machine is thrashing, which a plain watchdog reads as perfect health. It only sees the loop that kicks it, though. On the KIC-400 that is the Control task, which kicks the WWDG once per 1 ms cycle, so a control loop that runs early or late lands outside the window. A different task spinning fast is invisible to the WWDG, because it never touches it.
The monitor can hang too
The obvious hole. If the health monitor task is the only thing kicking the watchdog and the monitor itself blocks, nothing detects it.
The answer isn’t a second monitor, because that regresses infinitely. It’s that the IWDG period must be short enough that a hung monitor resets the device on its own. The monitor runs at 100 ms with an IWDG timeout of 1 second; the monitor gets ten chances, and if it’s genuinely stuck the hardware handles it without any software involvement. That’s why IWDG’s independence from the main clock matters: it’s the one recovery mechanism that doesn’t depend on any of your code being correct.
Three H7 details belong in the same calculation. The IWDG keeps counting while the core is halted at a breakpoint unless its debug freeze bit is set (DBG_IWDG1 in DBGMCU->APB4FZ1), so set it in debug builds or every breakpoint becomes a reset. It keeps running in Stop and Standby unless the FZ_IWDG_STOP and FZ_IWDG_SDBY option bits are cleared, which matters the day someone adds a low-power mode. And the LSI is an RC oscillator whose frequency moves with voltage and temperature, so compute the timeout at the fastest LSI the datasheet allows, which gives the shortest real timeout, and check that the monitor’s worst-case feed interval still fits inside it.
One interaction to check against article 02. A sector erase on the H7 stalls fetches from that bank until it completes, and the H743 datasheet (DS12110) specifies a 128 KB sector erase in seconds, not milliseconds: a maximum of about 4 s at x8 parallelism, with typical figures of roughly 1 to 2 s depending on the parallelism used. If an update ever writes metadata into the bank you are running from, a perfectly healthy device misses a 1 second IWDG window in the middle of an update. Keep update metadata out of the active bank, or run everything on the path to the feed from ITCM or RAM for the duration of the erase: the vector table, the tick interrupt, the scheduler, the health monitor and the code that kicks the IWDG, since any one of them fetching from the stalled bank stalls the feed. Failing both, size the IWDG above the datasheet maximum erase time, not the typical one, and write down why.
Preserving Evidence Across a Reset
A reset that destroys the evidence turns a diagnosable failure into a permanent mystery. Article 02 reserved 4 KB of backup SRAM for this. Here’s what goes in it.
The crash record
typedef struct __attribute__((aligned(32))) { /* whole cache lines */
uint32_t magic; /* validity marker */
uint32_t boot_count;
uint32_t reset_reason; /* latched from RCC */
uint8_t fault_class;
uint8_t fault_source;
uint16_t fault_code;
uint32_t pc; /* where it died */
uint32_t lr;
uint32_t exc_return; /* LR on entry: stack, mode */
uint32_t cfsr; /* configurable fault status*/
uint32_t hfsr; /* hard fault status */
uint32_t mmfar; /* MemManage fault address */
uint32_t bfar; /* bus fault address */
char task_name[8];
uint32_t stack_free[SUPERVISED_COUNT];
uint32_t crc;
} crash_record_t;Capturing it from a fault handler
You’re in a HardFault with a stack that may itself be the problem. If MSP has overflowed, the first C function that pushes a frame faults again and the core locks up, so the naked stub switches to a small reserved fault stack before it calls any C. From there the discipline is: drive the outputs safe, read the stacked frame, write fixed-size fields, reset. No allocation, no RTOS or HAL calls, nothing that can block or fault again. The only calls are short, bounded routines you own, here the record CRC and the CMSIS cache clean, and both run on the reserved stack.
/* Naked: no prologue, so the stack pointer is exactly where the
hardware left the exception frame. */
__attribute__((naked)) void HardFault_Handler(void)
{
__asm volatile (
"tst lr, #4 \n" /* EXC_RETURN bit 2 picks MSP or PSP */
"ite eq \n"
"mrseq r0, msp \n"
"mrsne r0, psp \n"
"mov r1, lr \n" /* EXC_RETURN, for the record */
"ldr r2, =_fault_stack_top \n" /* MSP may be the stack that overflowed, */
"mov sp, r2 \n" /* so switch to the reserved fault stack */
"b hardfault_capture \n"
);
}
void hardfault_capture(uint32_t *sp, uint32_t exc_return)
{
/* Safe state first, before any logging: direct register stores, no HAL */
outputs_force_safe();
/* A corrupted SP faults again here and locks the core up: check it first */
if (sp_in_ram(sp, 8u)) {
/* Stacked: R0 R1 R2 R3 R12 LR PC xPSR */
crash.pc = sp[6];
crash.lr = sp[5];
}
crash.exc_return = exc_return;
crash.cfsr = SCB->CFSR;
crash.hfsr = SCB->HFSR;
crash.mmfar = SCB->MMFAR;
crash.bfar = SCB->BFAR;
crash.magic = CRASH_MAGIC;
crash.crc = crc32_record(&crash);
/* Backup SRAM is cacheable by default on the M7: push the record out */
SCB_CleanDCache_by_Addr((uint32_t *)&crash, sizeof crash);
NVIC_SystemReset();
}The entry is naked for a reason. In an ordinary C handler the compiler pushes registers before your assembly runs, which moves MSP, and for a fault taken on the main stack sp[6] is then no longer the stacked PC. The stub also passes EXC_RETURN along and loads _fault_stack_top, a few hundred bytes the linker script reserves for this handler alone, into SP before any C runs. In Handler mode SP is MSP, so the capture function, the CRC and the cache clean all run on a stack nothing else can have overflowed, while r0 still points at the original frame. The crash record also has to live in backup SRAM with its clock enabled and backup domain writes unlocked at startup (BKPRAMEN in RCC->AHB4ENR, DBP in PWR->CR1), or these stores are silently dropped. To keep the record through Standby, or through loss of VDD on a board with VBAT fitted, also enable the backup regulator (BREN in PWR->CR2, then wait for BRRDY); without it, backup SRAM keeps its contents only while VDD is present.
outputs_force_safe() is a handful of direct stores to the output peripheral’s registers that drive both analog outputs to their safe values, written without the HAL so it cannot depend on driver state the fault may have corrupted. That keeps the rule from the top of this article, safe outputs before anything else including logging, true on the HardFault path. The reset that follows hands the pins back to their reset state, so the hardware has to make that state safe as well.
There is a second trap on the M7, and it is easier to miss. Backup SRAM sits at 0x38800000, inside the default memory map’s SRAM region, which is write-back cacheable once the D-cache is on. A crash record or a boot counter written there can sit in a dirty cache line, and NVIC_SystemReset() does not clean the cache. The record you were careful to capture never reaches the memory that survives. Either clean the lines before every reset, as the handler above does (which is why crash_record_t is aligned to 32 bytes, the M7 cache line size: the clean then covers whole lines and shares none with a neighbouring variable), or map the whole region non-cacheable in the MPU at startup, which also covers the boot counter and every other write:
/* Backup SRAM: Normal, non-cacheable (TEX=1, C=0, B=0), execute never, 4 KB */
ARM_MPU_SetRegionEx(MPU_REGION_BKPSRAM, 0x38800000u,
ARM_MPU_RASR(1u, ARM_MPU_AP_FULL, 1u, 0u, 0u, 0u, 0x00u, ARM_MPU_REGION_SIZE_4KB));Then verify it on hardware rather than trusting it: force a HardFault on a HIL rig, let the device reset, and read the record back through the diagnostic interface.
Two stacked words and CFSR usually locate the faulting instruction in a map file, with one exception that is common on the M7. Most write bus faults there are imprecise (IMPRECISERR set, BFARVALID clear): the store went through the write buffer, so BFAR holds no valid address and the stacked PC points somewhere past the store that caused it. The Cortex-M3/M4 trick of disabling write buffering with DISDEFWBUF in ACTLR has no equivalent on the M7, so while debugging, put a __DSB() after the suspect stores to make the fault arrive at the store, and narrow down from there. For precise faults, the stacked PC is usually the whole investigation.
Two refinements for production. The record captures MMFAR as well as BFAR; treat each address as meaningful only when its valid bit is set in CFSR (MMARVALID and BFARVALID). And enable MemManage, BusFault and UsageFault in SCB->SHCSR, routing all four handlers to the same capture, so the record says which fault fired instead of a forced HardFault.
Reading the reset reason, once
The reset flags accumulate until you explicitly clear them. Read them in startup, before anything else, then clear. Otherwise a watchdog reset from six boots ago sends you chasing a watchdog that isn’t firing.
STM32H7 register note: on this family the reset flags live in
RCC->RSRand are cleared with theRMVFbit, notRCC->CSRas on F1/F4.
The Boot-Failure Counter, and Its Hard Part
Preventing a reset loop is well understood in outline. Count boots, and if the count exceeds a threshold, enter recovery mode.
The hard part, and the one rarely spelled out, is when do you clear the counter?
Clear it at boot and you never detect a loop, because every boot resets the evidence. Never clear it and a unit that had three bad boots during commissioning in 2024 drops into recovery mode in 2029 on its eleventh unrelated reset.
The rule that works:
Increment before the risky part. Clear only after the system has demonstrated it’s actually working.
Demonstrated needs a definition, and it should be a real one:
/* early in startup. Backup SRAM is undefined after a cold boot or a
power loss without VBAT, so validate the record before trusting it. */
if (crash.magic != CRASH_MAGIC || crc32_record(&crash) != crash.crc) {
memset(&crash, 0, sizeof crash); /* cold boot or corrupt record */
crash.magic = CRASH_MAGIC;
}
crash.boot_count++;
crash.crc = crc32_record(&crash); /* keep the record valid */
if (crash.boot_count > BOOT_FAIL_THRESHOLD) {
enter_recovery_mode(); /* does not return */
}
/* later, from the health monitor, only after real evidence */
void health_monitor_tick(void)
{
uint32_t now = uptime_ms();
if (now > STABLE_UPTIME_MS /* 60 s, say */
&& all_healthy(now) /* every task alive */
&& control_loop_deadlines_met()) { /* doing the job */
crash.boot_count = 0;
crash.crc = crc32_record(&crash);
}
}Sixty seconds of every task checking in and the control loop meeting deadlines. That’s a device that’s working, not a device that reached main().
It also means a device that boots, runs for forty seconds, and crashes will accumulate its counter correctly and eventually stop trying, which is exactly the failure a boot counter exists to catch and exactly the one that “clear on successful boot” misses.
One gap remains. A counter in backup SRAM survives a reset, but not a power cycle unless VBAT is fitted, and brownout-driven boot loops are exactly the field case that power-cycles the part. Keep a second counter where the bootloader can see it across power loss: an EEPROM byte, or on the H7 one 32-byte flash word per increment in a sector the bootloader owns, erased when the application confirms it is healthy.
Recovery mode
Minimal firmware. Enough to be diagnosed and repaired remotely, and nothing else:
- Read out the crash records
- Accept a firmware update
- Reset configuration to factory defaults
- Roll back to the previous image
No control outputs, no cloud connection, no protocol stack beyond what the update needs. The smaller it is, the more likely it works when everything else didn’t. It shares its flash slot and signature verification with the bootloader, which is article 12.
The Recovery Matrix
One table, maintained as an architecture artifact, reviewed when anything on it changes. For the KIC-400:
| Failure | Detection | Class | First action | Escalation | Final state |
|---|---|---|---|---|---|
| Modbus CRC error | CRC check | Transient | Count, drop frame | none | Continue |
| SPI sensor timeout | Bus timeout | Recoverable | Retry once, reset SPI2 | 3 in 60 s → mark lost | Degraded (sensing) |
| Sensor not responding | NACK, no ID | Degrading | Mark offline | none | Degraded (sensing) |
| I2C EEPROM stuck | Bus timeout | Recoverable | 9-clock recovery | Fails → config read-only | Degraded (config) |
| Config CRC bad | CRC on load | Recoverable | Load second copy | Both bad → factory defaults | Continue + fault |
| Config sequence back | Sequence check | Fatal | Safe outputs, record | none | Reset |
| ADC DMA stalled | No buffer in 3 ms | Critical | Safe outputs | none | Safe state |
| Task stall | Health monitor | Critical | Record which task | none | Watchdog reset |
| HardFault | Fault handler | Fatal | Safe outputs, capture PC/LR/CFSR | none | Reset |
| 5 consecutive bad boots | Boot counter | Fatal | none | none | Recovery mode |
Two rows are worth looking at twice. Config CRC failure is recoverable because there’s a second copy, which is what article 02’s two-copy design bought. A sequence number moving backwards is fatal despite a valid CRC, because valid-but-impossible means something wrote memory it shouldn’t have, and nothing after that point is trustworthy.
Five Questions Per Error Path
If you can’t answer all five for a given failure, that path isn’t finished:
- How is it detected? A timeout, a CRC, a range check, a health deadline. If the answer is “it isn’t,” nothing else matters.
- How is it classified, and which table says so?
- What’s the first action, and what does it cost? In milliseconds, at which priority.
- What happens when that fails? Every recovery needs a defined next rung and a bounded attempt count.
- What survives to explain it later? A counter, a fault record, a crash record, a reset reason.
Question three is the one that gets skipped, and it’s the one that turns recovery into the outage.
Next
Everything above assumes failures are accidental. Peripherals glitch, rails dip, cosmic rays flip bits, and the architecture responds.
Some failures aren’t accidental. An attacker who can reach the Modbus TCP port, the MQTT connection, or the update endpoint is a fault source that reads your recovery documentation and picks the path that helps them. Retry limits become denial of service. Error messages become an oracle. Recovery mode becomes a way in, because it’s the smallest firmware with the fewest checks.
Article 08: Firmware Security Architecture: secure boot, chain of trust, key management, and designing for an adversary rather than an accident.
References
- Armv7-M Architecture Reference Manual
- Arm Cortex-M7 Technical Reference Manual
- STM32H743ZI documentation (RM0433 reference manual and datasheet)
- NXP UM10204, I2C-bus specification and user manual
Series Index
Overview, reading paths and related articles: Embedded Firmware Architecture series guide.
Phase 1: Decide
01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture
03. Firmware Architecture Anti-Patterns
Phase 2: Build
04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture
Phase 3: Fail safely
07. Error Handling & Recovery (you are here)
08. Firmware Security Architecture
Phase 4: Verify
09. Designing Firmware for Testability
Phase 5: Operate
10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates
