Firmware Error Handling and Recovery Architecture: Designing Embedded Systems That Know How to Fail

Every state machine in article 06 had a FAULT state. Every one of them was a box with nothing in it.

That’s not an accident of the writing. It’s how most firmware is actually built. The fault path is the least-exercised code in the product, written last, reviewed least, and executed only when conditions are already bad — which is precisely when you need it to be the most dependable code you have.

This article is about what goes in that box. Not error propagation — article 02 built result_t and the layered model, and article 03 covered what happens when errors get swallowed. This is the part after detection: classification that actually drives behaviour, recovery that’s budgeted rather than hoped for, and preserving enough evidence that the failure is diagnosable from a log file in another country.

---

Errors and Faults Are Different Things

Article 03 drew this line and deferred it here. It’s the most consequential distinction in failure handling, and conflating the two is why devices limp along in corrupted states.

An error is an expected outcome you have a plan for. The sensor timed out. A Modbus frame had a bad CRC. The socket closed. These are things the physical world does. Count them, handle them, carry on.

A fault is a broken invariant. The config CRC is valid but the sequence number moved backwards. A stack canary is gone. An MPU violation fired. A queue that the design says can never fill is full.

The test that separates them, and it’s a single question:

Can the system correctly continue?

If yes, it’s an error, and recovery is appropriate. If no — if continuing means operating on state you can’t trust — it’s a fault, and running recovery is the wrong move. Recovery assumes the system is sound and something external failed. When the system itself is unsound, recovery executes on a corrupted foundation and produces worse outcomes than stopping.

So faults get a different path entirely: record what you can, reach the safe state, reset. No retry, no degrade, no attempt to be clever. Article 02’s KIC-400 drives two analog outputs; a fault means those outputs go to their defined safe value before anything else happens, including logging.

---

Classification That Drives Behaviour

Five severity levels, and the reason for five rather than three is that each maps to a different first action. A classification that doesn’t change what happens next isn’t worth having.

Class	Meaning	First action	Example
Transient	May clear by itself	Count, continue	One bad CRC on a Modbus frame
Recoverable	Needs an action, system is sound	Retry once, then reset the peripheral	I2C timeout, sensor NAK
Degrading	Capability lost, product still useful	Isolate, mark capability unavailable	One of four sensors dead
Critical	Product can’t do its primary job	Safe state, keep diagnostics alive	Control feedback lost
Fatal	Invariant broken, state untrustworthy	Record, safe outputs, reset	Memory corruption, bad config sequence

Two things to get right here.

Classification belongs to the code that detects, but the severity isn’t fixed by the detector. The SPI bus manager knows a transfer timed out — that’s the detection, and it’s transient or recoverable. Whether one dead sensor is degrading or critical depends on what that sensor does, and only the application knows. So drivers report what happened; the fault policy maps it to severity.

That mapping should live in one table, at the application layer, where it can be read in a review:

static const fault_policy_t policy[] = {
    { SRC_SPI,    RES_TIMEOUT,   CLASS_RECOVERABLE, DOMAIN_SENSING },
    { SRC_SPI,    RES_NO_DEVICE, CLASS_DEGRADING,   DOMAIN_SENSING },
    { SRC_ADC,    RES_TIMEOUT,   CLASS_CRITICAL,    DOMAIN_CONTROL },
    { SRC_CONFIG, RES_SEQUENCE,  CLASS_FATAL,       DOMAIN_SYSTEM  },
};

Notice SRC_ADC, RES_TIMEOUT is critical while the same error from SPI is recoverable. Same failure, different meaning, because the ADC feeds the control loop and the SPI sensors don’t. That judgement is a product decision and it belongs in a product-layer table, not scattered across drivers.

Degraded is per capability, not global. A single DEGRADED flag tells an operator nothing and tells your telemetry less. What you need is which capability is gone:

uint32_t degraded_mask;   /* bit per capability */
#define CAP_SENSING_FULL   (1u << 0)
#define CAP_CLOUD_TELEM    (1u << 1)
#define CAP_MODBUS_TCP     (1u << 2)

One word, and a field engineer can tell the difference between “lost cloud connectivity” and “running on three of four sensors” without a site visit.

---

Recovery Has a Worst-Case Execution Time

This is the part that gets left out of every treatment of this topic, and it’s the one that turns a recovery mechanism into an outage.

Article 05 built a schedulability analysis from task periods and execution budgets. Every number in that table was a normal-path number. Then a peripheral fails, recovery runs, and recovery’s execution time is nowhere in the analysis.

Work it out for I2C bus recovery on the KIC-400’s EEPROM:

Detect timeout                       25 ms   (the configured bus timeout)
Retry once                           25 ms
Nine clock pulses at 100 kHz         90 µs
Generate STOP, settle               ~50 µs
Peripheral reset and reinit         ~200 µs
Re-read the page                     ~5 ms
                                   --------
Worst case                          ~55 ms

Fifty-five milliseconds. If that runs in a task at priority 5, every task below it is blocked for 55 ms, which includes Modbus TCP’s 100 ms deadline — surviving, but only just — and it means a bus glitch on a configuration EEPROM can affect protocol response times.

Three rules fall out, and they’re the practical content of this section:

Recovery runs at the priority of the thing being recovered, never higher. Config EEPROM recovery is a config-service concern and belongs at the config service’s priority. The temptation to raise it — “this is urgent, we need the bus back” — is exactly backwards. Recovery is never more urgent than the deadlines it would displace.

Recovery’s WCET goes in the task table. Add a column. If a task’s recovery path is longer than its period, that task cannot recover in-line and needs to hand off to a lower-priority recovery context.

The control task never recovers anything. Article 02 put it at the top with a hard 1 ms deadline and a never-blocks policy. That policy has no exception for recovery. If the control task’s own inputs fail, it uses last-known-good with an age stamp and posts a request. Someone else does the work.

Getting this wrong produces the most frustrating class of field failure: a device that was handling a minor peripheral glitch fine until the recovery logic caused a deadline miss, which triggered a watchdog, which reset a device that was never actually broken.

---

Progressive Recovery, Worked Properly

Article 02 gave the escalation ladder — count, log, retry once, reset peripheral, degrade, isolate, controlled reset. Here’s what one rung actually looks like when you build it, because “reset the peripheral” hides real work.

I2C bus recovery

I2C deserves the example because its failure mode is genuinely nasty: a slave interrupted mid-byte can hold SDA low indefinitely, and no amount of resetting the master peripheral fixes it, because the bus is stuck outside the master.

The sequence that actually works:

result_t i2c_bus_recover(i2c_bus_t *bus)
{
    /* 1. Release the peripheral's hold on the pins */
    i2c_peripheral_disable(bus);
    gpio_set_mode(bus->scl, GPIO_OUTPUT_OD);
    gpio_set_mode(bus->sda, GPIO_INPUT);

    /* 2. Nine clocks lets a stuck slave finish its byte and see a NACK */
    for (int i = 0; i < 9; i++) {
        gpio_write(bus->scl, 0); delay_us(5);
        gpio_write(bus->scl, 1); delay_us(5);
        if (gpio_read(bus->sda)) break;     /* released early */
    }

    /* 3. Manual STOP: SDA low→high while SCL is high */
    gpio_set_mode(bus->sda, GPIO_OUTPUT_OD);
    gpio_write(bus->sda, 0); delay_us(5);
    gpio_write(bus->scl, 1); delay_us(5);
    gpio_write(bus->sda, 1); delay_us(5);

    /* 4. Hand the pins back and reinitialise */
    gpio_set_mode(bus->scl, GPIO_AF_I2C);
    gpio_set_mode(bus->sda, GPIO_AF_I2C);
    i2c_peripheral_init(bus);

    return gpio_read(bus->sda) ? RES_OK : RES_NO_DEVICE;
}

Nine clocks because the slave may be anywhere in a byte plus its ACK, so nine guarantees it completes the transfer and sees a NACK. The manual STOP because the peripheral can’t generate one while the bus is held.

Two things worth saying plainly about this, because most write-ups of the nine-clock trick stop before them.

It doesn’t always work. If the slave has genuinely latched up — brownout mid-transaction, ESD event — clocking won’t recover it. Which means your architecture needs a rung below peripheral reset: power-cycling the device. If the EEPROM’s supply isn’t switchable, that rung doesn’t exist and the honest escalation is straight to marking the device offline. Find that out at schematic review, not in the field. A load switch on a bus-critical device is one BOM line and it’s the difference between a recoverable fault and a truck roll.

The recovery itself must be bounded. Those delay_us(5) calls total roughly 140 µs of busy-wait. Acceptable once. Called in a loop by a retry policy that doesn’t count, it’s a task that stops meeting its deadline while nominally working.

---

Recovery Storms

Correlated failure is underrated, and it produces outages that look inexplicable in the logs.

A brownout dips the rail for 40 ms. Three sensors NAK, the EEPROM times out, the Ethernet PHY drops link. Five subsystems detect failure within the same millisecond. Each starts its retry policy. Each logs. Each requests a degrade evaluation. The CPU, which was at 27% utilisation, is now saturated with recovery, and the control task — which was fine, because its ADC path is in DTCM and never noticed — misses a deadline because everything below it is thrashing.

The device resets. On reboot, the rail is fine, everything works, and the log shows five simultaneous failures and a watchdog reset with no obvious cause.

Three mitigations:

Jitter your backoff. Exponential backoff with a fixed constant makes correlated failures retry in lockstep forever — every subsystem retrying at 10 ms, then 50 ms, then 250 ms, together. Add randomness:

uint32_t backoff_ms(uint8_t attempt)
{
    uint32_t base = 10u << attempt;            /* 10, 20, 40, 80... */
    if (base > 2000u) base = 2000u;
    return base + (rand_u32() % (base / 2));   /* ±50% jitter */
}

This is the thundering-herd fix from distributed systems, and it applies directly to a device with several independent recovery loops.

Rate-limit recovery globally. One recovery action at a time, system-wide, arbitrated by whoever owns fault policy. Five subsystems needing recovery isn’t five parallel recoveries — it’s a queue.

Correlate before you act. Five failures in the same millisecond across unrelated domains is not five independent faults. It’s one cause — usually power, sometimes clock, occasionally a shared reset line. The right response is to record the correlation and treat it as a single event, not to run five recovery procedures for a problem none of them addresses.

That last one needs almost no code: a timestamp on each fault record and a check for how many arrived inside the same few milliseconds. It turns an incomprehensible log into a diagnosis.

---

The Watchdog, Supervised

A watchdog kicked from the main loop tells you the main loop is running. That’s all it tells you, and it’s rarely the thing you need to know.

for (;;) {
    do_work();
    watchdog_kick();   /* Task A died twenty minutes ago. Nobody noticed. */
}

Per-task deadlines, one gate

Each supervised task checks in. The health monitor kicks the hardware watchdog only if every supervised task has checked in within its own deadline:

typedef struct {
    uint32_t last_checkin_ms;
    uint32_t deadline_ms;      /* generous multiple of the task's period */
    uint32_t missed_count;
} task_health_t;

static bool all_healthy(uint32_t now)
{
    for (int i = 0; i < SUPERVISED_COUNT; i++) {
        if ((now - health[i].last_checkin_ms) > health[i].deadline_ms) {
            fault_emit(SRC_HEALTH, RES_FAULT, i);   /* record WHICH task */
            return false;
        }
    }
    return true;
}

Recording which task failed before the reset is the entire value. Without it, a watchdog reset tells you the device died. With it, you know the sensor task stopped checking in, which is a diagnosis.

Set deadlines generously — several times the task’s period — because a health monitor that trips on normal jitter is worse than no health monitor, and it will be disabled by the second engineer who gets paged about it.

Two watchdogs, two jobs

On an STM32H7 you have both, and they catch different failures.

IWDG runs from the LSI, an independent RC oscillator. It survives main clock failure, it can’t be stopped once started, and it’s your backstop. This is the one the health monitor feeds.

WWDG runs from the APB clock and has a window — kicking too early also resets you. That catches a different bug: a task that’s spinning fast through its loop because a state machine is thrashing, which a plain watchdog reads as perfect health.

The monitor can hang too

The obvious hole. If the health monitor task is the only thing kicking the watchdog and the monitor itself blocks, nothing detects it.

The answer isn’t a second monitor — that regresses infinitely. It’s that the IWDG period must be short enough that a hung monitor resets the device on its own. The monitor runs at 100 ms with an IWDG timeout of 1 second; the monitor gets ten chances, and if it’s genuinely stuck the hardware handles it without any software involvement. That’s why IWDG’s independence from the main clock matters — it’s the one recovery mechanism that doesn’t depend on any of your code being correct.

---

Preserving Evidence Across a Reset

A reset that destroys the evidence turns a diagnosable failure into a permanent mystery. Article 02 reserved 4 KB of backup SRAM; article 03 noted it was never used. Here’s what goes in it.

The crash record

typedef struct {
    uint32_t magic;              /* validity marker          */
    uint32_t boot_count;
    uint32_t reset_reason;       /* latched from RCC         */
    uint8_t  fault_class;
    uint8_t  fault_source;
    uint16_t fault_code;
    uint32_t pc;                 /* where it died            */
    uint32_t lr;
    uint32_t cfsr;               /* configurable fault status*/
    char     task_name[8];
    uint32_t stack_free[SUPERVISED_COUNT];
    uint32_t crc;
} crash_record_t;

Capturing it from a fault handler

You’re in a HardFault with a stack that may itself be the problem. The discipline is: no library calls, no allocation, nothing that could fault again. Read the stacked frame, write fixed-size fields, reset.

void HardFault_Handler(void)
{
    uint32_t *sp;
    __asm volatile (
        "tst lr, #4      \n"   /* EXC_RETURN bit 2 picks MSP or PSP */
        "ite eq          \n"
        "mrseq %0, msp   \n"
        "mrsne %0, psp   \n"
        : "=r" (sp) :: "memory");

    /* Stacked: R0 R1 R2 R3 R12 LR PC xPSR */
    crash.pc   = sp[6];
    crash.lr   = sp[5];
    crash.cfsr = SCB->CFSR;
    crash.magic = CRASH_MAGIC;
    crash.crc  = crc32_record(&crash);

    NVIC_SystemReset();
}

Two stacked words and CFSR, and you can locate the faulting instruction in a map file. That’s usually the whole investigation.

Reading the reset reason, once

The reset flags accumulate until you explicitly clear them. Read them in startup, before anything else, then clear — otherwise a watchdog reset from six boots ago sends you chasing a watchdog that isn’t firing.

STM32H7 register note: on this family the reset flags live in RCC->RSR and are cleared with the RMVF bit, not RCC->CSR as on F1/F4. Verify the exact flag names against RM0433 for your part.

---

The Boot-Failure Counter, and Its Hard Part

Preventing a reset loop is well understood in outline. Count boots, and if the count exceeds a threshold, enter recovery mode.

The hard part, which almost nobody states, is when do you clear the counter?

Clear it at boot and you never detect a loop — every boot resets the evidence. Never clear it and a unit that had three bad boots during commissioning in 2024 drops into recovery mode in 2029 on its eleventh unrelated reset.

The rule that works:

Increment before the risky part. Clear only after the system has demonstrated it’s actually working.

Demonstrated needs a definition, and it should be a real one:

/* early in startup */
crash.boot_count++;
if (crash.boot_count > BOOT_FAIL_THRESHOLD) {
    enter_recovery_mode();     /* does not return */
}

/* later, from the health monitor — only after real evidence */
void health_monitor_tick(void)
{
    if (uptime_ms() > STABLE_UPTIME_MS          /* 60 s, say         */
        && all_healthy(now)                      /* every task alive  */
        && control_loop_deadlines_met()) {       /* doing the job     */
        crash.boot_count = 0;
    }
}

Sixty seconds of every task checking in and the control loop meeting deadlines. That’s a device that’s working, not a device that reached main().

It also means a device that boots, runs for forty seconds, and crashes will accumulate its counter correctly and eventually stop trying — which is exactly the failure a boot counter exists to catch and exactly the one that “clear on successful boot” misses.

Recovery mode

Minimal firmware. Enough to be diagnosed and repaired remotely, and nothing else:

• Read out the crash records
• Accept a firmware update
• Reset configuration to factory defaults
• Roll back to the previous image

No control outputs, no cloud connection, no protocol stack beyond what the update needs. The smaller it is, the more likely it works when everything else didn’t — and it shares its flash slot and signature verification with the bootloader, which is article 12.

---

The Recovery Matrix

One table, maintained as an architecture artifact, reviewed when anything on it changes. For the KIC-400:

Failure	Detection	Class	First action	Escalation	Final state
Modbus CRC error	CRC check	Transient	Count, drop frame	none	Continue
SPI sensor timeout	Bus timeout	Recoverable	Retry once, reset SPI2	3 in 60 s → mark lost	Degraded (sensing)
Sensor not responding	NACK, no ID	Degrading	Mark offline	—	Degraded (sensing)
I2C EEPROM stuck	Bus timeout	Recoverable	9-clock recovery	Fails → config read-only	Degraded (config)
Config CRC bad	CRC on load	Recoverable	Load second copy	Both bad → factory defaults	Continue + fault
Config sequence back	Sequence check	Fatal	Safe outputs, record	—	Reset
ADC DMA stalled	No buffer in 3 ms	Critical	Safe outputs	—	Safe state
Task stall	Health monitor	Critical	Record which task	—	Watchdog reset
HardFault	Fault handler	Fatal	Capture PC/LR/CFSR	—	Reset
5 consecutive bad boots	Boot counter	Fatal	—	—	Recovery mode

Two rows are worth looking at twice. Config CRC failure is recoverable because there’s a second copy — that’s what article 02’s two-copy design bought. A sequence number moving backwards is fatal despite a valid CRC, because valid-but-impossible means something wrote memory it shouldn’t have, and nothing after that point is trustworthy.

---

Five Questions Per Error Path

If you can’t answer all five for a given failure, that path isn’t finished:

1. How is it detected? A timeout, a CRC, a range check, a health deadline. If the answer is “it isn’t,” nothing else matters.
2. How is it classified, and which table says so?
3. What’s the first action, and what does it cost? In milliseconds, at which priority.
4. What happens when that fails? Every recovery needs a defined next rung and a bounded attempt count.
5. What survives to explain it later? A counter, a fault record, a crash record, a reset reason.

Question three is the one that gets skipped, and it’s the one that turns recovery into the outage.

---

Next

Everything above assumes failures are accidental. Peripherals glitch, rails dip, cosmic rays flip bits, and the architecture responds.

Some failures aren’t accidental. An attacker who can reach the Modbus TCP port, the MQTT connection, or the update endpoint is a fault source that reads your recovery documentation and picks the path that helps them. Retry limits become denial of service. Error messages become an oracle. Recovery mode becomes a way in, because it’s the smallest firmware with the fewest checks.

Article 08: Firmware Security Architecture — secure boot, chain of trust, key management, and designing for an adversary rather than an accident.

---

Series Index

Phase 1 — Architecture

1. Embedded Firmware Architecture Fundamentals
2. Production-Grade Firmware Architecture
3. Firmware Architecture Anti-Patterns

Phase 2 — Implementation

4. HAL vs BSP vs Drivers vs Middleware
5. RTOS Architecture
6. State Machine Architecture

Phase 3 — Failure & Resilience

7. Error Handling & Recovery (you are here)
8. Firmware Security Architecture

Phase 4 — Verification

9. Designing Firmware for Testability

Phase 5 — Product Lifecycle

10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates