How to Design a Production-Grade Firmware Architecture

Part 2 of 12 in the Embedded Firmware Architecture series · Phase 1: Decide · Level: Advanced

Article 01 was about how to decide. This one is about the production firmware architecture you actually hand to a team.

So we’re going to build one. Not a diagram of one. A production firmware architecture with named tasks, assigned priorities, a memory map, a stack budget, an ownership matrix, and a defined answer to what happens when the SPI sensor stops responding at 3 a.m. in a substation.

The product is made up. Everything else is the way I’d really do it.

---

The Product

KIC-400 Industrial IoT Controller. Panel-mounted, DIN rail, sold into water treatment and building automation. Ten-year field life, no scheduled maintenance visits.

What it has to do:

• 8 × 4 to 20 mA analog inputs, 16-bit, sampled at 1 kHz
• 4 × SPI pressure and temperature sensors on a shared bus, 100 Hz
• One PID control loop at 1 kHz driving two analog outputs
• Modbus RTU slave on RS-485, 115200 baud
• Modbus TCP server on Ethernet
• MQTT over TLS to a cloud broker
• Configuration in external EEPROM, survives power loss
• OTA firmware update over Ethernet, with rollback
• Watchdog, diagnostics, field-readable fault history

Silicon: STM32H743ZI. Cortex-M7 at 480 MHz, which needs revision V silicon and VOS0. Revision Y caps at 400 MHz, and it is still in circulation, so check before you spend the cycles. 2 MB flash in two 1 MB banks, about 1 MB of RAM across six regions, built-in Ethernet MAC.

That part choice isn’t neutral, and I want to be honest about why. Dual-bank flash means A/B update lives entirely in internal flash with no external QSPI part and no extra BOM line. If you’d landed on a 1 MB single-bank H723 instead, the download slot goes to external QSPI, the bootloader gets a driver it didn’t need, and the update story gets meaningfully harder. That’s article 01’s flash-occupancy rule arriving as a purchase order.

---

Section 1: Start From Requirements

Nobody skips this step on purpose. They skip it because the schematic showed up and there was code to write.

Each arrow is a derivation. If you can’t point at the requirement that produced an architectural decision, you invented it, and you’ll defend it for years without knowing why.

The version that matters isn’t the marketing spec. It’s the timing and failure spec, written down with IDs, because those IDs are what your tests trace back to later.

ID	Requirement	Type	Architectural consequence
REQ-01	Control loop executes at 1 kHz, jitter < 50 µs	Hard timing	Timer-driven, highest priority, must never block
REQ-02	ADC sampled at 1 kHz, no missed samples	Hard timing	Timer-triggered DMA, zero CPU in the sample path
REQ-03	Modbus RTU frame detected after 1.75 ms silence	Hard timing	USART receiver timeout interrupt, not a polled timer
REQ-04	Modbus TCP response within 100 ms	Soft timing	Ordinary RTOS task
REQ-05	MQTT publish every 5 s, TLS 1.2 minimum	Soft, blocking	Needs a thread that can block for seconds
REQ-06	Sensor failure must not stop the control loop	Failure behaviour	Fault domains, last-known-good values
REQ-07	Failed OTA must not brick the unit	Failure behaviour	A/B banks, rollback, boot counter
REQ-08	Configuration survives power loss mid-write	Failure behaviour	Two-copy config with CRC and sequence number
REQ-09	Reset cause readable in the field after the fact	Observability	Backup SRAM, survives reset, not flash
REQ-10	Ten-year life, unattended	Lifecycle	Watchdog, health monitor, no unbounded resources

Ten requirements. Six of them are hard timing or failure behaviour. That ratio is normal for industrial gear, and it’s why an architecture built around features instead of deadlines and failure modes falls over in the second year.

Running the decision tree

Article 01 ended with a tree. Here it is on real inputs.

Tightest hard deadline? REQ-02 wants a sample every 1 ms with no misses, but the CPU never touches it: timer triggers ADC, ADC feeds DMA, DMA raises one interrupt per buffer. That path is settled in hardware. REQ-03’s 1.75 ms silence is the USART receiver timeout, also settled in hardware. The 1.75 ms is the Modbus spec’s recommended fixed t3.5 for anything above 19200 baud, not a computed one. At 115200 the true 3.5-character time is only about 334 µs, counting the 11 bits of an RTU character, so the fixed 1.75 ms is a minimum silence the spec sets for you, well above the computed value. People lose an afternoon trying to derive it. It is also not the idle-line interrupt: that fires after a single character time of idle, about 95 µs at 115200, which is far too early to call a frame complete. The tightest deadline left for software is REQ-01 at 1 ms, which is comfortably in the “open” branch.

On the H7 the receiver timeout counts in bit times, so the register value falls straight out of the spec figure:

/* t3.5 = 1.75 ms fixed above 19200 baud (Modbus over Serial Line spec) */
#define MODBUS_T35_BITS  ((uint32_t)((1750u * 115200u) / 1000000u))  /* 201 bit times */

USART3->RTOR = (USART3->RTOR & ~USART_RTOR_RTO_Msk) | MODBUS_T35_BITS;
USART3->CR2 |= USART_CR2_RTOEN;   /* receiver timeout enable */
USART3->CR1 |= USART_CR1_RTOIE;   /* RTOF marks end of frame */

Linux-class services? No filesystem, no UI, no package ecosystem. Skip.

How many operations must block concurrently? MQTT sits inside a TLS handshake that can take seconds. The OTA task blocks on flash erase, and on an H7 a sector erase runs to about a second, a bank erase to several. The config service blocks on I2C EEPROM writes at 5 ms per page. That’s three, and they overlap in normal operation.

Three concurrent blockers, 1 MB of RAM. FreeRTOS. Settled, and now the interesting work starts.

---

Section 2: The Production Firmware Architecture

Boxes are cheap. What makes this an architecture is the three tables underneath it.

The task table

This is the single most useful artifact in the whole design. If a firmware team can’t produce this on request, they don’t have an architecture, they have a build that happens to work.

Task	Prio	Stack	Trigger	Deadline	May block?
Control	6	1 KB	Timer notification, 1 ms	1 ms	Never
Sensor	5	1 KB	10 ms tick	10 ms	SPI only, bounded
Modbus RTU	5	1.5 KB	USART receiver timeout (RTOF)	5 ms	No
ethernetif_input	5	1 KB	ETH RX semaphore	None	No
lwIP tcpip_thread	4	2 KB	Mailbox	None	Yes
Modbus TCP	3	2 KB	Socket	100 ms	Yes
MQTT / Cloud	3	6 KB	5 s timer	Soft	Yes, seconds
Firmware Update	2	3 KB	Command event	None	Yes, long
Health Monitor	5	768 B	100 ms tick	500 ms	No
System (Config Service + Device Manager)	2	1.5 KB	Event queue	None	Yes, EEPROM page writes
Logging	1	1 KB	Queue	None	Yes
Idle	0	256 B	Scheduler	None	Never

Priorities are native FreeRTOS with configMAX_PRIORITIES at 7, higher number wins. Worth stating, because CubeMX generates CMSIS-RTOS v2 on this part by default, and that layer uses its own priority enumeration, with osPriorityNormal at 24 and configMAX_PRIORITIES raised to 56 to hold it. The older CMSIS-RTOS v1 wrapper offset priorities by three instead, so the same 3 landed on absolute priority 6. A task table that doesn’t say which scheme it’s written in isn’t a task table.

Idle and the FreeRTOS timer service task are kernel tasks, not among the eleven. The timer task runs only short software timer callbacks here, so it gets a 512 B stack, and that stack is counted in the RAM budget below.

The System task hosts the Configuration Service and the Device Manager. Both are event driven, both have no deadline, and the Configuration Service blocks on 5 ms EEPROM page writes, so by the rules below they sit with the other blocking, deadline-free work at priority 2.

Note ethernetif_input sitting above tcpip_thread. That ordering is recommended, and not just stylistic. ST’s generated ethernetif.c creates the input task above the TCP/IP thread by default, and the ordering is easy to lose when someone renumbers priorities by hand to fit a table like this one. Invert it and you rarely lose frames outright, because the DMA descriptors keep buffering them, but receive latency grows and bursts start overrunning the descriptor ring, which shows up as TCP retransmits that look like a network problem for about a week.

Two rules produced those priority numbers, and both get argued about in every design review I’ve sat in.

Priority is a function of deadline, not importance. Firmware Update is the most commercially important feature on the list. It runs at priority 2. It has no deadline, so it gets no priority. The engineer who bumps OTA to priority 6 because “the customer cares about it” has just given a flash erase the right to preempt a control loop.

A task that can block never sits above a task that can’t. Control and Health Monitor are above everything that touches the network, which means a stuck socket can’t starve them. This is also why lwIP’s thread sits at 4 rather than wherever the porting guide’s example put it.

That second rule is what fixes Health Monitor’s number, and it’s the one I get wrong most often when sketching quickly. Health Monitor feeds the watchdog against a 500 ms deadline. lwIP blocks at 4, so Health Monitor cannot sit below 5. Sort by importance instead and it drifts down to 2, next to Firmware Update, at which point a multi-second flash erase is competing with the task whose entire job is to prove the system is still alive.

Note the MQTT stack at 6 KB. TLS is expensive in stack, not just in flash: certificate chain parsing and the handshake state machine are deep, and the number you’ll find in the wild ranges from 4 KB to 16 KB depending on cipher suite and whether you verify a chain. Treat that range as folklore: mbedTLS publishes no stack figures at all. Budget generously, then measure with high-water marks and cut it down. Guessing low here produces a stack overflow that corrupts a neighbouring task and shows up two seconds later somewhere unrelated.

The memory map

On an STM32H7 this is architecture, not tuning, and getting it wrong costs you most of the performance you paid for.

Region	Size	Contents	Why there
ITCM	64 KB	Control loop, ADC ISR, flash driver	Zero wait state, no cache dependency, and it still executes while flash is being erased
DTCM	128 KB	Control task stack, PID state, control working set	Fastest data, but no DMA master can reach it, including the ones you’d like to
AXI SRAM (D1)	512 KB	lwIP pbuf pool, TLS arena, most task stacks	Big and cacheable
SRAM1/2/3 (D2)	288 KB	Ethernet and ADC DMA descriptors and buffers	Reachable by the ETH DMA and by DMA1/DMA2. DTCM is not.
SRAM4 (D3)	64 KB	BDMA buffers, low-power path	Reachable when D1/D2 are gated
Backup SRAM	4 KB	Crash record, reset reason, boot counter	Survives system reset; survives power loss only with VBAT and the backup regulator enabled. REQ-09 lives here.

The D2 line is the one that eats a week of somebody’s life. On this part only the CPU and the MDMA can reach ITCM and DTCM; every other master, the Ethernet DMA and DMA1/DMA2 included, is locked out. So Ethernet descriptors and buffers sit in D2, and they get marked non-cacheable in the MPU, or Device memory, because the DMA writes behind the D-cache’s back. Skip that second half and Ethernet works fine on the bench, then drops occasional frames under load, and the packets that go missing look random. No amount of protocol debugging finds it. It’s an MPU table entry.

D2 is a preference, not a law. AXI SRAM in D1 works too, at the cost of the D2-to-D1 bridge, and ST’s own lwIP guidance puts RX buffers there when D2 is full. DTCM is the one that’s actually forbidden.

Which has a consequence worth stating plainly, because it’s the opposite of what instinct suggests: the ADC’s DMA buffer can’t live in DTCM either. The ADC feeds DMA1 or DMA2, and those are D2 masters under exactly the restriction above. Put the buffer in DTCM because it’s the fastest RAM and the hard-deadline path deserves the fastest RAM, and it doesn’t degrade gracefully. It doesn’t work. So the ADC buffer goes to D2, non-cacheable, same as Ethernet, and the sample path pays the same cache discipline as everything else.

You can’t duck coherency by hiding in tightly coupled memory. You can only choose where to pay it, and write that down. What belongs in DTCM is what the CPU touches alone: the control task’s stack, the PID state, the working set of the one loop with a hard deadline.

Why the control loop lives in ITCM

REQ-01 wants 1 kHz with under 50 µs of jitter. REQ-07 wants firmware update. On this part those two requirements are in direct conflict, and the memory map is what resolves it.

Erasing flash on an H7 stalls reads from the bank being erased. Not for microseconds: a sector erase runs to roughly a second, a bank erase to several (check the erase-time table in the H743 datasheet for your voltage range and program parallelism). The dual-bank H743 lets you execute from one bank while erasing the other, which is exactly what a normal A/B update does, so the running image is not stalled in the common case. The cases that remain are the ones that touch the bank you are running from: writing update metadata into the active bank, and recovery paths that rewrite the running image. The bank swap itself is not one of them: SWAP_BANK is an option byte change that takes effect at the next reset, not an erase. Code fetched from that bank during the erase does not execute, and a 1 ms deadline does not survive a one-second stall.

The control loop runs from ITCM and its data lives in DTCM, so it keeps running whichever bank is busy, and its timing never depends on flash wait states or cache misses. That isn’t a performance optimisation, whatever the “zero wait state” column suggests. It’s what guarantees this product can accept an over-the-air update without dropping the loop it exists to run, including on the paths that touch the active bank. The flash driver has to be in ITCM for the same reason, and so does anything on the interrupt path you need during an erase. That includes every ISR left enabled while the active bank is being erased. If the ETH, USART or DMA interrupts stay unmasked and their handlers live in flash, the first one to fire stalls the core on the instruction fetch until the erase finishes, and the control loop stalls with it. Either relocate those handlers, and everything they call, to ITCM or RAM, or mask them for the duration of the erase.

Be precise about what “the control loop” means here, because the guarantee is only as strong as its weakest fetch. The loop needs the vector table, so VTOR points at a copy in RAM. It needs the timer ISR that wakes it, the kernel paths it passes through (xTaskNotifyFromISR, the PendSV and SysTick handlers, the scheduler), its constants in .rodata, and any compiler helpers it calls, such as memcpy or the float runtime. Every one of those sits in flash by default. Miss one and the control task still stalls on the erase, just less often, which is worse because it is harder to find. Check the map file: no symbol reachable from the control path, or from any ISR left enabled during the erase, should have a flash address.

Then prove it on hardware. Start a sector erase in the active bank while the control task toggles a GPIO every period, capture 10,000 periods on a scope or logic analyser, and compare the worst-case jitter against REQ-01’s 50 µs. That measurement is the acceptance test for this whole section. Run it twice: once with the system otherwise idle, and once with the lowest-priority runnable task executing code from the bank under erase, because a task stalled on a fetch or a literal load from that bank is the case the ITCM argument has to survive. If the second run breaks the 50 µs budget, only erase the active bank in a mode where every runnable path is in RAM.

It’s the clearest example I know of memory placement being architecture rather than tuning. Move those two things into flash because it’s convenient and the design still passes every bench test you have, right up until the first field update.

The RAM budget

Consumer	Bytes
Task stacks (eleven application tasks plus the 512 B timer service task, idle excluded)	~21 KB
TCBs and kernel objects	~4 KB
Queues and event pools	~8 KB
lwIP pbufs and PCBs	~48 KB
Ethernet DMA descriptors + buffers	~16 KB
mbedTLS arena (one session)	~40 KB
Config snapshots (three slots) + shadow	~6 KB
Log ring buffer	~8 KB
Sensor and ADC buffers	~4 KB
Total	~155 KB of 1 MB

Comfortable, which is the point. You want the budget written down before implementation so that the day someone proposes a second TLS session, the conversation is arithmetic instead of opinion.

No heap

configSUPPORT_DYNAMIC_ALLOCATION 0, configSUPPORT_STATIC_ALLOCATION 1. Every task, queue, timer and semaphore is declared with its storage at file scope. No malloc in the link map, and a linker assertion that fails the build if one appears. More on sizing all of this in the FreeRTOS memory management walkthrough.

Turning dynamic allocation off has two consequences worth knowing before you flip the switch. The whole xTaskCreate family disappears and you use the ...Static variants everywhere. And you must supply vApplicationGetIdleTaskMemory(), plus vApplicationGetTimerTaskMemory() if configUSE_TIMERS is on. Omit them and it’s a link error, which is the friendliest failure mode in this entire article.

Three traps are less friendly. The CMSIS-RTOS v2 wrapper that CubeMX generates still compiles with dynamic allocation off, but osThreadNew, osMessageQueueNew and the rest return NULL unless every attribute struct supplies cb_mem and stack_mem (or mq_mem) with their sizes, so check every handle at start-up. The stock lwIP sys_arch.c allocates as well: sys_thread_new and sys_mbox_new create their kernel objects dynamically and need reworking onto static storage. And newlib’s float formatting is a back door: printf with %f pulls in the dtoa code, which calls _malloc_r, so one debug print trips the no-malloc linker assertion. Format in fixed point, or use a printf implementation that does not allocate.

lwIP needs its own decision. MEM_LIBC_MALLOC off keeps it away from the C library, but on its own that just moves it to lwIP’s internal heap. MEM_USE_POOLS is the one that gets you fixed-size pools and bounded allocation behaviour, and bounded is the whole point.

One honest exception: mbedTLS effectively needs an allocator. You can fight it, and people do, but the maintainable answer is to give it a dedicated bounded arena through MBEDTLS_PLATFORM_MEMORY and let it allocate inside that. If the arena is exhausted, the handshake fails and the cloud connection retries later. Contained, observable, and it can’t take the control loop down with it.

Budget the arena, not just the stack. The 6 KB in the task table is thread stack. The 40 KB in the RAM budget is mostly MBEDTLS_SSL_IN_CONTENT_LEN and MBEDTLS_SSL_OUT_CONTENT_LEN, 16 KB each at their defaults and both living in the session context. Size only the stack and you are about 32 KB short on a design with a 155 KB budget. MBEDTLS_PLATFORM_MEMORY also needs MBEDTLS_PLATFORM_C alongside it. The simplest bounded arena is mbedTLS’s own MBEDTLS_MEMORY_BUFFER_ALLOC_C, initialised over a static buffer with mbedtls_memory_buffer_alloc_init(). Build once with MBEDTLS_MEMORY_DEBUG and read mbedtls_memory_buffer_alloc_max_get() after a full handshake against the real broker. With 16 KB records in both directions and a certificate chain to verify, the peak can exceed 40 KB, and the measured figure, not this table, is what sizes the arena.

That’s the distinction that matters. “No heap” isn’t a purity contest. It’s a requirement that no allocation failure anywhere can affect a subsystem with a deadline.

---

Section 3: Ownership

Here’s where most firmware architectures actually fail, and it never looks like an architecture failure at the time. It looks like a bug.

Every hardware resource and every piece of mutable state has exactly one owner. The owner is the only code that touches it. Everyone else asks.

Who owns the UART?

The RS-485 driver. Not the Modbus stack, and definitely not the application. That driver owns UART3, its DMA channels, and the DE/RE direction pin, and the direction pin is why. Turn the transceiver around a few microseconds early and you clip your own last byte; turn it late and you collide with the master. That timing has to live in one place, next to the code that knows when the last stop bit actually left the shift register.

Nobody else gets to call HAL_UART_Transmit on that instance. If a second call site appears, the bus has two owners and you now have a bug that only shows up under traffic.

Who owns the SPI bus?

Not the sensor drivers. This one trips up good engineers.

Four sensors share SPI2. If each sensor driver owns “its” SPI access, you have four owners of one bus and you’re relying on a mutex plus everyone remembering to take it. Instead, a bus manager owns SPI2 outright, along with all four chip selects, and exposes transactions:

/* drivers/spi_bus.h */
typedef struct {
    uint8_t        cs_id;
    const uint8_t *tx;
    uint8_t       *rx;
    size_t         len;
    uint32_t       timeout_ms;
} spi_txn_t;

result_t spi_bus_execute(spi_bus_t *bus, const spi_txn_t *txn);

Sensor drivers describe what they want. The bus manager serialises, drives CS, handles the DMA and applies the timeout. The part that matters: it is the only code that can decide to reset the peripheral after a stuck transfer. One owner, one recovery path.

The API also forces a timeout into every transaction. You can’t call it without one. Design that removes the option of forgetting beats a code review comment reminding people not to.

Who owns configuration?

The Configuration Service, and it’s a single-writer design. Everyone else reads a snapshot:

const config_t *config_get(void);        /* current snapshot, never NULL */
result_t        config_stage(const config_t *candidate);
result_t        config_commit(void);      /* validates, persists, swaps */

Readers get a const pointer to an immutable snapshot. No locks on the read path, which is what lets the control task read a setpoint without a mutex it can’t afford to wait on. That works because a naturally aligned pointer store is atomic on Cortex-M7, and it only works if the pointer is published with release semantics and read with acquire semantics. volatile alone is not enough: it stops the compiler hoisting the read out of a loop, but it does not stop the compiler moving the writes that fill the new snapshot to after the pointer store, which hands a reader a half-built configuration. Commit validates the candidate, writes it to EEPROM with a CRC and an incrementing sequence number, and only then swaps the pointer.

#include <stdatomic.h>

static config_t slots[3];                  /* three slots: a reader may still hold one */
static _Atomic(const config_t *) current;  /* readers never block */

const config_t *config_get(void)
{
    return atomic_load_explicit(&current, memory_order_acquire);
}

static void config_publish(const config_t *fresh)  /* single writer only */
{
    atomic_store_explicit(&current, fresh, memory_order_release);
}

Three slots, not two, and a rule for readers: copy what you need from the snapshot within one period and do not keep the pointer. With only two slots, two commits in quick succession can overwrite the buffer a slow reader is still walking. Three slots are safe only if every reader finishes with its snapshot within one commit interval. The control task does by construction. Low-priority readers such as Modbus TCP and MQTT can be preempted for longer than that, so either restrict pointer access to the control task and hand everyone else a copy through the Configuration Service, or give each slot a generation counter that readers check before and after copying, retrying if it changed. That second option is a seqlock.

One start-up rule keeps the never-NULL promise honest: config_init() loads or defaults the configuration and publishes it before vTaskStartScheduler(), so no reader can ever see the initial NULL. The three-slot scheme also assumes commits are rate-limited, at most one per control period; the 5 ms EEPROM page write already guarantees that on this design.

Two copies in EEPROM, alternating, highest valid sequence number wins. That’s REQ-08 satisfied by a design decision rather than by hoping power stays up during a 5 ms page write.

Who can modify system state?

The Device Manager, running an explicit state machine with five states: INIT, RUNNING, DEGRADED, UPDATING, FAULT. It’s the only component that transitions between them.

Everything else requests. The Firmware Update service doesn’t set the state to UPDATING; it posts a request and gets told yes or no. Which is what lets a single piece of code answer “can we accept an update right now?” by looking at whether the control loop is actively driving an output. If that decision were distributed, it’d be answered inconsistently in four places.

Who handles errors, and who restarts a peripheral?

The resource owner, in both cases. The SPI bus manager retries and resets SPI2. The sensor driver decides a sensor is dead. The Device Manager decides the product is degraded. Each layer handles what it can distinguish and escalates what it can’t. More on that in section 5.

The ownership matrix

Component	Owns	May call	Must not call
Application	Product behaviour, control law	Services	Drivers, HAL, registers
Service	Domain state, policy	Middleware, other services	Drivers, HAL
Middleware	Protocol and format state	HAL, drivers via BSP	Application, services
Driver	One peripheral, its DMA, its pins	HAL, RTOS primitives	Application, services
Bus manager	A shared bus and all its chip selects	HAL, RTOS primitives	Anything above drivers
ISR	Capturing an event, nothing more	...FromISR primitives	Application logic, blocking calls, logging

The ISR row is the one to enforce hardest. An ISR captures a timestamp and a value, posts it, and returns. It doesn’t format a string, doesn’t take a mutex, doesn’t call the logging service. Every one of those has appeared in production code I’ve had to fix, and the logging call is the most common by a wide margin.

---

Section 4: Dependency Direction

Arrows point down. Only down. When something below needs to tell something above, it does it through a callback or an event the upper layer registered, which is a dependency inversion, not a reversed arrow.

Four specific edges break this, and each one fails differently.

Driver → Application. The ADC driver includes control_manager.h so it can call the PID update directly from the DMA callback. Feels efficient. Now the driver can’t be compiled without the application, can’t be reused on the next product, can’t be tested on a host, and your PID math is running in interrupt context where only higher-priority interrupts can preempt it, and it holds off every interrupt at its own priority or below, and every task, while it runs.

Application → Driver. The control manager calls HAL_GPIO_WritePin to trip a relay, because it’s one line and the alternative is a service. Then the relay moves to an I2C expander in rev C, and the change lands in application code, which means product behaviour gets re-reviewed and re-tested to accommodate a schematic edit.

Service → Driver. The diagnostics service reads the ADC directly to grab a supply rail voltage. The service was the one component you could have moved to the next product unchanged, and now it’s board-specific.

Driver → Service. This is the dangerous one, because it doesn’t arrive as an architectural decision. It arrives as logging.

Someone adds log_error("SPI timeout") inside the SPI driver. Entirely reasonable-looking. But the logging service writes to the log store, the log store is middleware over the flash driver, and the flash driver is now reachable from inside the SPI driver’s error path, possibly while the log store is already in the middle of a flash transaction of its own. You’ve built a cycle:

The failure mode is exquisite. Everything works until the SPI bus times out during a flash write, at which point the error path re-enters a module that’s mid-transaction and was never written to be reentrant. You get a hang that reproduces roughly once a week in the field and never on a bench.

The fix is a rule, not discipline: drivers don’t call up, they emit.

/* drivers/fault_sink.h: ISR-safe, bounded critical section, no dependencies */
void fault_emit(uint16_t source_id, uint16_t code, uint32_t detail);

That writes one 8-byte record into a ring, and the Logging service drains it at priority 1. The driver depends on nothing above it, the record survives if the log store is busy, and it’s safe from an ISR.

One implementation note, since the ownership matrix guarantees you’ll have several drivers and several ISRs producing at once: the write index has to be reserved atomically. Single-producer lock-free is easy and this isn’t that.

Reserving the index is not the whole job, either. A record that has a slot but is only half written must not be visible to the Logging task draining the ring. The simplest correct version uses a critical section of about twenty instructions, which is bounded and easy to defend in a timing review:

typedef struct { uint16_t src, code; uint32_t detail; } fault_rec_t;  /* 8 bytes */

static fault_rec_t ring[32];
static uint32_t head, tail, drops;  /* head: producers, tail: Logging task */

void fault_emit(uint16_t src, uint16_t code, uint32_t detail)
{
    UBaseType_t s = taskENTER_CRITICAL_FROM_ISR();  /* masks up to the syscall priority */
    if (head - tail < 32u) {
        ring[head % 32u] = (fault_rec_t){ src, code, detail };
        head++;
    } else {
        drops++;                                    /* keep the oldest, count the rest */
    }
    taskEXIT_CRITICAL_FROM_ISR(s);
}

That is not lock-free, and it does not need to be. If you want true lock-free behaviour, use a reserve and commit scheme with a per-slot ready flag, and accept the extra complexity in exchange.

Two limits on who may call it, and the other half of the contract. Producers are tasks and ISRs at or below configMAX_SYSCALL_INTERRUPT_PRIORITY; an ISR above that ceiling is not masked by the critical section, can preempt it mid-write and corrupt the ring, so it must never call fault_emit(). The Logging task drains the ring under the same lock:

bool fault_take(fault_rec_t *out)   /* Logging task only */
{
    bool got = false;
    taskENTER_CRITICAL();
    if (tail != head) {
        *out = ring[tail % 32u];
        tail++;
        got = true;
    }
    taskEXIT_CRITICAL();
    return got;
}

Enforce it in CI, not in review

Layer rules that live in a wiki decay within two sprints. Make the build fail:

# tools/check_layers.sh: run in CI
set -e
fail=0
check() {   # check <dir> <forbidden-include-dir>
  if grep -rn "#include \"$2/" "src/$1" 2>/dev/null; then
    echo "LAYER VIOLATION: $1 must not include $2"
    fail=1
  fi
}
check application drivers
check application hal
check services    drivers
check services    hal
check middleware  application
check middleware  services
check drivers     services
check drivers     application
check_vendor() {   # vendor headers stay in the layers allowed to call the HAL
  if grep -rnE '#include[[:space:]]*["<]stm32h7xx[_a-z0-9]*\.h' "src/$1" 2>/dev/null; then
    echo "LAYER VIOLATION: $1 must not include vendor headers"
    fail=1
  fi
}
check_vendor application
check_vendor services
exit $fail

Under thirty lines. It catches every one of the four bad edges above on the commit that introduces them, when the fix costs ten minutes, rather than in year two when it costs a refactor. It also catches the convenience include below, a vendor header such as stm32h7xx_hal.h in application or service code, in either include form. Know what it misses: the layer check only catches the #include "dir/..." form, so angle-bracket includes and bare filenames resolved through include paths slip past, and the vendor check only knows stm32h7xx names. In CMake you can do the same thing properly with PRIVATE link visibility and separate include directories per layer, but the grep version works today on whatever build system you already have, and shipped beats elegant.

---

Section 5: Error Propagation

The mistake isn’t failing to handle errors. It’s handling them at every layer, identically.

Error codes, not status objects

No exceptions, no RTTI, no heap. One typed result across the codebase:

typedef enum {
    RES_OK = 0,
    RES_TIMEOUT,        /* transient, retry may help    */
    RES_BUSY,           /* transient, back off          */
    RES_CRC,            /* transient or permanent       */
    RES_NO_DEVICE,      /* permanent until reset        */
    RES_RANGE,          /* caller's fault, never retry  */
    RES_UNSUPPORTED,    /* caller's fault, never retry  */
    RES_FAULT,          /* invariant broken, do not continue */
} result_t;

The comments are the design. Every value carries its own retry policy, so callers don’t have to guess whether a failure is worth trying again. Compare that to returning -1, or an errno-style integer that tells you what happened but not what to do about it.

Mark every function that returns it, so ignoring a result is a compile warning. GCC’s warn_unused_result attaches to a function, not to the type:

#define MUST_CHECK __attribute__((warn_unused_result))
MUST_CHECK result_t sensor_read(sensor_id_t id, int32_t *out);

Then turn on -Werror=unused-result. Now dropping an error on the floor stops the build.

Errors versus faults

Two different things, and conflating them is why devices limp along in corrupted states.

An error is an expected outcome you have a plan for. Sensor timed out. Socket closed. CRC mismatch on a Modbus frame. Handle it, count it, carry on.

A fault is a broken invariant. Config CRC valid but the sequence number went backwards. A stack canary is gone. An MPU violation. A queue that should never be full is full. There’s no correct way to continue, and continuing anyway is how you end up driving an output from a corrupted setpoint. Record the reason to backup SRAM and reset.

That distinction is the whole seam between this article and article 07.

Fault domains

REQ-06 says a sensor failure must not stop the control loop. That’s a requirement about isolation, and isolation has to be designed in:

Domain	Contains	Failure effect	Isolation mechanism
Control	PID, ADC, analog out	Product stops controlling	Own task, DTCM data, no dependency on any other domain
Sensing	SPI bus, 4 sensors	Degraded mode, last-known-good with an age stamp	Bus manager resets; control reads a cached value
Fieldbus	RS-485, Modbus RTU	Master sees timeouts	Own task, own UART, own recovery
Cloud	Ethernet, TLS, MQTT	No telemetry. Local control unaffected.	Own tasks, own memory arena
Update	Download, verify, stage	Update fails, old image keeps running	A/B banks, nothing shared with control

The Cloud row deserves a moment. The TLS arena being separate isn’t a nicety. It’s what means a memory exhaustion in the cloud path produces a failed MQTT connection instead of an allocation failure that starves the control loop. That’s the payoff for bounding the arena back in section 2.

And “last-known-good” needs an age stamp attached, always. A stale reading that presents itself as fresh is worse than no reading, because the control law can’t tell it’s being lied to. Return the value and its age; let the caller decide what’s too old.

Retries and timeouts

Retry exactly once, at the lowest layer that can tell transient from permanent. That’s the SPI bus manager for a bus glitch, and nowhere else.

Retrying at every layer is the anti-pattern, and it multiplies. Three attempts in the driver, three in the middleware, three in the service is 27 attempts. At a 100 ms timeout each, that’s the better part of three seconds inside what your architecture diagram calls a “sensor read.” Meanwhile the watchdog is looking at a task that hasn’t checked in and drawing the reasonable conclusion.

Timeouts follow the same discipline. Every blocking call has one, timeouts shrink as you go down the stack, and no upper-layer timeout is ever shorter than the sum of what’s below it. Otherwise the upper layer gives up while the lower one is still legitimately working, and you get two components with different beliefs about whether an operation is in flight.

The escalation ladder

Escalate one rung at a time, and never skip to 7. A device that resets on the first SPI timeout is a device that reboots every eleven minutes in a noisy panel, and every reboot loses more state than the fault did.

Putting it together

/* services/sensor_service.c */
result_t sensor_service_sample(sensor_id_t id, sample_t *out)
{
    int32_t raw;
    result_t r = sensor_read(id, &raw);      /* driver already retried once */

    switch (r) {
    case RES_OK:
        cache_store(id, raw, now_ms());
        out->value = raw;
        out->age_ms = 0;
        return RES_OK;

    case RES_TIMEOUT:
    case RES_CRC:
        diag_count(DIAG_SENSOR_TRANSIENT, id);
        return cache_fetch(id, out);         /* stale but stamped */

    case RES_NO_DEVICE:
        diag_count(DIAG_SENSOR_LOST, id);
        fault_emit(SRC_SENSOR, r, id);
        device_request_degrade(DOMAIN_SENSING);
        return cache_fetch(id, out);

    case RES_BUSY:                       /* transient: back off, not a fault */
        return cache_fetch(id, out);     /* try again next period */

    case RES_FAULT:                      /* invariant broken: do not continue */
        fault_emit(SRC_SENSOR, r, id);
        device_request_fault(DOMAIN_SENSING, r);  /* records to backup SRAM, then resets */
        return r;

    default:
        fault_emit(SRC_SENSOR, r, id);
        return r;
    }
}

Notice what this function does not do. It doesn’t retry; the driver already did. It doesn’t reset the bus; the bus manager owns that. It doesn’t treat a busy sensor as a fault; it serves the stamped cached value and tries again next period. It doesn’t decide whether the product should degrade or reset; it requests a degrade, or reports a broken invariant, and lets the Device Manager decide. It doesn’t log a string.

One layer, one job, and every branch has a defined outcome. That’s what makes an architecture production-grade. Not the diagram.

---

Where This Goes Wrong

Five things I’d watch for on this exact design over the next two years. Article 03 covers structural anti-patterns properly; these are the ones this architecture is specifically exposed to.

The priority creep. Someone raises the MQTT task because telemetry looks laggy. Nobody notices that a TLS handshake now preempts the sensor task. Fix: put the priority table in a header with a comment explaining each number, and require a design note to change one.

The convenience include. #include "stm32h7xx_hal.h" appears in a service, once, for a delay. Then a second time. The CI check exists for exactly this.

The god struct. A system_state_t that everything reads and three things write. It starts as two fields. Ownership dies quietly.

Log inflation. The bounded ring buffer gets a “temporary” bypass so someone can debug a customer issue, and the bypass ships. Bounded means bounded.

MPU drift. A new DMA buffer gets added without an MPU region, and it happens to work because it landed in an already-non-cacheable area. Six months later a linker change moves it. Fix: put the MPU table and the linker script in the same review.

---

Frequently asked questions

What makes a firmware architecture production-grade?

A task table with priorities derived from deadlines, a memory map, a RAM budget written down before implementation, exactly one owner for every resource, dependencies that point down and are enforced in CI, and a defined outcome for every error path. If the team cannot produce the task table on request, there is no architecture yet, only a build that happens to work.

How should FreeRTOS task priorities be assigned?

By deadline, not importance, and a task that can block never sits above one that cannot. The control loop goes at the top, the health monitor sits above anything that blocks on the network, and firmware update runs low despite being commercially important, because it has no deadline.

Can an RTOS-based firmware run without a heap?

Yes. Set configSUPPORT_DYNAMIC_ALLOCATION to 0, use the static creation APIs, build lwIP with MEM_USE_POOLS, and add a linker assertion that fails if malloc appears. The practical exception is mbedTLS, which gets a dedicated bounded arena so that exhaustion fails a TLS handshake rather than a subsystem with a deadline.

Where should DMA buffers go on an STM32H7?

Not in DTCM, because only the CPU and the MDMA can reach the tightly coupled memories. Ethernet and ADC DMA buffers belong in D2 SRAM, or AXI SRAM if D2 is full, marked non-cacheable in the MPU so the DMA cannot write behind the D-cache.

How do you keep a control loop running during a flash erase on an STM32H7?

Run it from ITCM with its data in DTCM, together with everything it reaches: the vector table, the timer ISR, the kernel paths, compiler helpers, and every interrupt handler left enabled during the erase. Check the map file for flash addresses on those paths, then prove it with a scope measurement taken while a sector in the active bank is being erased.

---

Next

This architecture works. Right up until it’s been maintained by four engineers for three years.

Article 03 is about how architectures decay: the specific structural patterns that turn a design like this one into something nobody wants to touch, and what the early symptoms look like while it’s still cheap to fix.

Article 03: Firmware Architecture Anti-Patterns.

---

References

• STM32H743ZI documentation (RM0433 reference manual and datasheet)
• Arm Cortex-M7 Technical Reference Manual
• FreeRTOS kernel documentation
• Modbus specifications, including Modbus over Serial Line and Modbus/TCP Security
• Mbed TLS
• lwIP

---

Series Index

Overview, reading paths and related articles: Embedded Firmware Architecture series guide.

Phase 1: Decide

01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture (you are here)
03. Firmware Architecture Anti-Patterns

Phase 2: Build

04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture

Phase 3: Fail safely

07. Error Handling & Recovery
08. Firmware Security Architecture

Phase 4: Verify

09. Designing Firmware for Testability

Phase 5: Operate

10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates