How to Design a Production-Grade Firmware Architecture

Article 01 was about how to decide. This one is about what you actually hand to a team.

So we’re going to build one. Not a diagram of one. A production firmware architecture with named tasks, assigned priorities, a memory map, a stack budget, an ownership matrix, and a defined answer to what happens when the SPI sensor stops responding at 3 a.m. in a substation.

The product is made up. Everything else is the way I’d really do it.

The Product

KIC-400 Industrial IoT Controller. Panel-mounted, DIN rail, sold into water treatment and building automation. Ten-year field life, no scheduled maintenance visits.

What it has to do:

• 8 × 4–20 mA analog inputs, 16-bit, sampled at 1 kHz
• 4 × SPI pressure and temperature sensors on a shared bus, 100 Hz
• One PID control loop at 1 kHz driving two analog outputs
• Modbus RTU slave on RS-485, 115200 baud
• Modbus TCP server on Ethernet
• MQTT over TLS to a cloud broker
• Configuration in external EEPROM, survives power loss
• OTA firmware update over Ethernet, with rollback
• Watchdog, diagnostics, field-readable fault history

Silicon: STM32H743ZI. Cortex-M7 at 480 MHz, which needs revision V silicon and VOS0 — revision Y caps at 400 MHz, and it is still in circulation, so check before you spend the cycles. 2 MB flash in two 1 MB banks, about 1 MB of RAM across six regions, built-in Ethernet MAC.

That part choice isn’t neutral, and I want to be honest about why. Dual-bank flash means A/B update lives entirely in internal flash with no external QSPI part and no extra BOM line. If you’d landed on a 1 MB single-bank H723 instead, the download slot goes to external QSPI, the bootloader gets a driver it didn’t need, and the update story gets meaningfully harder. That’s article 01’s flash-occupancy rule arriving as a purchase order.

Section 1 — Start From Requirements

Nobody skips this step on purpose. They skip it because the schematic showed up and there was code to write.

Each arrow is a derivation. If you can’t point at the requirement that produced an architectural decision, you invented it, and you’ll defend it for years without knowing why.

The version that matters isn’t the marketing spec. It’s the timing and failure spec, written down with IDs, because those IDs are what your tests trace back to later.

Ten requirements. Six of them are hard timing or failure behaviour. That ratio is normal for industrial gear, and it’s why an architecture built around features instead of deadlines and failure modes falls over in the second year.

Running the decision tree

Article 01 ended with a tree. Here it is on real inputs.

Tightest hard deadline? REQ-02 wants a sample every 1 ms with no misses, but the CPU never touches it — timer triggers ADC, ADC feeds DMA, DMA raises one interrupt per buffer. That path is settled in hardware. REQ-03’s 1.75 ms silence is a UART idle-line interrupt, also settled at the ISR. That figure is the Modbus spec’s recommended fixed t3.5 for anything above 19200 baud, not a computed one — at 115200 the true 3.5-character time is about 304 µs, so the spec is handing you a floor. People lose an afternoon trying to derive it. The tightest deadline left for software is REQ-01 at 1 ms, which is comfortably in the “open” branch.

Linux-class services? No filesystem, no UI, no package ecosystem. Skip.

How many operations must block concurrently? MQTT sits inside a TLS handshake that can take seconds. The OTA task blocks on flash erase, and on an H7 a sector erase runs to about a second, a bank erase to several. The config service blocks on I2C EEPROM writes at 5 ms per page. That’s three, and they overlap in normal operation.

Three concurrent blockers, 1 MB of RAM. FreeRTOS. Settled, and now the interesting work starts.

Section 2 — The Production Firmware Architecture

Boxes are cheap. What makes this an architecture is the three tables underneath it.

The task table

This is the single most useful artifact in the whole design. If a firmware team can’t produce this on request, they don’t have an architecture, they have a build that happens to work.

Priorities are native FreeRTOS with configMAX_PRIORITIES at 7, higher number wins. Worth stating, because CubeMX generates CMSIS-RTOS v2 on this part by default and that layer renumbers — hand it 3 and you land on absolute priority 6. A task table that doesn’t say which scheme it’s written in isn’t a task table.

Note ethernetif_input sitting above tcpip_thread. That ordering is required, not stylistic, and the CubeMX default gets it backwards. Invert it and you get dropped frames and TCP retransmits that look like a network problem for about a week.

Two rules produced those priority numbers, and both get argued about in every design review I’ve sat in.

Priority is a function of deadline, not importance. Firmware Update is the most commercially important feature on the list. It runs at priority 2. It has no deadline, so it gets no priority. The engineer who bumps OTA to priority 6 because “the customer cares about it” has just given a flash erase the right to preempt a control loop.

A task that can block never sits above a task that can’t. Control and Health Monitor are above everything that touches the network, which means a stuck socket can’t starve them. This is also why lwIP’s thread sits at 4 rather than wherever the porting guide’s example put it.

That second rule is what fixes Health Monitor’s number, and it’s the one I get wrong most often when sketching quickly. Health Monitor feeds the watchdog against a 500 ms deadline. lwIP blocks at 4, so Health Monitor cannot sit below 5. Sort by importance instead and it drifts down to 2, next to Firmware Update — at which point a multi-second flash erase is competing with the task whose entire job is to prove the system is still alive.

Note the MQTT stack at 6 KB. TLS is expensive in stack, not just in flash — certificate chain parsing and the handshake state machine are deep, and the number you’ll find in the wild ranges from 4 KB to 16 KB depending on cipher suite and whether you verify a chain. Treat that range as folklore: mbedTLS publishes no stack figures at all. Budget generously, then measure with high-water marks and cut it down. Guessing low here produces a stack overflow that corrupts a neighbouring task and shows up two seconds later somewhere unrelated.

The memory map

On an STM32H7 this is architecture, not tuning, and getting it wrong costs you most of the performance you paid for.

The D2 line is the one that eats a week of somebody’s life. On this part only the CPU and the MDMA can reach ITCM and DTCM — every other master, the Ethernet DMA and DMA1/DMA2 included, is locked out. So Ethernet descriptors and buffers sit in D2, and they get marked non-cacheable in the MPU, or Device memory, because the DMA writes behind the D-cache’s back. Skip that second half and Ethernet works fine on the bench, then drops occasional frames under load, and the packets that go missing look random. No amount of protocol debugging finds it. It’s an MPU table entry.

D2 is a preference, not a law. AXI SRAM in D1 works too, at the cost of the D2-to-D1 bridge, and ST’s own lwIP guidance puts RX buffers there when D2 is full. DTCM is the one that’s actually forbidden.

Which has a consequence worth stating plainly, because it’s the opposite of what instinct suggests: the ADC’s DMA buffer can’t live in DTCM either. The ADC feeds DMA1 or DMA2, and those are D2 masters under exactly the restriction above. Put the buffer in DTCM because it’s the fastest RAM and the hard-deadline path deserves the fastest RAM, and it doesn’t degrade gracefully — it doesn’t work. So the ADC buffer goes to D2, non-cacheable, same as Ethernet, and the sample path pays the same cache discipline as everything else.

You can’t duck coherency by hiding in tightly coupled memory. You can only choose where to pay it, and write that down. What belongs in DTCM is what the CPU touches alone: the control task’s stack, the PID state, the working set of the one loop with a hard deadline.

Why the control loop lives in ITCM

REQ-01 wants 1 kHz with under 50 µs of jitter. REQ-07 wants firmware update. On this part those two requirements are in direct conflict, and the memory map is what resolves it.

Erasing flash on an H7 stalls the flash interface. Not for microseconds: a sector erase runs to roughly a second, a bank erase to several. Code fetched from flash during that window does not execute. A 1 ms deadline does not survive a one-second stall.

The control loop runs from ITCM and its data lives in DTCM, so it keeps running while the flash interface is busy. That isn’t a performance optimisation, whatever the “zero wait state” column suggests. It’s the reason this product can accept an over-the-air update without dropping the loop it exists to run. The flash driver has to be in ITCM for the same reason, and so does anything on the interrupt path you need during an erase.

It’s the clearest example I know of memory placement being architecture rather than tuning. Move those two things into flash because it’s convenient and the design still passes every bench test you have, right up until the first field update.

The RAM budget

Comfortable, which is the point. You want the budget written down before implementation so that the day someone proposes a second TLS session, the conversation is arithmetic instead of opinion.

No heap

configSUPPORT_DYNAMIC_ALLOCATION 0, configSUPPORT_STATIC_ALLOCATION 1. Every task, queue, timer and semaphore is declared with its storage at file scope. No malloc in the link map, and a linker assertion that fails the build if one appears. More on sizing all of this in the FreeRTOS memory management walkthrough.

Turning dynamic allocation off has two consequences worth knowing before you flip the switch. The whole xTaskCreate family disappears and you use the ...Static variants everywhere. And you must supply vApplicationGetIdleTaskMemory(), plus vApplicationGetTimerTaskMemory() if configUSE_TIMERS is on. Omit them and it’s a link error, which is the friendliest failure mode in this entire article.

lwIP needs its own decision. MEM_LIBC_MALLOC off keeps it away from the C library, but on its own that just moves it to lwIP’s internal heap. MEM_USE_POOLS is the one that gets you fixed-size pools and bounded allocation behaviour, and bounded is the whole point.

One honest exception: mbedTLS effectively needs an allocator. You can fight it, and people do, but the maintainable answer is to give it a dedicated bounded arena through MBEDTLS_PLATFORM_MEMORY and let it allocate inside that. If the arena is exhausted, the handshake fails and the cloud connection retries later. Contained, observable, and it can’t take the control loop down with it.

Budget the arena, not just the stack. The 6 KB in the task table is thread stack. The 40 KB in the RAM budget is mostly MBEDTLS_SSL_IN_CONTENT_LEN and MBEDTLS_SSL_OUT_CONTENT_LEN, 16 KB each at their defaults and both living in the session context. Size only the stack and you are about 32 KB short on a design with a 152 KB budget. MBEDTLS_PLATFORM_MEMORY also needs MBEDTLS_PLATFORM_C alongside it.

That’s the distinction that matters. “No heap” isn’t a purity contest. It’s a requirement that no allocation failure anywhere can affect a subsystem with a deadline.

Section 3 — Ownership

Here’s where most firmware architectures actually fail, and it never looks like an architecture failure at the time. It looks like a bug.

Every hardware resource and every piece of mutable state has exactly one owner. The owner is the only code that touches it. Everyone else asks.

Who owns the UART?

The RS-485 driver. Not the Modbus stack, and definitely not the application. That driver owns UART3, its DMA channels, and the DE/RE direction pin, and the direction pin is why. Turn the transceiver around a few microseconds early and you clip your own last byte; turn it late and you collide with the master. That timing has to live in one place, next to the code that knows when the last stop bit actually left the shift register.

Nobody else gets to call HAL_UART_Transmit on that instance. If a second call site appears, the bus has two owners and you now have a bug that only shows up under traffic.

Who owns the SPI bus?

Not the sensor drivers. This one trips up good engineers.

Four sensors share SPI2. If each sensor driver owns “its” SPI access, you have four owners of one bus and you’re relying on a mutex plus everyone remembering to take it. Instead, a bus manager owns SPI2 outright, along with all four chip selects, and exposes transactions:

/* drivers/spi_bus.h */
typedef struct {
    uint8_t        cs_id;
    const uint8_t *tx;
    uint8_t       *rx;
    size_t         len;
    uint32_t       timeout_ms;
} spi_txn_t;

result_t spi_bus_execute(spi_bus_t *bus, const spi_txn_t *txn);

Sensor drivers describe what they want. The bus manager serialises, drives CS, handles the DMA, applies the timeout, and — this is the part that matters — is the only code that can decide to reset the peripheral after a stuck transfer. One owner, one recovery path.

The API also forces a timeout into every transaction. You can’t call it without one. Design that removes the option of forgetting beats a code review comment reminding people not to.

Who owns configuration?

The Configuration Service, and it’s a single-writer design. Everyone else reads a snapshot:

const config_t *config_get(void);        /* current snapshot, never NULL */
result_t        config_stage(const config_t *candidate);
result_t        config_commit(void);      /* validates, persists, swaps */

Readers get a const pointer to an immutable snapshot. No locks on the read path, which is what lets the control task read a setpoint without a mutex it can’t afford to wait on. That works because a naturally aligned pointer store is atomic on Cortex-M7, and it only works if the pointer is volatile or read through an atomic load — otherwise the compiler is free to hoist it out of a loop and hand the control task a snapshot two commits old. Commit validates the candidate, writes it to EEPROM with a CRC and an incrementing sequence number, and only then swaps the pointer.

Two copies in EEPROM, alternating, highest valid sequence number wins. That’s REQ-08 satisfied by a design decision rather than by hoping power stays up during a 5 ms page write.

Who can modify system state?

The Device Manager, running an explicit state machine — INIT, RUNNING, DEGRADED, UPDATING, FAULT. It’s the only component that transitions between them.

Everything else requests. The Firmware Update service doesn’t set the state to UPDATING; it posts a request and gets told yes or no. Which is what lets a single piece of code answer “can we accept an update right now?” by looking at whether the control loop is actively driving an output. If that decision were distributed, it’d be answered inconsistently in four places.

Who handles errors, and who restarts a peripheral?

The resource owner, in both cases. The SPI bus manager retries and resets SPI2. The sensor driver decides a sensor is dead. The Device Manager decides the product is degraded. Each layer handles what it can distinguish and escalates what it can’t — more on that in section 5.

The ownership matrix

The ISR row is the one to enforce hardest. An ISR captures a timestamp and a value, posts it, and returns. It doesn’t format a string, doesn’t take a mutex, doesn’t call the logging service. Every one of those has appeared in production code I’ve had to fix, and the logging call is the most common by a wide margin.

Section 4 — Dependency Direction

Arrows point down. Only down. When something below needs to tell something above, it does it through a callback or an event the upper layer registered — which is a dependency inversion, not a reversed arrow.

Four specific edges break this, and each one fails differently.

Driver → Application. The ADC driver includes control_manager.h so it can call the PID update directly from the DMA callback. Feels efficient. Now the driver can’t be compiled without the application, can’t be reused on the next product, can’t be tested on a host, and your PID math is running in interrupt context where it can’t be preempted by anything and holds off every other ISR while it runs.

Application → Driver. The control manager calls HAL_GPIO_WritePin to trip a relay, because it’s one line and the alternative is a service. Then the relay moves to an I2C expander in rev C, and the change lands in application code, which means product behaviour gets re-reviewed and re-tested to accommodate a schematic edit.

Service → Driver. The diagnostics service reads the ADC directly to grab a supply rail voltage. The service was the one component you could have moved to the next product unchanged, and now it’s board-specific.

Driver → Service. This is the dangerous one, because it doesn’t arrive as an architectural decision. It arrives as logging.

Someone adds log_error("SPI timeout") inside the SPI driver. Entirely reasonable-looking. But the logging service writes to the log store, the log store is middleware over the flash driver, and the flash driver — on a part where flash and SPI share an interrupt priority band — is now reachable from inside the SPI driver’s error path. You’ve built a cycle:

The failure mode is exquisite. Everything works until the SPI bus times out during a flash write, at which point the error path re-enters a module that’s mid-transaction. You get a hang that reproduces roughly once a week in the field and never on a bench.

The fix is a rule, not discipline: drivers don’t call up, they emit.

/* drivers/fault_sink.h — lock-free, ISR-safe, no dependencies */
void fault_emit(uint16_t source_id, uint16_t code, uint32_t detail);

That writes one 8-byte record into a ring, and the Logging service drains it at priority 1. The driver depends on nothing above it, the record survives if the log store is busy, and it’s safe from an ISR.

One implementation note, since the ownership matrix guarantees you’ll have several drivers and several ISRs producing at once: the write index has to be reserved atomically. Single-producer lock-free is easy and this isn’t that.

Enforce it in CI, not in review

Layer rules that live in a wiki decay within two sprints. Make the build fail:

# tools/check_layers.sh — run in CI
set -e
fail=0
check() {   # check <dir> <forbidden-include-dir>
  if grep -rn "#include \"$2/" "src/$1" 2>/dev/null; then
    echo "LAYER VIOLATION: $1 must not include $2"
    fail=1
  fi
}
check application drivers
check application hal
check services    drivers
check services    hal
check middleware  application
check middleware  services
check drivers     services
check drivers     application
exit $fail

Twenty lines. It catches every one of the four bad edges above on the commit that introduces them, when the fix costs ten minutes, rather than in year two when it costs a refactor. Know what it misses: it only catches the #include "dir/..." form, so angle-bracket includes and bare filenames resolved through include paths slip past. In CMake you can do the same thing properly with PRIVATE link visibility and separate include directories per layer, but the grep version works today on whatever build system you already have, and shipped beats elegant.

Section 5 — Error Propagation

The mistake isn’t failing to handle errors. It’s handling them at every layer, identically.

Error codes, not status objects

No exceptions, no RTTI, no heap. One typed result across the codebase:

typedef enum {
    RES_OK = 0,
    RES_TIMEOUT,        /* transient, retry may help    */
    RES_BUSY,           /* transient, back off          */
    RES_CRC,            /* transient or permanent       */
    RES_NO_DEVICE,      /* permanent until reset        */
    RES_RANGE,          /* caller's fault, never retry  */
    RES_UNSUPPORTED,    /* caller's fault, never retry  */
    RES_FAULT,          /* invariant broken, do not continue */
} result_t;

The comments are the design. Every value carries its own retry policy, so callers don’t have to guess whether a failure is worth trying again. Compare that to returning -1, or an errno-style integer that tells you what happened but not what to do about it.

Mark the type so ignoring a result is a compile warning:

#define MUST_CHECK __attribute__((warn_unused_result))
MUST_CHECK result_t sensor_read(sensor_id_t id, int32_t *out);

Then turn on -Werror=unused-result. Now dropping an error on the floor stops the build.

Errors versus faults

Two different things, and conflating them is why devices limp along in corrupted states.

An error is an expected outcome you have a plan for. Sensor timed out. Socket closed. CRC mismatch on a Modbus frame. Handle it, count it, carry on.

A fault is a broken invariant. Config CRC valid but the sequence number went backwards. A stack canary is gone. An MPU violation. A queue that should never be full is full. There’s no correct way to continue, and continuing anyway is how you end up driving an output from a corrupted setpoint. Record the reason to backup SRAM and reset.

That distinction is the whole seam between this article and article 07.

Fault domains

REQ-06 says a sensor failure must not stop the control loop. That’s a requirement about isolation, and isolation has to be designed in:

The Cloud row deserves a moment. The TLS arena being separate isn’t a nicety — it’s what means a memory exhaustion in the cloud path produces a failed MQTT connection instead of an allocation failure that starves the control loop. That’s the payoff for bounding the arena back in section 2.

And “last-known-good” needs an age stamp attached, always. A stale reading that presents itself as fresh is worse than no reading, because the control law can’t tell it’s being lied to. Return the value and its age; let the caller decide what’s too old.

Retries and timeouts

Retry exactly once, at the lowest layer that can tell transient from permanent. That’s the SPI bus manager for a bus glitch, and nowhere else.

Retrying at every layer is the anti-pattern, and it multiplies. Three retries in the driver, three in the middleware, three in the service is 27 attempts. At a 100 ms timeout each, that’s the better part of three seconds inside what your architecture diagram calls a “sensor read.” Meanwhile the watchdog is looking at a task that hasn’t checked in and drawing the reasonable conclusion.

Timeouts follow the same discipline. Every blocking call has one, timeouts shrink as you go down the stack, and no upper-layer timeout is ever shorter than the sum of what’s below it — otherwise the upper layer gives up while the lower one is still legitimately working, and you get two components with different beliefs about whether an operation is in flight.

The escalation ladder

Escalate one rung at a time, and never skip to 7. A device that resets on the first SPI timeout is a device that reboots every eleven minutes in a noisy panel, and every reboot loses more state than the fault did.

Putting it together

/* services/sensor_service.c */
result_t sensor_service_sample(sensor_id_t id, sample_t *out)
{
    int32_t raw;
    result_t r = sensor_read(id, &raw);      /* driver already retried once */

    switch (r) {
    case RES_OK:
        cache_store(id, raw, now_ms());
        out->value = raw;
        out->age_ms = 0;
        return RES_OK;

    case RES_TIMEOUT:
    case RES_CRC:
        diag_count(DIAG_SENSOR_TRANSIENT, id);
        return cache_fetch(id, out);         /* stale but stamped */

    case RES_NO_DEVICE:
        diag_count(DIAG_SENSOR_LOST, id);
        fault_emit(SRC_SENSOR, r, id);
        device_request_degrade(DOMAIN_SENSING);
        return cache_fetch(id, out);

    default:
        fault_emit(SRC_SENSOR, r, id);
        return r;
    }
}

Notice what this function does not do. It doesn’t retry — the driver already did. It doesn’t reset the bus — the bus manager owns that. It doesn’t decide whether the product should stop — it requests a degrade and lets the Device Manager decide. It doesn’t log a string.

One layer, one job, and every branch has a defined outcome. That’s what makes an architecture production-grade. Not the diagram.

Where This Goes Wrong

Five things I’d watch for on this exact design over the next two years. Article 03 covers structural anti-patterns properly; these are the ones this architecture is specifically exposed to.

The priority creep. Someone raises the MQTT task because telemetry looks laggy. Nobody notices that a TLS handshake now preempts the sensor task. Fix: put the priority table in a header with a comment explaining each number, and require a design note to change one.

The convenience include. #include "stm32h7xx_hal.h" appears in a service, once, for a delay. Then a second time. The CI check exists for exactly this.

The god struct. A system_state_t that everything reads and three things write. It starts as two fields. Ownership dies quietly.

Log inflation. The bounded ring buffer gets a “temporary” bypass so someone can debug a customer issue, and the bypass ships. Bounded means bounded.

MPU drift. A new DMA buffer gets added without an MPU region, and it happens to work because it landed in an already-non-cacheable area. Six months later a linker change moves it. Fix: put the MPU table and the linker script in the same review.

Next

This architecture works. Right up until it’s been maintained by four engineers for three years.

Article 03 is about how architectures decay — the specific structural patterns that turn a design like this one into something nobody wants to touch, and what the early symptoms look like while it’s still cheap to fix.

Article 03: Firmware Architecture Anti-Patterns — coming next.

Series Index

Phase 1 — Architecture

01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture (you are here)
03. Firmware Architecture Anti-Patterns

Phase 2 — Implementation

04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture

Phase 3 — Failure & Resilience

07. Error Handling & Recovery
08. Firmware Security Architecture

Phase 4 — Verification

09. Designing Firmware for Testability

Phase 5 — Product Lifecycle

10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates