Part 11 of 12 in the Embedded Firmware Architecture series · Phase 5: Operate · Level: Intermediate
Ten thousand KIC-400 units are in the field. Twenty of them, two tenths of one percent, occasionally stop responding on Modbus TCP and recover after a power cycle. The customer noticed because their SCADA polling logged gaps, not because of any firmware observability on the device.
You cannot reproduce it. The units are in plant rooms across three countries. You cannot attach a debugger. The failure happened once per unit, weeks apart, and every one of them has rebooted since.
The only thing that can tell you what happened is the firmware itself, and only if somebody decided years ago that it should be able to.
Articles 07 and 09 built most of the machinery for this without calling it firmware observability. This article is about turning that machinery into evidence an engineer can act on, and about the two budgets that decide how much of it you can actually afford.
Firmware Observability Is Not Logging
Logging produces a record of what the code chose to mention. Observability is the property of being able to answer questions you did not anticipate.
A device can emit thousands of lines and be opaque:
ERROR
ERROR
ERROROne structured record beats all of them:
seq 4471 t=1834921ms SEV=ERROR SUBSYS=SENSOR STATE=CALIBRATING
code=RES_TIMEOUT detail=cs2 retry=3/3 action=PERIPHERAL_RESET
fw=5.3.1+a4f21c hw=REV-C cfg=8The difference is not verbosity. The record carries subsystem, state, count, what the firmware did about it, and enough identity to place the device in a fleet. The string does not.
Five categories, and they have different costs and different lifetimes:
| Category | Question it answers | Where it lives |
|---|---|---|
| Identity | What is this device running? | Every record, always |
| State | What was it doing? | RAM, snapshot on demand |
| Events | What sequence led here? | RAM ring, persisted selectively |
| Faults | What went wrong, how often? | Counters in RAM, records in flash |
| Metrics | Is it getting worse? | Aggregated, transmitted periodically |
What the Series Already Built for Field Diagnostics
Most of the parts exist. This article’s job is largely to connect them rather than invent them.
Reset reason, latched from the RCC reset status register in startup before anything clears it, from article 07.
The crash record in backup SRAM: boot count, reset reason, fault class and source, stacked program counter and link register, CFSR, task name, per task stack headroom. Article 07 built it inside a HardFault handler that does nothing risky.
The fault sink from article 03: fault_emit(source, code, detail) writing eight bytes into a bounded, ISR-safe ring (a short critical section, as article 02 shows) with no dependencies above the driver layer.
The four runtime numbers from article 05: runtime share per task, stack high water marks, peak queue depth, and drop counters per queue.
The transition trace from article 06: eight bytes per state machine transition, sixty four entries in backup SRAM, reconstructing the last sixty four behavioural decisions before a failure.
That is a working observability core that costs under a kilobyte of RAM. What remains is the harder part: making it identifiable, affordable, retrievable and useful across a fleet.
Identity Is About Slicing the Fleet
Every write up on this topic says to record the firmware version so you know what is running. That is true and it undersells the point.
With ten thousand units, the single most valuable diagnostic operation is not reading one device. It is asking which devices are affected:
Units on firmware 5.3.1 with hardware revision C show eight times the watchdog reset rate of every other combination.
That query answers the question in one step. It costs a hardware revision field in every record, and it is impossible to run retroactively if the field is not there.
So the identity block travels with the data rather than living in a separate page somebody has to look up:
Device ID KIC400-004821
Firmware 5.3.1+a4f21c
Bootloader 2.1.0
Hardware REV-C
Config schema 8
Calibration 3
Security version 6Two details that matter more than they look.
The device reports its own identity. Never trust the version the customer states. The most common wasted day in field support is debugging against the firmware someone believes is installed.
The version carries a build identifier, not just a semantic version. 5.3.1 is a marketing string, and if that release was rebuilt after a toolchain bump or a hotfix, two different binaries share it. A commit hash narrows it but does not close it, because the same commit built with two different toolchains is two binaries with one hash. Link with -Wl,--build-id, which adds a GNU build-id note (a hash of the linked output) to the ELF; keep that note in flash, copy it into the image header, and record it in every crash record. With article 10‘s reproducible builds, a rebuild of the same commit in the pinned environment is bit identical and carries the same build-id, so the case this guards against is the one that matters: a different binary behind the same version string. The next section is why this matters concretely.
Turning a Program Counter Into a Line Number
Article 07 captures a program counter in the crash record. That value is a bare number, and by itself it is useless.
The workflow that makes it useful:
arm-none-eabi-addr2line -f -i -e archive/5.3.1+a4f21c/firmware.elf 0x08004A2C
# modbus_frame_validate
# /src/middleware/modbus_frame.c:214Three things have to be true for that command to work, and each is a decision someone makes years earlier.
The unstripped ELF with symbols must still exist, which is article 10‘s release archive.
The exact build must be identifiable. This is why the build-id belongs in the crash record, and why the tool selects the archived ELF by build-id rather than by version string. If 5.3.1 was rebuilt, the map file you kept may correspond to a different layout than the binary that crashed, and addr2line will confidently give you the wrong function. A wrong answer here is worse than no answer, because people act on it.
And the address must be interpretable. Under article 12‘s bank swap, both images run at 0x08000000, so a slot base tells you nothing about which image faulted. What places the address is the image identity, the build-id, recorded next to it. A program counter without that context can land in the wrong image entirely.
Three details decide whether the line you get is the right one. Use -i, because optimised code is heavily inlined, and without it addr2line names only the outermost function. The stacked LR is a return address with bit 0 set for Thumb state, so clear bit 0 and look up the call instruction before it (LR minus 2 or 4) to get the call site rather than the line after it. And for an imprecise BusFault, with IMPRECISERR set and BFARVALID clear in the CFSR, the stacked PC is already past the faulting store, so the culprit is somewhere before the line it names. A precise fault with BFARVALID set gives you both the instruction and the address it touched.
Make it a tool your support engineers run rather than a procedure they follow. A script that takes a crash record and emits a symbolicated report removes the step where somebody picks the wrong ELF.
Alive Is Not Healthy
Article 07’s supervised watchdog checks that every task has checked in within its deadline. That catches a task that stopped. It does not catch a task that is running perfectly and accomplishing nothing.
COMM_TASK heartbeat OK, every 100 ms
COMM_TASK success rate 0% over the last 30 minutesThe task is alive. The subsystem is dead. A liveness check will never notice, and this is the failure mode behind a large share of field reports that say the device is on but not working.
So each subsystem declares a functional health indicator with a threshold, separate from its heartbeat:
| Subsystem | Functional indicator | Unhealthy when |
|---|---|---|
| Modbus TCP | Successful responses per hour | Zero while link is up |
| Cloud | Publishes acknowledged / attempted | Below 50% over an hour |
| Sensing | Valid samples / expected samples | Below 90% per sensor |
| Storage | Successful writes / attempted | Any failure in 24 hours |
| Control | Loop deadlines met | Any miss |
Two columns, both cheap. The heartbeat says the task exists. The indicator says it is doing its job. A device reporting all heartbeats healthy and three indicators red is telling you exactly where to look.
The Wear Budget
Here is the calculation that turns rate limiting from good manners into an architectural requirement, and it is the section I would keep if I had to cut the rest.
Persistent diagnostics consume flash endurance, and flash endurance is finite in a way that matters over ten years.
Work it for the KIC-400. Article 12’s layout gives the KIC-400 a 128 KB fault history sector at the end of each bank, outside the application slot, addressed by physical bank because the bank swap moves it. Use the two as a ping-pong pair: write 32 byte records in a log structured pattern into the active sector, and when it fills, copy the live records (the counters and the retained rings in the table further down, about 5.3 KB) into the other sector before erasing the full one. Erasing a single sector when it fills would wipe the entire fault history at that moment, which is exactly when you may need it. Take the part’s endurance as roughly 10,000 erase cycles.
Records per erase cycle 131072 / 32 = 4096
Live records migrated 5376 / 32 = 168
Net records per cycle 4096 − 168 = 3928
Total records in life 2 x 3928 x 10000 = 78,560,000
Product life 10 years = 3650 days
Daily record budget 78.56M / 3650 ≈ 21,500 records/dayTwenty thousand a day sounds generous, and for normal operation it is. A handful of boot events, a few configuration changes, occasional faults. You will use a few dozen.
Now consider a single stuck I2C sensor generating a timeout every 100 ms:
10 records/second x 86400 = 864,000 records/day
Lifetime of both sectors 78.56M / 864,000 ≈ 91 daysOne failed twenty cent component wears out both fault-history sectors in about three months, and it does so while faithfully recording that something is wrong. On an H7 there is a second cost: every one of those sector erases stalls fetches from that bank for the whole erase, around a second (DS12110 specifies a 128 KB sector erase of about 1 s typical, 2 s maximum, at x64 parallelism, and longer at narrower settings), as article 02 showed, so anything not running from ITCM or RAM stalls with it.
That is the whole argument. Unbounded persistent logging is not a resource inefficiency, it is a hardware failure mechanism with a calculable date. Rate limiting is the control, and it is not optional.
Run this arithmetic for your own part and your own partition, because the endurance figure and the sector size both change the answer by an order of magnitude. Then write the daily budget down next to the flash and RAM budgets from article 10, and gate it the same way.
The record size is not arbitrary either. H7 flash programs 256-bit flash words with ECC, so each 32 byte record is one flash-word aligned word, written once and never reprogrammed. Power loss in the middle of programming a word can leave it with inconsistent ECC. A single-bit error is corrected silently; a double-bit error sets DBECCERR and raises a bus fault on read. The log scanner that runs at boot must expect that: read the log with the bus fault handled, or check the ECC status per word, treat a bad word as a torn record at the end of the log, and resume appending after it.
The Bandwidth Budget
Connected products have a second budget, and it is usually the one somebody is paying for monthly.
A cellular device on a 5 MB per month plan:
5 MB / 30 days ≈ 170 KB/day
MQTT + TLS overhead on a small message roughly doubles it
Periodic TLS handshake a few KB each
Practical payload budget ≈ 60 to 80 KB/daySixty kilobytes a day is around three hundred small messages, or one modest snapshot plus a periodic heartbeat. It is not a stream of events.
That number shapes the architecture rather than the implementation. It means the device sends summaries and the detail stays local until requested:
Pushed periodically: identity, uptime, boot count, health indicator status, fault counters, update status. A few hundred bytes.
Pulled on request: the full snapshot, the event ring, the crash record, the transition trace. Kilobytes, sent once, when an engineer asks.
Never pushed: raw event streams, per sample data, anything proportional to runtime.
The device is the primary store. Telemetry is an index into it.
Retention, Stated Explicitly
Every category gets a home, a size, and a discard rule, decided at design time rather than discovered when the sector fills.
| Data | Where | Size | Discard rule |
|---|---|---|---|
| Crash record | Backup SRAM | ~128 B | Overwritten next crash |
| Transition trace | Backup SRAM | 512 B | Ring, 64 entries |
| Boot and reset history | Flash | 1 KB | Ring, last 32 boots |
| Critical faults | Flash | 4 KB | Ring, last 128 |
| Fault counters | Flash, periodic | 256 B | Cumulative, never reset |
| Event ring | RAM | 4 KB | Ring, lost on reset |
| Metrics | RAM, aggregated | 512 B | Transmitted, then reset |
The flash rows are logical limits inside the log structured fault history sector, not separate regions: the boot history and critical fault rings are the newest 32 and 128 records of their kind, and older ones are simply not migrated at the next sector switch. Every periodic counter write is also a 32 byte record, and it counts against the same daily budget.
Two things worth calling out.
Counters are cumulative and never reset. A counter that resets on reboot cannot distinguish a device that has seen four I2C errors from one that has seen four hundred thousand across a thousand reboots. Cumulative counters, written infrequently, are the cheapest degradation signal you have.
The event ring lives in RAM and that is acceptable. Detailed event history is valuable when you can retrieve it and not worth flash endurance to preserve across a reset. The events that must survive are the ones in the crash record and the persistent fault ring, which is a deliberately small set.
Rate Limiting and Aggregation
The mechanism that keeps the wear budget honest. Three tiers per fault type:
First occurrence: full record with all context, persisted.
Next N occurrences: counter only, in RAM.
Periodic summary: one aggregated record, persisted at a fixed interval.
I2C1 timeout, sensor 2
first t=1834921 state=CALIBRATING retry=3/3
count 1832
window t=1834921 to t=2483110
last t=2483110That record occupies the same 32 bytes as one raw event and carries strictly more information: it establishes onset, rate, and duration. Eighteen hundred raw records would have told you the same thing while costing eighteen hundred times the wear.
The first occurrence is the one to keep in full. It is the causal event. Everything after it is consequence, and consequence compresses into a count.
The Snapshot Is a Versioned Wire Format
Article 10 established that persistent structures are file formats with an indefinite readership. The diagnostic snapshot is the same thing, and it has a worse problem: your field tooling in year eight must decode a snapshot produced by firmware from year one.
So it gets a header, a schema version, and a decoder that never drops support for an old version:
typedef struct {
uint32_t magic;
uint16_t schema_version;
uint16_t length;
/* identity block, then sections */
} snapshot_header_t;Fixed record sizes, explicit field ordering, append only evolution. Never reorder, never reuse a field, and give every section its own length so a newer decoder can skip a section it does not recognise and an older one can skip a section that did not exist.
And keep the on device representation binary. Formatting to text on a constrained device costs code space, stack, and CPU at exactly the moment the system is unhealthy. Emit compact records, decode them in the tool.
Fleet Signal Beats Device Signal
A single device cannot tell you that its retry count is high, because it has no baseline. Three hundred communication retries per day might be normal for a plant with a long RS-485 run and alarming for a unit on a bench.
Trend analysis belongs off the device. What the device provides is comparable numbers: cumulative counters, uptime, boot count, health indicator states, all tagged with identity. What the fleet view provides is the comparison.
The queries that actually find problems are all fleet shaped:
- Which firmware and hardware combinations have elevated watchdog rates
- Which units show a rising retry count over the last thirty days
- Which units have boot counts increasing with no uptime accumulating
- Which units accepted the last update and which are still on the old image
That last one is the routine payoff and it is worth stating plainly. After a fleet update, the useful question is not whether the update worked. It is which two hundred devices did not take it, and what they have in common. Article 12’s rollback mechanism produces exactly the population you need to identify.
Keep on device trend logic minimal. A rising counter is a fact worth transmitting. Deciding whether the rise is significant needs ten thousand comparisons the device cannot make.
Diagnostics Must Not Destabilise the System
The failure mode is real and it is embarrassing: a device crashes, the crash handler tries to log, the logging path fails, and the device hangs instead of resetting.
Four rules.
No allocation in the diagnostic path. Static buffers, fixed size records, preallocated rings. A device under memory pressure is precisely when diagnostics are most needed and most likely to fail.
No blocking. fault_emit writes into a bounded ring under a critical section of a few dozen instructions and returns. It does not wait for flash, it does not take a mutex, it does not touch the network. The diagnostic task drains the ring at low priority.
A bounded CPU cost. Article 05 budgets task execution time, and the diagnostic task is a task. Under one percent in normal operation, and it must not grow under fault load, which is another way of saying the rate limiter is a real time requirement rather than a storage optimisation.
Every diagnostic operation can fail silently. Flash full, network down, ring full. Each failure increments a counter and the system carries on. A dropped diagnostic event is always better than a disturbed control loop.
The principle underneath: diagnostics observe the system and must never become a dependency of it.
Reading Is Not Controlling
Article 08 covered securing the interfaces. One split belongs here because it is a diagnostics design decision rather than a security one.
| Read | Control |
|---|---|
| Identity and versions | Reset the device |
| Health indicators | Change configuration |
| Fault counters | Trigger a firmware update |
| Snapshot and crash record | Clear fault history |
| Event ring | Change the diagnostic level |
Observing should require less privilege than controlling. A support engineer needs the left column routinely, and giving them the right column so they can read a counter is how a diagnostic interface becomes the easiest way into the product.
Two related constraints. Clearing fault history is a control operation, because an attacker who can erase evidence has defeated the point. And runtime diagnostic level changes must be authenticated and self expiring, because an elevated logging level left on permanently will consume the wear budget calculated above.
What never appears in diagnostic output, regardless of level: key material, credentials, private keys, and any customer process data the device happens to be carrying. A snapshot is an artifact that gets emailed around.
Design Firmware Observability During Architecture
Eight questions, answered at design review rather than after the first field failure:
- Which failures matter enough to survive a reset?
- What is the daily persistent write budget, given this part’s endurance?
- What is the daily telemetry budget, given this connectivity plan?
- Which state must a snapshot capture to be actionable?
- What is the functional health indicator for each subsystem, and its threshold?
- What must never appear in diagnostic output?
- Which queries will you run across the fleet, and what field does each require?
- Who symbolicates a crash address, with which tool, from which archive?
Question seven is the one that is impossible to retrofit. A field you did not record is a query you cannot run about the past.
Next
You can now detect that a fleet has a problem, identify which units, and get evidence about why.
Then you have to ship the fix. To ten thousand devices, over networks that drop, to units that may lose power mid write, some of which are running firmware from three releases ago, in plant rooms nobody will visit. Article 12 opens with what separates a failed update from a recall.
Article 12: OTA and Safe Firmware Updates covers designing updates that cannot brick the device.
References
- GNU binutils: addr2line
- STM32H743ZI documentation (RM0433 reference manual and datasheet)
- Arm Cortex-M7 Technical Reference Manual
- MQTT Version 5.0, OASIS Standard
Series Index
Overview, reading paths and related articles: Embedded Firmware Architecture series guide.
Phase 1: Decide
01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture
03. Firmware Architecture Anti-Patterns
Phase 2: Build
04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture
Phase 3: Fail safely
07. Error Handling & Recovery
08. Firmware Security Architecture
Phase 4: Verify
09. Designing Firmware for Testability
Phase 5: Operate
10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability (you are here)
12. OTA & Safe Firmware Updates
