Part 10 of 12 in the Embedded Firmware Architecture series · Phase 5: Operate · Level: Intermediate
Article 09 ended on a test suite that still passes, on a machine nobody can rebuild. Make that concrete. A customer reports a fault in firmware 3.7.2, released five years ago. You still have the git tag. You still have the test suite, and it still passes.
On a machine nobody can rebuild.
The compiler is three major versions on and generates different code. The vendor SDK download page returns a 404. The CI image was pinned to a tag that has since been repointed twice. The one engineer who set up the build left in year three, and the README says “install the usual toolchain.”
Nothing in that story is a firmware defect. Every part of it is an architecture decision that was never made, because nobody was asked to make it in year zero.
This article is about the decisions that only pay off years later: rebuilding a binary you shipped, reading configuration written by a firmware version that no longer exists, migrating to a replacement part, and handing the whole thing to engineers who were not there.
What Actually Changes
The KIC-400 is specified for ten years of field life with no scheduled maintenance. Over that span, a realistic sequence looks like this.
Year 0 Production release
Year 1 Feature release, flash usage passes 60%
Year 2 Hardware revision B, sensor second-sourced
Year 3 TLS library CVE, needs a version with a changed API
Year 4 Compiler upgrade forced by a toolchain support window
Year 5 Original firmware lead leaves
Year 6 MCU last-time-buy notice
Year 7 New regional communication requirement
Year 8 Migration to the replacement part
Year 10 End of support
That ratio, six firmware changes and one feature request, is the whole argument. An architecture optimised purely for adding features will be bad at five of the six firmware changes it actually has to make, and no help at all with the other three events.
Reproducible Builds for Long-Term Support
Start here, because it is the foundation everything else in the article sits on. If you cannot rebuild a shipped binary, you cannot bisect a field bug against it, you cannot prove what you shipped, and for regulated products you cannot answer an auditor.
The test is simple and almost no team has run it:
make clean && make && sha256sum build/firmware.bin
make clean && make && sha256sum build/firmware.binTwo builds, same machine, same minute. The hashes should match. The first time most teams try this, they do not.
Two refinements make the check meaningful. Compare the unsigned image: ECDSA signatures are randomised unless you sign deterministically (RFC 6979), so a signed binary differs every build by design. And once the same-machine check passes, make it harder: build in two different directories, on a second machine, with a different locale. Same-path builds hide exactly the path and environment leaks you are trying to find.
What breaks it
Timestamps compiled in. __DATE__ and __TIME__ anywhere in the source guarantees a different binary every build. The version banner is the usual culprit. Replace it with the git commit hash, which is meaningful and stable. If something must carry a date, set SOURCE_DATE_EPOCH; GCC uses it for __DATE__ and __TIME__, so the date comes from the commit rather than the clock.
Absolute paths in debug info. The build directory ends up embedded in DWARF sections and assertion strings, so a build from /home/chintan/work differs from one in /builds/ci/42. Fix with -ffile-prefix-map=$(PWD)=. so paths are recorded relative. The flag (GCC 8 and later) also covers __FILE__, which assert macros expand into every assertion string and which is the most common path leak in firmware.
Unordered source lists. A glob in the makefile returns files in filesystem order, which varies by machine, which changes link order, which changes the layout of the binary. Sort the list explicitly.
Gap fill and erased-state assumptions. The compiler emits padding deterministically, but the gaps between sections in a raw binary are filled by the conversion step, and different tools or versions choose different fill bytes. Set it explicitly (objcopy --gap-fill 0xFF to match erased flash) so the binary does not depend on a default.
The toolchain itself. The largest source of drift, and the one that needs infrastructure rather than a flag.
Pin the environment, by digest
A container image is the practical answer, with one detail that matters more than the rest:
# Pin by digest, never by tag. Tags get repointed.
FROM debian@sha256:a1b2c3d4...
# Pin the package mirror to a date as well, so apt resolves the same versions.
RUN rm -f /etc/apt/sources.list.d/debian.sources \
&& echo "deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/20260901T000000Z bookworm main" \
> /etc/apt/sources.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates xz-utils make cmake \
&& rm -rf /var/lib/apt/lists/*
# The toolchain comes from your release archive, not a vendor URL (see below).
COPY archive/arm-gnu-toolchain-13.3.rel1-x86_64-arm-none-eabi.tar.xz /tmp/tc.tar.xz
RUN echo "e1f2... /tmp/tc.tar.xz" | sha256sum -c - \
&& tar xf /tmp/tc.tar.xz -C /opt \
&& rm /tmp/tc.tar.xzFROM debian:bookworm is not a pin. That tag pointed at different content last year and will point at different content next year. The digest is the pin.
Then verify rather than assume. A CI job that builds twice and compares hashes catches reproducibility regressions on the commit that introduces them, which is the only time they are cheap to fix.
The Release Archive: Store Artifacts, Not URLs
Reproducibility gets you the ability to rebuild. The archive gets you the inputs, five years later, when half the internet you depended on has moved.
This is the rule that experience teaches expensively: vendor download links rot faster than your product lifetime. A URL to an SDK tarball is not an archive. It is a bet that a company you do not control will keep a file at a stable address for a decade, and it is a bet that loses often enough to plan around.
So store the bytes. Per release, in your own storage, immutable:
| What | Why the obvious version fails |
|---|---|
| Source at the exact commit | A tag can be moved or deleted |
| Container image by digest | The tag will be repointed |
| Vendor SDK tarball itself | The download page disappears |
| Every third-party source archive | Including transitive ones you did not choose |
| Full build log | Records the versions actually used, not the ones documented |
| The binary, and its hash | Lets you prove the rebuild matches |
| Map file and ELF with symbols | Without these, a field crash address is meaningless |
| Signing certificate chain | Certificates expire; you still need to verify old signatures |
| Test results and HIL logs | Evidence of what was actually verified |
| Architecture decision records as of that release | The reasoning, not just the outcome |
The map file and unstripped ELF deserve emphasis. Article 07 captures a program counter in the crash record. That value is a number, and turning it into a function name requires the exact symbols for the exact build. Discard the ELF and every field crash report from that release becomes permanently unreadable.
Persistent Data Is a File Format
Article 02 gave the config store two copies, a CRC and a sequence number, for power loss atomicity. Lifecycle adds a different requirement, and it runs in both directions.
A structure written to flash is a file format with an indefinite readership. Give it a real header:
typedef struct {
uint32_t magic; /* identifies the format */
uint16_t version; /* schema version, not firmware */
uint16_t length; /* total bytes: header + payload + CRC */
uint32_t sequence;
} config_header_t;
/* payload follows, then a trailer: */
typedef struct {
uint32_t crc; /* over header and payload */
} config_trailer_t;Backward compatibility is the easy direction
Firmware v2 reads a v1 record. Detect the version, run a migration function, write back at v2. Straightforward, and most teams get here.
Forward compatibility is the one that bites
Firmware v1 reads a v2 record.
This is not hypothetical, and it is the coupling most people miss. Article 12’s rollback exists precisely so a bad update can revert to the previous image. When it does, the old firmware boots and finds configuration written by the newer firmware it just replaced. If old firmware rejects unknown versions, your rollback path leads straight to factory defaults, and the device comes back from a failed update having also forgotten its settings. That turns a recoverable update failure into a site visit.
So old firmware must tolerate config it does not fully understand:
/* buf_len: bytes actually read from EEPROM. KNOWN_SIZE includes the header. */
if (buf_len < sizeof(config_header_t) + sizeof(config_trailer_t)) return CFG_CORRUPT;
memcpy(&hdr, buf, sizeof hdr);
if (hdr.length > buf_len) return CFG_CORRUPT;
/* Minimum length depends on the record's own version: older records are shorter. */
size_t min_len = (hdr.version >= CONFIG_VERSION_SUPPORTED)
? KNOWN_SIZE + sizeof(config_trailer_t)
: min_size_for_version(hdr.version); /* includes trailer, 0 if unknown */
if (min_len == 0 || hdr.length < min_len) return CFG_CORRUPT;
if (!crc_ok(buf, hdr.length)) return CFG_CORRUPT; /* never parse unverified data */
if (hdr.version > CONFIG_VERSION_SUPPORTED) {
/* Read the fields we know. Preserve the rest verbatim. */
size_t tail = hdr.length - KNOWN_SIZE - sizeof(config_trailer_t);
if (tail > sizeof cfg.unknown_tail) return CFG_TOO_NEW; /* keep the copy, refuse */
load_known_fields(&cfg, buf);
memcpy(cfg.unknown_tail, buf + KNOWN_SIZE, tail);
cfg.unknown_len = tail;
/* Do not rewrite unless the user changes something. */
}The checks come first for a reason: this record lives in external EEPROM, which article 08 lists as reachable by anyone with an I2C probe, so its length field is untrusted input. The minimum length is checked against the record’s own version: KNOWN_SIZE applies only to records at or above the version this firmware supports, and an older, shorter record is validated against its own size and handed to the migration path instead of being rejected as corrupt. The CRC catches corruption, but anyone who can rewrite the EEPROM can also compute a valid CRC, which is why the bounds checks stand on their own. Two rules make the rest work. Never reorder or reuse a field; only append. And when rewriting a record you only partly understand, carry the unknown tail through unchanged so a subsequent roll forward does not lose the newer settings. In practice the old firmware keeps the original version number and the tail verbatim and changes only the fields it knows, so the newer firmware reads back a record that is still in its own format. If the unknown tail does not fit (CFG_TOO_NEW), the old firmware leaves that newer record untouched, even though its sequence number is higher, and falls back to the older valid copy in the other slot of article 02’s two-copy store. Article 12 relies on the same rule.
Pin the layout as well. The struct is now a file format, and this article’s own warning about compiler upgrades applies to it: padding, alignment and byte order are ABI details, not guarantees. Assert them, so a toolchain change that moves a field fails the build instead of corrupting fielded units:
#include <stddef.h>
_Static_assert(sizeof(config_header_t) == 12u, "config header layout changed");
_Static_assert(offsetof(config_header_t, version) == 4u, "version moved");
_Static_assert(offsetof(config_header_t, length) == 6u, "length moved");
_Static_assert(offsetof(config_header_t, sequence) == 8u, "sequence moved");The stored format is little-endian, as every Cortex-M part here is. A host tool that reads these records should decode them field by field rather than casting a buffer to the struct.
The same discipline applies to your protocol wire format and your calibration data, both of which outlive individual firmware versions.
The Memory Budget Is a Governed Resource
The usual trajectory:
v1.0 Flash 55% RAM 48%
v2.0 Flash 64% RAM 55%
v3.0 Flash 73% RAM 61%
v4.0 Flash 81% RAM 74%
v5.0 Flash 87% RAM 82%
v5.1 "We can't add that, there's no flash left."Nobody caused that. Each release added a few percent, each one justified, and the constraint arrived as a surprise in a planning meeting four years in.
The number is smaller than you think
For the KIC-400 the ceiling is not 2 MB. Article 02 chose the H743ZI for dual-bank A/B updates, which means the application has one 1 MB bank, minus a 128 KB bootloader sector and a 128 KB fault-history sector, because H743 flash erases in 128 KB sectors and no independently erasable region can be smaller (article 12 shows the layout). Programming is finer grained: 256-bit flash words let small append-only records share a sector, which is how the fault history and the security version log work. The real application budget is 768 KB, and 87% of 2 MB was never available at all.
This is article 01’s flash occupancy rule arriving years later. If the application cannot fit twice, your update architecture changes, and by year four that is not a decision you can revisit.
Gate it in CI
Budgets that live in a spreadsheet are not budgets. Make the build fail:
FLASH_BUDGET=786432 # 768 KB application slot
RAM_BUDGET=262144 # 256 KB
# Flash: every allocated section with a load address in flash, including
# .isr_vector, .ARM.exidx and ITCM code that is copied from flash at boot.
arm-none-eabi-objcopy -O binary build/firmware.elf build/firmware.bin
flash=$(stat -c %s build/firmware.bin)
ram=$(arm-none-eabi-size -A build/firmware.elf \
| awk '/^\.(data|bss|dtcm|axi|noinit)/ {s+=$2} END {print s}')
[ "$flash" -le "$FLASH_BUDGET" ] || { echo "FLASH BUDGET EXCEEDED: $flash > $FLASH_BUDGET"; exit 1; }
[ "$ram" -le "$RAM_BUDGET" ] || { echo "RAM BUDGET EXCEEDED: $ram > $RAM_BUDGET"; exit 1; }Adjust the RAM section names to your linker script. Better still, put the same limits in the linker script itself with ASSERT(), so the build fails even when someone runs it without the CI wrapper. The linker check is also the more robust one for flash: objcopy -O binary spans the lowest to the highest load address, so a single section with a RAM load address, such as a .noinit region declared without NOLOAD, inflates the file to hundreds of megabytes and the gate fails for the wrong reason.
Then plot it per release. The slope tells you when you will run out, and a slope discovered in year two is a planning input rather than an emergency.
Article 05’s stack high-water marks and article 09’s soak counters belong on the same dashboard. Flash, RAM, stack and persistent storage are one budget viewed four ways.
Dependency Inventory, Including What You Did Not Choose
Every third-party component is a lifecycle liability, because each one can produce a CVE at a time of its choosing.
Track version, source, license, upgrade path and who owns the decision to upgrade. Article 08 noted that the CRA expects a software bill of materials, and this list is most of it. Emit it in a standard format, SPDX or CycloneDX, generated in CI from the release archive rather than maintained by hand, so the SBOM describes exactly the inputs that built the shipped binary.
The part that surprises people: your vendor SDK contains third-party code you never selected. An ST SDK ships with a specific lwIP version, a specific mbedTLS, sometimes FatFS and a USB stack. Those versions are in your binary, they are in your SBOM, and when one of them has a CVE your upgrade path runs through the SDK rather than through the upstream project. You may be waiting on a vendor release, or vendoring the fix yourself and maintaining a patched SDK.
Find that out during architecture, not during an incident. Enumerate the transitive components in your SDK, record their versions, and decide in advance whether you would patch locally or wait.
And treat a dependency update as an engineering change, never a version bump. A new TLS release can change stack depth, which changes article 05’s task sizing, which changes RAM. It can change timing, which changes the schedulability analysis. Run the same pipeline you would run for a feature.
Component Obsolescence and the Last-Time Buy
The timeline above puts an MCU last-time-buy notice in year six. That is not bad luck. Every part on the BOM has its own lifecycle, and in a ten-year product some of them will end before yours does. The work is noticing early and already having a plan with a cost attached.
Subscribe to PCNs for every part on the BOM. Product change and end-of-life notices go to whoever registered with the manufacturer or distributor, which is often someone who has since left. Register a shared address per manufacturer, and run a lifecycle monitoring service or a quarterly report against the full BOM, because a passive regulator going end of life stops production as surely as the MCU does.
Size the last-time buy from three numbers: the production forecast up to the point the replacement is qualified, spares for repairs across the support window, and replacements for field returns at your measured return rate, plus margin for yield loss. Buy too few and production stops before the migration ships; buy too many and you scrap inventory you paid for. The forecast needs an owner and a review, not a single guess.
Read the vendor’s longevity commitment per part. ST’s STM32 longevity programme is a ten-year commitment renewed annually, and each part number carries an “available until” date. Check that date for your exact H7 part at selection and every year after, rather than assuming a decade from your design start.
Plan the second source at selection time. Prefer a package with a pin-compatible path within the family, footprint alternates for sensors and memories, and record the intended replacement in an architecture decision record while the reasoning is fresh.
Firmware decides what the notice finally costs. If vendor headers stop at the HAL boundary and board specifics live in the BSP, as article 04 requires, the migration below is a bounded port of the lower layers. If they leak upward, the last-time buy has to cover the time it takes to rewrite the product.
MCU Migration: What Abstraction Actually Saves
Year six brings the last-time-buy notice the previous section planned for. Year eight you migrate. The replacement is pin-compatible, which tempts everyone into assuming this is a small job.
Here is the honest breakdown, assuming article 04’s layer rules held:
| Layer | Reusable | Why |
|---|---|---|
| Application | ~100% | No vendor headers by construction |
| Services | ~95% | Occasional assumptions about timing or storage geometry |
| Middleware | ~90% | Protocol logic ports; integration glue does not |
| Peripheral drivers | ~0% | Different registers, different DMA |
| BSP | 0% | Board and part specific by definition |
| Startup and linker | 0% | Different memory map |
| RTOS port | ~50% | Kernel unchanged; port layer, configuration and revalidation are the work |
| Bootloader | ~20% | Logic survives; flash controller and security do not |
Treat those percentages as a planning heuristic for a codebase that kept article 04’s layer rules, not as measured data. Abstraction localises the work. It does not eliminate it, and anyone promising otherwise has not done a migration.
What that table buys you is a migration that is bounded and estimable. Without the layer discipline, the reusable column collapses and the answer to “how long will the port take” becomes “we will find out.”
Two things get forgotten in migration planning, both of which cost more than the driver work.
Performance must be revalidated from scratch. Different cache, different wait states, different DMA arbitration, different interrupt latency. Every worst-case execution time in article 05’s task table is now unverified, and the schedulability analysis is a document about a part you no longer use. Re-measure, do not scale.
Security configuration does not port. Option bytes, readout protection levels, OTP layout and secure boot mechanisms differ between parts and sometimes between variants of the same family. Article 08’s decisions need to be made again against the new reference manual.
Toolchain Drift Invalidates Your Timing Analysis
This one is not obvious and it catches good teams.
A compiler upgrade is usually treated as maintenance. New version, rebuild, tests pass, ship it.
But a new compiler makes different inlining decisions, different register allocation, different loop transformations. Your control task that measured 60 microseconds may now measure 52, or 71. The binary is smaller or larger, which shifts the memory map, which changes which code is in which cache line.
Article 05’s entire schedulability analysis rests on measured execution times. A compiler change invalidates those measurements, which means:
- Re-measure worst-case execution time for every task with a deadline
- Re-measure the interrupt handler budget
- Re-check flash and RAM against the budget
- Re-run the soak with the counters from article 09
Treat a compiler upgrade as a hardware change with a software delivery mechanism. Plan it, schedule the revalidation, and never do it in the same release as a feature, because when the timing moves you want exactly one candidate explanation.
Feature Flags and the Shipped Configuration Matrix
Ten boolean flags create 1024 theoretical builds. You will not test 1024.
You do not need to. You need to test the combinations you actually ship, which means knowing what they are:
| Configuration | Region | Sensors | Cloud | Shipped since |
|---|---|---|---|---|
| STD-EU | EU | 4 | Yes | v1.0 |
| STD-NA | NA | 4 | Yes | v1.0 |
| LITE-NA | NA | 2 | No | v2.1 |
| OEM-X | EU | 4 | No | v3.0 |
Four configurations. Those get the full test matrix. Every other combination is unsupported, and saying so explicitly is the difference between four test runs and a combinatorial problem.
Then give each flag an expiry. A flag introduced to de-risk a rollout has a job that ends once the rollout succeeds, and the last step of that job is deleting the flag and the disabled branch. Article 03’s decay comes for feature flags the same as everything else: a temporary flag that survives five years is a permanent second code path that nobody tests.
Branching, Freezing and Backporting
Fielded products cannot all track your newest architecture. That is not a failure, it is the normal state of a fleet.
A workable shape:
main active development, next release
release/5.x current shipping, bug and security fixes
release/4.x security fixes only
release/3.x critical security only, sunset announcedThe discipline that makes this survivable is deciding, per branch, what is allowed in. Security fixes and field-critical bugs go to maintenance branches. Features and refactors do not, no matter how tempting, because every merge into an old branch is a merge into a codebase your current test environment may not fully reproduce.
Backporting is where the architecture pays out or does not. If the TLS integration sits behind an interface, a security fix touches the integration layer and the layer above is unaffected. If TLS calls are scattered through service code, the same fix is a different change on every branch, each needing its own testing.
Decide the support window in advance and publish it. “Security fixes for five years from end of sale” is a commitment you can architect around. Silence is a commitment you will be held to anyway, at a time and scope somebody else chooses.
The Turnover Test
Year five, the original firmware lead leaves. The relevant question is not whether the code is good. It is how long a competent new engineer needs before they can make a safe change.
Make it a measurement rather than an opinion. Hand a new senior hire the repository and a real, small task. Time it. That number is your onboarding cost, and it is the most honest maintainability metric you have.
The artifacts that shorten it are ones this series has already produced as side effects of doing other things well:
- The task table from article 05: what runs, when, at what priority, with what deadline
- The ownership matrix from article 02: who owns each resource and each piece of state
- The layer rules and CI check from articles 02 and 04: what may call what, enforced
- The recovery matrix from article 07: every failure, its classification and its escalation
- The architecture decision records from article 01: why, not just what
That last one matters most and is the one skipped most. Five years on, the decision is visible in the code and the reasoning is not. An engineer who cannot find the reasoning has two options, both bad: preserve a constraint they do not understand, or remove it and discover why it existed.
Three sentences per decision. What we chose, what we rejected, what would make us revisit. Keep them in the repository, versioned with the code, archived with each release.
End of Life Is a Deliverable
A product does not become unsupported because the team moved on. It becomes unsupported on a date you announce, with a final state you defined.
At end of support, hand over or archive:
- The final supported firmware version and its complete build archive
- The security posture: known unpatched CVEs, and the honest statement about them
- Signing key custody, including who can still produce a valid image and who should not be able to
- Manufacturing and calibration procedures, if spares are still built
- The recovery procedure, so a field unit can still be revived
- Customer communication: what stops, when, and what the migration path is
The signing key item is the one with teeth. A key that can still sign firmware for a fleet nobody maintains is a liability that outlives the product. Decide at end of life whether that key is retired, escrowed, or revoked, and record the decision.
Next
Everything in this article assumes you find out that something is wrong.
A ten-year fleet tells you almost nothing by default. Units run in plants and basements and substations, and the information reaching you is filtered through a customer who noticed something and an email that says the device is acting strange. Article 07 built a crash record. Article 09 logged soak counters. Neither of those helps if nothing gets the data off the device and in front of an engineer.
Article 11: Field Diagnostics and Firmware Observability covers designing firmware that can explain what went wrong, to someone who was not there, months later.
References
- Reproducible Builds: SOURCE_DATE_EPOCH
- RFC 6979, deterministic ECDSA
- Regulation (EU) 2024/2847, the Cyber Resilience Act
Series Index
Overview, reading paths and related articles: Embedded Firmware Architecture series guide.
Phase 1: Decide
01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture
03. Firmware Architecture Anti-Patterns
Phase 2: Build
04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture
Phase 3: Fail safely
07. Error Handling & Recovery
08. Firmware Security Architecture
Phase 4: Verify
09. Designing Firmware for Testability
Phase 5: Operate
10. Designing Firmware for 10-Year Products (you are here)
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates
