Article 09 ended with a product lifecycle scenario worth sitting with. A customer reports a fault in firmware 3.7.2, released five years ago. You still have the git tag. You still have the test suite, and it still passes.
On a machine nobody can rebuild.
The compiler is three major versions on and generates different code. The vendor SDK download page returns a 404. The CI image was pinned to a tag that has since been repointed twice. The one engineer who set up the build left in year three, and the README says “install the usual toolchain.”
Nothing in that story is a firmware defect. Every part of it is an architecture decision that was never made, because nobody was asked to make it in year zero.
This article is about the decisions that only pay off years later: rebuilding a binary you shipped, reading configuration written by a firmware version that no longer exists, migrating to a replacement part, and handing the whole thing to engineers who were not there.
What Actually Changes
The KIC-400 is specified for ten years of field life with no scheduled maintenance. Over that span, a realistic sequence looks like this.
Year 0 Production release
Year 1 Feature release, flash usage passes 60%
Year 2 Hardware revision B, sensor second-sourced
Year 3 TLS library CVE, needs a version with a changed API
Year 4 Compiler upgrade forced by a toolchain support window
Year 5 Original firmware lead leaves
Year 6 MCU last-time-buy notice
Year 7 New regional communication requirement
Year 8 Migration to the replacement part
Year 10 End of supportSeven of those nine events change the firmware, and only one of them is a feature request. That ratio is the whole argument. An architecture optimised purely for adding features will be bad at seven of the nine things it actually has to do.
Reproducible Builds, and the Check Nobody Runs
Start here, because it is the foundation everything else in the article sits on. If you cannot rebuild a shipped binary, you cannot bisect a field bug against it, you cannot prove what you shipped, and for regulated products you cannot answer an auditor.
The test is simple and almost no team has run it:
make clean && make && sha256sum build/firmware.bin
make clean && make && sha256sum build/firmware.binTwo builds, same machine, same minute. The hashes should match. The first time most teams try this, they do not.
What breaks it
Timestamps compiled in. __DATE__ and __TIME__ anywhere in the source guarantees a different binary every build. The version banner is the usual culprit. Replace it with the git commit hash, which is meaningful and stable.
Absolute paths in debug info. The build directory ends up embedded in DWARF sections and assertion strings, so a build from /home/chintan/work differs from one in /builds/ci/42. Fix with -ffile-prefix-map=$(PWD)=. so paths are recorded relative.
Unordered source lists. A glob in the makefile returns files in filesystem order, which varies by machine, which changes link order, which changes the layout of the binary. Sort the list explicitly.
Uninitialised padding. Structure padding and unwritten flash regions can carry whatever was in memory. Fill explicitly, and set the pad byte for your binary conversion step.
The toolchain itself. The largest source of drift, and the one that needs infrastructure rather than a flag.
Pin the environment, by digest
A container image is the practical answer, with one detail that matters more than the rest:
# Pin by digest, never by tag. Tags get repointed.
FROM debian@sha256:a1b2c3d4...
RUN curl -fsSL https://.../gcc-arm-none-eabi-13.2.tar.bz2 \
-o /tmp/tc.tar.bz2 \
&& echo "e1f2... /tmp/tc.tar.bz2" | sha256sum -c - \
&& tar xf /tmp/tc.tar.bz2 -C /optFROM debian:bookworm is not a pin. That tag pointed at different content last year and will point at different content next year. The digest is the pin.
Then verify rather than assume. A CI job that builds twice and compares hashes catches reproducibility regressions on the commit that introduces them, which is the only time they are cheap to fix.
The Release Archive: Store Artifacts, Not URLs
Reproducibility gets you the ability to rebuild. The archive gets you the inputs, five years later, when half the internet you depended on has moved.
This is the rule that experience teaches expensively: vendor download links rot faster than your product lifetime. A URL to an SDK tarball is not an archive. It is a bet that a company you do not control will keep a file at a stable address for a decade, and it is a bet that loses often enough to plan around.
So store the bytes. Per release, in your own storage, immutable:
| What | Why the obvious version fails |
|---|---|
| Source at the exact commit | A tag can be moved or deleted |
| Container image by digest | The tag will be repointed |
| Vendor SDK tarball itself | The download page disappears |
| Every third party source archive | Including transitive ones you did not choose |
| Full build log | Records the versions actually used, not the ones documented |
| The binary, and its hash | Lets you prove the rebuild matches |
| Map file and ELF with symbols | Without these, a field crash address is meaningless |
| Signing certificate chain | Certificates expire; you still need to verify old signatures |
| Test results and HIL logs | Evidence of what was actually verified |
| Architecture decision records as of that release | The reasoning, not just the outcome |
The map file and unstripped ELF deserve emphasis. Article 07 captures a program counter in the crash record. That value is a number, and turning it into a function name requires the exact symbols for the exact build. Discard the ELF and every field crash report from that release becomes permanently unreadable.
Persistent Data Is a File Format
Article 02 gave the config store two copies, a CRC and a sequence number, for power loss atomicity. Lifecycle adds a different requirement, and it runs in both directions.
A structure written to flash is a file format with an indefinite readership. Give it a real header:
typedef struct {
uint32_t magic; /* identifies the format */
uint16_t version; /* schema version, not firmware */
uint16_t length; /* total bytes, including unknown */
uint32_t sequence;
/* payload follows */
uint32_t crc;
} config_header_t;Backward compatibility is the easy direction
Firmware v2 reads a v1 record. Detect the version, run a migration function, write back at v2. Straightforward, and most teams get here.
Forward compatibility is the one that bites
Firmware v1 reads a v2 record.
This is not hypothetical, and it is the coupling most people miss. Article 12’s rollback exists precisely so a bad update can revert to the previous image. When it does, the old firmware boots and finds configuration written by the newer firmware it just replaced. If old firmware rejects unknown versions, your rollback path leads straight to factory defaults, and the device comes back from a failed update having also forgotten its settings. That turns a recoverable update failure into a site visit.
So old firmware must tolerate config it does not fully understand:
if (hdr.version > CONFIG_VERSION_SUPPORTED) {
/* Read the fields we know. Preserve the rest verbatim. */
load_known_fields(&cfg, buf);
preserve_tail(&cfg, buf + known_size, hdr.length - known_size);
/* Do not rewrite unless the user changes something. */
}Two rules make this work. Never reorder or reuse a field; only append. And when rewriting a record you only partly understand, carry the unknown tail through unchanged so a subsequent roll forward does not lose the newer settings.
The same discipline applies to your protocol wire format and your calibration data, both of which outlive individual firmware versions.
The Memory Budget Is a Governed Resource
The usual trajectory:
v1.0 Flash 55% RAM 48%
v2.0 Flash 64% RAM 55%
v3.0 Flash 73% RAM 61%
v4.0 Flash 81% RAM 74%
v5.0 Flash 87% RAM 82%
v5.1 "We can't add that, there's no flash left."Nobody caused that. Each release added a few percent, each one justified, and the constraint arrived as a surprise in a planning meeting four years in.
The number is smaller than you think
For the KIC-400 the ceiling is not 2 MB. Article 02 chose the H743ZI for dual bank A/B updates, which means the application has 1 MB, minus the bootloader, minus the config region, minus the log store. The real application budget is closer to 900 KB, and 87% of 2 MB was never available at all.
This is article 01’s flash occupancy rule arriving years later. If the application cannot fit twice, your update architecture changes, and by year four that is not a decision you can revisit.
Gate it in CI
Budgets that live in a spreadsheet are not budgets. Make the build fail:
FLASH_BUDGET=921600 # 900 KB of the 1 MB bank
RAM_BUDGET=262144 # 256 KB
flash=$(arm-none-eabi-size -A build/firmware.elf \
| awk '/\.text|\.rodata|\.data/ {s+=$2} END {print s}')
if [ "$flash" -gt "$FLASH_BUDGET" ]; then
echo "FLASH BUDGET EXCEEDED: $flash > $FLASH_BUDGET"
exit 1
fiThen plot it per release. The slope tells you when you will run out, and a slope discovered in year two is a planning input rather than an emergency.
Article 05’s stack high water marks and article 09’s soak counters belong on the same dashboard. Flash, RAM, stack and persistent storage are one budget viewed four ways.
Dependency Inventory, Including What You Did Not Choose
Every third party component is a lifecycle liability, because each one can produce a CVE at a time of its choosing.
Track version, source, license, upgrade path and who owns the decision to upgrade. Article 08 noted that the CRA expects a software bill of materials, and this list is most of it.
The part that surprises people: your vendor SDK contains third party code you never selected. An ST SDK ships with a specific lwIP version, a specific mbedTLS, sometimes FatFS and a USB stack. Those versions are in your binary, they are in your SBOM, and when one of them has a CVE your upgrade path runs through the SDK rather than through the upstream project. You may be waiting on a vendor release, or vendoring the fix yourself and maintaining a patched SDK.
Find that out during architecture, not during an incident. Enumerate the transitive components in your SDK, record their versions, and decide in advance whether you would patch locally or wait.
And treat a dependency update as an engineering change, never a version bump. A new TLS release can change stack depth, which changes article 05’s task sizing, which changes RAM. It can change timing, which changes the schedulability analysis. Run the same pipeline you would run for a feature.
MCU Migration: What Abstraction Actually Saves
Year six brings a last time buy notice. Year eight you migrate. The replacement is pin compatible, which tempts everyone into assuming this is a small job.
Here is the honest breakdown, assuming article 04’s layer rules held:
| Layer | Reusable | Why |
|---|---|---|
| Application | ~100% | No vendor headers by construction |
| Services | ~95% | Occasional assumptions about timing or storage geometry |
| Middleware | ~90% | Protocol logic ports; integration glue does not |
| Peripheral drivers | ~0% | Different registers, different DMA |
| BSP | 0% | Board and part specific by definition |
| Startup and linker | 0% | Different memory map |
| RTOS port | 0% | Vendor supplied, but needs configuration and validation |
| Bootloader | ~20% | Logic survives; flash controller and security do not |
Abstraction localises the work. It does not eliminate it, and anyone promising otherwise has not done a migration.
What that table buys you is a migration that is bounded and estimable. Without the layer discipline, the reusable column collapses and the answer to “how long will the port take” becomes “we will find out.”
Two things get forgotten in migration planning, both of which cost more than the driver work.
Performance must be revalidated from scratch. Different cache, different wait states, different DMA arbitration, different interrupt latency. Every worst case execution time in article 05’s task table is now unverified, and the schedulability analysis is a document about a part you no longer use. Re-measure, do not scale.
Security configuration does not port. Option bytes, readout protection levels, OTP layout and secure boot mechanisms differ between parts and sometimes between variants of the same family. Article 08’s decisions need to be made again against the new reference manual.
Toolchain Drift Invalidates Your Timing Analysis
This one is not obvious and it catches good teams.
A compiler upgrade is usually treated as maintenance. New version, rebuild, tests pass, ship it.
But a new compiler makes different inlining decisions, different register allocation, different loop transformations. Your control task that measured 60 microseconds may now measure 52, or 71. The binary is smaller or larger, which shifts the memory map, which changes which code is in which cache line.
Article 05’s entire schedulability analysis rests on measured execution times. A compiler change invalidates those measurements, which means:
- Re measure worst case execution time for every task with a deadline
- Re measure the interrupt handler budget
- Re check flash and RAM against the budget
- Re run the soak with the counters from article 09
Treat a compiler upgrade as a hardware change with a software delivery mechanism. Plan it, schedule the revalidation, and never do it in the same release as a feature, because when the timing moves you want exactly one candidate explanation.
Feature Flags and the Shipped Configuration Matrix
Ten boolean flags create 1024 theoretical builds. You will not test 1024.
You do not need to. You need to test the combinations you actually ship, which means knowing what they are:
| Configuration | Region | Sensors | Cloud | Shipped since |
|---|---|---|---|---|
| STD-EU | EU | 4 | Yes | v1.0 |
| STD-NA | NA | 4 | Yes | v1.0 |
| LITE-NA | NA | 2 | No | v2.1 |
| OEM-X | EU | 4 | No | v3.0 |
Four configurations. Those get the full test matrix. Every other combination is unsupported, and saying so explicitly is the difference between four test runs and a combinatorial problem.
Then give each flag an expiry. A flag introduced to de risk a rollout has a job that ends once the rollout succeeds, and the last step of that job is deleting the flag and the disabled branch. Article 03’s decay comes for feature flags the same as everything else: a temporary flag that survives five years is a permanent second code path that nobody tests.
Branching, Freezing and Backporting
Fielded products cannot all track your newest architecture. That is not a failure, it is the normal state of a fleet.
A workable shape:
main active development, next release
release/5.x current shipping, bug and security fixes
release/4.x security fixes only
release/3.x critical security only, sunset announcedThe discipline that makes this survivable is deciding, per branch, what is allowed in. Security fixes and field critical bugs go to maintenance branches. Features and refactors do not, no matter how tempting, because every merge into an old branch is a merge into a codebase your current test environment may not fully reproduce.
Backporting is where the architecture pays out or does not. If the TLS integration sits behind an interface, a security fix touches the integration layer and the layer above is unaffected. If TLS calls are scattered through service code, the same fix is a different change on every branch, each needing its own testing.
Decide the support window in advance and publish it. “Security fixes for five years from end of sale” is a commitment you can architect around. Silence is a commitment you will be held to anyway, at a time and scope somebody else chooses.
The Turnover Test
Year five, the original firmware lead leaves. The relevant question is not whether the code is good. It is how long a competent new engineer needs before they can make a safe change.
Make it a measurement rather than an opinion. Hand a new senior hire the repository and a real, small task. Time it. That number is your onboarding cost, and it is the most honest maintainability metric you have.
The artifacts that shorten it are ones this series has already produced as side effects of doing other things well:
- The task table from article 05: what runs, when, at what priority, with what deadline
- The ownership matrix from article 02: who owns each resource and each piece of state
- The layer rules and CI check from articles 02 and 04: what may call what, enforced
- The recovery matrix from article 07: every failure, its classification and its escalation
- The architecture decision records from article 01: why, not just what
That last one matters most and is the one skipped most. Five years on, the decision is visible in the code and the reasoning is not. An engineer who cannot find the reasoning has two options, both bad: preserve a constraint they do not understand, or remove it and discover why it existed.
Three sentences per decision. What we chose, what we rejected, what would make us revisit. Keep them in the repository, versioned with the code, archived with each release.
End of Life Is a Deliverable
A product does not become unsupported because the team moved on. It becomes unsupported on a date you announce, with a final state you defined.
At end of support, hand over or archive:
- The final supported firmware version and its complete build archive
- The security posture: known unpatched CVEs, and the honest statement about them
- Signing key custody, including who can still produce a valid image and who should not be able to
- Manufacturing and calibration procedures, if spares are still built
- The recovery procedure, so a field unit can still be revived
- Customer communication: what stops, when, and what the migration path is
The signing key item is the one with teeth. A key that can still sign firmware for a fleet nobody maintains is a liability that outlives the product. Decide at end of life whether that key is retired, escrowed, or revoked, and record the decision.
Next
Everything in this article assumes you find out that something is wrong.
A ten-year fleet tells you almost nothing by default. Units run in plants and basements and substations, and the information reaching you is filtered through a customer who noticed something and an email that says the device is acting strange. Article 07 built a crash record. Article 09 logged soak counters. Neither of those helps if nothing gets the data off the device and in front of an engineer.
Article 11: Field Diagnostics and Firmware Observability covers designing firmware that can explain what went wrong, to someone who was not there, months later.
Series Index
Phase 1, Architecture
01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture
03. Firmware Architecture Anti-Patterns
Phase 2, Implementation
04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture
Phase 3, Failure and Resilience
07. Error Handling & Recovery
08. Firmware Security Architecture
Phase 4, Verification
09. Designing Firmware for Testability
Phase 5, Product Lifecycle
10. Designing Firmware for 10-Year Products (you are here)
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates
