Part 12 of 12 in the Embedded Firmware Architecture series · Phase 5: Operate · Level: Advanced
Every article in this series has been building toward a device you can fix remotely. Article 08 established what makes an image trustworthy. Article 10 established that you must be able to ship security fixes for a decade. Article 11 established how you find out something is wrong.
None of it matters if the mechanism that delivers the fix can destroy the device.
An update that fails on two percent of ten thousand units is two hundred site visits. An update that bricks on two percent is a recall, and the difference between those two outcomes is not the quality of the new firmware. It is the order in which a handful of flags get written to flash.
That is what this article is about. Not what makes an image trustworthy, which article 08 settled, but what happens between a verified image sitting in a download buffer and a device confirmed healthy on the other side.
What Article 08 Settled, and What Remains
Worth being explicit, because these two topics get conflated constantly.
Article 08 owns the trust decision. Is this image authentic, is it for this hardware, is its security version acceptable, what is the root of trust, where do the keys live, how do they rotate. Those questions are answered before a single byte is committed, and this article assumes the answers.
This article owns the mechanics. Where the image is staged, in what order the commit happens, what boots after an interruption at each possible moment, what counts as confirmation, and what happens to your data when you roll back.
One place they interlock, stated in article 08 and worth repeating because it is the sharpest coupling in the series: burn the new security version only after the image is confirmed. Burn it at install and a failed update leaves the working image below the stored minimum, unable to boot. That is a self inflicted brick produced by two correct mechanisms in the wrong order.
The Flash Layout, and Why the H743 Choice Pays Off Here
Article 02 selected the H743ZI for dual bank flash, two banks of 1 MB. Article 10 budgets the application at 768 KB as a result. Here is the return on that decision.
Bank 1 (sectors 0 to 7) Bank 2 (sectors 0 to 7)
┌────────────────────────┐ ┌────────────────────────┐
│ Bootloader 128 KB │ │ Bootloader copy 128 KB │
├────────────────────────┤ ├────────────────────────┤
│ Application 768 KB │ │ Application 768 KB │
│ (image header first) │ │ (image header first) │
├────────────────────────┤ ├────────────────────────┤
│ Fault history 128 KB │ │ Fault history 128 KB │
└────────────────────────┘ └────────────────────────┘
Outside the swap: config and update state (external EEPROM), backup SRAMOne constraint sets every number in that diagram: H743 flash erases in 128 KB sectors, eight per bank, so no region can be smaller than a sector. The bootloader takes a full sector, the image header lives at the start of the application slot rather than in a separate metadata region, and the small, frequently rewritten update state sits in external EEPROM beside the configuration. That leaves 768 KB for the application, which is the figure article 10’s flash gate enforces.
The fault-history sector needs one more rule. It exists in both banks, and the bank swap described next moves it along with everything else, so the logger must address it by physical bank: read the current swap state from the flash option register and translate the address. Skip that and the fault history appears to jump between two different sectors every time an update swaps the banks.
On the H7, an option bit swaps which physical bank is mapped at the boot address. Both images are therefore linked to the same address, and that removes an entire category of difficulty that most A/B discussions gloss over.
On a single bank part, your two slots live at different addresses, which means either position independent code, with the performance and toolchain cost that carries, or two separate builds of the same source with different link scripts, which means the image you tested in slot A is not bit identical to the one that runs in slot B. Neither option is pleasant, and the second one is genuinely dangerous because your verification covered a different binary.
Bank swap removes the choice. One build, one address, two locations.
Two honest caveats. The bootloader has to exist in both banks, because the swap moves everything, so you carry a duplicate copy and updating it means updating both. And the option byte write that performs the swap triggers a reset of its own, which has to be accounted for in the boot flow rather than discovered during integration.
Confirm the exact option bit behaviour against the reference manual for your part before building on this, because the details differ across the H7 family.
The Update State Machine, Persisted
Article 06 argued for making behaviour explicit. The update path is the strongest case for it, because unlike most state machines this one has to survive a power cycle in every state.
IDLE
→ CHECKING manifest fetched, evaluated
→ DOWNLOADING writing to the inactive bank
→ VERIFYING hash and signature over the staged image
→ STAGED pending flag set, awaiting reboot
→ TRIAL booted, not yet confirmed
→ CONFIRMED healthy, committedWith failure transitions from each:
DOWNLOADING → DOWNLOAD_FAILED network, storage, or space
VERIFYING → REJECTED hash, signature, hardware, version
TRIAL → ROLLED_BACK failed to confirmThe distinguishing property is that the current state lives in non-volatile memory, not RAM, because the transition from STAGED to TRIAL passes through a reboot and the transition from TRIAL to CONFIRMED might pass through several.
On the KIC-400 that is the external EEPROM, next to the configuration and under article 02’s two-copy CRC scheme. Rewriting a few bytes of state there is one page write; in internal flash it would cost a 128 KB sector erase.
So the state machine’s persisted portion is small and deliberate:
typedef struct {
uint32_t magic;
uint8_t pending_slot; /* 0xFF = none */
uint8_t confirmed_slot;
uint8_t boot_attempts; /* of the pending slot */
uint8_t reserved;
uint32_t bytes_received; /* for resume */
uint32_t crc;
} update_state_t;Four meaningful bytes plus a resume counter. Everything else about the update is derivable from the slot metadata and the image headers.
Commit Ordering Is the Whole Problem
Here is the part that separates an OTA implementation that works from one that works when interrupted.
Every persistent write during an update is a commit point. Power can fail immediately before or immediately after any of them. The design question is not whether each write is correct. It is whether every possible stopping point leaves a state the bootloader can resolve.
The ordering that works
1. Erase the inactive bank
2. Write image data, sector by sector
3. Read back and verify each sector against what was written
4. Hash the complete staged image, verify the signature
5. Write the image metadata block into the inactive bank
6. Set pending_slot, boot_attempts = 0 ← the commit point
7. Reboot
8. Bootloader: attempts++, map the pending bank (swap only if needed), boot it
9. Application: initialise, self test, reach operational health
10. Application: confirm ← confirmed_slot = pending_slot
11. Record the new security version in the monotonic store
12. Clear pending_slotEvery step before 6 is reversible by doing nothing. The inactive bank is scratch space, and an interruption anywhere in steps 1 to 5 leaves the running image untouched and the update simply restartable.
Step 6 is the only commit point on the download side, and it is a single small write. That is deliberate. Reduce the number of moments where an interruption changes the outcome, and make each one as small as possible.
Three orderings that brick devices
Setting the pending flag before verification. Now an interrupted or corrupt download becomes a bootable candidate. The device reboots into garbage, and your only protection is whatever the bootloader independently re checks. Set the flag after verification, always.
Burning the security version at install. Article 08’s warning, in mechanical form. Step 11 sits after confirmation for exactly this reason. Move it to step 6 and a candidate that fails to boot cannot roll back, because the previous image is now below the stored minimum. Two correct mechanisms, wrong order, unrecoverable device.
Writing the resume counter before the flash write. A subtle one. If bytes_received is updated before the sector is programmed, an interruption between the two leaves the counter claiming data that is not there, and the resumed download writes the next sector over a gap. The image then fails verification, which is the good outcome, but you have wasted a full download cycle and on a metered connection that is real money. Update the counter after the write and the verify, never before.
The Boot Decision Table
The bootloader’s entire logic, as a table. This is the artifact worth having in your design document, because it is reviewable in a way that nested conditionals are not.
| Active slot | Pending slot | Attempts | Decision | Records |
|---|---|---|---|---|
| Valid | none | n/a | Boot active | normal boot |
| Valid | valid | 0 | Swap, boot pending, attempts = 1 | trial start |
| Valid | valid | 1 to 2 | Keep pending mapped, boot it, attempts++ | trial retry |
| Valid | valid | ≥ 3 | Clear pending, boot active | rollback |
| Valid | invalid | any | Clear pending, boot active | rejected |
| Invalid | valid | 0 to 2 | Boot pending, attempts++ | trial, no fallback |
| Invalid | valid | ≥ 3 | Recovery mode | both exhausted |
| Invalid | none | n/a | Recovery mode | no bootable image |
| Invalid | invalid | any | Recovery mode | no bootable image |
With bank swap, “boot active” and “boot pending” both mean setting the swap bit so the right bank sits at the boot address. The bootloader compares the current mapping with the persisted state and changes it only when the two differ. It never toggles blindly, because a retry after a watchdog reset would otherwise swap straight back to the old image.
Three things this table makes visible that prose hides.
Three attempts, not one. A single failed boot can be a transient, a brownout during initialisation, a one off peripheral hiccup. Three consecutive failures is a pattern. The number is a product decision, but it should be greater than one and small enough that a genuinely broken image does not spend minutes cycling.
Rollback is a recorded event, not a silent fallback. Article 11’s fleet telemetry needs OTA_ROLLBACK with the version that failed, because two hundred devices rolling back is the signal that pauses a campaign. A rollback nobody hears about is a campaign that keeps rolling.
The rows with an invalid active slot are the interesting ones. They should be unreachable, since a confirmed image was verified before it was confirmed. They happen anyway: flash wear, a bad sector, cosmic rays over ten years. The bootloader that handles them is the one that does not need a truck.
Every Interruption Point
Article 09 built a test that cuts power at every byte of a config write. The same discipline, applied to an update, produces a table you should be able to point at for any moment in the sequence.
| Interruption at | Inactive bank | Persisted state | On next boot |
|---|---|---|---|
| During download | Partial | pending = none | Boot active, restart or resume |
| After download, before verify | Complete, unverified | pending = none | Boot active, re verify or restart |
| After verify, before metadata | Complete | pending = none | Boot active, restart |
| After metadata, before pending flag | Complete, valid | pending = none | Boot active, restart |
| After pending flag | Complete, valid | pending = B, attempts 0 | Boot B as trial |
| During bank swap reset | Complete, valid | pending = B | Boot B, or A if swap did not take |
| During trial boot init | Complete, valid | pending = B, attempts 1 | Retry B, then roll back |
| After boot, before confirm | Complete, valid | pending = B, attempts n | Retry, then roll back |
| After confirm, before security version burn | n/a | confirmed = B | Boot B. Security version lags by one. |
| After security version burn | n/a | confirmed = B | Normal |
Two rows deserve attention.
The bank swap reset row is the one people miss. The option byte write itself triggers a reset, so there is a window where the swap may or may not have taken effect. The bootloader must therefore decide which image to run based on the persisted state rather than assuming the swap succeeded, and must be able to re apply the swap if the state says pending but the mapping says otherwise.
The confirm before security version burn row is a deliberate, acceptable inconsistency. The device is running the new image and the stored security version still reflects the old one. The consequence is that for a few milliseconds a downgrade to the previous version would be permitted. That is a far better failure than the alternative ordering, and the window closes on the next boot regardless: the bootloader sees a confirmed image whose security version is above the stored minimum and records it before jumping.
Confirmation Means Healthy, Not Started
The most common OTA defect that is not a bricking defect: confirming too early.
int main(void) {
ota_confirm(); /* wrong */
system_init();
}The image is now permanently committed, and it has demonstrated nothing except that the reset vector was correct. If initialisation fails three functions later, the device is stuck with an image that cannot run and no rollback path.
Confirmation belongs after the device has proven it can do its job. Article 11 already defined what that means, and this is where those definitions pay off. The functional health indicators are the confirmation criteria:
static bool ready_to_confirm(uint32_t uptime_ms)
{
return uptime_ms > CONFIRM_MIN_UPTIME_MS /* 60 s */
&& all_tasks_checked_in() /* art 07 */
&& control_deadlines_met() /* art 05 */
&& config_loaded_and_valid()
&& modbus_rtu_responded_at_least_once()
&& storage_write_succeeded_once();
}Sixty seconds, every task alive, control loop meeting deadlines, configuration valid, and at least one successful transaction on the fieldbus and in storage. That is a device doing its job, not a device that reached main().
Make the fieldbus condition match the installation. A unit commissioned without a Modbus master never sees a request, and an unconditional check would roll back every update it receives. Take the requirement from configuration, or require it only when the unit saw fieldbus traffic before the update.
Note the overlap with article 07‘s boot counter clear condition. They are the same criteria, and using one definition for both is the right call.
The timeout
The bootloader needs an upper bound on how long a trial can take, or a firmware that hangs during initialisation stays pending forever.
Secure boot verification 15 to 85 ms (art 08, measure yours)
Peripheral and RTOS init ~200 ms
Network link and first poll up to 30 s
Confirmation criteria window 60 s
Margin 60 s
----------
Independent watchdog timeout well under the above
Trial supervision ~3 minutes totalThe independent watchdog from article 07 handles the hang case without needing a separate mechanism. A firmware that locks up during init never feeds it, the watchdog resets, the attempt counter increments, and three of those produce a rollback. The bootloader does not need its own timer; it needs the attempt count and a watchdog that is genuinely independent of the application.
The watchdog covers the image that hangs. It does not cover the image that runs perfectly well and never meets the confirmation criteria, for example because no Modbus master is polling it. That case needs a trial deadline in the application: if the criteria are not met within the supervision window, record OTA_TRIAL_TIMEOUT and reset, so the attempt counter advances and three such trials end in a rollback.
Rollback Has a Data Problem
The mechanism above returns you to the previous image. It does not return you to the previous data, and that asymmetry is where rollback designs fail.
Article 10 covered the rule: old firmware must tolerate configuration written by newer firmware, preserving fields it does not understand rather than rejecting the record. Here is the mechanical consequence for the update sequence.
Migrate late, and journal it. Configuration migration should not run during the update. It should run on first boot of the new image, after the point where a rollback is still clean:
Trial boot
→ read config at schema v8
→ new firmware wants v9
→ write journal entry: MIGRATION_PENDING v8 → v9
→ write migrated copy to the inactive config slot
→ journal: MIGRATION_COMPLETE
→ switch active config slotArticle 02’s two copy configuration store is what makes this safe. The v8 copy remains intact in the other slot throughout, so a rollback during the trial period finds configuration it can read, and an interruption mid migration resolves from the journal on the next boot.
Decide the rollback policy for data explicitly. Three defensible answers, and the wrong move is not choosing:
Keep both schema versions alive during the trial, and discard the old one at confirmation. Costs a config slot, and it is the cleanest.
Make migrations additive only, so v9 is readable by v8 as a v8 record with extra fields. Costs schema discipline forever, and it is free at runtime.
Accept that rollback reverts configuration to the pre update state, and say so in the release notes. Sometimes correct, never acceptable as a surprise.
Updating the Bootloader
The genuinely dangerous case, because a failed application update rolls back and a failed bootloader update leaves nothing to roll back with.
Three approaches, in order of how much I would trust them.
Do not update it. Keep the bootloader small enough and stable enough that it never needs to change. Verification, slot selection, rollback, recovery. No protocol stacks, no drivers beyond flash and the recovery transport. Article 08’s argument for a small bootloader is also the argument for a bootloader you never have to touch, and over a ten year product this is frequently achievable if you plan for it.
Two stage. An immutable first stage whose only job is to verify and select between two copies of a second stage bootloader. The second stage does the real work and is updatable with the same A/B discipline as the application. You have moved the unupdatable part to something small enough to get right the first time, at the cost of an extra verification hop on every boot.
Redundant copies with a write ordering discipline. Bootloader in both banks, updated one at a time, with the invariant that at least one valid copy exists at every instant. Works, and it depends entirely on a flash controller that behaves exactly as documented during an interrupted erase of a boot critical sector. That is more trust than I like to place in an interrupted erase.
Whichever you choose, the bootloader update path needs its own power interruption campaign, run more thoroughly than the application’s, because there is no recovery path to catch a mistake.
Delta Updates: The Arithmetic and the Trap
A full KIC-400 image can be up to 768 KB, the size of its slot. A typical delta between adjacent releases might be 60 KB.
On Ethernet, that difference is not interesting. On the cellular link from article 11, with a usable payload budget of around 60 to 80 KB per day, it is decisive:
Full image 768 KB ≈ 11 days of the entire data budget
Delta 60 KB ≈ 1 dayThat is the calculation. If your devices are on Ethernet, delta updates are complexity with no return. If they are on metered cellular, they may be the difference between an update campaign that is feasible and one that is not.
What they cost, beyond the obvious patch generation step in your build pipeline:
A patch per source version. A delta from 5.3.1 to 5.4.0 is useless to a device on 5.2.0. With a fleet spread across six deployed versions, you generate and test six patches per release, or you maintain a chain and apply several in sequence, which multiplies the interruption points.
The wrong base image trap, which is the one that bites. A patch applied to the wrong source produces a plausible looking image of the correct size. It will fail signature verification, which is your safety net, but only if you verify the reconstructed image rather than the patch. Hash the source image before applying the patch, and hash the result after. Two checks, both mandatory, and the first one is the one people omit.
More RAM or a scratch area, depending on the patch algorithm, at exactly the moment the device is also running normally.
My default is full images unless the bandwidth arithmetic says otherwise. The complexity is real and it lands in the most safety critical path in the product.
Staged Rollout, and How Many Canaries You Actually Need
Never ship to the whole fleet at once. That much is standard advice. The part that is usually missing is how large the first stage has to be for its result to mean anything.
Suppose your new firmware fails on two percent of units, which would be a serious problem across ten thousand devices.
Canary group of 20 expected failures: 0.4 indistinguishable from zero
Canary group of 100 expected failures: 2 a weak signal
Canary group of 500 expected failures: 10 clearA twenty device canary cannot detect a two percent failure rate. It will pass, you will proceed, and the problem surfaces at scale. Size the first stage against the failure rate you need to catch, not against a comfortable looking percentage.
A workable ladder for ten thousand units:
Stage 1 500 units hold 48 hours
Stage 2 2,000 units hold 48 hours
Stage 3 whole fleetAnd the gate needs a metric, not a vibe. Article 11’s telemetry provides it:
Pause the campaign if, in the current stage:
rollback rate > 1%
watchdog resets/device > baseline x 2
confirmation failures > 0.5%
any device in recovery modeThose thresholds come from your own baseline, which is the argument for having article 11’s telemetry running well before your first update campaign. A canary programme with no baseline is measuring against an assumption.
Hold time matters as much as group size. A failure that appears after twelve hours of runtime is invisible in a two hour hold. Forty eight hours is a reasonable default, and longer for a release touching the control path.
Testing the Campaign
Article 09 built the technique. Here is the OTA specific application, and it is the highest value HIL work you will do.
The interruption campaign. Automate: start an update, cut power at a chosen point, restore, and assert the device is running a valid image with valid configuration. Repeat across the whole sequence.
Cut power at: 10%, 50%, 90% of download
during sector erase
during sector write
between verify and metadata write
between metadata write and pending flag
during the bank swap reset
at 1 s, 10 s, 59 s of the trial boot
between confirm and security version burnEvery one of those maps to a row in the interruption table above, and the assertion is the same each time: the device boots, it runs a valid image, and its configuration is readable. Run it overnight, hundreds of cycles.
The rollback campaign. Ship a deliberately broken candidate, one that hangs during init, one that crashes after thirty seconds, one that never meets the confirmation criteria. Assert that each produces a rollback within three attempts and emits OTA_ROLLBACK with the failed version.
The negative campaign. Article 08’s rejection cases, delivered through the real update path rather than tested in isolation: flipped signature bit, wrong hardware revision, lower security version, truncated image, correct image with a corrupted metadata block.
If you automate one thing from this entire series, automate the interruption campaign. It is the test that turns “we believe the update is safe” into evidence, and it is cheap to run once the rig from article 09 exists.
The Series, and What It Was For
Twelve articles, one product, one architecture carried through all of them.
The KIC-400 started in article 02 as ten requirements, six of them about timing and failure. It picked up a task table with derived priorities, a memory map, an ownership matrix, state machines with timeouts on every waiting state, an escalation ladder, a chain of trust, a dual target build, a release archive, a wear budget, and finally an update path with a commit ordering you can defend in a review.
None of those is a feature. Every one of them is a decision that was cheap in year zero and impossible in year four.
That is the argument the series has been making throughout. Firmware architecture is not a diagram of layers. It is the set of decisions that determine whether the next requirement, the next hardware revision, the next security fix and the next ten years are affordable, and the measure of an architecture is not how it looks but what it costs you later.
References
- STM32H743ZI documentation (RM0433 reference manual and datasheet)
- MCUboot documentation
- Arm Cortex-M7 Technical Reference Manual
- Regulation (EU) 2024/2847, the Cyber Resilience Act
Series Index
Overview, reading paths and related articles: Embedded Firmware Architecture series guide.
Phase 1: Decide
01. Embedded Firmware Architecture Fundamentals
02. Production-Grade Firmware Architecture
03. Firmware Architecture Anti-Patterns
Phase 2: Build
04. HAL vs BSP vs Drivers vs Middleware
05. RTOS Architecture
06. State Machine Architecture
Phase 3: Fail safely
07. Error Handling & Recovery
08. Firmware Security Architecture
Phase 4: Verify
09. Designing Firmware for Testability
Phase 5: Operate
10. Designing Firmware for 10-Year Products
11. Field Diagnostics & Observability
12. OTA & Safe Firmware Updates (you are here)
