Signed-firmware OTA pipeline · the 2026 default we ship

Push a new firmware binary to an S3 bucket, let devices poll · that works for a hackathon. For a 10k+ device fleet in production, OTA is a 4-stage pipeline. Skip any stage, you get bricked devices eventually.

Stage 1 · signing

Every firmware binary is signed with an offline-kept key. Devices refuse to boot unsigned or wrongly-signed images. We use cosign with a hardware-backed signing key (YubiKey) for the private side, rotating quarterly.

Stage 2 · distribution

Signed binary lands in a versioned CDN path. Devices fetch via HTTPS with certificate pinning. The metadata file (version, hash, release notes, min-firmware-required) is fetched first, used to decide whether to upgrade.

Stage 3 · staged rollout

Canary (0.5%) -> 5% -> 50% -> 100%, with a hold between each stage where we watch telemetry. A regression in device-reported metrics triggers automatic rollback.

Stage 4 · monitoring + rollback

• Boot-success rate per firmware version.
• Crash-free session rate.
• Network-connect success rate.
• User-reported error rate (if app-backed).

If any metric regresses more than 5% on the canary, the pipeline auto-rolls back to the previous version. We have never had to intervene manually on a canary regression · the pipeline catches it within 30 minutes.