Decisions

1. It can't run there. Talos has no shell and no package manager, so the Keylime agent can't be installed on the host.
2. What Keylime checks is fixed. IMA file-integrity drift detection is pointless on a read-only SquashFS image with no SSH.
3. The boot chain is enforced, not just measured. Talos Trusted Boot (a Secure Boot-signed UKI) refuses to boot a tampered image, and the measurements go into the TPM.
4. Keys are tied to a healthy state. The disk key is sealed in the TPM and released by the Sidero Labs KMS server, so a tampered or revoked node can't unlock its disk or join.
5. Identity is short-lived. SPIRE TPM attestation issues SVIDs that last 5–15 minutes.

Remaining risk: runtime and in-memory kernel attacks, covered by Tetragon. There's no verifier checking continuously while the node runs; we accept that, and revocation happens through SVID expiry and the KMS at next boot. Physical attacks are covered by the tamper switch and key wipe.

Chicken-and-egg problems found:

1. KMS versus Talos. The KMS can't run on the single Talos node it unlocks. Fix: run the Sidero Labs KMS server on the switch host. The switch host unlocks its own disk from its TPM alone, with no network.

2. The 802.1X gate versus the Talos node's network. Talos needs the network to reach the KMS, but the gate and the SPIRE server aren't up yet. Fix: a static infrastructure port, pinned to the Talos node's MAC address and TPM certificate fingerprint and set at FDO onboarding, giving a reachable infrastructure VLAN that reaches only the KMS.

3. Keylime on the switch versus the registrar and verifier on Talos. The switch must be running before Talos can boot, so Talos can't be the first to attest it. Fix: the switch boots with measured boot and TPM-sealed keys, which is its gate to start. Its SPIRE agent gets no SVID until the Keylime verifier on Talos passes it. Failure means the SVID is denied, the KMS releases no further keys, and a key wipe is optional. Accepted risk: a loop where Talos attests the switch that unlocks Talos. It's bounded because each side's TPM measurement is checked independently.

   Key release condition (amended by ADR-021). The Sidero Labs KMS server releases a Talos disk key only when all four of these hold:

   • (a) a fresh TPM quote of the switch host's own PCRs matches policy, so a tampered switch stops releasing keys
   • (b) the Talos node proves its UKI boot state with PCR 11 against a signed PCR policy
   • (c) the request is bound to the Talos node's TPM endorsement key (EK), recorded at FDO onboarding
   • (d) every release is written to an append-only, hash-chained log (ADR-025)

   Even if the switch is compromised and unlocks Talos, workload secrets are still gated by SPIRE.

4. Argo CD deploying itself. Fix: Talos inlineManifests install Argo CD and the local registry first. The images are seeded from the install or update volume.

5. NATS needing an identity before SPIRE exists. Fix: the first NATS leaf credentials (NKey and JWT) are delivered through FDO onboarding, then rotated to SPIRE-issued credentials.

Order: switch host (TPM-sealed unlock, SONiC, KMS, NATS leaf with FDO credentials), then Talos (unlocked through the KMS, Cilium and SPIRE server up, Argo CD and registry from inline manifests), then Keylime attests the switch and its SVID is issued, then the 802.1X gate opens for assets.

• Problem: there was no CA or key design for SPIRE, EAP-TLS, the FDO owner key, update signing or Core's import key. If one key leaks, every Edge is exposed.
• Decision:
  • An offline root CA signs one intermediate CA per trust domain (core, cust-a, cust-b). SPIRE uses it through UpstreamAuthority.
  • The signing keys and Core's import key live in an HSM.
  • Rotation follows a written key ceremony.
• POC: SoftHSM, with the root CA on an offline laptop. Prod: YubiHSM 2 or Nitrokey HSM, and a two-person ceremony.

• Problem: a valid signature alone accepts an old, vulnerable bundle (rollback) and doesn't notice that updates have stopped (freeze).
• Decision:
  • Images and bundles are signed with Sigstore cosign, with in-toto/SLSA provenance and an SBOM.
  • The bundle carries TUF metadata: version numbers, expiry, threshold roles, and the root held offline.
  • The Edge stores the highest installed version in a TPM NV monotonic counter.
  • Kyverno (or the Sigstore policy-controller) enforces signatures at deploy time.
• POC and Prod: build it.

• Problem: one switch host runs the border, the 802.1X gate, the Sidero Labs KMS server, the NATS leaf and the update port. If it fails or is compromised, the whole site is lost.
• Decision: this is an accepted risk in the POC. Production gets:
  • two switch hosts (MC-LAG)
  • the Sidero Labs KMS server on a separate small TPM appliance, or the key split across both switch hosts (Shamir)
  • an offline break-glass recovery key
• POC: documented. Prod: build.

• Problem: Talos attests the switch, and the switch unlocks Talos. A compromised switch could release keys before Keylime has checked it.
• Decision: release a key only with a fresh switch TPM quote, Talos PCR 11 checked against a signed PCR policy, and binding to the Talos EK recorded at FDO onboarding. Log every release to an append-only log. Details are in ADR-016.
• POC and Prod: build it.

• Problem: with no NTP, clock drift or a deliberately set clock breaks certificate validity, SVIDs, TUF expiry and event ordering.
• Decision:
  • chrony on the switch host uses a GNSS/PTP source, or a battery-backed RTC.
  • A time floor means the clock never goes earlier than the newest update volume's timestamp.
  • A drift alarm goes to the audit log.
• POC: RTC plus the time floor and alarm. Prod: add a GNSS/PTP grandmaster.

• Problem: 5–15 minute SVIDs are fine locally, but the intermediate CA and federation bundles can expire during a long air gap.
• Decision:
  • The local SPIRE server issues SVIDs on its own.
  • The Edge intermediate CA lasts 90 days, and that's the declared maximum air-gap period.
  • Renewals and federation bundles arrive on the update volume.
• POC and Prod: build it.

• Problem: "one account per customer" wasn't backed by an operator, JWT or subject design.
• Decision:
  • NATS operator mode, with the operator and account signing keys in the HSM (ADR-018).
  • Subject allow-lists per account and user, for example publish-only to cust-a.edge.*.telemetry.>.
  • Leaf credentials scoped per leaf and revocable.
  • JetStream limits per account.
  • Later: auth callout using SPIRE JWT-SVIDs.
• POC and Prod: build it.

• Problem: without a tamper-evident audit trail, there is no trustworthy NIS2 / IEC 62443 evidence.
• Decision:
  • Gate decisions, key releases, Keylime results, Argo CD syncs, Tetragon alerts and the Kubernetes audit log become signed, hash-chained events in JetStream.
  • They're shipped over NATS or on the export volume.
  • Core stores them write-once (GCS bucket lock).
• POC and Prod: build it.

• Problem: single-node etcd and Longhorn had no backup. A dead disk or a key wipe means data loss.
• Decision:
  • Scheduled Talos etcd snapshots and Longhorn backups, encrypted.
  • Sent to Core over the NATS object store, or on the export volume when air-gapped.
  • A restore runbook, plus a bare-metal rebuild test.
• POC and Prod: build it.

• Problem: no written vulnerability policy; some customers need one-way export; the kill switch wasn't defined.
• Decision:
  • The quarantine VM runs Syft (SBOM) and Grype/Trivy, and blocks any image with a CISA KEV entry or a critical CVE that has no recorded exception.
  • An optional hardware data diode provides one-way export.
  • Kill switch: revoke federation, disable the customer's NATS account, and stop Sidero Labs KMS server key releases.
• POC: the scan policy is built; the diode and kill switch are on paper. Prod: build the diode and kill switch.

• Problem: the stock Talos image and Sidero Labs' Secure Boot keys mean we trust whatever is published upstream, and system extensions could drift.
• Decision:
  • Build the Talos UKI with Sidero Labs Image Factory (a self-hosted copy in production), using a schematic with pinned system extensions and versions.
  • Sign it with our own Secure Boot keys, enrolled in each node's firmware.
  • Keep the schematic ID in Git.
• POC: Image Factory schematic plus our own keys enrolled in OVMF. Prod: a self-hosted Image Factory, with the keys in the HSM (ADR-018).

• Problem: booting from USB or PXE, or changing firmware settings, can bypass Trusted Boot.
• Decision: set a BIOS password, disable external and network boot, and include firmware versions (PCR 0–2) in the TPM measured-boot policy, so the Sidero Labs KMS server refuses to release keys after unexpected firmware.
• POC: OVMF settings plus the PCR policy. Prod: apply to real hardware, with a firmware update process.

• Problem: control traffic (Argo CD, Sidero Labs KMS server, SPIRE, the NATS system account) shared VLANs with customer data and assets.
• Decision:
  • A dedicated management/control VLAN (VLAN 90) on each SONiC switch host.
  • SONiC ACLs allow only the listed control ports between VLAN 90 and the others.
  • The OT asset VLANs can't reach control services.
• POC and Prod: build it.

• Problem: without default-deny, any pod can talk to any pod.
• Decision: every namespace gets a Cilium default-deny policy. Allowed flows are written per workload, and mutual authentication between workloads uses SPIFFE identities from SPIRE.
• POC and Prod: build it.

• Problem: revoking an Edge touches three systems (SPIRE federation, NATS, the Sidero Labs KMS server), and doing them by hand is slow and error-prone.
• Decision:
  • One scripted, audited action does all three: removes the federation bundle, revokes the NATS account and leaf credentials, and stops Sidero Labs KMS server key release.
  • The action writes to the audit log (ADR-025).
• POC and Prod: build it. This is also the building block for the end-state kill switch.

• Problem: applications need secrets (database passwords, API keys) at air-gapped Edges, without plain-text secrets in Git.
• Decision:
  • POC: SOPS-encrypted secrets in Git. Argo CD decrypts them at the Edge with an age key that is sealed by the node's TPM.
  • Production: OpenBao, where workloads authenticate with their SPIRE identity (JWT-SVID).
• POC: SOPS + age/TPM. Prod: OpenBao.

• Problem: the system must fail safe (closed, recoverable), not open, when things go wrong.

• Decision: scripted failure tests:

  • power loss in the middle of an update, which must roll back to the last good state
  • a full disk
  • a corrupt or truncated update volume
  • loss of the switch host
  • expired certificates

  Each test has an expected safe outcome.

• POC and Prod: build it.

• Problem: there was no structured threat model to show IEC 62443 and NIS2 assessors.
• Decision: a STRIDE threat model per component, mapped to IEC 62443 security levels (target SL 2, SL 3 for the root of trust) and to the ADRs that mitigate each threat. It's kept in docs/THREAT-MODEL.md and updated alongside the ADRs.
• POC and Prod: keep it maintained.

• Problem: export volumes and the root CA must stay safe for years. Data captured today could be decrypted later by a quantum computer ("harvest now, decrypt later").
• Decision:
  • Keep a crypto inventory: algorithm, key size, lifetime and owner for each use (TLS, SVIDs, EAP-TLS, cosign, TUF, the Sidero Labs KMS server, SOPS/age, NATS, export volumes).
  • Plan hybrid ML-KEM (encryption) and ML-DSA (signatures), long-lived data and the root CA first, as tool support arrives.
• POC: the inventory and the plan, documented. Prod: build the hybrid crypto.

• Problem: one person could sign an update, issue a CA, or revoke a customer on their own.
• Decision:
  • The root CA, update signing (TUF root and targets) and Edge revocation need an M-of-N quorum on the HSM, for example 2 of 3.
  • Every Git change to policy, manifests or ADRs needs a review from someone other than the author (branch protection).
• POC: Git reviews enforced, the quorum documented (SoftHSM). Prod: HSM quorum.

• Problem: if a control fires and nobody notices, it isn't much use. Operators need to see that the controls are working.
• Decision:
  • Grafana dashboards and alerts, fed from the audit stream (ADR-025).
  • They show port-gate denials, Keylime and SPIRE attestation failures, Sidero Labs KMS server key releases and refusals, revocations, signature and scan rejections, and per Edge the days left before the 90-day air-gap limit.
• POC and Prod: build it.

• Problem: Talos, Kubernetes, SONiC, SPIRE and NATS move independently. An untested combination, or a big-bang rollout, can take Edges down.
• Decision:
  • A release cadence (for example quarterly, plus security fixes on demand).
  • A compatibility matrix in Git, listing the tested version sets.
  • Staged rollout: lab, then a canary Edge, then the rest.
  • Argo CD health checks with automatic rollback to the last good version.
• POC and Prod: build it.

• Problem: not all data is equally sensitive, but export and encryption rules were the same for everything.
• Decision:
  • Each NATS subject has a class: public, internal, confidential or restricted.
  • The class decides whether data may leave the Edge, how it's encrypted, and whether it needs the one-way diode.
  • The classification lives in Git, and both the NATS permissions and the export job enforce it.
• POC and Prod: build it.

• Problem: retired hardware can leak keys or data, or be re-admitted.
• Decision: a scripted decommission does four things and writes them to the audit log:
  • crypto-wipe (destroy the disk keys)
  • clear the TPM
  • revoke the FDO voucher
  • remove the SPIRE entries (plus revoke NATS credentials and the EAP-TLS certificate)
• POC and Prod: build it.

• Problem: 50+ tests are too many for a customer demo.
• Decision: a demo set of 10 tests (T1, T4, T6, T10, T11, T12, T14, T15, T21, T40) that together tell the story: zero-trust admission, customer isolation, tamper-proof boot, minimal egress, autonomy, secure air-gapped updates, and one-action revocation. All other tests are "designed, not demoed": they're built and run as evidence, but aren't part of the live demo. The list is in TESTS-AND-DEMOS.md.
• POC: applies to the Terma demo.

• Problem: upstream SONiC has no configurable 802.1X, and it was unclear whether hostapd could see EAPOL frames on a SONiC switch.
• Decision and finding:
  • hostapd (driver=wired, eap_server=1) runs inside the SONiC switch host, one instance per access port, on the EthernetN netdevs.
  • A hostapd_cli action handler is the gate controller. On AP-STA-CONNECTED it moves the port from VLAN 20 to VLAN 70 by writing VLAN_MEMBER in CONFIG_DB. On AP-STA-DISCONNECTED it moves the port back.
  • Tested on docker-sonic-vs: an Edge A certificate is admitted (T1), and a customer B certificate (T4), a device with no certificate (T2) and a logged-off device (T53) stay in quarantine.
• POC: the phase-1 shell prototype was replaced in phase 2 by the Go controller lab/gatectl/ (ADR-048). Device keys move to the TPM in phase 3. Prod: the same pattern on hardware SONiC; verify that EAPOL is trapped to the CPU on the target ASIC.

• Problem: we need a lab that can be rebuilt with one command, inside the single cost-capped GCP VM.
• Decision:
  • talosctl cluster create qemu --presets iso-secureboot: Image Factory Secure Boot ISO, UKI, swtpm and OVMF, with TPM-sealed LUKS2 STATE and EPHEMERAL. One cluster per site.
  • containerlab runs the SONiC-VS switch hosts and Alpine test devices. The Talos bridges are attached to the switch ports.
  • Scripts live in lab/ (up.sh, down.sh, run-tests.sh, gcp.sh).
  • On Ubuntu, AppArmor needs a local usr.bin.swtpm override (T51).
• POC: this is the build. Prod: not applicable; production runs on real hardware.

• Problem: all trust domains chain to one offline root (ADR-018). If a relying party trusted the root, an Edge B intermediate could issue spiffe://edge-customer-a.bastion/... identities.
• Decision and finding:
  • Every intermediate carries a critical X.509 nameConstraints extension: permitted;URI:<its trust domain>. A forged cross-domain SVID fails with permitted subtree violation (T54).
  • With SPIRE UpstreamAuthority disk, the trust bundle SPIRE publishes is the site's intermediate, not the root. Relying parties therefore trust one trust domain per bundle, and verification uses partial-chain semantics (normal for SPIFFE).
  • Rotation: pki-root.sh rotate <td> issues a new intermediate from the HSM-held root; SPIRE picks it up on restart (T24 partial).
• POC: SoftHSM, lab PIN. Prod: YubiHSM 2 / Nitrokey, a ceremony, and overlapping intermediates so rotation needs no restart.

• Problem: the target is SPIRE TPM node attestation (ADR-004, ADR-007). Talos has no shell to install a TPM DevID, and the community TPM plugin needs files on the host.
• Decision: phase 2 uses the chart default k8s_psat (Kubernetes projected service-account tokens). TPM-based node attestation (DevID provisioned during FDO, or spire-tpm-plugin via a Talos system extension) is moved to phase 3 together with FDO.
• Risk until then: a compromised Kubernetes API could mint node identities; the node itself is still protected by Secure Boot and the TPM-sealed disk.

• Problem: in the lab, the Talos site networks are routed by the GCP VM's kernel, not through the SONiC-VS data plane, so a SONiC ACL would not see Edge-to-Core traffic.
• Decision: lab/scripts/border.sh installs a BASTION-BORDER chain (in DOCKER-USER and FORWARD): Edge A and its switch may reach Core only on TCP 7222 and 443, Core may not open connections to Edge A, and drops are logged. border.sh cut|restore simulates a WAN cut (T11, T12).
• POC: stand-in only. Prod: the same rules as SONiC ACLs on the Edge switch host (ADR-012); to be validated on hardware SONiC.

• Decision:
  • lab/gatectl (Go) attaches to each hostapd control socket and writes VLAN_MEMBER in CONFIG_DB. Every port starts in quarantine. ADMIT, QUARANTINE and DENY are written as hash-chained, Ed25519-signed JSON lines (ADR-025). On restart it resumes the chain from the last line.
  • A shipper publishes each line to the leaf's local JetStream (AUDIT_A), which Core sources into AUDIT_A_CORE. Nats-Msg-Id = event hash with a 30-day duplicate window, so re-shipping is idempotent (T60).
  • audit-verify.py checks hash, signature, seq and prev-linkage, reports exact re-deliveries as duplicates, and reports a new chain (seq 1) as GENESIS.
• Finding: a wiped switch log starts a new chain, which is indistinguishable from truncation unless Core anchors the last known head. Prod: Core records the chain head per site and alerts on GENESIS (--strict); the signing key moves to the switch TPM.

• Problem: SPIFFE federation normally polls a bundle endpoint over HTTPS, which would add a third Edge-Core connection, against the two-egress rule (ADR-012).
• Decision: each side publishes its bundle to cust-<x>.federation.bundle.<td> in its customer account. Core keeps the latest per subject (stream FED_A, one message per subject), and each side sets the peer bundle with spire-server bundle set (lab/scripts/fed-sync.sh). Revocation deletes the bundle at Core and revokes the leaf user (T40).
• POC: a periodic sync script. Prod: a small controller that watches the stream, checks a signature on the bundle, and refreshes before expiry.

• Decision and findings:
  • Talos 1.14 rejects the v1alpha1 cluster.network/proxy/inlineManifests style for Cilium. Cilium goes in through a multi-document patch (KubeFlannelCNIConfig removed, KubeProxyConfig enabled:false, KubeInlineManifestConfig), with kube-proxy replacement (T55).
  • Single-node sites need cluster.allowSchedulingOnControlPlanes: true; removing the taint with kubectl is reverted by Talos within a minute (T56).
  • Until Longhorn or another StorageClass exists, the SPIRE datastore is emptyDir (a restart loses federated bundles, so fed-sync re-runs; T57), and the NATS hub's JetStream uses a hostPath so streams survive hub restarts (T58).
  • Sourced streams across the leaf need the leaf user to subscribe to $JS.<domain>.API.>, $JS.FC.> and $JS.ACK.>, still limited to its own domain. A source created before the leaf settles can stay inactive, so setup re-creates it (T58).
  • Gitea runs as non-root and cannot bind 443; a sidecar forwards 443 to it (T59). Core services use host networking on the Core node in the lab.

• Decision: lab/kms (bastion-kms) is the Sidero Labs reference KMS server (github.com/siderolabs/kms-client, pkg/server) with a key handler that enforces ADR-016 and ADR-021. A key is released only if (1) the node UUID was enrolled at onboarding and is not revoked, (2) the request comes from the node's pinned bootstrap address, and (3) a fresh switch-host self-check (at most 120 s old) matches the golden measurement recorded at onboarding. Per-node keys are derived from a master key with HKDF. On a refusal the reference server returns random bytes, so the LUKS slot stays shut (fail closed). Every START, RELEASE and DENY (with reason) goes into a hash-chained JSON log with the Linux append-only attribute (T27).
• Which volume: the Edge Talos EPHEMERAL volume (all workload data, etcd, container images) has one key slot, the KMS, and no TPM fallback slot, so a KMS refusal really keeps it shut (T7, T22). STATE stays TPM-sealed: Talos cannot use a custom CA or machine-config networking for a KMS-encrypted STATE, because both live on STATE. The KMS CA reaches Talos through a TrustedRootsConfig document.
• Onboarding: the Edge Talos VM boots in maintenance mode. Setup reads its UUID, enrols it at the KMS, then applies the config (stand-in for FDO, ADR-016 point 2). The lab boot order is switch host, then KMS, then Edge Talos install and unlock.
• Findings and limits:
  • The Talos KMS API sends only the node UUID, over server-only TLS with no client certificate (verified in internal/pkg/encryption/keys/kms.go and kms-client/api/kms/kms.proto). The key can't be bound to the node's TPM EK, so the binding is the UUID, the pinned source address and the port gate.
• Status: Accepted for the POC (decided by Michael, 14 Oct 2026), with two compensating controls:
  1. The Talos node's infrastructure port is admitted with 802.1X EAP-TLS using a TPM-backed certificate, so only the genuine node can hold the address that reaches the KMS. This strengthens ADR-016 port pinning and T23, and is tested by T68. Finding: Talos has no native wired 802.1X supplicant, and the KMS is needed before EPHEMERAL unlocks. The supplicant (wpa_supplicant plus the tpm2 OpenSSL provider) must therefore run as a Talos system extension in our own image, so this control depends on ADR-028.
  2. SPIRE TPM node attestation still gates workload identity and secrets, so a stolen disk key opens a disk, not the system (ADR-046 and ADR-054).
• Production requires TPM-bound KMS requests. The request for enhancement to Sidero Labs is RFE-SIDERO-KMS-TPM-BINDING.md (not filed).
  • SONiC-VS has no TPM, and in the lab it has no routed port on the Talos bridge. The KMS therefore runs in the lab VM's network namespace on the Edge A bridge (10.5.20.1:4443), next to the sw-edge-a container, with iptables allowing only the pinned node (T23). The "fresh TPM check of the switch" is a SHA-256 measurement of the gate controller, hostapd configs, gate CA and KMS binary, refreshed every 30 s. It stands in for a Keylime or TPM quote.
  • After a refusal, Talos keeps retrying. Once the cause is fixed (restore, switch back to golden), the volume unlocks without another reboot.

• Problem: the SPIRE datastore lost federated bundles on restart (T57). Longhorn on Talos needs the iscsi-tools and util-linux-tools extensions in the image schematic, which means a new Secure Boot image, and it adds nothing on single-node sites.
• Decision: lab/scripts/storage.sh installs local-path-provisioner (default StorageClass) on a directory-type Talos UserVolumeConfig at /var/mnt/local-path-provisioner (T62: v1alpha1 kubelet extraMounts are rejected in multi-document configs). That path is on EPHEMERAL, so data at rest is covered by the KMS- or TPM-released LUKS2 key. The SPIRE server uses a 1 GiB PVC. T57 is fixed.
• Prod: Longhorn or Talos user volumes come with the self-built image (ADR-028) and with multi-node or replicated sites, if those are ever needed.

• Decision: lab/scripts/argocd-edge.sh installs Argo CD at Edge A (no Dex, no notifications, reconcile every 60 s). It watches bastion/edge-a-config on Core Git over the 443 border path, with auto-sync, prune and self-heal. The demo app apps/sensor is a Deployment in namespace demo. Nothing at Core pushes into the edge. During a WAN cut the edge keeps its last state, and it reconciles after reconnect (T12b).
• Lab limits: Core Git is plain HTTP on port 443 (ADR-050), and container images come from public registries. Prod: TLS with the Core CA, signed commits checked by Argo CD's GPG verification, and images only from the local registry or update volume (ADR-016 point 4).

• SPIRE TPM node attestation (ADR-046): still k8s_psat. Talos has no way to provision a TPM DevID, and spire-tpm-plugin needs the host TPM device and files inside the agent pod, which would need a Talos extension in a new signed image. Moves together with ADR-028.
• TPM-backed 802.1X admission for the Talos node port (moved to the top of the phase 3 queue, compensating control for ADR-051, T68): this needs a Talos system extension with a wired supplicant and a TPM-resident key, so it is blocked on ADR-028 (not started, by decision).
• TPM-backed 802.1X device keys: the Alpine test devices have no TPM. It needs one swtpm per device container plus the tpm2 OpenSSL provider for wpa_supplicant. Planned for phase 4; device keys stay software keys.
• Keylime on the switch host: the Keylime agent needs a TPM, and SONiC-VS has none. The KMS gate uses the measurement stand-in from ADR-051. T8 and T26, and the SVID part of T22, stay blocked.
• Own Secure Boot keys / self-built image (ADR-028), so T6 is blocked: it needs our own imager run with our own signing keys and those keys enrolled in OVMF. The Image Factory image is signed with Sidero's key, so a modified UKI can't be produced and tested against it.
• T9 (SVID loss within TTL): not run in phase 3; it needs a workload agent entry and a 10-minute wait. Planned with the TPM attestation work.

• Decision: the switch host never becomes a Kubernetes node and runs no Cilium. Each layer enforces in its own place:
  • SONiC enforces at the network level: VLANs, ACLs, the 802.1X port gate and the WAN border.
  • Cilium enforces between pods inside Talos, by identity (ADR-031).
  • Both rule sets are generated from one policy source in Git, so they can't drift apart.
• Optional: Tetragon on the switch host, for runtime visibility only and never for enforcement.
• Rationale:
  • Making the switch a Kubernetes node would widen the attack surface on the root of trust. The switch host also runs the Sidero Labs KMS server (ADR-051).
  • Cilium's eBPF hooks would conflict with SONiC's data path.
• Precedent: the Hedgehog architecture overview (docs.hedgehog.cloud) uses the same split. The switches run only a Fabric Agent that applies gNMI changes to CONFIG_DB. Kubernetes (K3s on Flatcar) runs on a separate control node, and only the gateway nodes join as workers.
• Prod: if the design is scaled up to multi-switch fabrics, look at Hedgehog again.
• Diagram: checked. v5.1 has no "SONiC + Cilium" label; Cilium appears only on the Talos clusters.

• Finding 1, root cause of T58, T12 (NATS part) and T31 failing:
  • The Core *_CORE streams source the Edge streams through $JS.edge-a.API. The hub sends the consumer-create request across the leaf, and in the lab's nats-server 2.15.1 the Edge answers on $JSC.R.<...> (server/jetstream_cluster.go, infoReplySubject), not on _INBOX or $JS.
  • The Edge A leaf user could not publish there, so the reply was dropped silently. No source consumer was created, the streams stayed active=-1, and no logs showed a permission violation.
  • Recreating the streams didn't help. Allowing everything did, which pinned it down to permissions.
• Decision 1: the leaf user also gets allow-pub $JSC.R.> (lab/scripts/nats-setup.sh), still only inside account CUSTA, so customer isolation (T30) is unchanged. Regression test T69 checks that the permission is there and the sources are active right after setup.
• Prod: pin the nats-server version, and re-check the leaf permission set on every NATS upgrade (ADR-039).
• Finding 2 (T24 and T66): with the SPIRE datastore on a PVC (ADR-052), a server restart keeps the existing X.509 CA, which is signed by the old intermediate, until it rotates on its own at half of caTTL. New SVIDs therefore didn't chain to the rotated intermediate.
• Decision 2: after an intermediate rotation, lab/scripts/spire-rotate-ca.sh pushes the new upstream secret, restarts the server, then forces a local-authority rotation (spire-server localauthority x509 prepare, then activate). This step is part of the CA rotation runbook (ADR-018) and covered by T24 and T67.