The Thermostat That Ate Infrastructure: How a Container Self-Healing Pattern Became a Deployment Engine

The Thermostat That Ate Infrastructure: How a Container Self-Healing Pattern Became a Deployment Engine — And Broke Everything

The people who built Kubernetes told us in 2016 that infrastructure dependency management was an unsolved problem their system was not built to handle. The industry built on top of it anyway. This article documents how.

This is the synthesis article for the series “Why Kubernetes Infrastructure Rots.” Parts 1–6 build the evidence from repo fragmentation to silent data loss. This article delivers the verdict: the platform’s computational model is the wrong one for the problem the industry is using it to solve.

The Quote Nobody Read

In 2016, the creators of Kubernetes — Burns, Grant, Oppenheimer, Hockin, and Wilkes — published “Borg, Omega, and Kubernetes” in ACM Queue. The paper describes the evolution from Google’s Borg cluster manager to the open-source system that now runs half the world’s production workloads. Buried in the final pages is this paragraph:

“Almost no system, however, captures, maintains, or exposes this kind of dependency information, so automating even common cases at the infrastructure level is nearly impossible. Unfortunately, the perceived complexity of systems that express, analyze, and use system dependencies has been too high, and so they haven’t yet been added to a mainstream container-management system. We still hope that Kubernetes might be a platform on which such tools can be built, but doing so remains an open challenge.”

— Borg, Omega, and Kubernetes, ACM Queue, 2016, pp. 90–91

Read that again. The people who designed Kubernetes:

  1. Acknowledged that infrastructure dependency management is fundamentally hard.
  2. Admitted that no container-management system handles it.
  3. Stated that the complexity of building such systems was “too high” even for Google.
  4. Called it an “open challenge” — unsolved, aspirational, future work.

Ten years later, the Kubernetes operator ecosystem uses that same container-management system as the primary mechanism for managing infrastructure dependencies across DNS, certificates, VPCs, load balancers, CRDs, network policies, rate-limiting configurations, and multi-step deployment workflows. The “open challenge” was never solved. The community just pretended it didn’t exist and built on top of the gap.

This article documents what that gap costs.

What Kubernetes Was Built to Do

Kubernetes descends from Google’s Borg — a cluster manager that scheduled containers across tens of thousands of machines. Borg’s scope was precise: take a container, find a machine, run it, restart it if it dies, scale it if demand increases. That’s it.

From the same ACM Queue paper:

“The original Borg system made it possible to run disparate workloads on shared machines to improve resource utilization.”

— Borg, Omega, and Kubernetes, p. 70

Everything else — naming, service discovery, load balancing, autoscaling, machine lifecycle management — was built around Borg, not inside it:

“The rapid evolution of support services in the Borg ecosystem, however, showed that container management per se was just the beginning of an environment for developing and managing reliable distributed systems. Many different systems have been built in, on, and around Borg to improve upon the basic container-management services that Borg provided.”

— Borg, Omega, and Kubernetes, pp. 79–80

Kubernetes inherited Borg’s scope. Brendan Burns (K8s co-founder) described the initial goal:

“The initial goal of this small team was to develop a ‘minimally viable orchestrator.’ From experience we knew that the basic feature set for such an orchestrator was: Replication to deploy multiple instances of an application; Load balancing and service discovery to route traffic to these replicated containers; Basic health checking and repair to ensure a self-healing system; Scheduling to group many machines into a single pool and distribute work to them.”

— Brendan Burns, The History of Kubernetes & the Community Behind It, 2018

Replication. Load balancing. Health checking. Scheduling. Four concerns. All container-centric. Not a single mention of infrastructure provisioning, dependency management, certificate lifecycle, DNS record creation, or multi-step orchestrated workflows.

The official Kubernetes documentation makes the scope even more explicit:

“Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services.”

— What is Kubernetes?

And what K8s explicitly is NOT:

“Does not provide nor adopt any comprehensive machine configuration, maintenance, management, or self-healing systems.”

— What Kubernetes is not

The system that half the industry uses for infrastructure management explicitly disclaims responsibility for infrastructure management.

The Reconciliation Loop: A Thermostat, Not a Pipeline

The core K8s innovation is the reconciliation loop — a controller that continuously compares desired state to actual state and converges the two. The official documentation uses the thermostat analogy:

“When you set the temperature, that’s telling the thermostat about your desired state. The actual room temperature is the current state. The thermostat acts to bring the current state closer to the desired state, by turning equipment on or off.”

— Kubernetes Controllers

This is a beautiful pattern for self-healing. A pod crashes — the ReplicaSet controller sees fewer pods than desired, creates a new one. A node dies — the scheduler reschedules. No human intervention. Continuous convergence. Elegant.

The design is explicitly single-concern, level-triggered, and order-agnostic:

“As a tenet of its design, Kubernetes uses lots of controllers that each manage a particular aspect of cluster state.”

“It’s useful to have simple controllers rather than one, monolithic set of control loops that are interlinked. Controllers can fail, so Kubernetes is designed to allow for that.”

— Kubernetes Controllers

The K8s controller guidelines enforce this:

  • “Operate on one item at a time.”
  • “Random ordering between resources.”
  • “Level driven, not edge driven.” — Your controller “may be off for an indeterminate amount of time before running again.”

And then the K8s documentation says something remarkable — it explicitly rejects being a workflow orchestrator:

“Kubernetes is not a mere orchestration system. In fact, it eliminates the need for orchestration. The technical definition of orchestration is execution of a defined workflow: first do A, then B, then C. In contrast, Kubernetes comprises a set of independent, composable control processes that continuously drive the current state towards the provided desired state.”

— What is Kubernetes?

Read that one more time. Kubernetes explicitly defines itself as the opposite of “first do A, then B, then C.” It is a set of independent loops, not an ordered pipeline. It converges state, it doesn’t orchestrate sequences.

And yet.

Every Kubernetes operator that provisions infrastructure does exactly “first do A, then B, then C.” Create the CRD before the controller. Create the namespace before the resources. Create the certificate before the ingress. Create the VPC before the subnet before the security group before the RDS instance. These are ordered, dependent, sequential operations — the exact thing K8s was designed to not do.

The reconciliation loop doesn’t know about ordering. It doesn’t know about dependencies. It doesn’t know that creating a RateLimitConfig CR requires the CRD to exist, which requires the Helm chart that installs it to be deployed, which requires the networking stack to be reconciled. It just watches one resource type and converges. And the “retries forever” isn’t a metaphor — it’s the literal behavior of the controller-runtime workqueue: when Reconcile() returns an error, the item is re-enqueued through a rate limiter (ItemExponentialFailureRateLimiter, default 5ms doubling to a 1000s cap) and retried, indefinitely, with no notion of “this dependency is permanently gone versus not yet here.” The queue has no dead-letter, no give-up, no dependency-satisfied signal — a missing prerequisite and a prerequisite-arriving-in-two-minutes are the same state to it: retry after backoff. So a dependency that’s permanently gone — deleted by another release, pruned by Flux, removed by a chart swap — is retried on the backoff schedule until a human intervenes. CrashLoopBackOff (and its controller-side twin, an item pinned at the 1000s backoff ceiling) is not a bug. It’s what happens when you use a thermostat to run a pipeline: the thermostat faithfully keeps trying to reach 40°C in a room with no heater.

The CRD: A Database Table with No Foreign Keys

Custom Resource Definitions were introduced as ThirdPartyResources — described in the original design proposal as:

“an easily extensible and simple mechanism for adding new APIs”

— ThirdPartyResources Design Proposal

The example was a CronTab. A simple spec: schedule, command, image. One controller watches one CRD, creates Pods on schedule. Clean.

CRDs have:

  • A schema (OpenAPI v3)
  • Spec and status subresource convention
  • Namespace or cluster scope
  • Standard CRUD via the API server

CRDs do not have:

  • Foreign keys
  • Referential integrity
  • Cross-CRD constraints
  • Lifecycle dependencies
  • Ownership boundaries between controllers
  • Schema compatibility validation across versions

These are not limitations that nobody got around to fixing. They are design consequences of a system built for independent, loosely-coupled resources. The K8s design principles state: “Functionality must be level-based” and “Object status must be 100% reconstructable by observation.” The spec (desired) / status (observed) convention was designed for a single controller converging a single resource type. Not for five controllers writing to the same CR with different typed views of the same schema.

The official K8s documentation says CRDs should NOT be used when:

  • The API “does not fit the Declarative model”
  • You need “synchronous completion”
  • Operations “are not CRUD-y”

— Should I use a ConfigMap or a custom resource?

Infrastructure provisioning is almost never purely CRUD-y. Creating an RDS instance requires creating a subnet group first. Creating a certificate requires validating DNS ownership first. Deleting a CRD requires ensuring no controllers depend on it first. These are all ordered, conditional, multi-step operations — the exact cases where K8s docs say “don’t use a CRD.”

The community used CRDs for all of them anyway.

Case in Point: Three Attempts in Five Days

A team building a CDN operator hit the most basic CRD ordering problem: installing CRDs and their custom resources in the same Helm chart. The operator manages CloudFront Multi-Tenant Distributions — cluster-scoped CRs that each provision a CloudFront distribution, cache policies, edge functions, and CloudWatch logging. The CRDs and the CRs that reference them are deployed together.

The problem: Helm validates templates before applying CRDs on upgrades. New CRD fields → new CR fields → validation fails because the CRD hasn’t been updated yet.

Three approaches in five days:

Day Approach Why it failed
1 Helm hook Jobs — pre-install/pre-upgrade hook with kubectl apply --server-side to install CRDs before templates render Too complex: requires ServiceAccount, ClusterRole, ClusterRoleBinding, ConfigMap with embedded CRDs, and a bitnami/kubectl image — all to solve “install A before B”
2 Separate subchart — CRDs in a dependency subchart’s templates/ so they update on upgrades Helm renders ALL templates (parent + subchart) before applying ANY. No install-order guarantee between parent and dependency
5 CRDs in templates + CRs in Go code — move CRDs to main chart’s templates/, move CR creation to operator startup code Removing the old Helm template deletes existing CRs before Go code can recreate them. Gap window where the operator’s own watched resources don’t exist

Installing a CRD before its custom resources is the most elementary infrastructure dependency. “First A, then B.” The Kubernetes documentation explicitly says the system “eliminates the need for orchestration” and is “the opposite of first do A, then B, then C.” Three engineers proved it right — they couldn’t express “first A, then B” in any combination of Helm, Flux, and controller-runtime without gaps.

The team’s own platform had a documented best-practice blueprint recommending separated Flux HelmReleases with dependsOn — essentially building a two-step pipeline outside Helm because Helm can’t express the dependency internally. The blueprint exists because the problem is universal. None of the three PRs referenced it.

The Operator Pattern: One SRE’s Knowledge, Universalized

In November 2016, Brandon Philips at CoreOS introduced the operator pattern with two examples: etcd-operator and prometheus-operator. Both managed a single stateful application’s lifecycle — backup, restore, scaling, upgrades. The K8s documentation describes the pitch:

An operator encodes “the key aim of a human operator who is managing a service or set of services.”

— Operator pattern

One SRE. One application. One concern. The operator watches a CR, does the thing the SRE would do, converges. Beautiful for etcd. Beautiful for Prometheus. Beautiful for any single application whose lifecycle fits the reconciliation model.

Then the community extrapolated. If an operator can manage etcd, it can manage a database. If it can manage a database, it can manage an entire cloud footprint. If it can manage a cloud footprint, it can manage the deployment pipeline itself. If it can manage the deployment pipeline, it can replace Terraform.

Each step felt small. Each step was locally rational. And each step moved further from the design assumptions of the underlying system — single-concern controllers, level-triggered convergence, independent loops, no cross-resource ordering, no dependency graph.

By 2024, the operator pattern was used for:

  • Provisioning VPCs, subnets, security groups (Crossplane)
  • Managing DNS records, certificates, CloudFront distributions (cert-manager, external-dns, custom operators)
  • Orchestrating multi-step deployments with ordering constraints (ArgoCD, Flux)
  • Running Terraform from inside K8s (tf-controller)
  • Managing the CRDs that other operators depend on (meta-operators)

A pattern designed for “restart etcd if it crashes” was now running “provision a multi-region, multi-account AWS infrastructure stack with cross-resource dependencies, ordered creation, conditional rollback, and compliance constraints.”

The thermostat was running the power grid.

What Terraform Gets Right (That K8s Operators Cannot)

Try deleting a VPC that has subnets in Terraform:

$ terraform plan -destroy -target=aws_vpc.main

Error: aws_subnet.private still depends on aws_vpc.main
  Delete aws_subnet.private first, or remove both.

It won’t let you. The DAG knows that aws_subnet.private references aws_vpc.main.id. Deleting the VPC without deleting the subnet first is not allowed. Not “allowed but risky.” Not “allowed with a warning.” Not allowed.

Try deleting a CloudFormation stack that exports a value consumed by another stack:

DELETE_FAILED: Export networking-stack:VpcId is referenced by stack gateway-stack.
Cannot delete export in use.

The stack won’t delete. CloudFormation’s cross-stack references create hard dependency edges — “After another stack imports an output value, you can’t delete the stack that is exporting the output value.” The DAG is the enforcement mechanism.

This is what predictable infrastructure looks like. Every destructive operation is previewed, validated, and either approved or blocked — before it touches real infrastructure. The DAG walks forward for creation (dependencies first) and backward for deletion (dependents first). Remove a resource from the config, and the tool shows you the cascade. The graph is not a visualization. It’s a constraint.

Now here’s what Helm does when you remove a subchart that installed a CRD consumed by another stack:

Release "gateway-stack" has been upgraded. Happy Helming!

Happy Helming. The CRD is gone. The controller in the other namespace will crash tomorrow when a new pod starts and tries to resolve the GVK. Nobody will connect the crash to the Helm upgrade that happened 24 hours earlier. The team will spend a morning debugging.

This happened. On a production platform. The ratelimitconfigs.ratelimit.solo.io CRD was deleted by a Helm subchart swap. The network operator (this article will call it NetOp for brevity, matching the shorthand used elsewhere in this series) entered CrashLoopBackOff on two rings. The Helm upgrade reported success. (Full incident, including the 24-hour detection delay and the exact error chain: Part 4, The Distributed Monolith.)

Terraform would have blocked the deletion. CloudFormation would have blocked the deletion. Helm didn’t know the deletion was dangerous because Helm doesn’t have a dependency graph. It has a per-release inventory — a flat list of resources this release installed. It doesn’t know what other releases depend on those resources. It doesn’t know that a CRD is a shared schema that controllers across the entire cluster rely on. Each release is an island. The blast radius is invisible until the blast happens.

The Controller ControllerRef: Single-Owner by Design

The K8s API was designed with a specific ownership model, codified in the ControllerRef design proposal as “The Three Laws of Controllers”:

  1. Take ownership: Controllers claim objects by adding a ControllerRef.
  2. Don’t interfere: Controllers avoid acting on objects they don’t own.
  3. Don’t share: Controllers don’t count unowned objects toward their desired state.

Each object can have at most one ControllerRef — strict single-owner semantics.

This was designed for the ReplicaSet-manages-Pods model. One controller, one set of pods, clear ownership. It breaks immediately when:

  • Two Helm releases install the same CRD (as happened with gloo-ee and legacy-gateway-crds both installing ratelimitconfigs.ratelimit.solo.io)
  • Two controllers reconcile the same CR (as happened with the Deployment Controller and NetOp both writing to ApplicationDeployment)
  • An admission webhook in one stack blocks resource creation by a controller in another stack (as happened with Gatekeeper blocking Gateway API Services)

The ownership model is single-owner, single-concern, within-release. Real infrastructure is multi-owner, multi-concern, cross-release. The model doesn’t scale because it was never meant to.

And note the two ownership mechanisms K8s actually ships don’t compose into a solution here. ownerReferences (the ControllerRef above) governs lifecycle — who garbage-collects whom — and enforces one controller-owner per object. Server-Side Apply managedFields governs field-level writes — which field manager owns which field — and does allow multiple co-owners. But managedFields only protects fields when every writer uses SSA; a single client-side Update() (a PUT) replaces the whole object and rewrites managedFields wholesale, blowing the co-ownership away (this is exactly Part 6’s field-stripping incident). So K8s has a single-owner mechanism for lifecycle and an all-or-nothing mechanism for fields, and neither expresses “these N controllers legitimately share this object, in this order, with these responsibilities.” Multi-owner coordination is the case both mechanisms structurally decline to model.

Every K8s-Native Project Hits the DAG Wall

The evidence is in the issue trackers.

Crossplane — the most ambitious attempt to use K8s operators for infrastructure provisioning — received a feature request for depends_on (Terraform-style dependency ordering) in Compositions (crossplane/crossplane#2072). The maintainer’s response:

“I’d strongly prefer to avoid doing so and continue leaning into loose coupling and eventual consistency.”

— Nic Cope, crossplane/crossplane#3393

The issue was closed as NOT_PLANNED in December 2024. Crossplane explicitly chose not to add a dependency graph. When users reported that deleting an EKS cluster before its Helm releases causes permanent controller failure (the controller can’t connect to the deleted cluster to clean up), the project added a Usage type for deletion ordering only — with the maintainer again stating he’d “strongly prefer to avoid” using it for creation ordering.

KubeVela built a separate workflow engine on top of K8s — with step-by-step execution, DAG sub-steps, suspension, and notification — because the native controller model couldn’t express “run A, then B and C in parallel, then D after both complete.”

Argo Workflows built a complete DAG executor as a CRD, providing loops, conditionals, timeouts, retries, and cancellation. It exists specifically because the reconciliation model can’t do ordered workflows.

Flux added dependsOn between Kustomizations — a readiness gate, not a dependency graph. No cycle detection (just deadlocks). No plan-time validation. No atomic execution. No rollback. A bolt-on sequencing hint for a system that has no concept of sequence.

Every significant K8s-native infrastructure project has encountered the same wall. The reconciliation loop can’t express dependencies. The projects either refuse to add them (Crossplane), build a completely separate engine (KubeVela, Argo), or bolt on a minimal workaround (Flux). None achieve what Terraform has had for a decade: plan-time dependency validation, topological execution, and refusal to perform unsafe deletions.

The Smallest Proof: A Node Boots Up

The missing DAG isn’t only a cross-stack or cross-controller problem. It exists at the most basic level of the system: a single node joining a cluster.

When a new EC2 instance joins an EKS cluster, the startup sequence looks like this:

  1. Instance boots, kubelet starts
  2. Kubelet registers the Node object with the API server
  3. The kube-scheduler (since 1.12, DaemonSet pods go through the default scheduler, not a special kubelet path) binds the node’s DaemonSet pods to it; the kubelet, seeing pods bound to its node, immediately starts their containers

The correction matters for accuracy but not for the point: nothing gates step 3 on the node’s API-server connectivity being usable by the workload. The pod is bound and the kubelet starts its containers as soon as the node is Ready — there is no readiness gate that says “don’t start pods that need the API server until the API server is reachable from this pod,” no dependency declaration, no After= directive. The scheduler’s job is “place the pod”; the kubelet’s job is “run the placed pod.” Neither has a concept of “this workload’s own dependency on a control-plane endpoint isn’t satisfied yet.”

Now consider secrets-store-csi-driver — a CSI DaemonSet that manages secret volume mounts for every pod on the node. It runs on every node in every cluster. On startup, it needs to build a RESTMapper — a client-side cache of the API server’s resource types. This requires a working API server connection.

The dependency is trivial: CSI driver → API server. One arrow. Systemd would express it in one line: After=kube-apiserver-ready.target. Kubernetes has no equivalent.

What happens in practice, observed on a production Ring 3 environment:

T+0:00  Node ip-10-64-152-13.ec2.internal created
T+0:24  secrets-store-csi-driver-t8b7n starts
T+2:45  secrets-store: "initializing metrics backend"
T+3:15  secrets-store: "failed to start manager: could not create RESTMapper from config"
        → Exit code 1. Container restarts.
T+3:43  node-driver-registrar: "Attempting to open a gRPC connection with /csi/csi.sock"
T+4:13  node-driver-registrar: "error connecting to CSI driver: context deadline exceeded"
        → Exit code 1. Container restarts.
        ... 4 more secrets-store crashes, 5 more registrar crashes ...
T+4:06  secrets-store finally starts (API server reachable)
T+5:45  node-driver-registrar connects to csi.sock
        Pod stable. All containers Running/Ready.

Nine restarts. Five minutes of crash-looping. The alert fires: [FIRING:4] Container Restarts on a production ring. The pod’s tolerations include op=Exists for node.kubernetes.io/not-ready — by design, because CSI drivers need to start early so other pods can mount volumes. The pod is supposed to run on not-ready nodes. It just can’t, because the thing it depends on isn’t ready either.

The registrar’s failure is downstream — it can’t connect to the gRPC socket because the main container hasn’t started the listener yet. A cascade from one missing edge in a graph that doesn’t exist.

Why nobody fixes this

This is a solvable problem. An init container that polls the API server health endpoint. A startup probe with a generous failureThreshold. A kubelet flag that gates DaemonSet scheduling on node readiness. Any of these would work. None exist in the upstream project, and no KEP proposes adding node-level startup ordering to Kubernetes.

The reason is the operator mindset’s signature phrase: “good enough.”

The pod crashes 9 times, then stabilizes. The node is healthy after 5 minutes. The alert fires, someone glances at it, recognizes the pattern, ignores it. Over time, the team learns that Container Restarts on new nodes is noise. They adjust the alert threshold, or add a silence rule, or just stop looking. The crash-restart loop is the solution in the operator’s mental model. Kubelet restart is the retry mechanism. The system “self-heals.”

A software engineer sees a different thing: a pod that crashes 9 times on every node boot, across every cluster, across every ring, for the entire lifetime of the platform. Thousands of nodes, tens of thousands of unnecessary restarts, an alert channel trained to ignore production signals because the baseline noise is too high. A dependency that would be one line in any system that models dependencies — unrepresented, unfixed, and invisible to the people who consider it solved.

This is the perceptual gap that Part 3 (The Abstraction Instinct) describes: the operator sees resilience where the engineer sees a missing dependency edge. And the ecosystem reinforces the operator’s perception — the CSI driver project won’t add the init container because “pods should handle transient API server unavailability.” The Kubernetes project won’t add node-level startup ordering because “DaemonSets are intentionally scheduled eagerly.” Every layer defers to the layer below. Nobody owns the dependency.

The pattern at every layer

The CSI bootstrap race is the smallest instance of a pattern that repeats at every level of the Kubernetes stack:

Layer Dependency How it’s “handled”
Node CSI driver → API server Crash-restart until API server is reachable
Release CR → CRD Helm install fails, upgrade succeeds — teams add retry scripts
Stack Controller → CRD from another stack Flux dependsOn — no cycle detection, deadlocks silently
Controller Operator → watched GVK Controller crashes on startup if CRD missing — restart loop as “discovery”
Cross-stack HelmRelease health → Terraform status Stale status field blocks deployment chain for 7 days, zero alerts

Every row is the same structural failure: a real dependency, unmodeled, with crash-restart or retry-until-timeout substituting for explicit ordering. The workarounds accumulate — startup probes, readiness gates, dependsOn, health checks, force-reconcile annotations — each one an accidental complexity layer added because the platform won’t model the one thing every layer needs: a graph of what depends on what.

Systemd solved this for Linux services in 2010. Terraform solved it for cloud resources in 2014. Make solved it for build targets in 1976. The Kubernetes community has had the concept available for fifty years and chosen, at every opportunity, not to implement it. Not because it’s hard — because “good enough” is the community’s engineering standard, and the ecosystem’s economic incentives (as documented in Part 3) reward navigating complexity over eliminating it.

Six Incidents, One Missing DAG

One platform team hit six incidents in four weeks, caused by the same architectural gap. Different symptoms. Same root cause: independent deployment units modifying shared cluster state with no dependency graph. (For the full incident narratives, see Part 4: The Distributed Monolith and Part 6: The Shared Mutable State.)

Week What happened Failure class Blast radius Detection delay
Day 1 Gatekeeper admission policy blocked Service creation by a controller in another stack Cross-stack policy dependency Infinite retry loop, no alert Hours (discovered by reading pod logs)
Day 3 Helm subchart swap deleted a CRD that another stack’s controller depends on Cross-release CRD lifecycle Controller CrashLoopBackOff on 2 rings 24 hours (cached informers masked the blast)
Day 4 Two controllers wrote to the same CR with incompatible typed structs; fields silently erased Cross-controller schema versioning Majority of CRs had null fields Unknown (discovered during unrelated investigation)
Week 2 CDN operator manages 3 rings from one pod; no canary, no per-ring rollback Forced distributed monolith 3 rings of customer domains Architectural (not yet an incident — inevitable)
Week 3 Wrong FQDN prefix propagated through 4 controllers; 503 for all default apps Semantic invalidity in reconciliation chain Ring 1 apps down 4 days, 9+ engineers across 5 teams
Week 4 Stale Reconciling: True on a Terraform resource blocked all deployments for 7 days Convergence succeeded, status didn’t update 7 environments, zero alerts 7 days (discovered by accident)

Every incident shares three properties:

  1. CI said green. Flux said green. The cluster was broken. No tool in the pipeline detected the failure because no tool models cross-release dependencies.

  2. The reconciliation loop did exactly what it was told. Each controller converged its resource toward its desired state. The system-level failure was invisible to each controller individually — it existed only in the relationships between resources that no controller tracks.

  3. Terraform or CloudFormation would have caught it at plan time. Delete a CRD with dependents? Blocked. Create a resource that admission policy will deny? Shown in the plan. Write fields that another controller will overwrite? Visible in the state diff. The DAG is the enforcement mechanism.

The Week 3 incident illustrates the reconciliation loop’s deepest blind spot. Every loop did its job perfectly — the Deployment Controller wrote the CR, NetOp created the VirtualService, Istio synced the Envoy config. The system converged to a state where every request returned 503. The thermostat was set to 40°C and faithfully held it there. Four controllers, four successful reconciliations, one broken system. No controller in the chain validates that its input is semantically correct — only that it’s syntactically present. The thermostat doesn’t know that 40°C will melt the wax figures. It just holds the temperature.

The Week 4 incident reveals the most insidious failure mode: “convergence succeeded and nobody was told.” The Terraform resource was healthy — plans ran every 12 hours with no changes, Ready: True. But a stale Reconciling: True condition (the controller forgot to clear it) caused the Kustomization health check to read it as “still converging,” blocking the entire deployment chain. Seven environments. Seven days. Zero alerts. A stuck “heating” indicator light preventing the building management system from approving any HVAC changes — while the room was already at the right temperature.

Six incidents. One missing DAG. The same gap that lets a CSI driver crash-loop on node boot lets a Helm upgrade silently delete a CRD, lets two controllers corrupt the same CR, and lets a stale boolean freeze deployments for a week.

The Avalanche: How Twelve Locally Rational Decisions Built an Undeployable System

The CDN operator didn’t start as a 696-line reconciler managing six rings from two pods with no canary mechanism and three failed deployment approaches in five days. It started as a reasonable idea: replace per-tenant CDN distributions with a shared multi-tenant CDN to reduce cost and provisioning time. Every decision along the way was individually defensible. The result is an operator that its own team can’t deploy safely.

Here’s the cascade, reconstructed from design documents, PRs, and team discussions:

Decision 1: “The CDN provider’s multi-tenant API requires managing distributions at the account level, not per-cluster.” Reasonable. The API aggregates tenants into shared distributions — that’s the whole point. The operator needs account-wide access.

Decision 2: “We’ll deploy to management environments, since that’s where cross-account IAM roles exist.” Reasonable. Management environments already have the trust relationships. No new infrastructure needed.

Decision 3: “Two CDN accounts — one for non-production rings, one for production.” Inherited. The legacy CDN used the same account split. Nobody questioned whether the new architecture should inherit the old topology.

Decision 4: “DNS wildcard records are per-hosted-zone, and hosted zones are per-account. So all three non-production rings share one DNS zone.” Physics. Route53 constraint. Can’t be worked around without customer-visible DNS changes.

Decision 5: “So one operator deployment manages three rings.” Follows from 2+3+4. The team investigated per-ring isolation. Rejected it. The distributed monolith is architecturally mandatory.

Decision 6: “The operator needs CRDs for its multi-tenant distribution and domain resources.” Standard K8s pattern. Two CRDs, two controllers, one Helm chart.

Decision 7: “CRDs and custom resources in the same chart — the CRDs go in /crds for ordering.” Standard Helm pattern. Works on first install.

Decision 8: “CRDs don’t update from /crds on upgrades. Move to templates.” Helm limitation. Documented in Helm docs, known for years. The team moves CRDs to templates/.

Decision 9: “Templates fail validation because CRDs aren’t registered yet. Try a subchart.” Reasonable. Separate chart for CRDs as a dependency. Merged same day.

Decision 10: “Subchart doesn’t solve ordering either — Helm renders all templates before applying any. Revert. Move CR creation to Go startup code.” Discovery. The team’s third approach in five days. The operator now creates its own watched resources at startup.

Decision 11: “The environment field on the CRD is redundant with ring. Remove it.” Cleanup. But now a CRD schema migration is needed across all clusters.

Decision 12: “The operator also needs ACM certificates, Route53 records, CloudFront functions, CloudWatch log groups, cross-account IAM provider configs, key pair rotation, and status synchronization with the legacy operator.” Requirements. Each individually necessary. Collectively, the reconciler for a single distribution is 696 lines of strictly ordered AWS SDK calls wrapped in a reconciliation loop that retries the entire sequence on any failure.

Lay these out as a dependency graph:

Route53 wildcard constraint (physics)
  → Shared CDN accounts (inherited)
    → One operator manages 3 rings (forced)
      → No per-ring canary (consequence)
      → No per-ring rollback (consequence)
      → CRDs + CRs in same chart (implementation)
        → CRDs don't upgrade (Helm limitation)
          → Move to templates (workaround)
            → Validation fails (Helm limitation)
              → Subchart (workaround #2)
                → Still fails (Helm limitation)
                  → Go startup code (workaround #3)
                    → CR deletion gap (new problem)
      → 696-line sequential reconciler (accumulation)
        → No checkpoint, no partial progress (K8s limitation)
      → Cross-account IAM chain (requirement)
        → Silent failure on trust policy changes (consequence)
      → several interdependent PRs across multiple repos (accumulation)
        → No merge order, no transactional boundary (consequence)

Every arrow is locally rational. No single decision is wrong in isolation. But the graph has a property that no individual decision reveals: the system cannot be deployed safely. There is no sequence of PR merges, Helm upgrades, and ring promotions that moves from the current state to the desired state without a window where customer traffic is at risk across three rings simultaneously.

The team knows this. Their lead called for a “mini task force to clear the PR backlog” — an implicit acknowledgment that the PRs form a single atomic change scattered across repos. A team member who researches the deployment tooling concluded: “this is a domain modeling problem, not a tooling problem — from CPU threads to database transactions to deployment dependencies, none of the tools can solve it for users.”

He’s right. The tools can’t solve it because the problem isn’t “how do we deploy CRDs before CRs.” The problem is: “should one operator manage CDN resources across three rings from a single pod?” That question was never asked. It was answered implicitly by decisions 1 through 5, each of which was reasonable, none of which was evaluated for its downstream architectural consequences.

The platform’s own best-practice blueprint recommends separated Flux HelmReleases with explicit dependsOn ordering — essentially building a two-node DAG outside Helm. The CDN operator team iterated through three Helm approaches without referencing it. Not because they’re unaware of the blueprint, but because the blueprint solves “CRDs before CRs” — a symptom. The avalanche started at “two CDN accounts” — a cause that no deployment pattern can fix.

This is the thermostat problem at its most insidious. Each decision is a small temperature adjustment. The room feels fine after each one. But the cumulative effect is a building with no fire exits — and the thermostat has no concept of “fire.”

The Honest Diagnosis

The operator/CRD model is not broken. It is misapplied.

It works for what it was designed for:

  • Restarting crashed pods (ReplicaSet)
  • Scaling Deployments (HPA)
  • Rolling updates (Deployment controller)
  • Running batch jobs (Job controller)
  • Managing a single stateful application (etcd-operator, prometheus-operator)

All of these are single-concern, single-resource-type, order-agnostic, and continuously convergent. The reconciliation loop is perfect.

It fails for what it was never designed for:

  • Multi-step infrastructure provisioning with ordering constraints
  • Cross-release dependency management (CRD A depends on CRD B existing)
  • Cross-controller schema compatibility (Controller A and B write to the same CR)
  • Cross-stack policy enforcement (Stack A’s resources must satisfy Stack B’s admission rules)
  • Transactional deployment (either all of this deploys or none of it does)
  • Rollback across multiple independently reconciling components

All of these require a dependency graph — a DAG that knows what depends on what, validates the graph before mutation, and refuses unsafe operations. The thermostat pattern cannot provide this. Not because the implementation is bad, but because the computational model is wrong for the problem.

In the Ladder of Abstraction framework, the reconciliation loop is a Layer -1 engine — a stateful, continuous control loop that makes reality match a desired state. Terraform operates at Layer 2 — declarative composition with a graph database, dependency edges, and topological traversal. The entire K8s infrastructure ecosystem is attempting to solve Layer 2 problems (ordered, multi-resource provisioning with dependency constraints) using a Layer -1 engine (continuous, single-resource convergence with no dependency model). The CRD ordering problem, the cross-stack blast radius, the missing DAG — all are symptoms of layer-skipping: forcing a lower layer to recreate the capabilities of a higher layer it was never designed to provide.

A thermostat doesn’t know that turning off the heater will freeze the pipes. It knows the temperature. A reconciliation loop doesn’t know that deleting a CRD will crash a controller in another namespace. It knows the desired state of its own release.

The industry took a thermostat and asked it to run a power grid. It’s been doing its best. The incidents are the result.

Why Nobody Connects the Dots

Here’s how the CRD deletion incident actually played out in the team’s Slack channel:

  1. The network operator is crash-looping. Someone asks “what changed?”
  2. The team checks its deployment. An HPA PR was merged this morning. That’s what changed.
  3. The PR’s author offers to revert it. The team spends an hour investigating whether those changes caused it.
  4. Eventually someone reads the actual error log: no matches for kind "RateLimitConfig". The CRD is missing.
  5. Another engineer force-reconciles the gloo-ee HelmRelease. CRD comes back. The operator recovers.
  6. The team moves on to the next fire.

The incident is “resolved.” Nobody writes a root cause analysis. Nobody asks why the CRD was missing. Nobody connects it to the gateway stack subchart swap from the previous day. Nobody notices that the same architectural pattern — independent deployment units modifying shared cluster state with no dependency graph — caused the Gatekeeper block two days earlier, the field-stripping bug the day before, and would cause a 4-day cross-team investigation the following week when a wrong FQDN prefix silently propagated through four controllers. Six incidents, six separate Slack threads, six separate “fixes,” zero pattern recognition.

This is not laziness. This is the operator mindset doing exactly what it was trained to do.

The operator’s RCA stops at “what changed.” The deployment broke after someone’s PR. Force-reconcile fixed it. Root cause: something went wrong with the deployment. Fix: force-reconcile. Prevention: be more careful next time. Done.

A software engineer’s RCA asks “why was this possible.” Why could a Helm subchart swap silently delete a CRD that another stack depends on? Why does Helm not track cross-release dependencies? Why does Flux report green when a controller is crash-looping? Why did the blast take 24 hours to become visible? These are architecture questions. They require abstraction — seeing three concrete incidents as instances of one abstract pattern.

Operators don’t abstract because their tools don’t require abstraction. kubectl, helm, and flux are concrete tools for concrete operations. Force-reconcile this. Restart that. Check the logs. Patch the resource. The feedback loop is: something broke → find the broken thing → fix the broken thing. There is no step for “identify the structural pattern that made this class of breakage possible.”

Terraform forces abstraction. You can’t use Terraform without declaring dependencies. The depends_on block, the implicit reference graph, the terraform plan output — all require you to think about what depends on what. The tool requires you to model the relationship between resources. If you don’t, terraform apply fails. The abstraction is mandatory.

Helm requires no abstraction. You declare resources. Helm applies them. If they depend on something in another release, that’s your problem. The tool doesn’t ask. The tool doesn’t know. The tool doesn’t care. You can use Helm for years without ever thinking about cross-release dependencies — until the day one of them breaks, and you spend a morning debugging.

The six incidents are treated as six unrelated events because the operator mindset has no category for “cross-release dependency failure.” The vocabulary doesn’t exist. There’s no word for “a CRD that two Helm releases installed independently, where deleting one breaks the other.” There’s no word for “two controllers writing to the same CR with incompatible typed structs.” There’s no word for “an admission policy in one stack blocking a resource created by a controller in another stack.” Each incident gets its own Slack thread, its own ad-hoc fix, and its own forgetting.

A software engineer would call all three “violations of referential integrity across deployment boundaries.” One concept. One root cause. One fix: a dependency graph that makes these violations impossible. But that concept requires the ability to see three concrete incidents as instances of one abstract pattern — and the operator training pipeline doesn’t develop that skill. It develops kubectl skills. It develops “debug the broken pod” skills. It develops “trace the HelmRelease conditions” skills. It does not develop “identify the structural pattern across three independent failures” skills.

This is why the same class of incident recurs every few months on every platform team using Kubernetes for infrastructure management. Not because the teams are incompetent — they’re often excellent at what they do. But what they do is react to concrete failures, not prevent abstract failure classes. The force-reconcile is fast, effective, and forgotten. The architectural fix — a dependency graph, a deployment DAG, referential integrity across releases — would prevent the entire class of failure. But nobody proposes it because nobody sees the class. They see three bugs. They fix three bugs. They go home.

The K8s ecosystem produces this outcome by design. The tools are concrete. The training is concrete. The job postings ask for “5 years of Kubernetes experience,” not “5 years of infrastructure modeling experience.” The incentive structure rewards the person who can force-reconcile at 2am, not the person who can design a system that never needs force-reconciling.

And when someone does see the pattern? When a software engineer on the team points out that three incidents share a root cause, that the tooling lacks a dependency graph, that Terraform solved this a decade ago?

They’re told: “That’s not how Kubernetes works.”

Not “that’s a good point, but here’s why we can’t do it.” Not “let’s evaluate the tradeoff.” Just: “that’s not how we do things here.” The proposal dies in a Slack thread. The next incident happens. The force-reconcile is applied. The cycle continues.

This is what happens when a tool becomes an identity. Engineers who have spent five years mastering K8s reconciliation loops, Helm release debugging, Flux Kustomization tracing, and kubectl incantations have built their career on navigating the system’s complexity. A proposal to replace that complexity with a dependency graph isn’t heard as “let’s make the system better.” It’s heard as “your five years of expertise are unnecessary.” The reaction is predictable and human — reject the premise, defend the tool, reframe the limitation as a feature.

“Eventual consistency is a feature, not a bug.” “The reconciliation model is more resilient than a DAG.” “Terraform has its own problems.” “You just need to understand how Helm works.” Every one of these responses treats Kubernetes as the fixed frame of reference — the laws of physics within which all solutions must operate. The possibility that K8s is the wrong tool for this class of problem is outside the universe of discourse. It’s not considered and rejected. It’s not considered.

This is the deepest layer of the problem. The tools shape the thinking. The thinking shapes what solutions are conceivable. And when you’ve worked inside K8s long enough, the boundaries of the tool become the boundaries of your imagination. A dependency graph across Helm releases? “That’s not how Helm works.” Referential integrity across CRDs? “CRDs don’t do that.” Plan-time validation of cross-stack dependencies? “Flux doesn’t support that.” Each response is technically correct — and completely misses the point. The point isn’t what the tools support. The point is what the problem requires.

Terraform’s DAG exists because HashiCorp asked “what does infrastructure provisioning require?” and built the tool to match. The K8s operator ecosystem exists because the community asked “what can we build with reconciliation loops?” and stretched the tool to fit. The first approach produces tools that match the problem. The second produces problems that match the tool — and engineers who can’t see the difference because they’ve never used anything else.

The Crossplane maintainer who closed the depends_on issue with “I’d strongly prefer to continue leaning into loose coupling and eventual consistency” wasn’t being unreasonable. Within the K8s worldview, eventual consistency is the answer. The reconciliation loop is how you manage infrastructure. The question “should we add a dependency graph?” is answered with “no, because that’s not how Kubernetes works.” The premise — that Kubernetes might be the wrong computational model for this problem — is never examined. It can’t be. Examining it would mean questioning the foundation that the entire ecosystem is built on.

This is how a container self-healing pattern ends up running infrastructure provisioning for a decade. Not because anyone decided it was the right tool. But because the people using it can no longer imagine using anything else.

Why This Persists

The K8s operator ecosystem persists for the same reason any misapplied but entrenched technology persists: the people who understand its limitations are the ones whose expertise depends on it.

The engineer who carries the mental dependency graph — who knows which CRDs are shared, which controllers watch which GVKs, which Helm labels to check before removing a subchart — is the irreplaceable expert. Their value comes from navigating a system that a better architecture would make navigable by anyone.

Terraform’s DAG externalizes this knowledge. The dependency graph is in the tool, not in someone’s head. Anyone can run terraform plan and see the blast radius. You don’t need five years of tribal knowledge to know that deleting a VPC will break the subnets.

The K8s ecosystem keeps the graph implicit. You need the tribal knowledge. You need to know to run kubectl get crd -l helm.toolkit.fluxcd.io/name=<release> before removing a subchart. You need to know that gloo-ee also installs the same CRD. You need to know that NetOp watches ratelimit.solo.io/v1alpha1 at startup. None of this is in any tool. It’s all in the heads of the people who’ve been burned before.

This is not a conspiracy. It’s an incentive structure. The ecosystem rewards “Kubernetes experience” — the ability to debug reconciliation failures, trace HelmRelease conditions, force-reconcile stuck resources. It does not reward “infrastructure modeling experience” — the ability to design a system where those failures can’t happen. The first creates job security. The second eliminates it.

The K8s founders told us in 2016 that this was an open challenge. The community’s response was to not solve it — and to build a $10 billion ecosystem on top of the gap.

The Ceiling Nobody Sees

Kubernetes does blue/green deployment perfectly — for pods. Two ReplicaSets, readiness gates, traffic shift. The old ReplicaSet scales down after the new one passes health checks. Clean, elegant, battle-tested. This is container orchestration. This is what Kubernetes was built for.

The industry saw this and extrapolated: if Kubernetes can blue/green pods, it can blue/green infrastructure. But infrastructure blue/green means atomically switching a CloudFront distribution, its ACM certificate, its Route53 records, its WAF rules, its origin config, and the operator that manages all of them. That’s not a ReplicaSet swap. That’s a transactional state machine across five AWS services and three Kubernetes controllers — exactly the kind of ordered, multi-step, dependency-aware operation that the 2016 paper called an “open challenge.”

The same extrapolation applies to every advanced infrastructure concept:

Immutable infrastructure at pod level means “delete the pod, create a new one from the same image.” At infrastructure level, it means replacing an entire deployment unit atomically — the StatefulSet, its PVCs, its CRDs, its ConfigMaps, its network policies, its watched resources. Kubernetes makes the pod immutable. Everything the pod depends on — the CRD schema that carries forward, the ConfigMap that’s mutated in place, the PVC that persists across recreations — is mutable shared state. The unit of immutability is the container. The unit of deployment is the dependency graph. The platform offers the first and has no concept of the second.

Blue/green at pod level means two ReplicaSets behind a Service. At infrastructure level, it means two complete stack instances — each with its own CRDs, its own controllers, its own Helm releases, its own Flux Kustomizations. But both instances share the same CRDs (cluster-scoped, no namespacing), the same Gatekeeper policies, the same network policies, the same Flux dependency chain. You can’t switch traffic atomically because you can’t even enumerate what “the green stack” consists of. There’s no boundary to swap — just a collection of independently reconciling resources that happen to be labeled the same way.

Canary at pod level means routing 5% of traffic to new pods behind the same Service. At infrastructure level, it means running a new operator version against 5% of the clusters while the old version handles the rest. But both versions reconcile against the same CRDs, the same DynamoDB tables, the same NATS subjects, the same shared CRs. The data plane can be canary’d — Istio handles weighted routing. The control plane is all-or-nothing — two controller versions writing to the same CR will corrupt it, not canary it.

Self-healing at pod level means the kubelet restarts a crashed container. At infrastructure level, it means detecting that a CloudFront distribution is misconfigured and correcting it — which requires knowing the desired state of the distribution, the certificate, the DNS record, and the WAF association as a unit. The reconciliation loop heals one resource at a time. If the distribution is “healed” before the certificate is ready, the distribution serves traffic with an invalid cert. Self-healing without a dependency graph is self-damaging — each resource converges independently toward a state that may be inconsistent with its dependencies.

The pattern is always the same: the concept requires graph-level reasoning, the platform provides resource-level primitives, and the gap is filled with tribal knowledge and hope.

Every one of these concepts works at pod level because pods are the unit Kubernetes was designed around. The dependency graph for a pod is trivially simple: image → container → readiness probe → Service endpoint. Four nodes, linear, fully managed by the kubelet and kube-proxy. The moment you step outside that scope — the moment “the thing to deploy” is not a container but a constellation of interdependent cloud resources managed by independent controllers — every concept breaks, because every concept presupposes a graph that doesn’t exist.

This is the ceiling. Not a technical limitation that will be solved in the next Kubernetes release. A category error embedded in the platform’s computational model. Reconciliation loops converge individual resources toward individual desired states. Infrastructure deployment requires converging groups of resources toward consistent collective states, in dependency order, with transactional rollback on failure. These are fundamentally different computational problems. The first is a thermostat. The second is a workflow engine. Kubernetes is the thermostat. The industry keeps asking it to be the workflow engine, and then building accidental complexity layers — Flux dependsOn, Argo Workflows, Crossplane Compositions, operator-internal sequential reconcilers — to paper over the fact that it can’t.

The 2016 paper didn’t just identify a missing feature. It identified a missing computational primitive. Ten years later, the primitive is still missing. The layers are ten deep. And the community calls it “cloud-native.”

The One Mismatch This Series Is About

Step back from the individual incidents and there is a single sentence underneath all of them:

Kubernetes models, schedules, health-checks, and rolls back exactly one unit — the container. Every unit an enterprise application actually cares about — a capability, a contract, a domain boundary, a failure unit, a deployment unit — is a composition that spans containers, controllers, and cloud resources, and Kubernetes has no object for it.

That is the mismatch. Not “K8s has sharp edges.” Not “operators are hard.” One category error: the platform’s unit of representation is the container, and the enterprise’s unit of meaning never is. Everything else in this series is that one mismatch showing up on a different axis, discovered the same way every time — not from theory, but from inside, by people using the platform in earnest and hitting the seam where the thing they had to keep correct had no representation in the thing they were operating.

The evidence is deliberately first-person-plural. Each article is a real incident or a real design argument a team lived through: six near-identical CRDs nobody meant to write; a CI dashboard green while the cluster was half-broken for 24 hours; a pod reporting ready for a week while the capability it backed was dead. None of it is an outsider’s critique of Kubernetes. All of it is the record of insiders discovering, one axis at a time, that the unit they were handed is not the unit they needed.

Here is the same mismatch, per axis:

Axis How the container-vs-capability mismatch shows up The article
Modeling A domain is one thing; K8s can only slice it by CRD type → one domain becomes six repos of near-identical CRDs. Part 1
Abstraction Shared repos and even CDK don’t fix it, because the tool’s unit of grouping isn’t the domain’s unit of cohesion. Parts 2–3
Dependency A capability spans five repos; K8s has no DAG and the blast radius is always the whole cluster. Part 4
Verification / rollback The failure unit is a capability, not a container, so canary and rollback primitives that only reach a container can’t see it fail — pod green, rate limiting silently off. Part 5
Consistency The CR is shared mutable state; multiple controllers overwrite each other because ownership was never a container-level concept. Part 6
Computational model A container self-healing loop (reconciliation) is pressed into being a deployment engine it has no primitive for. This article
Cognition Long enough inside the container-runtime boundary and it becomes the boundary of what you can imagine — “that’s not how K8s works.” Stockholm · Auto-Approve · Front-run

The Staging Mindset piece closes the last escape hatch. The usual defense of Kubernetes concedes the infrastructure cases but retreats to “fine — for running a container, it’s exactly right.” The Redis incident there is a lone container, textbook-shaped, every pod-level signal green — and the enterprise capability it served was dead in production the whole time. If the mismatch bites even the most container-native workload imaginable, the safe harbor was never real: an enterprise application’s failure unit is essentially never a container, so the platform whose only unit is the container is measuring the wrong thing at every layer, not just the infrastructure layer.

What This Series Documents

Each article examines one axis of that single mismatch — the operator mindset applied to problems that need an architect’s mindset, using tools whose only unit is the container, producing systems whose complexity grows faster than the team’s ability to manage it:

  • Part 1: The Operator Mindset — Why one domain becomes six repositories. The repo-per-problem anti-pattern as a consequence of thinking in procedures instead of models.

  • Part 2: The Cargo Cult — Why shared repos and better tools don’t fix it. The failed abstraction phase.

  • Part 3: The Abstraction Instinct — What no tool can provide. CDK in the hands of an operator is still operator thinking.

  • Part 4: The Distributed Monolith — Why your GitOps is a monolith wearing a microservices costume. Five repos, five teams, zero transactional boundary, and six incidents in four weeks.

  • Part 5: The Staging Mindset — Routing is atomic. Deployment is not. Why feature flags are what happens when the infrastructure can’t express version coexistence.

  • Part 6: The Shared Mutable State — The CR is a database table with no foreign keys, shared between controllers with no ownership model. Silent data loss as a design consequence.

  • Part 7: Topology First — Why ops cuts operations in the wrong place, then pays a backlog to rejoin them. Placement drawn before the boundary, and the free isolation it destroys.

  • Part 8: The Slicing Machine — Why the horizontal cuts become departments, and departments become toll booths. Conway’s law applied to aspects, and the equity-not-equality reason discretion is never surrendered.

  • Aside: Operator Stockholm Syndrome — When the K8s control plane becomes the universe. Routing every cloud API through a cluster CR even when the cluster has no semantic role.

  • Aside: The Cron and the Gate — When the operator models itself instead of the domain. One Reconcile() hook, triggered identically by create/resync/requeue, becomes the only place policy can live.

  • Aside: The Configuration Problem — One business rule sliced across Helm, ConfigMap, Flux substitution, and Calico’s dataplane — zero cohesion, load-bearing tribal knowledge.

  • Aside: The Auto-Approve — When the reconcile loop swallows terraform plan. Wrapping a tool with a human-in-the-loop gate in a loop that structurally can’t hold one.

  • Aside: You Can’t Front-Run the Composition Gap — Why correct first-principles reasoning must crash once before it can diagnose.

  • Lab: Verify It Yourself — Copy-pasteable, real-output reproductions of every cluster mechanism the series cites (foreign keys, CEL scope, ownerRefs, SSA, PUT-strips-fields, resourceVersion, CRD versioning, kstatus).

  • Synthesis: The Thermostat That Ate Infrastructure — How a container self-healing pattern became a deployment engine. The missing DAG from node boot to infrastructure blue/green. (this article)

The root cause is the same in every article, on every axis: the only unit Kubernetes can represent is the container, and no enterprise application’s capability, contract, or failure ever fits inside one. A container self-healing mechanism is being used to model, deploy, verify, and roll back things that are not containers — by an industry that was trained to operate it, not to question whether the unit it hands you is the unit you need.

The K8s founders asked the right question in 2016. The industry answered by not asking it again.

“We still hope that Kubernetes might be a platform on which such tools can be built, but doing so remains an open challenge.”

— Burns, Grant, Oppenheimer, Hockin, Wilkes. “Borg, Omega, and Kubernetes.” ACM Queue, Vol. 14, No. 1, January-February 2016.

Cite This Article

Gary Yang. "The Thermostat That Ate Infrastructure: How a Container Self-Healing Pattern Became a Deployment Engine." ONDEMANDENV.dev, July 10, 2026. https://ondemandenv.dev/articles/k8s-thermostat-not-a-deployment-engine/.

📝
Source History
🤖
Analyze with AI