Skip to main content
Version: main 🚧

Resolve pods stuck at Ready=False with control plane and tenant version skew

When the tenant clusterTenant clusterA fully isolated Kubernetes environment provisioned for a single tenant. Each tenant cluster has its own API server, controller manager, and resource namespace, backed by a virtualized control plane hosted on a control plane cluster. From the tenant's perspective it behaves exactly like a standard Kubernetes cluster.Related: Control plane cluster, Tenant cluster runs Kubernetes 1.33 or earlier and the control plane clusterControl plane clusterThe Kubernetes cluster that hosts the virtualized control planes for tenant clusters. The control plane cluster is operated by the platform provider and is completely invisible to tenants. There are no shared control plane nodes, no in-cluster agent pods, and no lateral path between tenant environments. With shared nodes, this cluster also runs tenant workloads alongside the control plane pods — the same node pool is used for both.Related: Tenant cluster, Control plane cluster, Tenant cluster runs Kubernetes 1.34 or later, synced pods can get stuck reporting Ready=False even though the pod itself is running. Deployments and StatefulSets never show as ready, because they wait on that pod condition.

Symptoms​

A pod's containers are all Running, but its owning Deployment or StatefulSet never reaches the desired ready count:

Pod is Running, but the Deployment is not Ready
kubectl get pods
# NAME                          READY   STATUS    RESTARTS   AGE
# my-app-7cf4ddcbbf-k5m2n       1/1     Running   0          127m

kubectl get deployments
# NAME     READY   UP-TO-DATE   AVAILABLE   AGE
# my-app   0/1     1            0           127m

The syncerSyncerA component in vCluster that synchronizes resources between the tenant cluster and the control plane cluster, enabling tenant clusters to function while maintaining isolation.Related: vCluster, Tenant cluster logs a reconcile error for the pod, similar to:

Syncer log
ERROR   controller/controller.go:474    Reconciler error    {"component": "vcluster", "controller": "pod", ..., "error": "sync: patch host object: update object status: Pod \"my-app-7cf4ddcbbf-k5m2n--b21ec34921\" is invalid: status.qosClass: Invalid value: \"BestEffort\": field is immutable"}

Restarting the control plane's API serverAPI ServerThe core component of Kubernetes that exposes the Kubernetes API. It is the front-end for the Kubernetes control plane and handles all REST operations, validating and configuring data for API objects.Related: Control Plane, rate-limiting pod can temporarily clear some of these errors, but they come back after creating or updating deployments.

Cause​

This is a known issue tracked in vCluster GitHub Issue #3578, with two contributing root causes:

  • QoS class immutability: the pod syncer copies the tenant pod's QoS class onto the host object before patching. Kubernetes 1.32 and later rejects that patch because status.qosClass is immutable once set.
  • observedGeneration mismatch: starting in Kubernetes 1.34, the host kubelet sets observedGeneration on pod status and conditions. A tenant API server on Kubernetes older than 1.34 drops that field on write, since PodObservedGenerationTracking is alpha or off by default there. The syncer then sees the object cache and the informer disagree on every reconcile, reads that as a real condition change, and keeps overwriting the tenant's Ready=True condition with stale data.

Both root causes only trigger when the tenant cluster runs Kubernetes 1.33 or earlier and the control planeControl PlaneThe container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. In vCluster, each tenant cluster has its own control plane components.Related: API Server, vCluster cluster runs Kubernetes 1.34 or later. See the Kubernetes compatibility matrix for the specific version combinations flagged with this known issue.

Workaround​

There is no reliable workaround available today. Restarting the control plane's API server pod clears the immediate errors, but they recur as soon as new pods sync. The most effective mitigation is to keep the tenant cluster on Kubernetes 1.34 or later whenever the control plane cluster runs Kubernetes 1.34 or later, until the fix ships.

Fix status​

A fix is in progress in GitHub Issue #4037, which stops the syncer from writing the immutable QoS class to the host and ignores observedGeneration when the tenant cluster doesn't persist it.