Resolve embedded etcd defragmentation timeouts on startup
Symptoms​
The control plane doesn't start. Its logs contain this line:
Etcd defragmentation did not finish in time, increase controlPlane.backingStore.etcd.embedded.defragTimeout
The startup error that follows contains defragment: and a deadline error from etcd.
Cause​
Embedded etcd defragments its database each time it starts, when at least 16 MiB and 25% of the database are reclaimable. The control plane waits for defragmentation to finish before it serves requests. If it takes longer than controlPlane.backingStore.etcd.embedded.defragTimeout, which defaults to 5m, the control plane fails to start.
This happens with large databases, slow storage, or both. It applies only to clusters that use embedded etcd.
Fix​
Raise the timeout, or skip defragmentation on startup. The control plane reads the setting at startup, so applying the change restarts it.
Set a longer timeout in your
vcluster.yaml. To skip defragmentation instead, setdefragTimeout: 0s.vcluster.yamlcontrolPlane:backingStore:etcd:embedded:enabled: truedefragTimeout: 15mApply the change with the same method you used to deploy the cluster, such as the CLI or a Helm upgrade.
If you skip defragmentation, the database file doesn't shrink after compaction. Watch its size, because a database that reaches its quota raises the NOSPACE alarm.
Verify​
The control plane pods reach Running, and the logs show one of these lines instead of the timeout:
Defragmenting etcd database
Skipping etcd defragmentation, not enough reclaimable space
Skipping etcd defragmentation, disabled via controlPlane.backingStore.etcd.embedded.defragTimeout