Persisting vCluster data

vCluster uses k3s as Kubernetes distribution for the virtual control plane. The ability to run Kubernetes using a datastore other than etcd sets K3s apart from other Kubernetes distributions. This feature provides flexibility to vCluster operators. The available datastore options allow you to select a datastore that best fits your use case.

K3s supports the following datastore options:

• Embedded SQLite (default with Persistent Volume)
• PostgreSQL
• MySQL
• MariaDB
• etcd

For more information, please take a look at the k3s documentation directly.

Embedded SQLite without Persistent Volume

By default vCluster will deploy k3s to use a persistent volume claim to store the data in. You can also instead use an emptyDir to store the virtual cluster data.

In order to use an emptyDir to store the data instead of a persistent volume, please create a values.yaml with the following contents:

syncer:
  storage:
    persistence: false

Then upgrade or recreate the vCluster with:

vcluster create my-vcluster -n my-vcluster --upgrade -f values.yaml

Potential Data Loss

This method should only be used for testing purposes, as data will be lost upon pod recreation. If you are having problems with k3s in general, please consider using another Kubernetes distribution such as k0s or vanilla k8s

Datastore Options

If you want to use an external datastore such as PostgreSQL, MySQL, or etcd you must set the K3S_DATASTORE_ENDPOINT environment variable of the vCluster container so that K3s knows how to connect to it. You may also specify environment variables to configure the authentication and encryption of the connection. The following environment variables are available:

• K3S_DATASTORE_ENDPOINT: Specify a PostgreSQL, MySQL, or etcd connection string. This is a string used to describe the connection to the datastore. The structure of this string is specific to each backend and is detailed below.
• K3S_DATASTORE_CAFILE: TLS Certificate Authority (CA) file used to help secure communication with the datastore. If your datastore serves requests over TLS using a certificate signed by a custom certificate authority, you can specify that CA using this parameter so that the K3s client can properly verify the certificate.
• K3S_DATASTORE_CERTFILE: TLS certificate file used for client certificate based authentication to your datastore. To use this feature, your datastore must be configured to support client certificate based authentication. If you specify this parameter, you must also specify the K3S_DATASTORE_KEYFILE parameter.
• K3S_DATASTORE_KEYFILE: TLS key file used for client certificate based authentication to your datastore. See the previous K3S_DATASTORE_CERTFILE parameter for more details.

Configuration & Examples

As mentioned, the format of the value passed to the K3S_DATASTORE_ENDPOINT environment variable is dependent upon the datastore backend.

Etcd

In its most common form, the K3S_DATASTORE_ENDPOINT environment variable for etcd has the following format:

https://etcd-host-1:2379,https://etcd-host-2:2379,https://etcd-host-3:2379

The above assumes a typical three node etcd cluster. The environment variable can accept one more comma separated etcd URLs.

MySQL / MariaDB

In its most common form, the K3S_DATASTORE_ENDPOINT environment variable for MySQL and MariaDB has the following format:

mysql://username:password@tcp(hostname:3306)/database-name

More advanced configuration parameters are available. For more information on these, please see https://github.com/go-sql-driver/mysql#dsn-data-source-name

Note that due to a known issue in K3s, you cannot set the tls parameter. TLS communication is supported, but you cannot, for example, set this parameter to "skip-verify" to cause K3s to skip certificate verification.

If you specify a database name and it does not exist, the server will attempt to create it.

Postgresql

In its most common form, the K3S_DATASTORE_ENDPOINT environment variable for PostgreSQL has the following format:

postgres://username:password@hostname:port/database-name

More advanced configuration parameters are available. For more information on these, please see https://godoc.org/github.com/lib/pq.

If you specify a database name and it does not exist, the server will attempt to create it.

Postgres Example

The following example could be used to launch a vCluster that connects to a PostgresSQL database named k3s. Create a values.yaml with:

vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: postgres://username:password@hostname:5432/k3s

Create the vCluster with:

vcluster create my-vcluster -n my-vcluster -f values.yaml

MySQL Example

The following example could be used to connect the vCluster to a MySQL database using client certificate authentication. Create a values.yaml with:

vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: 'mysql://username:password@tcp(hostname:3306)/k3s'
  - name: K3S_DATASTORE_CERTFILE
    value: '/path/to/client.crt'
  - name: K3S_DATASTORE_KEYFILE
    value: '/path/to/client.key'
  volumeMounts:
    - mountPath: /data
      name: data
    - mountPath: /path/to
      name: datastore-tls
volumes:
- name: datastore-tls
  secret:
    secretName: my-datastore-secret
    items:
    - key: tls.key
      path: client.key
    - key: tls.crt
      path: client.crt

Create the vCluster with:

vcluster create my-vcluster -n my-vcluster -f values.yaml