Terraform
| Enterprise | ||||
|---|---|---|---|---|
| Available in these plans | Free | Dev | Prod | Scale |
| Terraform Node Provider | ||||
Free, Dev, Prod, and Scale are vCluster Platform license plans. Open source does not need a license or a Platform connection. See Compare open source and free tiers.
The terraform provider allows you to use Terraform scripts to automatically create nodes and the infrastructure behind network environments. When a cluster requests a new node, the platform runs the specified Terraform script to create the node and join it to the cluster. If the node or cluster is later deleted, the platform will destroy the node using Terraform.
We use OpenTofu to execute the terraform scripts. vCluster Platform v5.0 includes OpenTofu 1.13.2 in its image. The platform doesn't check the OpenTofu version that a template requires.
Tenant access​
Terraform templates, and the git commands that fetch them, run inside the platform pod with the platform's credentials. Only platform admins work with Terraform node providers:
- A tenant can't create or update a Terraform node provider.
- A tenant can't run Test Templates or check access to a template's git repository, in the platform UI or through the API. The API returns
Forbidden.
A Terraform node provider that belongs to a tenant, through the tenant.vcluster.com/owner label, can remain from an earlier version. The platform doesn't run Terraform for it:
- A
NodeClaimor network environment that uses the provider fails with the reasonTenantOwnedProvider. Nothing is provisioned. - Deleting a
NodeClaimor network environment that uses the provider skipstofu destroy. The platform logsSkipped Terraform destroy, anything the template created must be removed by hand. Remove those resources from the infrastructure yourself.
Platform v5.0 has no migration path for a tenant-owned Terraform node provider.
Node templates​
Node templates are terraform scripts either defined inside the node provider object directly or referenced from a Git repository.
We recommend to use git for storing your Terraform node and network environment templates. This way you have a complete audit trail of who has changed what and you can even create multiple node providers based off different branches, tags or commits.
Use git​
When you reference a Git repository, the platform clones it into a local cache and reuses that copy. Before each use, it fetches remote changes if the last fetch is older than fetchInterval, which defaults to 1 minute. The platform removes a cached repository after 24 hours without use, or when the cache holds more than 20 repositories, starting with the least recently used.
To change these cache defaults for every node provider, set the following environment variables through the env value of the platform Helm chart:
| Variable | Default | Description |
|---|---|---|
TERRAFORM_GIT_CACHE_FETCH_INTERVAL | 1m | Minimum time between fetches of a cached repository, as a Go duration. A node template's fetchInterval overrides it. |
TERRAFORM_GIT_CACHE_TTL | 24 | Hours a repository stays cached without use. Takes a whole number. |
TERRAFORM_GIT_CACHE_REPOS | 20 | Maximum number of cached repositories. |
TERRAFORM_GIT_CACHE_CLONE_TIMEOUT | 10m | Maximum time for a clone or fetch, as a Go duration. |
env:
TERRAFORM_GIT_CACHE_TTL: "48"
TERRAFORM_GIT_CACHE_REPOS: "50"
The following example specifies how to use a repository at https://github.com/my-org/my-user.git to create a node:
apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: terraform-example
spec:
terraform:
nodeTemplate:
git:
repository: https://github.com/my-org/my-repo.git
Besides repository you can use the following options for git:
branch: branch to checkoutcommit: commit sha256 to checkoutsubPath: the path within the repositoryusername.secretName,username.secretNamespaceandusername.keyto specify the username for a private git repositorypassword.secretName,password.secretNamespaceandpassword.keyto specify the password (or token) for a private git repositoryfetchInterval: specifies the interval to fetch the repository. Refetching only checks for remote changes but does not do a complete repull
Use inline​
As an alternative to git you can also specify the template inline in the node provider object.
The following example specifies a very simple inline template to create an AWS ec2 instance for a vCluster node:
apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: terraform-example
spec:
terraform:
nodeTemplate:
inline: |-
terraform {
required_version = ">= 1.5"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
}
}
provider "aws" {
region = var.vcluster.requirements["region"]
}
resource "aws_instance" "node" {
ami = "my-ami"
instance_type = "my-instance-type"
subnet_id = "my-subnet-id"
vpc_security_group_ids = ["my-security-group-id"]
user_data = var.vcluster.userData
}
Available terraform vCluster variables​
Besides the user-defined credentials specified below, the platform passes a var.vcluster object to each node script. The following values are available:
var.vcluster.name: Name of the vCluster instancevar.vcluster.namespace: Project namespace of the vCluster instancevar.vcluster.userData: The cloud-init script generated that should be passed to the instance to join the node into the vClustervar.vcluster.requirements: A key-value map of the specified and resolved requirements of the node claimvar.vcluster.instance: Full cluster instance objectvar.vcluster.nodeClaim: Full node claim objectvar.vcluster.nodeType: Full node type objectvar.vcluster.properties: Merged properties from the NodeProvider, NodeType, NetworkEnvironment, and NodeClaimvar.vcluster.networkEnvironment.outputs: If the node claim resolves a network environment, the outputs of its infrastructure template
Node types​
Node types are used by vCluster and the platform to decide what exact machine type should get provisioned.
Each node type can define their own node template or reuse the central node type under spec.nodeType.
Properties of node types will be passed via the var.vcluster.requirements field to the Terraform script.
Use properties of node types to define custom terraform variables which can be used for info like instance type or region.
An example could look like this:
apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: terraform-example
spec:
terraform:
nodeTemplate:
git:
repository: https://github.com/my-org/my-repo.git
subPath: generic
nodeTypes:
- name: medium
resources:
cpu: "2"
memory: "4Gi"
properties:
instance-type: medium # can be accessed via 'var.vcluster.requirements["instance-type"]' inside terraform
- name: large
resources:
cpu: "4"
memory: "8Gi"
properties:
region: "*" # allow all values for region
cpu-generation: "old,new" # allow old or new as cpu-generation
nodeTemplate: # override the spec.terraform.nodeTemplate
git:
repository: https://github.com/my-org/my-repo.git
subPath: large
Max capacity​
The node type field maxCapacity allows you to specify an upper limit of how many nodes should be created for the node type.
Network environment template​
A network environment template provisions shared infrastructure, such as a VPC, for a network environment. Configure it inline or using git, the same way as node templates. When the template is set, every node claim from this provider needs a network environment.
The platform provisions the infrastructure when you create a NetworkEnvironment that references the provider, not when the first node claim arrives. It destroys the infrastructure when you delete the environment. Deletion waits while any node claim still uses the environment.
The following example specifies a simple inline template that creates an AWS VPC:
apiVersion: management.loft.sh/v1
kind: NodeProvider
metadata:
name: terraform-example
spec:
terraform:
networkEnvironmentTemplate:
infrastructure:
inline: |-
terraform {
required_version = ">= 1.5"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.0"
}
}
}
provider "aws" {
region = var.vcluster.properties["region"]
}
resource "aws_vpc" "network" {
cidr_block = "10.0.0.0/16"
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "${var.vcluster.name}-vpc"
}
}
output "vpc_id" {
value = aws_vpc.network.id
}
nodeTemplate:
inline: |-
...
Then create a network environment that references the provider. To lend it to a Tenant, add the tenant.vcluster.com/exclusive-to: <tenant> label. Without the label, it serves platform workloads:
apiVersion: management.loft.sh/v1
kind: NetworkEnvironment
metadata:
name: aws-network
spec:
providerRef: terraform-example
properties:
region: us-east-1
Node claims select it through the vcluster.com/network-environment property, or use the provider's default environment.
The node template reads the vpc_id output as var.vcluster.networkEnvironment.outputs["vpc_id"].
Earlier releases used spec.terraform.nodeEnvironmentTemplate, a separate kubernetes template, and the var.vcluster.nodeEnvironment variable.
The field is now networkEnvironmentTemplate, and the variable is now var.vcluster.networkEnvironment. Node templates can still read var.vcluster.nodeEnvironment as a deprecated alias, but update them to the new name.
The kubernetes template is no longer supported.
Available network environment variables​
Besides the user-defined credentials specified below, the platform passes a var.vcluster object to each network environment script. The following values are available:
var.vcluster.name: Name of the network environmentvar.vcluster.namespace: Namespace of the platform, because network environments are cluster scopedvar.vcluster.properties: Merged properties from the NodeProvider and the network environmentvar.vcluster.nodeProvider: Full node provider objectvar.vcluster.networkEnvironment: Full network environment object
Credentials​
Credentials to the Terraform scripts can be defined using Kubernetes secrets. The key and values of these secrets can then be passed as either environment variables or regular variables to the Terraform commands.
Within the vCluster you can select the credentials using the terraform.vcluster.com/credentials property. This allows you to specify different credentials for different teams or users and select the dynmically.
If possible we recommend to use dynamic credentials for providers instead of long-living hardcoded credentials using Kubernetes secrets. Almost all Terraform providers support this and marks the additional configuration using Kubernetes secrets obsolete.
Environment variables secret​
You can create a secret in the following form:
apiVersion: v1
kind: Secret
metadata:
name: aws-credentials
namespace: any-namespace
labels:
terraform.vcluster.com/provider: my-node-provider
data:
# Base64 encoded value
AWS_ACCESS_KEY_ID: ...
# Base64
AWS_SECRET_ACCESS_KEY: ...
This will lead to the platform passing the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to the tofu init and tofu apply commands.
Terraform variables secret​
You can create a secret in the following form:
apiVersion: v1
kind: Secret
metadata:
name: aws-credentials
namespace: any-namespace
annotations:
terraform.vcluster.com/type: vars
labels:
terraform.vcluster.com/provider: my-node-provider
data:
# Base64 encoded value
aws_access_key_id: ...
# Base64
aws_secret_access_key: ...
This will lead to the platform passing the Terraform variables aws_access_key_id and aws_secret_access_key to the tofu init and tofu apply commands using -var aws_access_key_id=VALUE.
Terraform backend​
By default, the platform will use the Kubernetes backend to store the state of the nodes and network environments in Kubernetes secrets in the vcluster-platform namespace.
The platform will create a separate workspace per node and network environment in the following form:
- Node:
NODE_CLAIM_NAME.PROJECT_NAME.node - Network environment:
NETWORK_ENVIRONMENT_NAME.env
Use a custom backend​
The platform will parse the Terraform script files before initializing Terraform and search for backend blocks. If none is found, it will use the above described default backend.
If it finds a user-defined backend it will not use the default backend and instead try to use the specified backend.