Skip to main content
Version: main 🚧

NVIDIA Dynamo

NVIDIA Dynamo is an open source framework for serving AI models across GPU nodes. Its operator runs the models you describe with DynamoGraphDeployment resources.

vCluster Platform v4.12.5 and later bundle the nvidia-dynamo StackTemplate. It installs the Dynamo runtime into one cluster or control plane cluster, so that destination can serve models.

Understand the architecture​

The nvidia-dynamo StackTemplate has one App task, runtime. It installs the dynamo-platform Helm chart from the NVIDIA NGC ai-dynamo repository into the destination you select.

The release contains:

  • the Dynamo operator, which runs cluster-wide with validating and mutating admission webhooks
  • NATS, from the chart's bundled subchart
  • etcd, from the chart's bundled subchart

The runtime owns the whole Dynamo stack in its destination, so it installs NATS and etcd instead of connecting to external instances. The chart's other optional components, such as Grove, KAI Scheduler, and Snapshot, stay disabled.

Choose a destination​

Install the runtime at most once per destination. The operator, its CustomResourceDefinitions (CRDs), and its webhooks are cluster-scoped. The chart refuses to install when the destination already runs a cluster-wide Dynamo operator.

DestinationWhat it gives youUse case
ClusterA Dynamo runtime inside one cluster's API, with its own operator, NATS, and etcdEach tenant runs and manages its own models
Control plane clusterOne Dynamo runtime on the control plane clusterYour team runs the models and exposes them to customers through an API

If you choose the cluster destination so tenants can deploy their own models, those tenants have Kubernetes access and run their own workloads. The worker node model then decides how isolated they are.

Shared nodes are for trusted tenants

Shared nodes are a supported, common model for trusted tenants. They give each tenant control-plane, API, and namespace isolation, but tenant workloads share the same kernel and physical nodes. They aren't a security boundary for untrusted tenants with Kubernetes access or arbitrary workload execution. Good fits are development, testing, CI/CD, and internal engineering teams.

Use private nodes for external, resale, regulated, or otherwise untrusted tenant offerings, optionally with vNode for runtime isolation.

NetworkPolicy is an added isolation layer worth enabling even for trusted tenants. vCluster can create the policies for you through policies.networkPolicy, and your control plane cluster's CNI enforces them. Confirm your CNI supports enforcement, since some accept NetworkPolicy resources without acting on them. See the security baseline.

If you accept shared nodes for a trusted-tenant use case, follow the shared-node hardening guide as the required next step.

Before you begin​

Make sure that you have:

  • vCluster Platform v4.12.5 or later. The template uses App tasks, so it requires the Apps feature, which is included on every plan.
  • Kubernetes 1.30 or later in the destination. This is the chart's kubeVersion constraint.
  • A default StorageClass in the destination. NATS requests a 10Gi volume and etcd requests a 1Gi volume.
  • Outbound access from vCluster Platform to helm.ngc.nvidia.com. Platform runs the Helm install itself, so it downloads the chart.
  • Outbound access to nvcr.io and Docker Hub from the nodes that run the destination's pods. Those nodes pull the operator, NATS, and etcd images.
  • GPU nodes with the NVIDIA device plugin for the models you deploy later. The runtime itself doesn't need GPUs.

Review parameters​

ParameterDefaultDescription
version1.6.2Version of the dynamo-platform Helm chart. Must be a semantic version, such as 1.6.2.
namespacedynamo-systemNamespace the runtime is installed into. Must be a valid Kubernetes namespace name.

Defaults are pinned to each platform release. Check the bundled StackTemplate if you use a different release.

The template doesn't expose other Helm values. To change the chart configuration, see Customize the template.

Install the runtime​

To install from the Platform UI, follow Use a Stack and select the NVIDIA Dynamo template. To install the runtime in every cluster created from a template, add a deploy.stacks entry as described on the same page.

To install through the management API, create a StackInstance in the project namespace that owns the destination. This example installs the runtime into a cluster:

Modify the following with your specific values to generate a copyable command:
nvidia-dynamo.yaml
apiVersion: management.loft.sh/v1
kind: StackInstance
metadata:
name: nvidia-dynamo
spec:
displayName: NVIDIA Dynamo
owner:
user: admin
destination:
virtualCluster:
name: my-cluster
templateRef:
name: nvidia-dynamo
parameters:
version: "1.6.2"
namespace: dynamo-system
prunePolicy: Prune

To install into a control plane cluster instead, replace the virtualCluster block with cluster, and set name to the control plane cluster's name.

Apply the StackInstance:

Modify the following with your specific values to generate a copyable command:
vcluster platform connect management
kubectl apply -n p-default -f nvidia-dynamo.yaml

Verify the installation​

Helm waits up to 20 minutes for the release to become ready, which leaves room for slow image pulls. If the release isn't ready by then, the runtime task fails with Helm's timeout error. The task's own 25 minute timeout is longer, so you see Helm's error instead of a generic task timeout.

Watch the StackInstance until both the StackInstance phase and the runtime task report Healthy:

Modify the following with your specific values to generate a copyable command:
kubectl get stackinstance nvidia-dynamo -n p-default \
-o jsonpath='{.status.phase}{"\n"}{range .status.tasks[*]}{.name}{"\t"}{.phase}{"\t"}{.message}{"\n"}{end}'

Then connect to the destination. For a cluster, use the vCluster CLI. For a control plane cluster, use its own kubeconfig.

Modify the following with your specific values to generate a copyable command:
vcluster platform connect vcluster my-cluster --project default

Check that the operator, NATS, and etcd pods are Running, and that the Dynamo CRD exists:

kubectl get pods -n dynamo-system
kubectl get crd dynamographdeployments.nvidia.com

After the runtime is healthy, deploy a model with a DynamoGraphDeployment. See the NVIDIA Dynamo documentation for model recipes and backends.

Upgrade the runtime​

Change the version parameter on the StackInstance to move to another chart release. The operator applies its CRDs from its image at startup, so CRD changes roll out with the new operator version. Read the Dynamo release notes before you upgrade across minor versions.

Remove the runtime​

Delete every DynamoGraphDeployment in the destination first. The operator manages finalizers on its resources, so removing the operator first can leave those resources stuck in deletion.

Then delete the StackInstance. Platform deletes the AppInstance it created for the runtime task, which uninstalls the Helm release. The prunePolicy setting applies when a template update removes a task, not when you delete the StackInstance.

These resources remain after removal:

  • the Dynamo CRDs, because the operator applies them and the Helm release doesn't own them
  • the NATS and etcd PersistentVolumeClaims created from the StatefulSet volume claim templates

Delete them manually if you don't plan to reinstall the runtime.

Customize the template​

The bundled template is read-only in the platform UI and runs NATS and etcd as single, bundled instances. To expose more Helm values or point the operator at external NATS or etcd, copy the template. See Customize a certified stack.

Troubleshoot​

SymptomCauseResolution
The runtime task fails, and the Helm error includes Only one cluster-wide Dynamo operator should be deployed per clusterThe destination already runs a Dynamo operator, from another StackInstance or a manual install.Use the existing runtime, or remove it before you install this one.
The runtime task fails after about 20 minutes with a Helm timeout errorImages didn't pull, or a NATS or etcd volume didn't bind.Check image access to nvcr.io and Docker Hub, and confirm that the destination has a default StorageClass. Platform retries the install after 1, 5, and then 15 minutes, and the task turns Healthy if a retry succeeds.
A DynamoGraphDeployment stays in deletionThe operator was removed before the resource.Reinstall the runtime. The operator processes the resource's finalizer and completes the deletion. Then remove the runtime again.

For general Stack failures, see Troubleshoot Stacks.

Next steps​