What Is a Neocloud and How to Build One: GPUs, Fabrics, Tenancy

A neocloud is a cloud provider built mainly to rent out GPU and accelerator compute for AI training and inference, rather than to offer hundreds of general services. To build one you need dense GPU servers, a low latency RDMA fabric such as InfiniBand or RoCE, fast shared storage, a scheduler (Kubernetes, Slurm or both), tenant isolation, metering and billing, and enough contracted power to run it all. The hard parts are usually power, isolation between tenants who do not trust each other, and keeping a deep software stack patched.

"Neocloud" is industry shorthand, not a term defined by a standards body. There is no NIST or ISO definition, and the companies that get the label vary a lot in size and design. This post does not name or rank any of them. It explains the idea from first principles, walks through each building block with the official documentation for the open source and vendor components involved, gives you real configuration for a small multi tenant GPU cluster, and ends with what tenants should expect on security and compliance.

Key takeaways

• A neocloud is defined by its product, not its size. It sells GPU time (whole servers, whole GPUs or GPU slices) plus the networking and storage that make that time useful.
• The network is half the machine. Distributed training moves gradients between GPUs constantly, so the scale out fabric (InfiniBand, RoCE or Ultra Ethernet) is a first class design decision.
• Power is the binding constraint. A single 8 GPU server of the class NVIDIA documents for its DGX H100 draws up to 10.2 kW. LBNL estimates US data centers used about 4.4 percent of US electricity in 2023 and could reach 6.7 to 12 percent by 2028.
• Scheduling has two traditions. Slurm comes from HPC and is built around batch jobs, partitions and accounting. Kubernetes comes from cloud native services and reaches GPUs through device plugins and Dynamic Resource Allocation, with Kueue adding quotas and fair sharing for batch work.
• GPU sharing is not the same as GPU isolation. NVIDIA's own documentation says time slicing gives no memory or fault isolation between replicas, while MIG partitions a supported GPU into up to seven instances with separate memory paths.
• Containers alone are a weak boundary between strangers. The Kubernetes multi tenancy guide recommends sandboxing (virtual machines or userspace kernels) when you run untrusted code, plus node isolation and network policy.
• The software stack is long and moves fast. Drivers, container toolkits, operators, firmware, schedulers, images and model servers all ship security fixes, and some, like the container toolkit, sit right on the tenant boundary.
• Billing is a metering problem first. Slurm's TRES accounting and Kubernetes quota objects give you the raw numbers; the commercial layer sits on top.

Milestones that shaped the neocloud stack

There is no single founding date for neoclouds. What changed is that a set of open standards and open source components matured to the point where a smaller team could assemble a GPU cloud from parts. The table lists the dated or versioned milestones we could confirm on primary sources.

What exactly is a neocloud?

Strip away the marketing and a neocloud is an infrastructure as a service provider whose main product is accelerated compute. NIST SP 800-145 describes infrastructure as a service as the capability to provision processing, storage, networks and other fundamental computing resources on which the customer runs arbitrary software. A neocloud does exactly that, but the "processing" is mostly GPUs and the customers are mostly training or serving machine learning models.

A few traits show up again and again in how the term is used:

• Narrow catalog. The core offer is GPU servers, GPU clusters, or GPU containers, often with managed Kubernetes or Slurm on top. There may be object storage and a few managed inference endpoints, but not hundreds of services.
• Fabric included. For training, the provider sells a cluster connected by a high bandwidth RDMA fabric, not just a set of individual machines. That fabric is part of the product.
• Capacity contracts. Many customers reserve clusters for months or years, so a neocloud's revenue looks more like a lease book than pure pay as you go usage.
• Bare metal or thin virtualization. Tenants often want direct access to the GPUs and the RDMA network cards for performance. That shapes the isolation model, which we cover below. Our earlier post on bare metal versus virtualized GPU cloud looks at the evidence on that trade off.

Because the word is informal, contracts and procurement documents should describe the service you need (GPU model, fabric, storage, isolation, support) rather than rely on the label.

How is a neocloud different from a hyperscaler?

The differences are about emphasis, not a hard line. Large general purpose clouds also rent GPUs, and some neoclouds are large. The table describes typical tendencies, not rules, and it does not describe any particular company.

For a buyer, a neocloud can be simpler and more GPU focused, but you may bring more of the surrounding platform yourself: identity, logging, data pipelines and policy.

What are the building blocks of a neocloud?

Think of a neocloud as seven layers. Each one has its own vendor documentation, its own update cycle and its own failure modes.

The rest of this post goes through them in order.

What compute do you need?

The compute unit in most GPU clouds is a multi GPU server. NVIDIA's DGX H100 user guide is a useful public reference because it documents the whole box: 8 H100 GPUs with 640 GB of total GPU memory (or 8 H200 GPUs with 1,128 GB), fourth generation NVLink providing 900 GB/s of GPU to GPU bandwidth inside the server, 2 TB of system memory, 8 single port ConnectX-7 cards for the InfiniBand compute fabric at up to 400 Gb/s each, plus 2 dual port ConnectX-7 cards for Ethernet. It is an 8U chassis weighing up to 130.45 kg, with six 3.3 kW power supplies in a 4+2 redundant arrangement and a maximum system power of 10.2 kW.

Those numbers matter even if you buy elsewhere. One fabric NIC per GPU is a common training pattern, four such servers put more than half a tonne in one rack, and power per server drives the whole facility design.

AMD Instinct accelerators are the other widely used datacenter GPU family. AMD's GPU Operator documentation lists support for Instinct MI355X, MI350X, MI325X, MI300X, MI250 and MI210, among others, on Ubuntu 22.04 and 24.04, Debian 12, SUSE and Red Hat CoreOS, with Kubernetes 1.29 to 1.36 validated at the time of writing. We do not rank accelerators. The right choice depends on the software your tenants run, the frameworks they have tuned, and what you can actually get delivered and powered.

Every server also carries a baseboard management controller with its own firmware and network port. BMCs belong on an isolated management network and need their own patch schedule, separate from the host operating system and the GPU drivers.

Which network fabric: InfiniBand, RoCE or Ultra Ethernet?

Training a large model splits the work across many GPUs. After each step they exchange gradients through collective operations such as AllReduce. NVIDIA's NCCL library, which most training frameworks use for this, describes itself as providing topology aware inter GPU communication over PCIe, NVLink, InfiniBand Verbs and IP sockets. If the fabric is slow or congested, expensive GPUs sit idle waiting for the slowest flow.

There are three main options for the scale out network. Our post on scale up versus scale out AI networks goes deeper on NVLink and other scale up links inside a server or rack.

InfiniBand is a switched fabric architecture defined by the InfiniBand Trade Association. The IBTA describes reliable messaging and RDMA "without software intervention in the data movement path", and cites measured end to end latency of 600 ns. Its site says NDR 400 Gb/s products are shipping, its FAQ mentions link speeds as high as 800 Gb/s, and its roadmap projects GDR 1.6 Tb/s products around 2028. It brings its own subnet management and partitioning model.

RoCE (RDMA over Converged Ethernet) is also an IBTA standard. The IBTA describes it as a transport that lets RDMA run on ordinary Ethernet layer 2 and 3 networks, and says the first RoCE specification was released in 2010, defined within the overall InfiniBand Architecture specification. RoCE lets you use Ethernet switching and operations skills, but lossless or near lossless behavior and congestion control need careful tuning.

Ultra Ethernet is the newest option. The Ultra Ethernet Consortium, a Joint Development Foundation project under the Linux Foundation, released Specification 1.0 on 11 June 2025. The consortium describes it as a full communications stack for AI and HPC across NICs, switches, optics and cables, including what it calls modern RDMA for Ethernet and IP, with attention to multipathing, congestion control and tail latency. Its site now offers a 1.1 specification for download. Because the specification is young, check what your NIC and switch vendors actually implement before you design around it.

Whichever you pick, keep three networks separate: the GPU compute fabric, a storage and front end network, and an out of band management network for BMCs, switches and PDUs. Mixing management traffic into a tenant reachable network is one of the most common and most avoidable design mistakes. For a broader look at the optics and transceivers involved, see AI data center networking, optics and transceivers.

What storage does a GPU cloud need?

Storage in a GPU cloud has three jobs, and they pull in different directions:

1. Feed training data fast enough. Data loaders read many small files or large shards in parallel. Slow reads leave GPUs idle.
2. Absorb checkpoints. Long training runs write model and optimizer state regularly so a failure does not lose days of work. Checkpoints are large, bursty writes from many nodes at once.
3. Hold datasets and model artifacts durably. That is usually object storage, which is cheaper per terabyte but slower.

A common answer is tiers: local NVMe as scratch and cache (the DGX H100 has eight 3.84 TB NVMe drives in RAID 0), a shared parallel file system for active work, and object storage for retention.

From a tenancy angle, storage is where isolation mistakes hide. The Kubernetes multi tenancy guide recommends dynamic volume provisioning and avoiding volume types that use node resources, and suggests a separate StorageClass per tenant to strengthen isolation. Our post on object storage keys and tenant isolation makes the related point that a bucket naming convention is not an access control. Also plan for wiping local NVMe between tenants, because scratch disks remember.

Kubernetes or Slurm: how do you schedule GPUs?

Most neoclouds offer one or both. They come from different traditions and solve slightly different problems.

Slurm

The Slurm documentation describes it as an open source, fault tolerant and highly scalable cluster management and job scheduling system. A central manager, slurmctld, tracks resources and work, each compute node runs a slurmd daemon, and an optional slurmdbd records accounting for one or more clusters. GPUs are managed as generic resources (GRES): you declare them in slurm.conf with GresTypes and Gres, and describe each node's devices in gres.conf, or let Slurm detect them with AutoDetect=nvml, AutoDetect=nvidia, AutoDetect=rsmi and other options. Slurm notes that AutoDetect=nvidia, added in 24.11, does not need the NVML library but does not detect MIG instances or NVLinks. Jobs only get GPUs when they ask for them, and Slurm can confine a job to its allocated GPUs using Linux cgroups.

Slurm suits tenants who already run HPC style batch training and expect sbatch and srun. Its weak spot in a commercial multi tenant setting is that it was built for one organization's users sharing one cluster. Login nodes, shared file systems and the MUNGE authentication key all assume a degree of mutual trust. Our posts on Slurm security basics and who patches a shared HPC cluster cover those assumptions. Many GPU clouds therefore give each tenant its own Slurm cluster on dedicated nodes rather than sharing one Slurm instance between unrelated customers.

Kubernetes

Kubernetes reaches GPUs through device plugins. The Kubernetes documentation says the administrator installs the vendor's GPU drivers on the nodes and runs the vendor's device plugin, after which the cluster exposes a schedulable resource such as nvidia.com/gpu or amd.com/gpu. GPUs are requested in the limits section; if you also set requests they must be equal. Node labels, and Node Feature Discovery for automatic labelling, steer pods to the right GPU type.

Two newer pieces matter for a neocloud:

• Dynamic Resource Allocation (DRA) lets pods request devices through ResourceClaims and DeviceClasses, with drivers attaching devices through the Container Device Interface. The Kubernetes documentation marks DRA stable since v1.35, first available in v1.30. AMD's GPU Operator lists DRA support as an alternative to its device plugin.
• Kueue is a Kubernetes native system that manages quotas and decides when a job waits, starts or is preempted. Its documentation lists fair sharing among tenants, cohorts, preemption policies, all or nothing admission, topology aware scheduling and quota for DRA devices.

The NVIDIA GPU Operator bundles the node software a GPU cluster needs: drivers, the Kubernetes device plugin, the NVIDIA Container Toolkit, GPU Feature Discovery labelling and DCGM based monitoring. The AMD GPU Operator does the same job for AMD Instinct and Radeon GPUs, including driver management, its device plugin, node labelling and metrics.

Which to choose

How do you isolate tenants who do not trust each other?

This is the part that separates a hobby GPU cluster from a cloud. The Kubernetes multi tenancy guide is blunt that there is no single definition of "hard" and "soft" multi tenancy and that isolation is a spectrum. It frames the problem in two planes.

Control plane isolation stops tenants from seeing or changing each other's API objects. The tools are namespaces, RBAC scoped to those namespaces, and quotas. The guide calls authorization the most important control plane isolation and recommends resource quotas to stop one tenant from monopolising the cluster.

Data plane isolation stops tenants' workloads from reaching each other. The guide lists network policy (by default all pods can talk to all pods, and NetworkPolicy objects are ignored unless your network plugin supports them), storage isolation, sandboxing, and node isolation. It notes that containers use operating system level virtualization and so offer a weaker boundary than hardware virtualization, and recommends sandboxing when running untrusted code, with virtual machines and userspace kernels as the two common approaches.

For GPU clouds there are extra layers.

GPU sharing versus GPU isolation

NVIDIA's GPU Operator documentation states plainly that with time slicing "there is no memory or fault-isolation between replicas". That is fine inside one team. It is not a boundary between two paying customers. MIG, by contrast, gives each instance separate and isolated paths through the memory system, which NVIDIA says provides defined quality of service with fault isolation between clients such as VMs, containers or processes.

Sandboxed runtimes and dedicated nodes

Kubernetes RuntimeClass, stable since v1.20, lets a pod select a different container runtime configuration. That is how you route untrusted tenant pods to a sandboxed runtime. Kata Containers, an open source runtime launched in December 2017, runs each pod in a lightweight virtual machine with its own kernel and uses hardware virtualization as a second layer of defense. Passing GPUs and RDMA NICs through to such sandboxes needs support from your hardware, drivers and runtime, so test the exact combination before you promise it.

Many GPU clouds take the simpler route the Kubernetes guide also describes: node isolation. Each tenant gets whole servers, enforced with taints, tolerations and node selectors, so no two tenants share a kernel, a GPU or a local disk. Add fabric partitioning and separate storage and you have the usual bare metal answer to "how are customers kept apart". Our Kubernetes multi tenancy decision guide walks through when to choose namespaces, dedicated nodes or dedicated clusters.

Confidential computing

NVIDIA's confidential computing documentation describes three deployment modalities: confidential bare metal (tenant controls the host and the full stack), confidential virtual machines (the VM boundary isolates the workload from privileged host software) and confidential containers (a confidential pod boundary for Kubernetes native workloads). Attestation lets a workload verify the hardware and security configuration before releasing secrets. For a neocloud this is a way to reduce how much tenants must trust the operator, not just each other. Check NVIDIA's platform compatibility matrix for which GPUs, CPUs and drivers support each mode, and note NVIDIA's own licensing requirements for production use.

Zero trust for the operator side

NIST SP 800-207 (August 2020) defines zero trust as granting no implicit trust based on network location or asset ownership, with authentication and authorization performed before each session. In a neocloud that applies most to the operator's own access: engineers reaching BMCs, switches, the Kubernetes API and the billing system should authenticate per session with strong identity, not because they are on the management VLAN.

The software stack is the attack surface

A GPU cloud runs a long chain of software from many sources, and every link ships updates:

• Server firmware, BMC firmware, NIC and switch firmware
• The host operating system and kernel
• GPU drivers and management libraries
• The container runtime and the GPU container toolkit
• GPU operators, device plugins and DRA drivers
• Kubernetes or Slurm and their plugins
• Monitoring agents such as DCGM exporters
• Base images and framework images (CUDA or ROCm userland, PyTorch and others)
• Model serving stacks and the tenant's own code

NIST SP 800-190, the Application Container Security Guide, covers the image, registry, orchestrator, container and host risks that apply to most of that list. NVIDIA publishes security bulletins for its drivers and software and recommends following their guidance on driver and package updates; it offers email notifications for new and revised bulletins.

CVE-2024-0132 shows why this matters for tenancy. NVD describes it as a time of check, time of use flaw in NVIDIA Container Toolkit 1.16.1 and earlier, in the default configuration, where "a specifically crafted container image may gain access to the host file system". NVD records a CVSS 3.1 base score of 9.0 from NVIDIA and 8.3 from NVD's own analysis, and notes that use cases relying on the Container Device Interface (CDI) are not affected. In a GPU cloud the container toolkit sits exactly on the line between a tenant's image and the host, so a flaw there is a tenant boundary problem, not just a local bug. We cover it in more detail in our CVE-2024-0132 explainer and in enabling GPUs in Docker with the NVIDIA Container Toolkit safely.

The lesson: know which version of each component runs on every node, subscribe to vendor advisories, have a tested way to drain and patch nodes without killing long jobs, and rebuild images rather than patch them in place.

How do metering and billing work?

Billing starts with measurement. Both schedulers give you the raw numbers.

In Slurm, trackable resources (TRES) are the unit of accounting. The Slurm TRES documentation lists billing, CPU, energy, memory and node among the defaults, and you can add GPUs with AccountingStorageTRES=gres/gpu, which also tracks gres/gpumem and gres/gpuutil. A partition's TRESBillingWeights turn usage into a single billing number. Slurm's own example weights CPU at 1.0, memory at 0.25 per gigabyte and each GPU at 2.0, so you can price a GPU hour differently from a CPU hour. Slurm also documents that if no billing weights are set, the billing TRES equals the node's CPU count, which is a trap if you forget to configure it.

In Kubernetes there is no built in billing, but there are building blocks. ResourceQuota objects cap what each namespace can request; for extended resources such as GPUs only the requests prefix is allowed, so a quota of requests.nvidia.com/gpu set to 4 limits a namespace to four GPUs. DRA claims can be capped per device class. Kueue adds admission, fair sharing and Prometheus metrics. DCGM provides job level statistics and continuous GPU telemetry. A billing service then joins pod or job lifetimes with the GPUs they held.

Common pricing shapes, described generically:

Why is power the hardest constraint?

Power decides how many GPUs you can run, where, and when. Some arithmetic using NVIDIA's published figure:

• One 8 GPU server at its 10.2 kW maximum.
• Four of them in a rack is about 41 kW of IT load before switches.
• A 1,024 GPU cluster (128 servers) is about 1.3 MW of server load at maximum, before networking, storage, cooling and conversion losses.

That is illustrative arithmetic from one vendor's maximum rating, not a measured average, but it shows why air cooled rooms designed for a few kilowatts per rack do not work for dense GPU racks without major changes. Our post on data center cooling, chillers, liquid cooling and CDUs explains the cooling side.

At national scale, the US Department of Energy announced on 20 December 2024 that Lawrence Berkeley National Laboratory's 2024 report estimates US data centers used about 4.4 percent of total US electricity in 2023, rising from 58 TWh in 2014 to 176 TWh in 2023, with 325 to 580 TWh projected by 2028, or roughly 6.7 to 12 percent of US electricity. DOE's older buildings program page still gives a figure of about 2 percent of US electricity, which reflects an earlier estimate; use the 2024 report for planning.

For a builder, that means the timeline is usually set by the utility, not the GPU vendor. Grid connection studies, substation work and transformer lead times can take longer than the hardware order. See what a data center needs to connect to the grid and how long it takes to build an AI data center. All electrical design and work belongs with licensed electrical engineers and electricians; nothing in this post is a substitute for that.

Build, colocate or lease

Most new GPU clouds start in colocation, renting powered and cooled space that can handle high density racks; our colocation data center security checklist covers what to ask. Build to suit leases and owned builds give more control at the cost of time and capital.

How are neoclouds financed?

Financing patterns fall into a few generic types. Our post on how AI data centers are financed goes deeper, and AI data center cost per megawatt covers the capital side.

The structural tension is the same in every model: GPUs depreciate on a hardware generation cycle, while facility leases and power contracts run much longer. Utilization and contract length decide whether the numbers work.

What security and compliance should tenants expect?

If you are buying from a neocloud, or building one and selling to enterprises, these are the questions that come up in every security review. They are framed as what to ask, not as a certification checklist.

For builders, frameworks such as SOC 2 and ISO/IEC 27001 are the usual ways to show those controls exist, and NIST SP 800-190 and the Kubernetes Pod Security Standards give concrete technical baselines. The Pod Security Standards define three profiles: Privileged (unrestricted), Baseline (prevents known privilege escalations) and Restricted (current pod hardening practice). A sensible default is Restricted for tenant namespaces, and Privileged only for the system namespaces that need it, such as the GPU operator's.

Step by step: a small multi tenant GPU cluster on Kubernetes

This walkthrough sets up the skeleton of a GPU cloud on a Kubernetes cluster you already run, using commands from the official NVIDIA and Kubernetes documentation. It is a lab sized starting point, not a production design. Replace version numbers with the current ones from the vendor documentation before you run anything.

1. Prepare the GPU nodes

Install the GPU vendor's driver on each node using the vendor's instructions, then confirm the GPUs are visible:

nvidia-smi

If your nodes manage the container runtime themselves rather than through the operator, install the NVIDIA Container Toolkit from NVIDIA's repository and configure containerd. Per NVIDIA's install guide, this writes a drop in file under /etc/containerd/conf.d and updates the containerd config:

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

Check the installed toolkit version against NVIDIA's security bulletins before going further. Versions 1.16.1 and earlier are affected by CVE-2024-0132.

2. Install the GPU Operator into its own privileged namespace

NVIDIA's getting started guide labels the operator namespace as privileged, because the operator manages drivers and host level components:

kubectl create ns gpu-operator
kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v26.7.2

Confirm that nodes now advertise GPUs:

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia\\.com/gpu

For AMD Instinct nodes, install the AMD GPU Operator with Helm following AMD's documentation instead; it exposes amd.com/gpu.

3. Create a tenant namespace with a hard baseline

Each tenant gets a namespace that enforces the Restricted Pod Security Standard, a GPU quota, and default deny networking:

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    pod-security.kubernetes.io/enforce: restricted
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: tenant-a
spec:
  hard:
    requests.nvidia.com/gpu: "4"
    requests.cpu: "64"
    requests.memory: 512Gi
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Remember that the NetworkPolicy only takes effect if your network plugin supports network policy. Add explicit allow rules for DNS, the storage network and anything else the tenant needs.

4. Pin the tenant to its own nodes

For tenants who do not trust each other, dedicate nodes. Taint and label them:

kubectl label nodes gpu-node-01 gpu-node-02 tenant=tenant-a
kubectl taint nodes gpu-node-01 gpu-node-02 tenant=tenant-a:NoSchedule

Then a tenant workload requests GPUs in limits, tolerates its own taint and selects its own nodes:

apiVersion: v1
kind: Pod
metadata:
  name: train-smoke-test
  namespace: tenant-a
spec:
  nodeSelector:
    tenant: tenant-a
  tolerations:
    - key: tenant
      operator: Equal
      value: tenant-a
      effect: NoSchedule
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: cuda
      image: registry.example.internal/base/cuda-runtime:pinned-digest
      command: ["nvidia-smi"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        limits:
          nvidia.com/gpu: 1
  restartPolicy: Never

In production, enforce the node pinning with an admission policy rather than trusting each tenant to set the selector, so a tenant cannot schedule onto another tenant's nodes.

5. Add a sandboxed runtime for untrusted inference pods (optional)

If you offer shared nodes for small inference workloads, route them to a sandboxed runtime with RuntimeClass. The handler name must match the runtime configured on the node, for example a Kata Containers handler:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: sandboxed
handler: kata

Pods then set runtimeClassName to sandboxed. Test GPU passthrough for your exact GPU, driver and runtime combination; do not assume it works.

6. Decide your GPU sharing policy

Per product tier: whole servers for training, whole GPUs or MIG instances for shared inference, and time slicing only inside one tenant's own nodes.

7. Meter usage

Scrape DCGM exporter metrics, record which GPUs each pod held and for how long, and join the two in your billing service. Kueue's metrics add pending demand.

Step by step: the same idea on Slurm

For tenants who want Slurm, give each tenant its own Slurm controller and dedicated nodes. A minimal GPU configuration from the Slurm documentation looks like this.

In slurm.conf:

GresTypes=gpu
NodeName=gpu[01-04] Gres=gpu:8 CPUs=112 RealMemory=2000000 State=UNKNOWN
PartitionName=train Nodes=gpu[01-04] Default=YES MaxTime=INFINITE State=UP TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0"
AccountingStorageTRES=gres/gpu
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup

In gres.conf on each node:

AutoDetect=nvml

In cgroup.conf, constrain devices so a job only sees the GPUs it was given:

ConstrainDevices=yes

Submit and check accounting:

sbatch --gpus-per-node=8 --nodes=2 train.sh
sacct --format=JobID,User,Partition,AllocTRES%60,Elapsed

Set real values for CPUs and memory from your hardware, and take the billing weights from your own price list. If you leave TRESBillingWeights out, Slurm bills by CPU count, which will badly underprice GPU time.

Common mistakes

• Treating time slicing as isolation. NVIDIA says there is no memory or fault isolation between time sliced replicas. Use it inside one tenant only.
• Sharing one Slurm cluster between unrelated customers. Slurm's design assumes users of one organization. Give strangers separate clusters and nodes.
• Leaving NetworkPolicy objects on a plugin that ignores them. Kubernetes accepts them silently. Test that traffic is actually blocked.
• Putting BMCs on a network tenants can reach. Keep the management network separate and authenticate every session.
• Not wiping local NVMe between tenants. Scratch disks and RAID 0 caches keep data after a job ends.
• Patching drivers and toolkits "later". Components such as the container toolkit sit on the tenant boundary. Track versions on every node.
• Forgetting Slurm billing weights. Without TRESBillingWeights, the billing TRES equals CPU count and GPUs look free.
• Planning around GPU delivery dates instead of power dates.
• Assuming a new fabric specification is fully implemented by your NICs and switches.
• Running tenant namespaces at the Privileged pod security level because the GPU operator needed it. Only the operator's namespace needs that.

FAQ

What is a neocloud in simple terms?

A cloud company whose main business is renting GPU and accelerator computing for AI rather than a broad catalog of services, usually as GPU servers or clusters with fast networking and storage, often on reserved contracts.

How is a neocloud different from the big public clouds?

General purpose clouds offer hundreds of services, GPUs among them. A neocloud focuses on GPU compute and the fabric and storage around it, usually in fewer sites sized around available power. The line is not sharp.

Can I build my own neocloud?

You can build a GPU cloud from standard parts: GPU servers, an InfiniBand or RoCE fabric, storage, Kubernetes or Slurm, and a billing layer. The hard parts are securing enough power, isolating tenants properly, keeping the software stack patched, and financing hardware that depreciates quickly.

Do I need InfiniBand to run a GPU cloud?

No. InfiniBand is widely used for training clusters, but RoCE brings RDMA to Ethernet, and the Ultra Ethernet specification released in June 2025 targets AI and HPC on Ethernet. Inference workloads that fit on one server may not need an RDMA fabric at all.

Should a GPU cloud use Kubernetes or Slurm?

Use Slurm if tenants run HPC style batch training and expect sbatch and srun. Use Kubernetes, with Kueue for quotas and fair sharing, if you also run inference services and pipelines. Many providers run Kubernetes as the control plane and offer Slurm per tenant on dedicated nodes.

Is it safe to share one GPU between customers?

Only with real isolation. NVIDIA MIG partitions supported GPUs into up to seven instances with separate memory paths and fault isolation. Time slicing does not isolate memory or faults, so it should not be used between different customers.

How much power does a GPU cloud need?

It depends on scale. NVIDIA documents a maximum of 10.2 kW for an 8 GPU DGX H100 system, so a 1,024 GPU cluster of such servers is roughly 1.3 MW of server load at maximum before networking, storage and cooling. Use measured data and engineering advice for real designs.

What security questions should I ask a neocloud provider?

Ask about the isolation model, disk wiping between tenants, fabric partitioning, patch speed for drivers and toolkits, staff access to hosts and BMCs, and which audit reports cover which sites.

How do neoclouds bill for GPUs?

Usually per GPU hour on demand, by reserved cluster for a fixed term, by committed spend, by preemptible capacity, or per request for managed inference. Under the hood, Slurm TRES accounting or Kubernetes quotas and GPU telemetry provide the usage numbers.

Do neoclouds offer confidential computing?

Some may. NVIDIA documents confidential bare metal, confidential virtual machines and confidential containers on supported GPUs, with attestation. Ask a provider which modality it supports, on which hardware, and how you verify the attestation yourself.

Fix it without waiting for upstream

Most of a neocloud's risk outside the facility sits in software someone else wrote: drivers, the container toolkit, operators, Kubernetes, Slurm, base images and model servers, much of it open source and all of it rebuilt every time a node is reimaged. Safeguard does not patch GPU firmware, drivers or a provider's hosts, but it does help with the open source dependencies and container images that you and your tenants build and ship on top of them.

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Safeguard's Griffin models open a tested, policy gated pull request that moves the affected dependency to the safe version. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation covers eight categories, including dependencies, licenses, security posture and findings. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Autonomous remediation ships switched off, with every category set to Manual, so your team decides how much runs on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds. The safeguard-sh/gold-scan-action GitHub Action can fail a build on a known exploited vulnerability, on any known vulnerability, or never, so a version you have just removed cannot quietly come back in next week's merge.

3. Gold Registry: start from components that are already fixed

registry.safeguard.sh provides hardened, drop in replacements for upstream packages and container images. Every Gold artifact is held continuously to zero exploitable critical or high CVEs, zero malware and a completed license check, with SLSA L3 or higher provenance, a Sigstore signature and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened, compatibility tested build of a specific package. This fork and patch flow is in early access today, so ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

The first demo is a real terminal session with the Safeguard CLI. It reads a vulnerability finding and its advisory, asks before each command, upgrades a dependency, runs the tests with your approval and pushes nothing. In a GPU cloud the same flow applies to the code you own around the GPUs: provisioning tools, billing services, tenant portals and the inference services you build on framework images.

Click through it at your own pace, or open it full screen.

The second demo sets up the local runner on your own machine, approves it with a device code and runs scans from the terminal. That suits an operator workstation inside a cluster's management network, where you want to scan a container image or an SBOM for a node image before it rolls out, without sending source anywhere else.

Click through it at your own pace, or open it full screen.

The third demo connects Claude to the Safeguard MCP server with OAuth and lets you choose which tools Claude can use. Once connected, you can ask which of your projects contain a component named in a new driver, toolkit or framework advisory, as long as that software is part of your Safeguard inventory.

Click through it at your own pace, or open it full screen.

To try it on your own projects, download Safeguard at safeguard.sh/download and read the CLI documentation.

Quick check

Sources

• NIST SP 800-145, The NIST Definition of Cloud Computing (September 2011)
• NIST SP 800-190, Application Container Security Guide (September 2017)
• NIST SP 800-207, Zero Trust Architecture (August 2020)
• Kubernetes documentation, Multi-tenancy
• Kubernetes documentation, Schedule GPUs
• Kubernetes documentation, Dynamic Resource Allocation
• Kubernetes documentation, Resource Quotas
• Kubernetes documentation, Runtime Class
• Kubernetes documentation, Pod Security Standards
• Kueue overview
• Slurm overview, SchedMD
• Slurm Generic Resource (GRES) scheduling
• Slurm Trackable Resources (TRES)
• NVIDIA Multi-Instance GPU User Guide, Introduction
• NVIDIA GPU Operator overview
• NVIDIA GPU Operator, Getting started
• NVIDIA GPU Operator, Time-slicing GPUs
• NVIDIA Container Toolkit installation guide
• NVIDIA DCGM user guide
• NVIDIA NCCL overview
• NVIDIA DGX H100/H200 user guide, Introduction
• NVIDIA Confidential Computing documentation
• NVIDIA Product Security and Security Bulletins
• NIST NVD, CVE-2024-0132
• AMD GPU Operator documentation
• InfiniBand Trade Association, About InfiniBand
• InfiniBand Trade Association, Specification FAQ including RoCE
• Ultra Ethernet Consortium
• Ultra Ethernet Consortium, UEC launches Specification 1.0 (11 June 2025)
• US Department of Energy, DOE releases new report evaluating increase in electricity demand from data centers (20 December 2024)
• US Department of Energy, Data centers and servers
• Kata Containers

Last reviewed: 23 October 2026.