GPU types
Each GPU brings a fixed share of vCPUs and memory, so the machine’s size comes from the GPU type and count, and you cannot set the vCPUs or memory yourself.
To give a machine more than one GPU, add the count after the type. Counts are powers of two, so a
host’s GPUs always split evenly between machines:
l4, a100, and t4 machines take 1, 2, or 4
GPUs, and an h100 machine takes one.
no_capacity. Add --queue to
wait in line for a GPU instead: the machine starts when one frees up
(Wait for a GPU).
What is preinstalled
InstaBox picks the template from the GPU type:gpu-metal for h100, and gpu for the other
types. Both have the CUDA 13.0 toolkit (nvcc), PyTorch 2.8 with CUDA in /opt/venv, and
instabox-gpu-smoke, which checks that the GPU works. On gpu, the NVIDIA driver and nvidia-smi
come from the host. An h100 machine gets the whole GPU, so gpu-metal brings its own driver.
Check the GPU once the machine runs:
Model weights
A GPU machine’s/data disk is 200 GiB at create, whatever its GPU count, and your home is
/data/home, so a model you download lands on the disk and is still there after a
stop, a start, a pause, or a snapshot. Keep weights out of /: the root disk is the template’s
19 GiB, with little room left.
The hf CLI is preinstalled, with Hugging Face’s Xet transfers in high-performance mode
(HF_XET_HIGH_PERFORMANCE=1), so a download uses all the machine’s bandwidth:
~/.cache/huggingface, where transformers, vLLM, and diffusers find it on the next
run without another download. Pass --local-dir /data/models/qwen3-8b for a plain directory
instead, for llama.cpp or a tool that wants a path.
When a model needs more room, grow the disk while the machine runs, up to your plan’s cap
():
--gpu a100:4) or a smaller quantization.
Start GPU machines from a snapshot
Install your models and libraries once, save a snapshot, and start new GPU machines from it. A machine created from a GPU machine’s snapshot keeps its GPU type. It gets the snapshot’s GPU count unless you ask for another:What works differently
A GPU machine is a Linux machine, so the pages under Configuration and Access apply to it, except for these:- Fork: a GPU machine cannot be forked. The fork is refused with
fork_unavailable. To copy one, save a snapshot and create new machines from it, as above. - Snapshots: a running GPU machine cannot be saved live. Stop it first, or add
--restarttoinstabox snapshot save(Snapshots). A GPU machine’s snapshot always keeps/data, so--root-onlyis refused. - Size: the vCPUs and memory come from the GPU type and count, so a resize of the vCPUs or
memory is refused. You can still grow the disk (
instabox resize train --disk 200). For more GPUs, save a snapshot and create from it with another count (--gpu l4:4). - Pause on
h100: anh100machine cannot be paused, because its GPU memory cannot be saved. Idle time and the max run time stop it instead, which keeps its files but not its running processes (Auto-pause and max-run).
Memory inside a GPU machine
Memory inside a GPU machine
A GPU machine runs in a sandbox whose Set such budgets from that estimate, or explicitly. A model whose weights do not fit in the
machine’s memory is read from disk again and again while it runs. Give it more GPUs, and so more
memory:
/proc/meminfo differs from a Linux kernel’s in one way that
matters for large models: file pages you map or read (model weights opened with mmap, for
example) count as used memory, and MemAvailable equals MemFree. So once the files you map are
larger than the machine’s memory, MemFree and MemAvailable read 0 and Cached can be larger
than MemTotal, even though most of that memory is file cache that can be reclaimed. Tools that
size buffers from “available” memory (psutil’s virtual_memory().available, and the CPU offload or
pinned-memory budgets of some inference servers) then see none.MemTotal is the machine’s memory, and AnonPages is the memory programs hold (shared memory and
tmpfs included). To estimate what is free, treat file cache as reclaimable:--gpu l4:2 has twice the memory of --gpu l4.GPU workloads
Serve a model, train a model, generate images, and more, step by step.
Wait for a GPU
Queue a create or start when every GPU of a type is in use.