Skip to main content
Run a GPU job over a large input and pay only while it runs. This page has two jobs: batch inference with an open model on one GPU, and a sweep split across many GPU machines. You need the CLI installed and signed in (Quickstart) and a paid plan.

Batch inference with an open model

Run a model over a file of prompts, without sending the data to a hosted API. The steps run a 7B model with vLLM on one l4 GPU.
1

Create a GPU machine

2

Install vLLM

Install vLLM in a virtual environment of its own, as a background job. The command prints the job’s ID.
3

Wait for the install

Follow the job’s output with that ID. It ends with Successfully installed after about four minutes.
4

Write the prompts and the script

On your computer, put one prompt per line in prompts.jsonl.
prompts.jsonl
Save this script as batch.py. It loads the model once, runs the prompts 512 at a time, and writes each batch’s answers to out.jsonl.
batch.py
5

Copy both files to the machine

6

Start the batch

Start the script as a background job. The command prints the job’s ID.
7

Wait for the batch

Follow the job’s output with that ID. The first run downloads the model’s weights, about 15 GB, and the first prompt runs after about three minutes.
Near its end, the log shows the progress of the batch. In this run, 64 prompts took 38 seconds, at 492 output tokens a second.
8

Copy the results to your computer

out.jsonl has one line for each prompt, with the prompt and the model’s answer.
9

Stop or delete the machine

Stop the machine to keep the weights on its disk for the next batch. A stopped machine pays only for its disk.
Delete the machine when you no longer need it.
To go faster, split prompts.jsonl into parts and run each part on a machine of its own, as in the next section.

A sweep across many GPUs

A sweep runs the same GPU program many times, once for each combination of parameters and seeds. The runs do not depend on each other, so each machine takes one shard of the sweep. The steps assume a project folder sim on your computer with a requirements.txt and a program sweep.py that takes --shard and --shards, runs its share of the combinations, and writes its results under results/.
1

Set up one machine

Create a GPU machine.
Push your project to it. The folder lands in /data/home/sim.
Install what the project needs. The machine’s /opt/venv already has PyTorch with CUDA.
2

Save it as a snapshot

Stop the machine, because a GPU machine is saved while it is stopped.
Save the snapshot, so that every shard starts with the same files and packages.
The snapshot holds everything now, so delete the base machine.
3

Start one machine per shard

Start eight machines from the snapshot, and start one shard as a background job on each.
Each machine starts in a few seconds, and each exec prints the ID of the job that it started.
4

Check on the shards

List each machine’s jobs. A job shows running until it ends, and then exited(0) when it succeeded.
5

Collect the results and delete the machines

When every job has exited, copy each shard’s results to your computer, and delete the machines.
Each shard’s results are now in shard-<n>/results. The snapshot stays, so the next sweep starts from it.

Limits

Two limits cap how many shards run at once: your workspace’s GPU limit, and its limit of running machines, which counts every running machine. When a create is refused with concurrency_limit, run the loop over the shards that fit, collect and delete those machines, and then run the loop over the remaining shard numbers.

GPU workloads

What a GPU machine comes with, and the other guides.

GPU machines

GPU types, counts, and what the gpu template includes.

Run commands

Background jobs, timeouts, and following output.