Batch inference with an open model
Run a model over a file of prompts, without sending the data to a hosted API. The steps run a 7B model with vLLM on onel4 GPU.
1
Create a GPU machine
2
Install vLLM
Install vLLM in a virtual environment of its own, as a background job. The command prints the
job’s ID.
3
Wait for the install
Follow the job’s output with that ID. It ends with
Successfully installed after about four
minutes.4
Write the prompts and the script
On your computer, put one prompt per line in Save this script as
prompts.jsonl.prompts.jsonl
batch.py. It loads the model once, runs the prompts 512 at a time, and
writes each batch’s answers to out.jsonl.batch.py
5
Copy both files to the machine
6
Start the batch
Start the script as a background job. The command prints the job’s ID.
7
Wait for the batch
Follow the job’s output with that ID. The first run downloads the model’s weights, about 15 GB,
and the first prompt runs after about three minutes.Near its end, the log shows the progress of the batch. In this run, 64 prompts took 38 seconds,
at 492 output tokens a second.
8
Copy the results to your computer
out.jsonl has one line for each prompt, with the prompt and the model’s answer.9
Stop or delete the machine
Stop the machine to keep the weights on its disk for the next batch. A stopped machine pays
only for its disk.Delete the machine when you no longer need it.
prompts.jsonl into parts and run each part on a machine of its own, as in the
next section.
A sweep across many GPUs
A sweep runs the same GPU program many times, once for each combination of parameters and seeds. The runs do not depend on each other, so each machine takes one shard of the sweep. The steps assume a project foldersim on your computer with a requirements.txt and a program
sweep.py that takes --shard and --shards, runs its share of the combinations, and writes its
results under results/.
1
Set up one machine
Create a GPU machine.Push your project to it. The folder lands in Install what the project needs. The machine’s
/data/home/sim./opt/venv already has PyTorch with CUDA.2
Save it as a snapshot
Stop the machine, because a GPU machine is saved while it is stopped.Save the snapshot, so that every shard starts with the same files and packages.The snapshot holds everything now, so delete the base machine.
3
Start one machine per shard
Start eight machines from the snapshot, and start one shard as a background job on each.Each machine starts in a few seconds, and each
exec prints the ID of the job that it started.4
Check on the shards
List each machine’s jobs. A job shows
running until it ends, and then exited(0) when it
succeeded.5
Collect the results and delete the machines
When every job has exited, copy each shard’s results to your computer, and delete the machines.Each shard’s results are now in
shard-<n>/results. The snapshot stays, so the next sweep
starts from it.Limits
Two limits cap how many shards run at once: your workspace’s GPU limit, and its limit of running machines, which counts every running machine. When acreate is refused with
concurrency_limit, run the loop over the shards that fit, collect and delete those machines, and
then run the loop over the remaining shard numbers.
GPU workloads
What a GPU machine comes with, and the other guides.
GPU machines
GPU types, counts, and what the
gpu template includes.Run commands
Background jobs, timeouts, and following output.