> ## Documentation Index
> Fetch the complete documentation index at: https://instabox.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Run batch jobs

> Run batch LLM inference over a file of prompts, or split a simulation sweep across many GPU machines.

Run a GPU job over a large input and pay only while it runs. This page has two jobs: batch
inference with an open model on one GPU, and a sweep split across many GPU machines.

You need the CLI installed and signed in ([Quickstart](https://instabox.sh/docs/quickstart.md)) and a paid plan.

## Batch inference with an open model

Run a model over a file of prompts, without sending the data to a hosted API. The steps run a 7B
model with [vLLM](https://docs.vllm.ai) on one `l4` GPU.

**1. Create a GPU machine**

```bash theme={null}
instabox create --name infer --gpu l4
```

**2. Install vLLM**

Install vLLM in a virtual environment of its own, as a background job. The command prints the
job's ID.

```bash theme={null}
instabox exec --background infer -- \
  bash -c 'python3 -m venv /data/home/vllm && /data/home/vllm/bin/pip install vllm'
```

**3. Wait for the install**

Follow the job's output with that ID. It ends with `Successfully installed` after about four
minutes.

```bash theme={null}
instabox jobs logs infer <job-id> -f
```

**4. Write the prompts and the script**

On your computer, put one prompt per line in `prompts.jsonl`.

```json prompts.jsonl theme={null}
{"prompt": "Write a Python function that reverses a singly linked list."}
{"prompt": "Write a Python function that checks whether a string is a palindrome."}
```

Save this script as `batch.py`. It loads the model once, runs the prompts 512 at a time, and
writes each batch's answers to `out.jsonl`.

```python batch.py theme={null}
import itertools, json
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-Coder-7B-Instruct", max_model_len=8192)
params = SamplingParams(temperature=0.2, max_tokens=1024)

with open("prompts.jsonl") as src, open("out.jsonl", "w") as f:
    lines = (json.loads(line)["prompt"] for line in src)
    while batch := list(itertools.islice(lines, 512)):
        chats = [[{"role": "user", "content": p}] for p in batch]
        for prompt, out in zip(batch, llm.chat(chats, params)):
            f.write(json.dumps({"prompt": prompt, "output": out.outputs[0].text}) + "\n")
        f.flush()
```

**5. Copy both files to the machine**

```bash theme={null}
instabox cp prompts.jsonl infer:/data/home/prompts.jsonl
```

```bash theme={null}
instabox cp batch.py infer:/data/home/batch.py
```

**6. Start the batch**

Start the script as a background job. The command prints the job's ID.

```bash theme={null}
instabox exec --background infer --cwd /data/home -- \
  /data/home/vllm/bin/python batch.py
```

**7. Wait for the batch**

Follow the job's output with that ID. The first run downloads the model's weights, about 15 GB,
and the first prompt runs after about three minutes.

```bash theme={null}
instabox jobs logs infer <job-id> -f
```

Near its end, the log shows the progress of the batch. In this run, 64 prompts took 38 seconds,
at 492 output tokens a second.

```text theme={null}
Processed prompts: 100%|██████████| 64/64 [00:38<00:00,  1.05s/it, est. speed input: 73.05 toks/s, output: 491.68 toks/s]
```

**8. Copy the results to your computer**

```bash theme={null}
instabox cp infer:/data/home/out.jsonl ./out.jsonl
```

`out.jsonl` has one line for each prompt, with the prompt and the model's answer.

**9. Stop or delete the machine**

Stop the machine to keep the weights on its disk for the next batch. A stopped machine pays
only for its disk.

```bash theme={null}
instabox stop infer
```

Delete the machine when you no longer need it.

```bash theme={null}
instabox rm infer --yes
```

To go faster, split `prompts.jsonl` into parts and run each part on a machine of its own, as in the
next section.

## A sweep across many GPUs

A sweep runs the same GPU program many times, once for each combination of parameters and seeds.
The runs do not depend on each other, so each machine takes one shard of the sweep.

The steps assume a project folder `sim` on your computer with a `requirements.txt` and a program
`sweep.py` that takes `--shard` and `--shards`, runs its share of the combinations, and writes its
results under `results/`.

**1. Set up one machine**

Create a GPU machine.

```bash theme={null}
instabox create --name sweep-base --gpu l4
```

Push your project to it. The folder lands in `/data/home/sim`.

```bash theme={null}
instabox push sweep-base ./sim
```

Install what the project needs. The machine's `/opt/venv` already has PyTorch with CUDA.

```bash theme={null}
instabox exec sweep-base --cwd /data/home/sim -- \
  /opt/venv/bin/pip install -r requirements.txt
```

**2. Save it as a snapshot**

Stop the machine, because a GPU machine is saved while it is stopped.

```bash theme={null}
instabox stop sweep-base
```

Save the snapshot, so that every shard starts with the same files and packages.

```bash theme={null}
instabox snapshot save sweep-base --name sim-ready
```

The snapshot holds everything now, so delete the base machine.

```bash theme={null}
instabox rm sweep-base --yes
```

**3. Start one machine per shard**

Start eight machines from the snapshot, and start one shard as a background job on each.

```bash theme={null}
for i in 0 1 2 3 4 5 6 7; do
  instabox create --name sweep-$i --snapshot sim-ready --gpu l4
  instabox exec --background sweep-$i --cwd /data/home/sim -- \
    /opt/venv/bin/python sweep.py --shard $i --shards 8
done
```

Each machine starts in a few seconds, and each `exec` prints the ID of the job that it started.

```text theme={null}
NAME     STATE    TEMPLATE  CREATED
sweep-0  Running  gpu       11:23
c7b22b24
```

**4. Check on the shards**

List each machine's jobs. A job shows `running` until it ends, and then `exited(0)` when it
succeeded.

```bash theme={null}
for i in 0 1 2 3 4 5 6 7; do instabox jobs ls sweep-$i; done
```

```text theme={null}
ID        STATUS     STARTED              COMMAND
e16dfeb7  exited(0)  2026-11-20 11:25:31  '/opt/venv/bin/python' sweep.py --shard 7 --shards 8
```

**5. Collect the results and delete the machines**

When every job has exited, copy each shard's results to your computer, and delete the machines.

```bash theme={null}
for i in 0 1 2 3 4 5 6 7; do
  mkdir -p shard-$i
  instabox pull sweep-$i /data/home/sim/results shard-$i
  instabox rm sweep-$i --yes
done
```

```text theme={null}
Pulled 10 files (1.1 KiB compressed) from sweep-0:/data/home/sim/results to shard-0/results
```

Each shard's results are now in `shard-<n>/results`. The snapshot stays, so the next sweep
starts from it.

## Limits

Two limits cap how many shards run at once: your workspace's [GPU limit](https://instabox.sh/docs/machines/gpu.md), and its
limit of running machines, which counts every running machine. When a `create` is refused with
`concurrency_limit`, run the loop over the shards that fit, collect and delete those machines, and
then run the loop over the remaining shard numbers.

> [card] [GPU workloads](https://instabox.sh/docs/use-cases/gpu)
>
> What a GPU machine comes with, and the other guides.

> [card] [GPU machines](https://instabox.sh/docs/machines/gpu.md)
>
> GPU types, counts, and what the `gpu` template includes.

> [card] [Run commands](https://instabox.sh/docs/machines/run-commands.md)
>
> Background jobs, timeouts, and following output.
