Skip to main content
Train a model on a GPU machine as a background job. You can pause the machine and the run continues from the same step when you resume, or stop it and continue from the last checkpoint. This guide trains a 25M-parameter GPT on Shakespeare with PyTorch on one l4 GPU. PyTorch is already installed, and the whole run takes about 10 minutes.
1

Create a GPU machine

2

Copy the training script

Save this script as train.py on your computer. It downloads the text, trains the model, prints a line every 50 steps, and saves a checkpoint to ckpt.pt every 200 steps. When ckpt.pt exists, the script continues from it.
train.py
Copy the script to the machine. It lands in your home directory, /data/home.
3

Start the training run

Start the script as a background job, so that it keeps running after the command returns, and send its output to a log file.
The command prints the job’s ID. Read the end of the log to see the run’s progress.
4

Pause the machine and resume it

Pause the machine when you want to stop paying for the GPU for a while. A pause saves the machine’s memory, the GPU’s memory included, so the training process is kept as it is.
Resume the machine when you want the run to go on.
Read the log again. The run continues with the next step, in the same process.
The pause took 8 seconds and the resume took 4 seconds. Step 300 shows a lower rate because its 50 steps include the paused time.
5

Stop the machine and continue from the checkpoint

A stop shuts the machine down. Its disk stays, so the checkpoint and the log are there when you start it again, but the training process is gone.
Start the script again. It finds ckpt.pt and continues from the step that the checkpoint was saved at.
Read the start of the new log. Its first line says which step the run continued from.
This run was stopped just after step 250, so it continued from the checkpoint of step 200.
6

Use the trained model

The run ends at step 3000 and prints done. Then ask the model to write some text from the checkpoint.
To keep the checkpoint, copy it to your computer.
7

Delete the machine

Delete the machine when you no longer need it. Its disk, with the checkpoint, is deleted too.

Pause or stop

An h100 machine cannot be paused, so on that GPU type use checkpoints.
A machine pauses by itself after idle minutes by default, and a background job with no open connection can count as idle. The run is kept and continues on resume. To keep a long run going, turn the idle pause off for the machine. See Auto-pause.

Train your own model

The steps are the same for your own code: push your project with instabox push, install its packages, and start your training command as a background job. Keep datasets and checkpoints under /data/home, which is the machine’s large disk.

GPU machines

GPU types and counts, and what the template includes.

Run commands

Background jobs, timeouts, and following output.