l4 GPU. PyTorch is
already installed, and the whole run takes about 10 minutes.
1
Create a GPU machine
2
Copy the training script
Save this script as Copy the script to the machine. It lands in your home directory,
train.py on your computer. It downloads the text, trains the model, prints
a line every 50 steps, and saves a checkpoint to ckpt.pt every 200 steps. When ckpt.pt
exists, the script continues from it.train.py
/data/home.3
Start the training run
Start the script as a background job, so that it keeps running after the command returns, and
send its output to a log file.The command prints the job’s ID. Read the end of the log to see the run’s progress.
4
Pause the machine and resume it
Pause the machine when you want to stop paying for the GPU for a while. A pause saves the
machine’s memory, the GPU’s memory included, so the training process is kept as it is.Resume the machine when you want the run to go on.Read the log again. The run continues with the next step, in the same process.The pause took 8 seconds and the resume took 4 seconds. Step 300 shows a lower rate because its
50 steps include the paused time.
5
Stop the machine and continue from the checkpoint
A stop shuts the machine down. Its disk stays, so the checkpoint and the log are there when you
start it again, but the training process is gone.Start the script again. It finds Read the start of the new log. Its first line says which step the run continued from.This run was stopped just after step 250, so it continued from the checkpoint of step 200.
ckpt.pt and continues from the step that the checkpoint was
saved at.6
Use the trained model
The run ends at step 3000 and prints To keep the checkpoint, copy it to your computer.
done. Then ask the model to write some text from the
checkpoint.7
Delete the machine
Delete the machine when you no longer need it. Its disk, with the checkpoint, is deleted too.
Pause or stop
An
h100 machine cannot be paused, so on that GPU type use checkpoints.
A machine pauses by itself after idle minutes by default, and a background job
with no open connection can count as idle. The run is kept and continues on resume. To keep a long
run going, turn the idle pause off for the machine. See Auto-pause.
Train your own model
The steps are the same for your own code: push your project withinstabox push, install its
packages, and start your training command as a background job. Keep datasets and checkpoints under
/data/home, which is the machine’s large disk.
GPU machines
GPU types and counts, and what the template includes.
Run commands
Background jobs, timeouts, and following output.