> ## Documentation Index
> Fetch the complete documentation index at: https://instabox.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Train a model

> Train a model on a GPU machine as a background job, pause the machine without losing the run, and continue from a checkpoint after a stop.

Train a model on a GPU machine as a background job. You can pause the machine and the run
continues from the same step when you resume, or stop it and continue from the last checkpoint.

This guide trains a 25M-parameter GPT on Shakespeare with PyTorch on one `l4` GPU. PyTorch is
already installed, and the whole run takes about 10 minutes.

**1. Create a GPU machine**

```bash theme={null}
instabox create --name train --gpu l4
```

**2. Copy the training script**

Save this script as `train.py` on your computer. It downloads the text, trains the model, prints
a line every 50 steps, and saves a checkpoint to `ckpt.pt` every 200 steps. When `ckpt.pt`
exists, the script continues from it.

```python train.py expandable theme={null}
"""Train a small GPT on Shakespeare, one character at a time, on the machine's GPU.

  python train.py            train, and save a checkpoint every 200 steps
  python train.py --sample   write text with the saved checkpoint
"""
import os, sys, time, urllib.request, warnings
warnings.filterwarnings('ignore')
import torch, torch.nn as nn, torch.nn.functional as F

URL = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'
if not os.path.exists('input.txt'):
    urllib.request.urlretrieve(URL, 'input.txt')
text = open('input.txt').read()
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text], device='cuda')
V, T, B, D, H, L = len(chars), 256, 96, 512, 8, 8


class Block(nn.Module):
    def __init__(self):
        super().__init__()
        self.n1, self.n2 = nn.LayerNorm(D), nn.LayerNorm(D)
        self.qkv, self.proj = nn.Linear(D, 3 * D), nn.Linear(D, D)
        self.mlp = nn.Sequential(nn.Linear(D, 4 * D), nn.GELU(), nn.Linear(4 * D, D))

    def forward(self, x):
        b, t, _ = x.shape
        q, k, v = self.qkv(self.n1(x)).view(b, t, 3, H, D // H).permute(2, 0, 3, 1, 4)
        a = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        x = x + self.proj(a.transpose(1, 2).reshape(b, t, D))
        return x + self.mlp(self.n2(x))


class GPT(nn.Module):
    def __init__(self):
        super().__init__()
        self.tok, self.pos = nn.Embedding(V, D), nn.Embedding(T, D)
        self.blocks = nn.Sequential(*[Block() for _ in range(L)])
        self.norm, self.head = nn.LayerNorm(D), nn.Linear(D, V)

    def forward(self, i):
        pos = torch.arange(i.shape[1], device='cuda')
        return self.head(self.norm(self.blocks(self.tok(i) + self.pos(pos))))


model = GPT().cuda()
opt = torch.optim.AdamW(model.parameters(), lr=6e-4)
scaler = torch.amp.GradScaler()
step = 0
if os.path.exists('ckpt.pt'):
    ck = torch.load('ckpt.pt')
    model.load_state_dict(ck['model']); opt.load_state_dict(ck['opt']); step = ck['step']

if '--sample' in sys.argv:
    out = torch.tensor([[stoi['\n']]], device='cuda')
    with torch.no_grad():
        for _ in range(400):
            p = F.softmax(model(out[:, -T:])[:, -1] / 0.8, dim=-1)
            out = torch.cat([out, torch.multinomial(p, 1)], dim=1)
    print(f'step {step}:' + ''.join(chars[i] for i in out[0].tolist()))
    sys.exit()

print(f'{torch.cuda.get_device_name(0)}, {sum(p.numel() for p in model.parameters()) / 1e6:.1f}M '
      f'parameters, process {os.getpid()}, from step {step}', flush=True)
t0, seen = time.time(), 0
while step < 3000:
    ix = torch.randint(len(data) - T - 1, (B,), device='cuda')
    x = torch.stack([data[i:i + T] for i in ix])
    y = torch.stack([data[i + 1:i + T + 1] for i in ix])
    with torch.autocast('cuda', dtype=torch.float16):
        loss = F.cross_entropy(model(x).view(-1, V), y.view(-1))
    opt.zero_grad(set_to_none=True)
    scaler.scale(loss).backward(); scaler.step(opt); scaler.update()
    step += 1; seen += B * T
    if step % 50 == 0:
        dt = time.time() - t0
        print(f'{time.strftime("%H:%M:%S")}  step {step:4d}  loss {loss.item():.3f}  '
              f'{seen / dt:7,.0f} tokens/s', flush=True)
        t0, seen = time.time(), 0
    if step % 200 == 0:
        torch.save({'model': model.state_dict(), 'opt': opt.state_dict(), 'step': step}, 'ckpt.pt')
print('done', flush=True)
```

Copy the script to the machine. It lands in your home directory, `/data/home`.

```bash theme={null}
instabox cp train.py train:
```

**3. Start the training run**

Start the script as a background job, so that it keeps running after the command returns, and
send its output to a log file.

```bash theme={null}
instabox exec --background train -- \
  'python -u train.py > train.log 2>&1'
```

The command prints the job's ID. Read the end of the log to see the run's progress.

```bash theme={null}
instabox exec train -- tail -n 5 train.log
```

```text theme={null}
NVIDIA L4, 25.4M parameters, process 321, from step 0
18:53:34  step   50  loss 2.474  112,532 tokens/s
18:53:44  step  100  loss 2.419  127,608 tokens/s
18:53:53  step  150  loss 2.283  127,403 tokens/s
18:54:03  step  200  loss 2.087  127,009 tokens/s
```

**4. Pause the machine and resume it**

Pause the machine when you want to stop paying for the GPU for a while. A pause saves the
machine's memory, the GPU's memory included, so the training process is kept as it is.

```bash theme={null}
instabox pause train
```

Resume the machine when you want the run to go on.

```bash theme={null}
instabox resume train
```

Read the log again. The run continues with the next step, in the same process.

```bash theme={null}
instabox exec train -- tail -n 4 train.log
```

```text theme={null}
18:54:13  step  250  loss 1.906  123,402 tokens/s
18:54:38  step  300  loss 1.758   50,352 tokens/s
18:54:47  step  350  loss 1.657  125,883 tokens/s
18:54:57  step  400  loss 1.550  125,102 tokens/s
```

The pause took 8 seconds and the resume took 4 seconds. Step 300 shows a lower rate because its
50 steps include the paused time.

**5. Stop the machine and continue from the checkpoint**

A stop shuts the machine down. Its disk stays, so the checkpoint and the log are there when you
start it again, but the training process is gone.

```bash theme={null}
instabox stop train
```

```bash theme={null}
instabox start train
```

Start the script again. It finds `ckpt.pt` and continues from the step that the checkpoint was
saved at.

```bash theme={null}
instabox exec --background train -- \
  'python -u train.py > train.log 2>&1'
```

Read the start of the new log. Its first line says which step the run continued from.

```bash theme={null}
instabox exec train -- head -n 3 train.log
```

```text theme={null}
NVIDIA L4, 25.4M parameters, process 164, from step 200
20:16:53  step  250  loss 1.897  114,569 tokens/s
20:17:03  step  300  loss 1.778  124,244 tokens/s
```

This run was stopped just after step 250, so it continued from the checkpoint of step 200.

**6. Use the trained model**

The run ends at step 3000 and prints `done`. Then ask the model to write some text from the
checkpoint.

```bash theme={null}
instabox exec train -- python train.py --sample
```

```text theme={null}
step 3000:
Which longs and pulinant I spare my hand, my lord,
Thou hadst bewraven Margaret march and freely.

WARWICK:
What comfort me? O, no! the tribunes calls
```

To keep the checkpoint, copy it to your computer.

```bash theme={null}
instabox cp train:ckpt.pt ckpt.pt
```

**7. Delete the machine**

Delete the machine when you no longer need it. Its disk, with the checkpoint, is deleted too.

```bash theme={null}
instabox rm train --yes
```

## Pause or stop

| | Pause | Stop |
| - | - | - |
| GPU billing | Stops | Stops |
| Disk, with checkpoints and data | Kept | Kept |
| Running processes | Kept, and continue on resume | Ended |
| GPU memory, with the loaded model | Kept | Lost |

An `h100` machine cannot be paused, so on that GPU type use checkpoints.

> [!NOTE]
>
> A machine pauses by itself after 30 idle minutes by default, and a background job
> with no open connection can count as idle. The run is kept and continues on resume. To keep a long
> run going, turn the idle pause off for the machine. See [Auto-pause](https://instabox.sh/docs/machines/auto-pause.md).

## Train your own model

The steps are the same for your own code: push your project with `instabox push`, install its
packages, and start your training command as a background job. Keep datasets and checkpoints under
`/data/home`, which is the machine's large disk.

> [card] [GPU machines](https://instabox.sh/docs/machines/gpu.md)
>
> GPU types and counts, and what the template includes.

> [card] [Run commands](https://instabox.sh/docs/machines/run-commands.md)
>
> Background jobs, timeouts, and following output.
