> ## Documentation Index
> Fetch the complete documentation index at: https://instabox.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Serve an open model

> Serve an open LLM with vLLM on a GPU machine and call it with any OpenAI client, from the machine or from your computer.

Serve an open-weights model on a GPU machine and call it like a hosted model API. This guide serves
[Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) with [vLLM](https://docs.vllm.ai) on one `l4` GPU
and ends with an OpenAI-compatible HTTPS endpoint that answers only requests with your token.

You need the CLI installed and signed in ([Quickstart](https://instabox.sh/docs/quickstart.md)), a paid plan, and `jq` and
Python 3 on your computer.

## Start the model

**1. Create a GPU machine**

```bash theme={null}
instabox create --name llm --gpu l4
```

**2. Install vLLM**

Install vLLM in a virtual environment of its own, as a background job. The command prints the
job's ID.

```bash theme={null}
instabox exec --background llm -- bash -c \
  'uv venv -p 3.12 /data/home/vllm && uv pip install -p /data/home/vllm vllm'
```

**3. Wait for the install**

Follow the job's output with that ID. The command returns when the install ends, after about a
minute.

```bash theme={null}
instabox jobs logs llm <job-id> -f
```

**4. Start the model server**

Start vLLM as a background job. The first start downloads the model's weights, about 16 GB.

```bash theme={null}
instabox exec --background llm -- \
  /data/home/vllm/bin/vllm serve Qwen/Qwen3-8B \
  --served-model-name qwen3-8b --max-model-len 16384 \
  --reasoning-parser qwen3
```

**5. Wait until the server is ready**

A start takes 3 to 4 minutes. This command returns when the server answers on port 8000.

```bash theme={null}
instabox exec llm --timeout 600 -- bash -c \
  'until curl -sf localhost:8000/health; do sleep 5; done; echo ready'
```

```text theme={null}
ready
```

**6. Ask the model a question**

Send a chat request from inside the machine, and print the answer with `jq`. Qwen3 thinks before
it answers by default, and `enable_thinking` turns that off.

```bash theme={null}
instabox exec llm --timeout 120 -- \
  curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3-8b",
       "messages": [{"role": "user", "content": "Name three uses of a GPU."}],
       "chat_template_kwargs": {"enable_thinking": false}}' \
  | jq -r '.choices[0].message.content'
```

```text theme={null}
1. **Graphics Rendering**: GPUs are primarily used to render images, videos, and animations for gaming, movies, and graphic design.
...
```

## Call the model from your computer

Publish the server's port as a private port, which answers only requests that carry its token. Run
these steps in one terminal, because they share two variables.

**1. Publish the port**

The command saves the URL and the token to a file. The token is shown only once.

```bash theme={null}
instabox port publish llm 8000 --private --json > llm-port.json
```

**2. Read the URL and the token**

```bash theme={null}
export LLM_URL=$(jq -r .url llm-port.json)
```

```bash theme={null}
export LLM_TOKEN=$(jq -r .token llm-port.json)
```

**3. Call the endpoint**

Send the token as a bearer token.

```bash theme={null}
curl -s $LLM_URL/v1/chat/completions \
  -H "Authorization: Bearer $LLM_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3-8b",
       "messages": [{"role": "user", "content": "Hello"}],
       "chat_template_kwargs": {"enable_thinking": false}}' \
  | jq -r '.choices[0].message.content'
```

```text theme={null}
Hello! How can I assist you today? 😊
```

**4. Check that the endpoint is private**

A request without the token gets `401`.

```bash theme={null}
curl -s -o /dev/null -w '%{http_code}\n' $LLM_URL/v1/models
```

```text theme={null}
401
```

**5. Install the OpenAI client**

OpenAI clients send the API key as a bearer token, so the port's token works as the API key.
Make a Python environment for the client.

```bash theme={null}
python3 -m venv venv
```

```bash theme={null}
venv/bin/pip install openai
```

**6. Call the model from Python**

Save this script as `chat.py`.

```python chat.py theme={null}
import os
from openai import OpenAI

client = OpenAI(base_url=os.environ["LLM_URL"] + "/v1",
                api_key=os.environ["LLM_TOKEN"])
reply = client.chat.completions.create(
    model="qwen3-8b",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(reply.choices[0].message.content)
```

Run it.

```bash theme={null}
venv/bin/python chat.py
```

```text theme={null}
Silicon whispers,
Cores blaze in silent dance—
Power in a chip.
```

> [!WARNING]
>
> The port answers `502` when the server has sent nothing for about 20 seconds. For a long answer,
> or an answer with thinking on, ask for a streamed response (`stream=True` in the OpenAI client),
> so that the first tokens arrive at once.

## Pause the machine

Pause the machine when you do not need the model for a while. The server keeps the model loaded,
and a paused machine does not pay for its GPU. In this run the pause took 50 seconds.

```bash theme={null}
instabox pause llm
```

Resume it to use the model again. The same URL and token answer about 12 seconds later.

```bash theme={null}
instabox resume llm
```

## Stop and start the machine

A stop shuts the machine down. The weights and the vLLM install stay on its disk, and the port
keeps its URL and its token.

```bash theme={null}
instabox stop llm
```

```bash theme={null}
instabox start llm
```

After a start, run the commands of the steps "Start the model server" and "Wait until the server is
ready" again. The server is ready after 2 to 3 minutes, with no download.

## Clean up

Remove the URL, and delete the machine with its disk.

```bash theme={null}
instabox port unpublish llm 8000
```

```bash theme={null}
instabox rm llm --yes
```

> [accordion] Run a larger model
>
> A model needs about 2 GB of GPU memory per billion parameters in 16-bit precision, plus room for
> its cache. To run a model that does not fit one GPU, create the machine with more GPUs and split
> the model across them with `--tensor-parallel-size`.
>
> ```bash theme={null}
> instabox create --name llm-2 --gpu l4:2
> ```
>
> ```bash theme={null}
> instabox exec --background llm-2 -- \
>   /data/home/vllm/bin/vllm serve <model> --tensor-parallel-size 2
> ```

> [card] [Run batch jobs](https://instabox.sh/docs/use-cases/gpu/batch-jobs.md)
>
> Run a model over a large file of prompts.

> [card] [Public and private ports](https://instabox.sh/docs/access/ports.md)
>
> Rotate a port's token, make it public, or unpublish it.
