Skip to main content
Serve an open-weights model on a GPU machine and call it like a hosted model API. This guide serves Qwen3-8B with vLLM on one l4 GPU and ends with an OpenAI-compatible HTTPS endpoint that answers only requests with your token. You need the CLI installed and signed in (Quickstart), a paid plan, and jq and Python 3 on your computer.

Start the model

1

Create a GPU machine

2

Install vLLM

Install vLLM in a virtual environment of its own, as a background job. The command prints the job’s ID.
3

Wait for the install

Follow the job’s output with that ID. The command returns when the install ends, after about a minute.
4

Start the model server

Start vLLM as a background job. The first start downloads the model’s weights, about 16 GB.
5

Wait until the server is ready

A start takes 3 to 4 minutes. This command returns when the server answers on port 8000.
6

Ask the model a question

Send a chat request from inside the machine, and print the answer with jq. Qwen3 thinks before it answers by default, and enable_thinking turns that off.

Call the model from your computer

Publish the server’s port as a private port, which answers only requests that carry its token. Run these steps in one terminal, because they share two variables.
1

Publish the port

The command saves the URL and the token to a file. The token is shown only once.
2

Read the URL and the token

3

Call the endpoint

Send the token as a bearer token.
4

Check that the endpoint is private

A request without the token gets 401.
5

Install the OpenAI client

OpenAI clients send the API key as a bearer token, so the port’s token works as the API key. Make a Python environment for the client.
6

Call the model from Python

Save this script as chat.py.
chat.py
Run it.
The port answers 502 when the server has sent nothing for about 20 seconds. For a long answer, or an answer with thinking on, ask for a streamed response (stream=True in the OpenAI client), so that the first tokens arrive at once.

Pause the machine

Pause the machine when you do not need the model for a while. The server keeps the model loaded, and a paused machine does not pay for its GPU. In this run the pause took 50 seconds.
Resume it to use the model again. The same URL and token answer about 12 seconds later.

Stop and start the machine

A stop shuts the machine down. The weights and the vLLM install stay on its disk, and the port keeps its URL and its token.
After a start, run the commands of the steps “Start the model server” and “Wait until the server is ready” again. The server is ready after 2 to 3 minutes, with no download.

Clean up

Remove the URL, and delete the machine with its disk.
A model needs about 2 GB of GPU memory per billion parameters in 16-bit precision, plus room for its cache. To run a model that does not fit one GPU, create the machine with more GPUs and split the model across them with --tensor-parallel-size.

Run batch jobs

Run a model over a large file of prompts.

Public and private ports

Rotate a port’s token, make it public, or unpublish it.