l4 GPU
and ends with an OpenAI-compatible HTTPS endpoint that answers only requests with your token.
You need the CLI installed and signed in (Quickstart), a paid plan, and jq and
Python 3 on your computer.
Start the model
1
Create a GPU machine
2
Install vLLM
Install vLLM in a virtual environment of its own, as a background job. The command prints the
job’s ID.
3
Wait for the install
Follow the job’s output with that ID. The command returns when the install ends, after about a
minute.
4
Start the model server
Start vLLM as a background job. The first start downloads the model’s weights, about 16 GB.
5
Wait until the server is ready
A start takes 3 to 4 minutes. This command returns when the server answers on port 8000.
6
Ask the model a question
Send a chat request from inside the machine, and print the answer with
jq. Qwen3 thinks before
it answers by default, and enable_thinking turns that off.Call the model from your computer
Publish the server’s port as a private port, which answers only requests that carry its token. Run these steps in one terminal, because they share two variables.1
Publish the port
The command saves the URL and the token to a file. The token is shown only once.
2
Read the URL and the token
3
Call the endpoint
Send the token as a bearer token.
4
Check that the endpoint is private
A request without the token gets
401.5
Install the OpenAI client
OpenAI clients send the API key as a bearer token, so the port’s token works as the API key.
Make a Python environment for the client.
6
Call the model from Python
Save this script as Run it.
chat.py.chat.py
Pause the machine
Pause the machine when you do not need the model for a while. The server keeps the model loaded, and a paused machine does not pay for its GPU. In this run the pause took 50 seconds.Stop and start the machine
A stop shuts the machine down. The weights and the vLLM install stay on its disk, and the port keeps its URL and its token.Clean up
Remove the URL, and delete the machine with its disk.Run a larger model
Run a larger model
A model needs about 2 GB of GPU memory per billion parameters in 16-bit precision, plus room for
its cache. To run a model that does not fit one GPU, create the machine with more GPUs and split
the model across them with
--tensor-parallel-size.Run batch jobs
Run a model over a large file of prompts.
Public and private ports
Rotate a port’s token, make it public, or unpublish it.