Skip to main content
Run speech and vision models over your own audio and video files on a GPU machine, without sending the files to another service. This guide does both jobs on one l4 GPU: it transcribes speech with Whisper, and it describes a video with Qwen2.5-VL.

Transcribe speech

Whisper turns speech into text, with a timestamp for each part.
1

Create a GPU machine

2

Install ffmpeg and Transformers

Install ffmpeg with apt. This takes about a minute.
Then install Hugging Face Transformers in /opt/venv.
3

Copy the script

Save this script as transcribe.py on your computer. It loads Whisper on the GPU, transcribes the file that you name, and prints each part with its start time.
transcribe.py
Copy it to the machine.
4

Transcribe a recording

Download a sample recording, a part of a speech by Martin Luther King Jr.
Run the script on it. The first run downloads Whisper.
The output is the speech as text, one line for each part, with its start time in seconds. This 13-second recording is one part.
The first run took 28 seconds with the download, and the second run took 11 seconds.
5

Transcribe the sound of a video

To transcribe a video of your own, copy it to the machine.
Take the sound out of the video as a WAV file first, because Whisper cannot read every MP4 directly.
Then transcribe the WAV file.

Describe a video

A vision-language model answers a question about a video. vLLM serves the model with the OpenAI API, so you send the video in a chat request. These steps use the same machine.
1

Install vLLM

Install vLLM in a virtual environment of its own under /data/home. The install takes about a minute, so run it as a background job.
Follow the job’s output until it ends, with the job ID that the command printed.
2

Put a video on the machine

This guide uses a 10-second clip of a jellyfish. Download it to the machine.
3

Start the vision model

Start vLLM with the model as a background job. --allowed-local-media-path lets a request name a file under /data/home, and --gpu-memory-utilization 0.6 leaves GPU memory free for Whisper.
Wait until the server answers on port 8000. The first start takes 3 to 4 minutes.
4

Ask about the video

Save this request as describe.json on your computer. It sends the video as a video_url part, next to the question.
describe.json
Send the request to the server with --stdin, and print the answer with jq.
The answer comes back in a few seconds.
To ask about another file, change the url to its path under /data/home.
Check a model’s answers on your own files before you rely on them. A small vision model can describe things that are not in a clip, and it does so more often when the clip shows little.

What it uses

Both models fit one l4 at the same time: together they used 14.4 GB of its 24 GiB at their peak. For a large library, split the files across several GPU machines, as in Run batch jobs.

Stop the machine

Stop the machine when you are done. The models, ffmpeg, and your files stay on its disk.
After instabox start video, start the vision model again with the same vllm serve command.

Serve an open model

Call a model on the machine from your computer through a private port.

Run batch jobs

Split a large job across many GPU machines.