> ## Documentation Index
> Fetch the complete documentation index at: https://instabox.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Transcribe audio and describe video

> Transcribe speech with Whisper and describe a video with a vision model, on one GPU machine, without sending your files to another service.

Run speech and vision models over your own audio and video files on a GPU machine, without sending
the files to another service. This guide does both jobs on one `l4` GPU: it transcribes speech
with [Whisper](https://huggingface.co/openai/whisper-large-v3-turbo), and it describes a video with
[Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct).

## Transcribe speech

Whisper turns speech into text, with a timestamp for each part.

**1. Create a GPU machine**

```bash theme={null}
instabox create --name video --gpu l4
```

**2. Install ffmpeg and Transformers**

Install `ffmpeg` with apt. This takes about a minute.

```bash theme={null}
instabox exec video --timeout 300 -- bash -c \
  'sudo apt-get update && sudo apt-get install -y ffmpeg'
```

Then install Hugging Face Transformers in `/opt/venv`.

```bash theme={null}
instabox exec video --timeout 300 -- /opt/venv/bin/pip install transformers
```

**3. Copy the script**

Save this script as `transcribe.py` on your computer. It loads Whisper on the GPU, transcribes
the file that you name, and prints each part with its start time.

```python transcribe.py theme={null}
import sys
from transformers import pipeline

asr = pipeline("automatic-speech-recognition",
               model="openai/whisper-large-v3-turbo", device="cuda")
result = asr(sys.argv[1], return_timestamps=True)
for chunk in result["chunks"]:
    print(f"[{chunk['timestamp'][0]:7.1f}s] {chunk['text'].strip()}")
```

Copy it to the machine.

```bash theme={null}
instabox cp transcribe.py video:/data/home/transcribe.py
```

**4. Transcribe a recording**

Download a sample recording, a part of a speech by Martin Luther King Jr.

```bash theme={null}
instabox exec video --cwd /data/home -- curl -fsSLo mlk.flac \
  https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac
```

Run the script on it. The first run downloads Whisper.

```bash theme={null}
instabox exec video --cwd /data/home --timeout 300 -- \
  /opt/venv/bin/python transcribe.py mlk.flac
```

The output is the speech as text, one line for each part, with its start time in seconds. This
13-second recording is one part.

```text theme={null}
[    1.0s] I have a dream that one day this nation will rise up and live out the true meaning of its creed.
```

The first run took 28 seconds with the download, and the second run took 11 seconds.

**5. Transcribe the sound of a video**

To transcribe a video of your own, copy it to the machine.

```bash theme={null}
instabox cp ./clip.mp4 video:/data/home/clip.mp4
```

Take the sound out of the video as a WAV file first, because Whisper cannot read every MP4
directly.

```bash theme={null}
instabox exec video --cwd /data/home -- \
  ffmpeg -y -i clip.mp4 -vn -ac 1 -ar 16000 clip.wav
```

Then transcribe the WAV file.

```bash theme={null}
instabox exec video --cwd /data/home --timeout 300 -- \
  /opt/venv/bin/python transcribe.py clip.wav
```

## Describe a video

A vision-language model answers a question about a video. vLLM serves the model with the OpenAI
API, so you send the video in a chat request. These steps use the same machine.

**1. Install vLLM**

Install vLLM in a virtual environment of its own under `/data/home`. The install takes about a
minute, so run it as a background job.

```bash theme={null}
instabox exec --background video -- bash -c \
  'uv venv -p 3.12 /data/home/vllm && uv pip install -p /data/home/vllm vllm'
```

Follow the job's output until it ends, with the job ID that the command printed.

```bash theme={null}
instabox jobs logs video <job-id> -f
```

**2. Put a video on the machine**

This guide uses a 10-second clip of a jellyfish. Download it to the machine.

```bash theme={null}
instabox exec video --cwd /data/home -- curl -fsSLo jellyfish.mp4 \
  https://test-videos.co.uk/vids/jellyfish/mp4/h264/360/Jellyfish_360_10s_1MB.mp4
```

**3. Start the vision model**

Start vLLM with the model as a background job. `--allowed-local-media-path` lets a request name
a file under `/data/home`, and `--gpu-memory-utilization 0.6` leaves GPU memory free for Whisper.

```bash theme={null}
instabox exec --background video -- \
  /data/home/vllm/bin/vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
  --served-model-name qwen-vl --max-model-len 16384 \
  --gpu-memory-utilization 0.6 --allowed-local-media-path /data/home
```

Wait until the server answers on port 8000. The first start takes 3 to 4 minutes.

```bash theme={null}
instabox exec video --timeout 600 -- bash -c \
  'until curl -sf localhost:8000/health; do sleep 5; done; echo ready'
```

**4. Ask about the video**

Save this request as `describe.json` on your computer. It sends the video as a `video_url` part,
next to the question.

```json describe.json theme={null}
{
  "model": "qwen-vl",
  "max_tokens": 400,
  "messages": [{
    "role": "user",
    "content": [
      {"type": "video_url", "video_url": {"url": "file:///data/home/jellyfish.mp4"}},
      {"type": "text", "text": "Describe what happens in this video."}
    ]
  }]
}
```

Send the request to the server with `--stdin`, and print the answer with `jq`.

```bash theme={null}
instabox exec video --stdin --timeout 120 -- \
  curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' -d @- \
  < describe.json | jq -r '.choices[0].message.content'
```

The answer comes back in a few seconds.

```text theme={null}
The video shows a close-up of a jellyfish swimming in the ocean. The jellyfish has a yellowish body with red tentacles and white spots on its surface. The jellyfish is moving slowly through the water, its tentacles trailing behind it. The background is dark blue, indicating that the jellyfish is in deep water.
```

To ask about another file, change the `url` to its path under `/data/home`.

> [!WARNING]
>
> Check a model's answers on your own files before you rely on them. A small vision model can
> describe things that are not in a clip, and it does so more often when the clip shows little.

## What it uses

Both models fit one `l4` at the same time: together they used 14.4 GB of its 24 GiB at their peak.
For a large library, split the files across several GPU machines, as in
[Run batch jobs](https://instabox.sh/docs/use-cases/gpu/batch-jobs.md).

## Stop the machine

Stop the machine when you are done. The models, `ffmpeg`, and your files stay on its disk.

```bash theme={null}
instabox stop video
```

After `instabox start video`, start the vision model again with the same `vllm serve` command.

> [card] [Serve an open model](https://instabox.sh/docs/use-cases/gpu/run-a-model.md)
>
> Call a model on the machine from your computer through a private port.

> [card] [Run batch jobs](https://instabox.sh/docs/use-cases/gpu/batch-jobs.md)
>
> Split a large job across many GPU machines.
