Whisper is an open-source speech recognition model capable of accurate transcription across many languages. This guide covers running it on a VPS for self-hosted, private speech-to-text.
Why Self-Host Speech-to-Text
- Keep audio data private, never sent to a third-party API
- Avoid per-request costs of commercial transcription APIs at high volume
- Full control over the specific model version and configuration
Prerequisites
- Python installed
- ffmpeg for audio processing
Step 1 — Install ffmpeg
sudo apt install ffmpeg -yStep 2 — Install Whisper
pip install openai-whisper --break-system-packagesDespite the package name referencing OpenAI, this runs entirely locally — no API calls or external dependency once installed.
Step 3 — Transcribe an Audio File
whisper audio-file.mp3 --model baseProduces a text transcript, along with subtitle-format outputs (.srt, .vtt) by default.
Choosing a Model Size
| Model | Speed | Accuracy | RAM Needed |
|---|---|---|---|
| tiny | Fastest | Lower | ~1 GB |
| base | Fast | Good balance | ~1 GB |
| small | Moderate | Better | ~2 GB |
| medium | Slower | High | ~5 GB |
| large | Slowest | Highest | ~10 GB |
Using Whisper Programmatically (Python)
import whisper
model = whisper.load_model("base")
result = model.transcribe("audio-file.mp3")
print(result["text"])Specifying a Language (Improves Accuracy and Speed)
result = model.transcribe("audio-file.mp3", language="en")Whisper auto-detects language by default, but specifying it explicitly (if known) skips detection and can slightly improve accuracy.
Building a Simple Transcription API
from fastapi import FastAPI, UploadFile
import whisper
import shutil
app = FastAPI()
model = whisper.load_model("base")
@app.post("/transcribe")
async def transcribe(file: UploadFile):
with open("temp_audio.mp3", "wb") as f:
shutil.copyfileobj(file.file, f)
result = model.transcribe("temp_audio.mp3")
return {"text": result["text"]}See How to Serve a Machine Learning Model with FastAPI for deploying this as a complete production API.
Processing Time Expectations
On CPU-only hardware, transcription typically takes longer than the audio's actual duration, especially with larger models — test with your specific hardware and model choice before assuming real-time or near-real-time performance is achievable.
Common Errors
"ffmpeg not found" — Whisper depends on ffmpeg for audio decoding; verify it's installed and accessible in the system PATH.
Transcription is very slow — try a smaller model size, or verify sufficient RAM is available without swapping to disk.
Continue Reading
- How to Serve a Machine Learning Model with FastAPI
- VPS Requirements for Running AI and Machine Learning Workloads
- How to Install Python on Ubuntu & Debian
Browse more articles in AI & Machine Learning on a VPS.
