How to Set Up Whisper for Self-Hosted Speech-to-Text

Whisper is an open-source speech recognition model capable of accurate transcription across many languages. This guide covers running it on a VPS for self-hosted, private speech-to-text.

Why Self-Host Speech-to-Text

Keep audio data private, never sent to a third-party API
Avoid per-request costs of commercial transcription APIs at high volume
Full control over the specific model version and configuration

Prerequisites

Python installed
ffmpeg for audio processing

Step 1 — Install ffmpeg

sudo apt install ffmpeg -y

Step 2 — Install Whisper

pip install openai-whisper --break-system-packages

Despite the package name referencing OpenAI, this runs entirely locally — no API calls or external dependency once installed.

Step 3 — Transcribe an Audio File

whisper audio-file.mp3 --model base

Produces a text transcript, along with subtitle-format outputs (.srt, .vtt) by default.

Choosing a Model Size

Model	Speed	Accuracy	RAM Needed
tiny	Fastest	Lower	~1 GB
base	Fast	Good balance	~1 GB
small	Moderate	Better	~2 GB
medium	Slower	High	~5 GB
large	Slowest	Highest	~10 GB

Using Whisper Programmatically (Python)

import whisper

model = whisper.load_model("base")
result = model.transcribe("audio-file.mp3")
print(result["text"])

Specifying a Language (Improves Accuracy and Speed)

result = model.transcribe("audio-file.mp3", language="en")

Whisper auto-detects language by default, but specifying it explicitly (if known) skips detection and can slightly improve accuracy.

Building a Simple Transcription API

from fastapi import FastAPI, UploadFile
import whisper
import shutil

app = FastAPI()
model = whisper.load_model("base")

@app.post("/transcribe")
async def transcribe(file: UploadFile):
    with open("temp_audio.mp3", "wb") as f:
        shutil.copyfileobj(file.file, f)
    result = model.transcribe("temp_audio.mp3")
    return {"text": result["text"]}

See How to Serve a Machine Learning Model with FastAPI for deploying this as a complete production API.

Processing Time Expectations

On CPU-only hardware, transcription typically takes longer than the audio's actual duration, especially with larger models — test with your specific hardware and model choice before assuming real-time or near-real-time performance is achievable.

Common Errors

"ffmpeg not found" — Whisper depends on ffmpeg for audio decoding; verify it's installed and accessible in the system PATH.

Transcription is very slow — try a smaller model size, or verify sufficient RAM is available without swapping to disk.

Continue Reading

Browse more articles in AI & Machine Learning on a VPS.

مكتبة الشروحات

أقسام الشروحات

أقسام الشروحات