Whether self-hosting AI infrastructure or calling third-party APIs, tracking usage and cost is essential for both budgeting and identifying optimization opportunities. This guide covers setting up practical AI usage monitoring.
Why AI Usage Monitoring Deserves Dedicated Attention
AI inference costs (whether compute time for self-hosted, or per-token API costs for third-party services) can scale unpredictably with usage patterns — without deliberate monitoring, cost surprises and capacity issues are genuinely more likely than with typical, more predictable infrastructure.
Tracking Self-Hosted Inference Compute Usage
import time
def logged_inference(prompt, model):
start = time.time()
response = model.generate(prompt)
duration = time.time() - start
log_metric("inference_duration_seconds", duration)
log_metric("prompt_tokens", count_tokens(prompt))
log_metric("completion_tokens", count_tokens(response))
return responseSee How to Set Up Prometheus and Grafana for VPS Monitoring for the base monitoring infrastructure this feeds into — track duration and token counts as core metrics for understanding actual resource consumption.
Tracking Third-Party API Costs
def logged_api_call(prompt, model="gpt-4"):
response = openai_client.chat.completions.create(model=model, messages=[{"role": "user", "content": prompt}])
cost = calculate_cost(response.usage.prompt_tokens, response.usage.completion_tokens, model)
log_metric("api_cost_dollars", cost)
return responseMost third-party AI API responses include token usage data — calculate and log actual cost per request based on your provider's current pricing, building a genuine picture of spend attribution.
Attributing Usage to Specific Users/Features
log_metric("inference_cost", cost, labels={"user_id": user_id, "feature": "chatbot"})Beyond aggregate totals, per-user and per-feature attribution reveals which parts of your application are actually driving cost — important for informed decisions about where optimization effort is genuinely warranted.
Building a Cost Dashboard
See How to Build a Monitoring Dashboard for Your Whole Team — a dedicated AI cost/usage dashboard, showing trends over time and breakdown by feature/user, gives your team visibility into what's often a significant and variable cost category.
Setting Up Budget Alerts
See How to Set Up Effective Server Alerting (Without Alert Fatigue) — alert when spending trends toward exceeding a defined budget threshold, giving you advance warning rather than discovering an unexpected large bill after the fact.
Comparing Self-Hosted vs API Cost Over Time
Track both approaches' effective cost (self-hosted infrastructure cost amortized per request, vs per-token API pricing) to periodically reassess which approach is genuinely more cost-effective as your usage volume/patterns evolve.
Identifying Optimization Opportunities from Usage Data
Usage data often reveals optimization opportunities — a high volume of similar/repeated queries suggests caching (see How to Cache LLM Responses to Reduce Compute Costs) would be valuable; unusually long prompts might warrant investigation for unnecessary verbosity.
Monitoring for Anomalous Usage (Potential Abuse)
See How to Rate Limit and Secure a Self-Hosted AI API — unusual usage spikes can indicate abuse of an insufficiently protected endpoint; usage monitoring serves both cost-management and security-relevant purposes simultaneously.
Common Errors
Actual bill differs significantly from tracked usage estimates — verify your cost calculation logic uses genuinely current pricing (provider pricing changes periodically) and correctly accounts for all billable components (some providers bill prompt and completion tokens at different rates).
Continue Reading
- How to Cache LLM Responses to Reduce Compute Costs
- How to Set Up Prometheus and Grafana for VPS Monitoring
- How to Rate Limit and Secure a Self-Hosted AI API
Browse more articles in AI & Machine Learning on a VPS.
