Returning an entire dataset in a single API response doesn't scale — pagination breaks large result sets into manageable pages. This guide covers the common pagination approaches and how to choose between them.
Why Pagination Is Essential
Without pagination, an endpoint returning "all records" becomes progressively slower and more resource-intensive as your dataset grows, eventually becoming genuinely unusable — pagination is a foundational, not optional, API design consideration for any endpoint returning a list.
Offset-Based Pagination (Simplest, Most Common)
GET /api/products?limit=20&offset=40SELECT * FROM products LIMIT 20 OFFSET 40;Simple to implement and understand — but has a genuine performance problem at scale: a large offset requires the database to scan and discard many rows before reaching the requested page, becoming progressively slower for later pages.
Cursor-Based Pagination (Better for Large Datasets)
GET /api/products?limit=20&after=eyJpZCI6MTIzfQ==SELECT * FROM products WHERE id > 123 ORDER BY id LIMIT 20;Uses a reference point (often an encoded ID or timestamp) rather than a numeric offset — avoids the offset performance problem entirely, since the database can seek directly to the cursor position rather than scanning/discarding preceding rows.
Comparing the Two Approaches
| Approach | Pros | Cons |
|---|---|---|
| Offset-based | Simple, supports jumping to arbitrary page numbers | Slow for large offsets, can show duplicates/gaps if data changes between requests |
| Cursor-based | Consistent performance at any scale, stable results even with concurrent data changes | Can't jump to an arbitrary page number, slightly more complex to implement |
Including Pagination Metadata in Responses
{
"data": [...],
"pagination": {
"next_cursor": "eyJpZCI6MTQzfQ==",
"has_more": true
}
}Include enough metadata for clients to easily request the next page without needing to construct cursor values themselves — a well-designed API returns the exact value/URL needed for the next request.
Setting Sensible Default and Maximum Page Sizes
const limit = Math.min(parseInt(req.query.limit) || 20, 100);Provide a reasonable default (so clients don't need to always specify) and enforce a maximum (preventing a client from requesting an unreasonably large page that strains your server) — both important for API stability.
Documenting Your Pagination Approach
See How to Version an API Without Breaking Existing Clients and general API documentation practices — clearly document which pagination style your API uses and exactly how to use it, since inconsistent or undocumented pagination behavior is a common source of integration friction for API consumers.
Handling "Total Count" Requests Carefully
Providing a total record count alongside paginated results is convenient for clients but can itself be an expensive query on very large tables — consider whether an exact total is genuinely necessary, or whether an approximate/estimated count (or omitting it) is acceptable for your use case.
Common Errors
Duplicate or missing records when using offset pagination with frequently-changing data — a known limitation of offset-based pagination when the underlying dataset changes between page requests; cursor-based pagination is more resilient to this specific issue.
Continue Reading
- How to Build and Secure a REST API on a VPS
- How to Cache API Responses Effectively
- How to Version an API Without Breaking Existing Clients
Browse more articles in Object Storage, Messaging & APIs.
