Knowledgebase

Redis, primarily known as a cache, also works well as a lightweight message queue for simpler use cases — avoiding the operational overhead of a dedicated broker like RabbitMQ when your queuing needs are modest.

Redis as a Queue vs a Dedicated Message Broker

Prerequisites

• Redis installed — see How to Install and Secure Redis on Ubuntu & Debian

Simple Queue Pattern with Redis Lists

Producer (adding a job to the queue):

const redis = require('redis');
const client = redis.createClient();

await client.lPush('job_queue', JSON.stringify({ task: 'send_email', to: '[email protected]' }));

Consumer (processing jobs):

while (true) {
  const result = await client.brPop('job_queue', 0);
  const job = JSON.parse(result.element);
  await processJob(job);
}

brPop blocks until a job is available, rather than continuously polling — efficient for a simple worker loop.

Using a Dedicated Job Queue Library (Recommended for Production)

Rather than implementing queue logic manually, use a mature library built on Redis — for Node.js, BullMQ is a popular choice offering retries, delayed jobs, and monitoring out of the box:

npm install bullmq

const { Queue, Worker } = require('bullmq');

const emailQueue = new Queue('emails', { connection: { host: 'localhost', port: 6379 } });

await emailQueue.add('send-welcome', { to: '[email protected]' });

new Worker('emails', async (job) => {
  console.log('Processing:', job.data);
}, { connection: { host: 'localhost', port: 6379 } });

For Python, Celery with Redis as the Broker

pip install celery redis

# tasks.py
from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def send_email(to):
    print(f"Sending email to {to}")

celery -A tasks worker --loglevel=info

Handling Failed Jobs

Configure retry logic and a dead-letter mechanism (a separate list/queue for jobs that repeatedly fail) so failures are visible and can be investigated, rather than silently lost:

await emailQueue.add('send-welcome', { to: '[email protected]' }, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 1000 },
});

Monitoring the Queue

redis-cli LLEN job_queue

For BullMQ specifically, a dashboard UI (Bull Board) can visualize queue status, job history, and failures.

When to Upgrade from Redis to a Dedicated Broker

• You need guaranteed delivery with acknowledgments and complex routing
• Message volume grows large enough that Redis's simpler model becomes a bottleneck
• You need features like message priority, delayed delivery at scale, or multi-consumer patterns beyond what your chosen Redis-based library supports

Common Errors

Jobs processed twice — if running multiple workers without proper coordination, ensure your queue library correctly handles atomic job claiming (built-in libraries like BullMQ handle this correctly; naive manual implementations often don't).

Jobs lost on Redis restart — ensure Redis persistence (RDB or AOF) is enabled if job durability matters; see How to Install and Secure Redis on Ubuntu & Debian.

Best Practices

• Use a mature queue library rather than implementing queue logic manually
• Configure retry logic and monitor for failed jobs
• Enable Redis persistence if job loss on restart is unacceptable

Continue Reading

• How to Install and Secure Redis on Ubuntu & Debian
• Message Queue Basics: When and Why to Use One
• How to Install and Configure RabbitMQ on a VPS

Browse more articles in Object Storage, Messaging & APIs.