Apache Kafka is a distributed event streaming platform, well-suited for high-throughput, durable message processing at scale beyond what simpler message queues typically handle. This guide covers basic setup on a VPS.
When Kafka Is the Right Choice
See Message Queue Basics: When and Why to Use One — Kafka specifically excels at high-throughput event streaming with durable, replayable message logs; if your needs are simpler (basic task queuing), RabbitMQ or Redis-based queuing (see How to Install and Configure RabbitMQ on a VPS) may be simpler, more appropriate choices.
Resource Requirements
Kafka is genuinely resource-intensive, particularly for memory and disk I/O — a minimum of 4 vCPU, 8 GB+ RAM is a reasonable starting point for meaningful production use; smaller instances can work for development/testing only.
Step 1 — Install Java
sudo apt install openjdk-17-jre -yStep 2 — Download Kafka
wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0Step 3 — Start Kafka in KRaft Mode (No Longer Requires ZooKeeper)
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/kraft/server.properties
bin/kafka-server-start.sh config/kraft/server.propertiesModern Kafka versions support KRaft mode, eliminating the previously-required separate ZooKeeper dependency — significantly simplifies deployment compared to older Kafka setups.
Step 4 — Create a Topic
bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092Step 5 — Test Producing and Consuming Messages
bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092Running Kafka as a systemd Service
[Unit]
Description=Apache Kafka
After=network.target
[Service]
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
Restart=on-failure
[Install]
WantedBy=multi-user.targetSecuring Kafka
By default, Kafka has no authentication — configure SASL authentication and TLS for any production deployment, particularly if Kafka is reachable beyond localhost; never expose an unauthenticated Kafka broker to the public internet.
Configuring Retention
log.retention.hours=168Kafka retains messages for a configured duration (or size limit) regardless of consumption — unlike traditional queues where consumed messages are typically removed, Kafka's log-based model retains data for the configured window, enabling replay/multiple independent consumers.
Monitoring Kafka
See How to Monitor Message Queue Health and Performance — Kafka exposes JMX metrics suitable for integration with Prometheus (see How to Set Up Prometheus and Grafana for VPS Monitoring) for comprehensive monitoring of throughput, consumer lag, and broker health.
Common Errors
High disk usage growing continuously — review your retention configuration; without appropriate limits, Kafka's durable log storage will continue accumulating disk usage until retention policy or manual cleanup addresses it.
Continue Reading
- Message Queue Basics: When and Why to Use One
- How to Monitor Message Queue Health and Performance
- How to Install and Configure RabbitMQ on a VPS
Browse more articles in Object Storage, Messaging & APIs.
