How to Set Up Centralized Logging with the ELK Stack (Elasticsearch, Logstash, Kibana)

מאגר מידע

The ELK Stack (Elasticsearch, Logstash, Kibana) is a mature, powerful centralized logging solution — letting you collect, search, and visualize logs from multiple servers/services in one place rather than SSHing into individual machines.

What Each Component Does

Elasticsearch — stores and indexes log data, enabling fast search
Logstash — collects, parses, and forwards logs into Elasticsearch
Kibana — the web interface for searching, visualizing, and building dashboards from log data

Prerequisites

Ubuntu 22.04/24.04 VPS: 4 vCPU, 8 GB RAM minimum (Elasticsearch is resource-intensive)
Docker installed (simplifies deployment considerably)

Step 1 — Run Elasticsearch

docker run -d \
  --name elasticsearch \
  --restart unless-stopped \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -v elasticsearch-data:/usr/share/elasticsearch/data \
  docker.elastic.co/elasticsearch/elasticsearch:8.13.0

Disabling security simplifies initial setup for a single-node evaluation; enable proper authentication before any production use handling sensitive log data.

Step 2 — Verify Elasticsearch Is Running

curl http://localhost:9200

Step 3 — Run Kibana

docker run -d \
  --name kibana \
  --restart unless-stopped \
  -p 5601:5601 \
  -e "ELASTICSEARCH_HOSTS=http://elasticsearch:9200" \
  --link elasticsearch \
  docker.elastic.co/kibana/kibana:8.13.0

Step 4 — Access Kibana

http://YOUR_SERVER_IP:5601

Step 5 — Install Filebeat on Servers You Want to Collect Logs From

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.13.0-amd64.deb
sudo dpkg -i filebeat-8.13.0-amd64.deb

Filebeat is a lightweight log shipper — install it on each server whose logs you want centralized, rather than running the full ELK stack on every machine.

Step 6 — Configure Filebeat

sudo nano /etc/filebeat/filebeat.yml

filebeat.inputs:
  - type: log
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log

output.elasticsearch:
  hosts: ["YOUR_ELK_SERVER_IP:9200"]

Step 7 — Start Filebeat

sudo systemctl enable --now filebeat

Step 8 — Create an Index Pattern in Kibana

In Kibana, under Stack Management → Index Patterns, create a pattern matching your Filebeat indices to start exploring the collected logs.

Step 9 — Explore Logs in Kibana

Use the Discover section to search and filter logs across all connected servers, and build dashboards under Visualize for recurring monitoring needs.

Securing the ELK Stack

Never leave Elasticsearch/Kibana exposed to the public internet without authentication — enable Elasticsearch's built-in security features, and restrict network access to trusted sources only.

Common Errors

Elasticsearch fails to start ("max virtual memory areas too low") — a common Linux kernel setting issue; increase vm.max_map_count per Elasticsearch's official troubleshooting documentation.

No logs appearing in Kibana — verify Filebeat is running and successfully connecting to Elasticsearch (sudo systemctl status filebeat, check Filebeat's own logs for connection errors).

Continue Reading

Browse more articles in Advanced Observability & Incident Management.

מאגר מידע

קטגוריות

קטגוריות