知識庫

When running multiple servers, checking logs individually on each one is slow and error-prone during an incident. Centralized logging collects logs from all your servers into one searchable location.

Why Centralize Logs

• Search across all servers from one place during an incident
• Logs survive even if the originating server is compromised or destroyed
• Correlate events across multiple services more easily
• Retain logs longer than local disk space would otherwise allow

Option 1 — rsyslog Forwarding (Simple, Built-In)

On the central log server, configure rsyslog to accept remote logs:

sudo nano /etc/rsyslog.conf

module(load="imtcp")
input(type="imtcp" port="514")

sudo systemctl restart rsyslog

Allow the port:

sudo ufw allow from YOUR_APP_SERVER_IP to any port 514

On each application server, configure rsyslog to forward logs:

sudo nano /etc/rsyslog.conf

*.* @@YOUR_LOG_SERVER_IP:514

sudo systemctl restart rsyslog

Option 2 — Docker Logging Driver to a Central Destination

For containerized applications, configure the syslog driver directly:

services:
  web:
    image: myapp
    logging:
      driver: syslog
      options:
        syslog-address: "udp://YOUR_LOG_SERVER_IP:514"

Option 3 — Grafana Loki (Modern, Purpose-Built for Logs)

Loki is designed to work alongside Grafana/Prometheus for a unified observability stack.

docker run -d --name loki -p 3100:3100 grafana/loki:latest

Install Promtail on each server to ship logs to Loki:

docker run -d --name promtail \
  -v /var/log:/var/log \
  grafana/promtail:latest \
  -config.file=/etc/promtail/config.yml

Add Loki as a data source in Grafana to search and visualize logs alongside your existing metrics dashboards.

Searching Centralized Logs

With rsyslog-based centralization, logs typically land in /var/log/ on the central server, searchable with standard tools:

grep "ERROR" /var/log/remote/app-server-1.log

With Loki, use Grafana's Explore view with LogQL queries for more powerful filtering.

Securing Log Transmission

Plain rsyslog forwarding over UDP/TCP isn't encrypted by default — for sensitive log data, use TLS-enabled rsyslog forwarding, or tunnel traffic through a private network/VPN (see How to Set Up a Private Network Between Multiple VPS Instances).

Log Retention Planning

Set explicit retention policies on the central log server to avoid unbounded disk growth — rotate and archive or delete logs older than your required retention period.

Common Errors

Logs not arriving at the central server — verify the firewall allows the log port between servers, and check rsyslog's own error output: sudo journalctl -u rsyslog.

Central log server disk filling up — implement log rotation and retention policies proactively, not after the disk is already full.

Best Practices

• Transmit logs over a private network or encrypted connection, not plain UDP over the public internet
• Set explicit retention policies on the central server
• Include enough context (hostname, service name) in forwarded logs to identify the source at a glance

Continue Reading

• How to Read and Analyze Linux Logs with journalctl
• How to Set Up a Private Network Between Multiple VPS Instances
• Managing Docker Logs

Browse more articles in Performance & Monitoring.