L16. Centralized Logging: Shipping Logs Off-Box

Why Centralized Logging Matters

Local logs on a compromised server are unreliable. An attacker with root access can:

• Delete log files entirely
• Edit log entries to remove evidence of their activity
• Disable the logging daemon
• Truncate logs to hide their entry point

Centralized logging solves this by forwarding log data to a separate, hardened log server in real time. Even if the attacker wipes the local disk, the evidence is already stored elsewhere.

Beyond incident response, centralized logging provides:

• Correlation: seeing events across multiple servers in one place
• Retention: meeting compliance requirements for log storage duration
• Alerting: triggering notifications when specific patterns appear

rsyslog Remote Forwarding

The simplest way to centralize logs on Linux is with rsyslog, which supports remote log forwarding out of the box.

Configuring the Log Server (Receiver)

On the server that will receive logs, enable TCP or UDP listening:

# /etc/rsyslog.conf on the LOG SERVER
# Enable UDP reception (port 514)
module(load="imudp")
input(type="imudp" port="514")
# Enable TCP reception (port 514, more reliable)
module(load="imtcp")
input(type="imtcp" port="514")
# Store remote logs in per-host directories
template(name="RemoteLogs" type="string"
  string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
*.* ?RemoteLogs

Restart rsyslog on the log server:

sudo systemctl restart rsyslog

Configuring the Source Server (Sender)

On each server that should forward its logs:

# /etc/rsyslog.conf on the SOURCE SERVER
# Forward all logs to the log server via TCP
*.* @@logserver.example.com:514
# Use @@ for TCP, @ for UDP
# TCP is preferred: it guarantees delivery

Restart rsyslog on the source server:

sudo systemctl restart rsyslog

TCP vs UDP for Log Forwarding

Always use TCP (@@) for security-relevant logs. Losing a single authentication event could mean missing the attacker's initial access.

Securing Log Transport with TLS

By default, rsyslog sends logs in plain text. Anyone on the network path can read them. For production environments, encrypt the transport:

# On the log server: add TLS settings to rsyslog.conf
# Load the TLS module
module(load="imtcp"
  StreamDriver.Name="gtls"
  StreamDriver.Mode="1"
  StreamDriver.Authmode="x509/name"
)
global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/rsyslog-certs/ca.pem"
  DefaultNetstreamDriverCertFile="/etc/rsyslog-certs/server-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog-certs/server-key.pem"
)
input(type="imtcp" port="6514")

TLS-encrypted syslog typically uses port 6514 instead of 514.

The ELK Stack Concept

For organizations that need more than simple log storage, the ELK stack (Elasticsearch, Logstash, Kibana) provides a full log analytics platform:

The ELK stack is powerful but resource-intensive. A small deployment typically requires at least 8 GB of RAM for Elasticsearch alone. It is worth the investment when you have more than a handful of servers or need complex log analysis.

Lightweight Alternatives: Fluent Bit and Fluentd

If the ELK stack is too heavy for your environment, consider lighter-weight log shippers:

Fluent Bit

Fluent Bit is an extremely lightweight log forwarder (a few MB of memory). It can collect logs from systemd journal, syslog, and files, then forward them to various destinations.

# /etc/fluent-bit/fluent-bit.conf
[INPUT]
    Name        systemd
    Tag         host.*
    Read_From_Tail  on
[OUTPUT]
    Name        forward
    Match       *
    Host        logserver.example.com
    Port        24224

Fluentd

Fluentd is the more feature-rich sibling. It supports hundreds of input and output plugins and is commonly used in Kubernetes environments.

syslog-ng

syslog-ng is a direct alternative to rsyslog. It offers a different configuration syntax and some features like built-in pattern matching and database output that rsyslog handles through modules.

# syslog-ng source and destination example
source s_network {
    tcp(ip("0.0.0.0") port(514));
};
destination d_remote {
    file("/var/log/remote/${HOST}/${PROGRAM}.log");
};
log {
    source(s_network);
    destination(d_remote);
};

Choosing the Right Approach

Log Server Hardening

Your centralized log server is now a high-value target. Harden it accordingly:

• Restrict SSH access to the log server to a small set of administrators
• Use a firewall to allow only syslog traffic (port 514 or 6514) from known source IPs
• Store logs on a separate partition so they cannot fill the root filesystem
• Set up its own FIM (AIDE) and auditd rules
• Back up logs to a separate location or immutable storage
• Apply the same CIS benchmarks you use on other servers

A centralized log server that is easier to compromise than the servers it monitors provides no security benefit. Treat it as the most critical infrastructure in your environment.