L13. Syslog and journald: Where Linux Logs Everything

Why Logs Matter for Security

Logs are your evidence trail. When an attacker gains access to a server, the first questions are always: when did they get in, what did they do, and how did they do it? Without logs, you cannot answer any of these.

From a security perspective, logs serve three purposes:

1. Detection: Identifying suspicious activity in real time or near-real time
2. Investigation: Reconstructing what happened after an incident
3. Compliance: Proving that security controls are in place and functioning

Traditional Syslog: /var/log/

The original Linux logging system uses syslog (or its modern implementation, rsyslog). Services write log messages to the syslog daemon, which routes them to files under /var/log/.

Key Log Files

Reading Log Files

# View the last 50 lines of the auth log
tail -50 /var/log/auth.log
# Search for failed SSH logins
grep "Failed password" /var/log/auth.log
# Follow a log file in real time
tail -f /var/log/syslog

rsyslog Configuration

rsyslog's main config file is /etc/rsyslog.conf. It uses a facility.priority syntax to route messages:

# Log all authentication messages to auth.log
auth,authpriv.*    /var/log/auth.log
# Log kernel messages at warning level or higher
kern.warning       /var/log/kern-warnings.log
# Log everything except mail and auth to syslog
*.*;auth,authpriv.none,mail.none    /var/log/syslog

The facility identifies the source (auth, kern, mail, cron, etc.), and the priority sets the minimum severity level (emerg, alert, crit, err, warning, notice, info, debug).

systemd Journal: journalctl

Modern distributions running systemd collect logs through the journal, which is a binary log managed by the systemd-journald service. Unlike syslog's plain text files, the journal is structured and indexed, making it faster to query.

Basic journalctl Commands

# View all logs (newest at the bottom)
journalctl
# View logs from the current boot only
journalctl -b
# View logs for a specific service
journalctl -u sshd.service
# View logs from the last hour
journalctl --since "1 hour ago"
# View logs between two timestamps
journalctl --since "2026-06-20 08:00" --until "2026-06-20 12:00"

Filtering by Priority

# Show only errors and above (err, crit, alert, emerg)
journalctl -p err
# Show only warnings from the SSH daemon
journalctl -u sshd -p warning

Following Logs in Real Time

# Follow new log entries as they arrive (like tail -f)
journalctl -f
# Follow only SSH-related entries
journalctl -u sshd -f

Syslog vs journald

Most modern systems run both. journald captures all systemd-managed service output, while rsyslog may still receive messages and write them to traditional files. They are complementary, not competing:

Log Rotation with logrotate

Text log files grow indefinitely. Without management, they will fill your disk. logrotate handles this automatically.

# View the logrotate configuration
cat /etc/logrotate.conf
# Per-service configs live in /etc/logrotate.d/
ls /etc/logrotate.d/

A typical logrotate config for rsyslog:

/var/log/syslog {
    rotate 7        # Keep 7 rotated copies
    daily           # Rotate once per day
    compress        # gzip old logs
    delaycompress   # Compress the previous rotation, not the current
    missingok       # Do not error if the log file is missing
    notifempty      # Do not rotate if the file is empty
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Security Considerations for Log Rotation

• Keep enough history to support incident investigations (30 to 90 days is common)
• Compressed logs still contain evidence: do not delete them prematurely
• Set appropriate file permissions on log files (640 or stricter)

Practical Security Checks

# Find all failed login attempts today
journalctl -u sshd --since today | grep "Failed"
# Count failed login attempts per IP
grep "Failed password" /var/log/auth.log | \
  awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
# Check for recent sudo usage
journalctl -u sudo --since "24 hours ago"

These queries should become part of your regular security review routine.