L19. Patch Management: Automated Updates and Rollback

Why Patching Matters for Security

Unpatched systems are consistently one of the top attack vectors in real-world breaches. When a vulnerability is publicly disclosed, attackers begin scanning for unpatched systems within hours. The window between disclosure and exploitation keeps shrinking: in many cases, working exploits appear within days of a CVE being published.

Patching is not just about installing new features. Security patches fix known vulnerabilities, close privilege escalation paths, and address bugs that attackers can exploit remotely. A well-maintained patch process is one of the simplest and most effective defenses you can implement.

Manual vs Automated Updates

Manual Updates

Manual patching gives you full control over what gets installed and when:

# Debian/Ubuntu: update package lists and upgrade
sudo apt update
sudo apt upgrade -y
# RHEL/Fedora: check for and install updates
sudo dnf check-update
sudo dnf upgrade -y

Security-only updates let you skip feature changes and focus on patches:

# Debian/Ubuntu: security updates only
sudo apt update
sudo unattended-upgrades --dry-run  # preview what would be installed
sudo unattended-upgrades            # apply security updates
# RHEL/Fedora: security updates only
sudo dnf upgrade --security

Manual patching works for small environments, but it does not scale. If you manage more than a handful of servers, missed patches become inevitable.

Automated Updates on Debian/Ubuntu: unattended-upgrades

The unattended-upgrades package automatically installs security updates on a schedule:

# Install
sudo apt install -y unattended-upgrades apt-listchanges
# Enable
sudo dpkg-reconfigure -plow unattended-upgrades

Configuration

The main configuration file controls which updates are installed automatically:

# Edit the configuration
sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

Key settings to review:

// Enable security updates (enabled by default)
Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
// Automatically reboot if a kernel update requires it
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";
// Email notification on errors
Unattended-Upgrade::Mail "admin@example.com";
Unattended-Upgrade::MailReport "on-change";
// Remove unused dependencies after upgrade
Unattended-Upgrade::Remove-Unused-Dependencies "true";

Verifying It Works

# Run a dry run to see what would be updated
sudo unattended-upgrades --dry-run --debug
# Check the log for past activity
cat /var/log/unattended-upgrades/unattended-upgrades.log

Automated Updates on RHEL/Fedora: dnf-automatic

The equivalent tool for Red Hat-based systems is dnf-automatic:

# Install
sudo dnf install -y dnf-automatic
# Enable and start the timer
sudo systemctl enable --now dnf-automatic-install.timer

Configuration

# Edit the configuration
sudo nano /etc/dnf/automatic.conf

Key settings:

[commands]
# What to do: download-only, apply, or security
upgrade_type = security
apply_updates = yes
[emitters]
# How to notify: email, stdio, or motd
emit_via = email
[email]
email_from = root@example.com
email_to = admin@example.com

Checking the Timer

# Verify the timer is active
sudo systemctl status dnf-automatic-install.timer
# Check logs
sudo journalctl -u dnf-automatic-install.service

Kernel Updates and Reboots

Most package updates take effect immediately, but kernel updates require a reboot. This creates a challenge: the system is running a vulnerable kernel until you restart.

Checking if a Reboot Is Needed

# Debian/Ubuntu: check if reboot is required
cat /var/run/reboot-required 2>/dev/null && echo "Reboot needed" || echo "No reboot needed"
# Or check which packages triggered the reboot requirement
cat /var/run/reboot-required.pkgs 2>/dev/null
# RHEL/Fedora: compare running vs installed kernel
uname -r                      # running kernel
rpm -q kernel | tail -1       # latest installed kernel

Scheduling Reboots

For production servers, schedule reboots during maintenance windows:

# Schedule a reboot for 4 AM
sudo shutdown -r 04:00 "Scheduled reboot for kernel update"
# Cancel a scheduled reboot
sudo shutdown -c

Live Patching (No Reboot Required)

Some organizations use live kernel patching to apply security fixes without rebooting:

• Ubuntu: Canonical Livepatch (free for up to 5 machines)
• RHEL: kpatch (included with RHEL subscription)

These solutions patch the running kernel in memory. They are useful for critical vulnerabilities when a maintenance window is not available soon enough.

Rollback Strategies

Sometimes an update breaks something. Having a rollback plan before you patch is essential.

Package-Level Rollback

# RHEL/Fedora: rollback the last transaction
sudo dnf history list          # find the transaction ID
sudo dnf history undo <ID>     # undo that specific transaction
# Debian/Ubuntu: downgrade a specific package
apt list --installed | grep <package-name>
sudo apt install <package-name>=<previous-version>

Filesystem Snapshots

For more comprehensive rollback, take a filesystem snapshot before patching:

# If using LVM
sudo lvcreate --size 5G --snapshot --name pre-patch-snap /dev/vg0/root
# If the update causes problems, restore the snapshot
sudo lvconvert --merge /dev/vg0/pre-patch-snap

VM Snapshots

If your Linux server runs as a virtual machine, take a VM snapshot before applying updates. This provides the fastest and most reliable rollback option.

Patch Testing Workflow

For production environments, never patch directly without testing. A practical workflow looks like this:

Key Practices

• Stagger rollouts: Patch a subset of servers first, monitor for 24-48 hours, then proceed
• Maintain a package mirror: Use a local repository mirror so all environments install the exact same package versions
• Document exceptions: If a patch must be delayed, record the reason, the risk, and the planned remediation date
• Monitor after patching: Watch system logs, application performance, and error rates after each patch cycle