Unattended upgrades (DRAFT)

Unattended upgrades are on by default in any Forge configured server but will only update and upgrade a small number of packages. To have all packages updated some adjustments will need to be made to the configurations.

Configuration

Configurations for this file /etc/apt/apt.conf.d/50unattended-upgrades

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}:${distro_codename}-backports";
    "${distro_id}:${distro_codename}-updates";
    "LP-PPA-ondrej-php:${distro_codename}";
    "LP-PPA-ondrej-nginx:${distro_codename}";
    "Ubuntu ${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Mail "[email protected]";
Unattended-Upgrade::Sender "Unattended-upgrades [hostname] <[email protected]>";
Unattended-Upgrade::Package-Blacklist {
    //
};
Dpkg::Options {
   "--force-confdef";
   "--force-confold";
}

Teams notifications

So we know what has been updated and when we want to send emails from the server to our dedicated Unattended Upgrades Log teams channel. This requires the use of msmtp purely for ease of use when configuring SendGrid SMTP on the server

Install msmtp and msmtp-mta for sending out report emails

apt update
apt install -y msmtp msmtp-mta

Create the config file at /etc/msmtprc

# Set default values for all following accounts.
defaults
auth           on
tls            on
tls_starttls   on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp
aliases /etc/aliases
set_from_header on

# server
account        server
host           smtp.sendgrid.net
port           587
from           "[email protected]"
user           apikey
password       [api key available in bitwarden]

# Set a default account
account default : server

‼️ If msmtp is already installed and there are already details in /etc/msmtprc be sure to add any additional configurations without affecting existing functionality.

You may also need to add the following to /etc/mail.rc

set sendmail="/usr/bin/msmtp -t"

Test your configurations

echo "Subject: Testing msmtp" | sendmail -v [email protected]

You should receive an email into your inbox. If not use /var/log/msmtp and /var/log/mail.log to troubleshoot the issue.

Scheduling Unattended Upgrades

As unattended upgrades are triggered as part of the apt schedule you can control when they are triggered by overriding 2 key files:

1. /lib/systemd/system/apt-daily.timer
2. /lib/systemd/system/apt-daily-upgrade.timer

Do not edit the files directly you can create an override file

systemctl edit apt-daily.timer

Update the time section, changing the time within OnCalendar to an appropriate time

[Timer]
OnCalendar=*-*-* 2:00
RandomizedDelaySec=60m
Persistent=true

Once you have saved the file run the following

systemctl restart apt-daily.timer
systemctl status apt-daily.timer

The same can be completed for apt-daily-upgrade.timer

systemctl edit apt-daily-upgrade.timer
systemctl restart apt-daily-upgrade.timer
systemctl status apt-daily-upgrade.timer

Linode scheduled backups

As Linode snapshots can be scheduled with the Linode console it would be good to arrange time timers to trigger just after a snapshot has been completed.

For example you could set the apt timers to trigger at 2:00 and set the Linode snapshots to complete between 00:00 - 02:00. You will then have a backup that can be restored to that will be reasonably recent relative to the upgrade if anything goes wrong.