My First 60 Minutes With A Server

eric | Jan. 3, 2023, 5:25 p.m.

There are a few posts with titles like "My First 5 Minutes With A Server", perhaps even 10 or 15 minutes. Most are helpful for a minimal setup, but I can't remember spending less than 60 to get everything up and going and secured. Maybe I am just slow, but a minimal setup involves: A working firewall, passwordless ssh, automatic banning of malicious IPs, unattended security upgrades, malware scanning, an up-to-date security auditing system, systematic backups, and a monitoring system.

Recipes for:

- AlmaLinux

- Debian

- openSUSE

- Ubuntu

AlmaLinux / Rocky Linux

Preparation

Set up cron and crontab [1]:

> dnf install cronie-noanacron

and

> dnf remove cronie-anacron

Firewall With firewall-cmd

The firewall should be enabled as part of the installation process, as you presumably have direct terminal access. You can do anything in this list via ssh, but the firewall must be active to let ssh through, which means that it will initially block ssh. Catch 22!

Get the firewall going:

> sudo systemctl start firewalld.service

> sudo systemctl enable firewalld.service

> sudo systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2023-01-02 12:28:18 CET; 2 days ago
       Docs: man:firewalld(1)
   Main PID: 1177 (firewalld)
      Tasks: 2 (limit: 4915)
     CGroup: /system.slice/firewalld.service
             └─1177 /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid

Now, you can add the ssh service (and perhaps also other services, such as http, https, etc):

> sudo firewall-cmd --zone=public --add-service=ssh --permanent

> sudo firewall-cmd --reload

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

> ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Befor you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

> sudo systemctl restart sshd.service

Next, log in from the client with:

> ssh user@server

You should be logged directly in without being prompted for a password.

System Audit

For most systems, lynis is the go-to package for system audits. However, the package isn't currently available for Almalinux / Rocky Linux because of a problem with the key used to sign the repository. Hopefully, the lynis developers will get this fixed soon. In the meantime, you can get quite a few things done with auditd. Check out this excellent article to get going.

Malware Scan With clamav

Install, enable and start clamav packages. Freshclam keeps the clamav-database up to date and should run continuously as a service:

> sudo dnf install clamav clamd clamav-update

> sudo systemctl enable clamav-freshclam

> sudo systemctl start clamav-freshclam

> sudo systemctl status clamav-freshclam

Clamav scans for malware with clamscan, which should run at regular intervals. This can be done with cron. Add clamscan to cron:

> sudo crontab -e

# m h  dom mon dow   command
*       3       *       *       1,3,5   nice -n 15 clamscan && clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

So what is going on here?

*       3       *       *       1,3,5

means that the command runs every Monday, Wednesday and Friday at 3 am.

nice -n 15 clamscan

Clamscan is resource intensive. nice makes clamscan run with an adjusted niceness; It affects process scheduling, and a positive number is unfavorable to the process, thus avoiding that the process hogs the server.

clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

is a mouthful. We only want warnings of infected files (i), recursive search (r), and all findings logged to /var/log/clamav.log. We exclude directories that are unlikely to contain malware by using exclude-dir, and we set the maximum file size for individual files to 50M.

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box [2]. Simply:

> sudo yum install fail2ban

Unattended Security Upgrades

Run the following command to update security-related packages only:

> sudo yum update --security

To run the command on a daily basis, add it to cron.

> sudo crontab -e

# m h  dom mon dow   command
*       3       *       *       *       1,3,5       yum update --security

Backup

The best backup solution for you will depend a lot on your setup. My favorite solution is - hands down - BorgBackup. It includes deduplication, compression, and authenticated encryption. Moreover, it works well for both desktops and servers. For the former, the Vorta GUI is really handy. With a separate machine holding the backups, the installation looks like this on both server and remote backup server (if both are Almalinux, Rocky Linux, or similar - for other distributions, see [3]):

> sudo dnf install epel-release
> sudo dnf config-manager --set-enabled epel
> sudo config-manager --set-enabled powertools
> sudo yum install borgbackup

First, initialize a repository on the remote backup server:

> borg init --encryption=repokey /path/to/repo

Next, enter a passphrase when prompted, and then copy the repo key (you will need both to access the repo):

> borg key export /path/to/repo text.txt
> cat text.txt
BORG_KEY e036...

Now, you can create a backup using the repo. Let's call it Tuesday:

$ borg create ssh://user@server//path/to/repo::Tuesday ~/Documents/
Enter passphrase for key ssh://user@server//path/to/repo:

> borg list /path/to/repo/
Enter passphrase for key /path/to/repo:
Tuesday                              Tue, 2023-01-10 20:03:47 [3b096c...........]

Monitoring With Grafana, Loki and Promtail

Services such as clamav don't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started really quick, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

Debian

Preparation

Most of the following commands require root / sudo access, so on Debian it makes sense to change to root user (as users are by default not sudoers on Debian installations):

$ su root

Firewall

There are a lot of firewalls to choose from [4], but the simplest choice is to go with Ubuntu`s Uncomplicated Firewall (ufw) [5].

# apt update && sudo apt upgrade -y

# apt install ufw -y

When using the root user, the path to ufw may not be included in PATH. Add it to PATH, or use the whole path when running the command (ufw is usually in /usr/sbin ).

If you are accessing the server via ssh, you'd better allow ssh before enabling the firewall:

# ./usr/sbin/ufw allow ssh

Next, enable ufw:

# systemctl enable ufw --now

See the status:

# systemctl status ufw

List current rules:

# ufw status numbered

Other typical services to add are http and https. For a full list of available apps:

# ufw app list

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

# ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Befor you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

# systemctl restart sshd.service

Next, log in from the client with:

# ssh user@server

You should be logged directly in without being prompted for a password.

System Audit With lynis

Install lynis by adding the lynis repository. First, make a directory for the public key for the repository:

# mkdir -p /etc/apt/keyrings/

Next, download the key:

# wget -O - https://packages.cisofy.com/keys/cisofy-software-public.key | sudo tee /etc/apt/keyrings/cisofy-software-public.key > /dev/null

# cat /etc/apt/keyrings/cisofy-software-public.key

Then, add the repository to apt sources:

# echo "deb [signed-by=/etc/apt/keyrings/cisofy-software-public.key] https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list

Now, you are ready to install lynis:

# apt update

# apt install lynis

# lynis show version

Time to run lynis and get som results:

# lynis audit system

You will get a long list of results and recommendations. The most important part is the the beginning of the result list. If you are lucky, you get this message:
...
  -[ Lynis 3.?.? Results ]-

  Great, no warnings
...

However, if you get warnings instead, immediate action is probably required.

Scanning For Malware With clamav

Install clamav:

# apt install clamav

Test that clamscan works with:

# nice -n 15 clamscan && clamscan -ir

clamscan can be really resource intensive, and nice ensures that clamscan doesn't hog the server.

Freshclam keeps the clamav-database up to date and should run continuously as a service:

# systemctl enable clamav-freshclam

# systemctl start clamav-freshclam

# systemctl status clamav-freshclam

Clamav scans for malware with clamscan, which should run at regular intervals. This can be done with cron. Add clamscan to cron:

# crontab -e

# m h  dom mon dow   command
*       3       *       *       1,3,5   nice -n 15 clamscan && clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

So what is going on here?

*       3       *       *       1,3,5

means that the command runs every Monday, Wednesday and Friday at 3 am.

nice -n 15 clamscan

Clamscan is resource intensive. nice makes clamscan run with an adjusted niceness; It affects process scheduling, and a positive number is unfavorable to the process, thus avoiding that the process hogs the server.

clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

is a mouthful. We only want warnings of infected files (i), recursive search (r), and all findings logged to /var/log/clamav.log. We exclude directories that are unlikely to contain malware by using exclude-dir, and we set the maximum file size for individual files to 50M.

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box []. Simply:

# apt install fail2ban

Unattended Security Upgrades

Minimum setup:

# vim /etc/apt/apt.conf.d/50unattended-upgrades

Change:

Unattended-Upgrade::Mail "";

Unattended-Upgrade::MailReport "only-on-error";

Create an apt configuration file:

# vim /etc/apt/apt.conf.d/02periodic

with the following contents:

// Control parameters for cron jobs by /etc/cron.daily/apt-compat //

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade --download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "21";

// Send report mail to root
//     0:  no report             (or null string)
//     1:  progress report       (actually any string)
//     2:  + command outputs     (remove -qq, remove 2>/dev/null, add -d)
//     3:  + trace on
APT::Periodic::Verbose "2";

Check status:

# systemctl restart unattended-upgrades.service

# systemctl status unattended-upgrades.service

Monitoring With Grafana, Loki and Promtail

Services such as clamav doesn't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started really quick, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

openSUSE

Firewall With firewall-cmd

The firewall should be enabled as part of the installation process, as you presumably have direct terminal access. You can do anything in this list via ssh, but the firewall must be active to let ssh through, which means that it will initially block ssh. Catch 22!

Get the firewall going:

$ sudo systemctl start firewalld.service

$ sudo systemctl enable firewalld.service

$ sudo systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2023-01-02 12:28:18 CET; 2 days ago
       Docs: man:firewalld(1)
   Main PID: 1177 (firewalld)
      Tasks: 2 (limit: 4915)
     CGroup: /system.slice/firewalld.service
             └─1177 /usr/bin/python3 /usr/sbin/firewalld --nofork --nopid

Now, you can add the ssh service (and perhaps also other services, such as http, https, etc):

$ sudo firewall-cmd --zone=public --add-service=ssh --permanent

$ sudo firewall-cmd --reload

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

$ ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Befor you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

# systemctl restart sshd.service

Next, log in from the client with:

$ ssh user@server

You should be logged directly in without being prompted for a password.

lynis

Scanning For Malware With clamav

Install clamav:

$ sudo zypper install clamav

Next, run freshclam in order to get virus definitions to work with:

$ sudo freshclam

Test that clamscan works with:

$ sudo nice -n 15 clamscan && clamscan -ir

clamscan can be really resource intensive, and nice ensures that clamscan doesn't hog the server.

Freshclam keeps the clamav-database up to date and should run continuously as a service:

$ sudo systemctl enable clamav-freshclam

$ sudo systemctl start clamav-freshclam

$ sudo systemctl status clamav-freshclam

Clamav scans for malware with clamscan, which should run at regular intervals. This can be done with cron. Add clamscan to cron:

$ sudo crontab -e

# m h  dom mon dow   command
*       3       *       *       1,3,5   nice -n 15 clamscan && clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

So what is going on here?

*       3       *       *       1,3,5

means that the command runs every Monday, Wednesday and Friday at 3 am.

nice -n 15 clamscan

Clamscan is resource intensive. nice makes clamscan run with an adjusted niceness; It affects process scheduling, and a positive number is unfavorable to the process, thus avoiding that the process hogs the server.

clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

is a mouthful. We only want warnings of infected files (i), recursive search (r), and all findings logged to /var/log/clamav.log. We exclude directories that are unlikely to contain malware by using exclude-dir, and we set the maximum file size for individual files to 50M.

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box [2]. Simply:

$ sudo zypper install fail2ban

Unattended Security Upgrades

Install the yast2 online update tool:

> sudo zypper install yast2-online-update-configuration

Run the tool:

> sudo yast2 online_update_configuration

My choices look like this:

yast2 online update configurations

These choices givesyou a weekly update of non-interactive security patches. For other options, tweak the configuraiton with the tool or dit the configuration file.

If you want automatic reboots whenever a package installation require a reboot, you can do this using the Reboot Manager:

> sudo zypper install rebootmgr

> sudo systemctl enable --now rebootmgr.service
Created symlink /etc/systemd/system/multi-user.target.wants/rebootmgr.service → /usr/lib/systemd/system/rebootmgr.service.

Monitoring With Grafana, Loki and Promtail

Services such as clamav doesn't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started really quick, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

Ubuntu

Firewall With ufw

Ubuntu`s Uncomplicated Firewall (ufw) is really easy to use []. If you are accessing the server via ssh, you'd better allow ssh before enabling the firewall:

$ sudo ufw allow ssh

Next, enable ufw:

$ sudo systemctl enable ufw --now

See the status:

$ sudo systemctl status ufw

List current rules:

$ sudo ufw status numbered

Other typical services to add are http and https. For a full list of available apps:

$ sudo ufw app list

 

Block ssh Password Login

Allowing only certificate-based login to ssh will make it much harder to gain access for unauthorized users. If you think this is overkill, check the logs of a public-facing server, and see how many login attempts there are on port 22.

First, copy public keys from all clients that are supposed to have access to the authorized_keys file. From the client (with the user you will use to connect to the server):

$ ssh-copy-id -i ~/.ssh/id_rsa.pub username@server
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/eric/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
(username@server) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'username@server'"
and check to make sure that only the key(s) you wanted were added.

Befor you try to log in from the client, restart ssh on the server to make sure that the changed authorized_keys file is loaded:

# systemctl restart sshd.service

Next, log in from the client with:

$ ssh user@server

You should be logged directly in without being prompted for a password.

System Audit With lynis

Install lynis by adding the lynis repository. First, make a directory for the public key for the repository:

$ sudo mkdir -p /etc/apt/keyrings/

Next, download the key:

$ sudo wget -O - https://packages.cisofy.com/keys/cisofy-software-public.key | sudo tee /etc/apt/keyrings/cisofy-software-public.key > /dev/null

$ cat /etc/apt/keyrings/cisofy-software-public.key

Then, add the repository to apt sources:

$ echo "deb [signed-by=/etc/apt/keyrings/cisofy-software-public.key] https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list

Now, you are ready to install lynis:

$ sudo apt update

$ sudo apt install lynis

$ sudo lynis show version

Time to run lynis and get som results:

$ sudo lynis audit system

You will get a long list of results and recommendations. The most important part is the the beginning of the result list. If you are lucky, you get this message:
...
  -[ Lynis 3.0.8 Results ]-

  Great, no warnings
...

However, if you get warnings instead, immediate action is required.

Scanning For Malware With clamav

Try it out. From the terminal, you could do something like this:

$ sudo clamscan -ir -l /var/log/clamav.log --exclude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

It limits CPU hogging by excluding directories that are unlikely to contain problems and by limiting file size. Running it after-hours also helps. Set up regular scans after-hours with cron and cpulimit:

$ sudo crontab -e

# m h  dom mon dow   command
*       3       *       *       *       1,3,5   cpulimit -l 30 -- clamscan -ir -l /var/log/clamav.log --ex
clude-dir="^/proc/|^/sys/|^/dev/" --max-scansize=50M //.

"*       3       *       *       *       1,3,5" means that cron runs the command at 3 am every Monday, Wednesday, and Friday.

The l-optionfor cpulimit limits the maximum resources a clamscan process can use. 30 limits the amount of CPU usage to 30%. Use a value that works for you.

Make sure that cpulimit is installed:

$ sudo apt install cpulimit

Filtering Malicious IPs With fail2ban

Fail2ban scans log files and bans IPs that show certain signs of malicious intent. Filters for many services are ready out of the box [2]. Simply:

$ sudo yum install fail2ban

Unattended Security Upgrades

To get going with unattended securoty upgrades in Ubuntu is really easy:

$ sudo apt update && sudo apt upgrade

$ sudo apt install unattended-upgrades -y

For more options, have a look at the configuration file:

$ sudo vi /etc/apt/apt.conf.d/50unattended-upgrades

Furthermore, this article provides a lot of detail on commands and options.

Monitoring With Grafana, Loki and Promtail

Services such as clamav doesn't do much good if you are not keeping an eye on logs. That also goes for the server maintenance as such. There are many ways to look at logs. Personally, I really like the grafana log-stack, with grafana as a front end, combined with the loki database, and the promtail client-side agent that ships logs to loki.

A normal setup is to have loki and grafana on a separate machine, and then use promtail to shovel logs into loki from each client. To get started really quick, look at this webinar on loki and grafana. To get going with promtail, all the details you need are here.

References

[1] Cron in Rocky Linux - https://docs.rockylinux.org/en/guides/automation/cron_jobs_howto/

[2] Fail2ban Wiki - http://www.fail2ban.org/wiki/index.php/Main_Page

[3] Borg 1.24 Installation - https://borgbackup.readthedocs.io/en/1.2-maint/installation.html

[4] Firewalls (for Debian) - https://wiki.debian.org/Firewalls

[5] Install/Configure UFW Firewall on Debian 11 Bullseye - https://www.linuxcapable.com/how-to-setup-and-configure-ufw-firewall-on-debian-11-bullseye/

Other useful references:

How to Use the ssh-copy-id Command - https://bytexd.com/how-to-use-the-ssh-copy-id-command/

How to Disable SSH Login With Password - https://linuxhandbook.com/ssh-disable-password-authentication/

CISOFY Software Repository (Community) - https://packages.cisofy.com/community/

How to stop ClamAV/Clamscan from hogging the CPU/ draining the battery in Ubuntu 18.04? https://askubuntu.com/questions/1158514/how-to-stop-clamav-clamscan-from-hogging-the-cpu-draining-the-battery-in-ubuntu

UFW Essentials: Common Firewall Rules and Commands - https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands

 

About Me

Experienced dev and PM. Data science, DataOps, Python and R. DevOps, Linux, clean code and agile. 10+ years working remotely. Polyglot. Startup experience.
LinkedIn Profile

By Me

Statistics & R - a blog about - you guessed it - statistics and the R programming language.
R-blog

Erlang Explained - a blog on the marvelllous programming language Erlang.
Erlang Explained