Use external mail server for mdadm

mdadm is the utility to check and report failures on RAID disks. The usual way this Linux application send its message is a plain old e-mail. In this howto you’ll find the instruction to use an external mail server with mdadm.

First, replace sendmail with an external email account. After you’ve configured and tested msmtp you’re ready to configure mdadm.

Configure mdadm with the new SMTP

Change /etc/mdadm/mdadm.conf to

# instruct the monitoring daemon where to send mail alerts
# MAILADDR root
MAILADDR alertrecipient@example.com
MAILFROM senderaddress@example.com

Where:

  • senderaddress@example.com is your FROM e-mail, the email or alias you’re sending emails from.
  • alertrecipient@example.com is your recipient TO e-mail. It must be a frequently-used e-mail since alerts of failures are sent there.

Actually, using /etc/aliases and assigning root to the right recipient should allow you to avoid this step but you’ve to test yourself.

Send test message with mdadm

Type this command to emulate a disk failure message from mdadm:

sudo mdadm --monitor --scan --test -1

If you receive the message in the alertrecipient@example.com this job is finally done!

This is an automatically generated mail message from mdadm
running on YOURSERVERHOSTNAME

A TestMessage event had been detected on md device /dev/md/1.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid...

md1 : active raid...

unused devices:
...

See also

Use external SMTP server for system mails on Linux

To use an external SMTP for all system e-mails, you have to install these:

sudo apt-get install msmtp msmtp-mta

Where msmtp-mta transform the external reference in the sendmail command usable by any application using sendmail. In this way you haven’t to install and configure Postfix since you’ll rely on an external SMTP service.

Create the config file for msmtp

This is an example based on the popular Gmail by Google:

# Example for a system wide configuration file
# /etc/msmtprc

# A system wide configuration file is optional.
# If it exists, it usually defines a default account.
# This allows msmtp to be used like /usr/sbin/sendmail.
account default
aliases /etc/aliases

# The SMTP smarthost.
# host mailhub.oursite.example

# Construct envelope-from addresses of the form "user@oursite.example".
#auto_from on
#maildomain oursite.example

# Construct envelope-from addresses of the form "user@oursite.example".
# this fix the error: msmtp: account default from /etc/msmtprc-php: envelope-from address is missing
### auto_from on

# Use TLS.
tls on
auth on
tls_trust_file /etc/ssl/certs/ca-certificates.crt

# Syslog logging with facility LOG_MAIL instead of the default LOG_USER.
syslog LOG_MAIL

host smtp.gmail.com
port 587
from senderaddress@example.com
user realaccount@example.com
password GMAIL-AUTH-PASSWORD

Replace these with the real data from your e-mail account.

In this example, the realaccount@example.com in GMail is the user that created the Gmail app password. He has to have the senderaddress@example.com configured as sender address alias in GMail.

Add aliases

To match local users with sender address, create the aliases file:

nano /etc/aliases

# See man 5 aliases for format
# postmaster: root
postmaster: senderaddress@example.com
root: senderaddress@example.com
default: senderaddress@example.com

If you’ve any process sending emails using a specific username, add to this list with the right email to use. Any occurrence of the original address will be translated to the right address.

First test

Type this to test the new configuration:

sendmail alertrecipient@example.com
WRITE YOUR TEXT
Ctrl+D

Additional step

If you need to use the mail command, install mailutils without installing postfix:

sudo apt-get install --no-install-recommends mailutils

Then you can use something like:

echo -e Print a variable here $MYVAR.\n\n– \nSign here | mail -s Type your subject here $EMAIL

Here some variables are added as you can use them in a custom script.

Using msmtp command directly

If you already use the base mail command and you don’t want to replace it with mail, you can use this:

(echo -e Subject: Type your subject here; echo; echo -e “Print a variable here $MYVAR.\n\n– \nSign here) | msmtp -a default $EMAIL

Done

Now any application using sendmail will actually use your external SMTP service. Use a mail server supporting TLS to avoid transmitting clear text filled with system information.

Tested on Ubuntu Linux 18

Automate log cleanup for GDPR: the Sentry case

With the General Data Protection Regulation (GDPR) enforced by European Union logs have to be cleaned regularly to delete IP addresses and other information about visitors. This can be interpreted as a way to protect an emerging and discussed right, the right to be forgotten.

This new regulation is impacting every automated log system out of there. Since Sentry is a good open source error monitoring software* and it’s widely used, this guide will show how to clean Sentry logs on Linux systems according to GDPR using the sentry cleanup command line utility.

Set a time limit for logs

Before starting discover the maximum time limit a log can be kept according to the service policy you’re working on.

In the below examples, the max time a log can be kept is 26 months, one of the sizes proposed by Google Analytics on cleanup settings.

A 26 months limit for stored logs in sentry are set like this:

env SENTRY_CONF='/usr/local/etc/sentry' sentry cleanup --days 749

where /usr/local/etc/sentry is the directory where config.yml and sentry.conf.py are located or

env SENTRY_CONF='/usr/local/etc/sentry' sentry cleanup --days 749 --project 5

where 5 is the id of the project you can find in Project settings > Client Keys (DSN) as the very last part of the DSN path (always an integer number).

749 days are calculated like this:

30 days × 26 month = 780 days – 31 days = 749

31 days are a margin to safely delete logs the same day of each month.

Apparently, sentry cleanup needs to be root to access to postgres user and thus all sentry database tables so we have to put it on the cron for root.

Schedule the cleanup

  1. Login as root with su – or sudo bash
  2. crontab -e
  3. add a command line like this
. /usr/local/etc/virtualenvs/sentry/bin/activate && env SENTRY_CONF='/usr/local/etc/sentry' sentry cleanup --days 758 --project 5 && deactivate

leading dot . is an alternative for source available on /bin/sh (environment of cron) and not only by /bin/bash. This avoid to set the environment variable SHELL=’/bin/bash’ on crontab.

The resulting cron entry would be:

20 3 28 * * . /usr/local/etc/virtualenvs/sentry/bin/activate && env SENTRY_CONF='/usr/local/etc/sentry' sentry cleanup --days 749 --project 5 && deactivate

It isn’t a bad idea to add a fallback cleanup command the day after, so if you forget to cleanup logs for a specific project it will be done automatically:

20 3 29 * * . /usr/local/etc/virtualenvs/sentry/bin/activate && env SENTRY_CONF='/usr/local/etc/sentry' sentry cleanup --days 749 && deactivate

Now even your Sentry logs are GDPR compliant. The power of this method is that you can set a different cleanup limit for every project, according to its policies. And you haven’t to use any proprietary software to do this, just free/libre open source software.

If you are in a hurry to publish privacy policies and you have a dedicated hosting, give a try to JournaKit legalazy on GitHub.

* Plus it’s written on top of Django.

Cannot connect to wired connection on Ubuntu (SOLVED)

When your Wireless interface is working and the ethernet isn’t working on Ubuntu, here’s a quick howto to check and fix a misconfiguration. It doesn’t solve any ethernet issues but you can give a try and on an Asus laptop (with JMicron chipset) I worked on it makes the job done.

Tested on Ubuntu 16.04 LTS

First steps

To detect Ethernet interface:

ifconfig

To check and configure connection:

apt-get install ethtool

To save the current status of network interface:

ethtool ens5f5 > ethernet_before.txt

Make ethernet interface works

ethtool -s ens5f5 speed 1000 duplex full autoneg on

or:

ethtool -s ens5f5 speed 100 duplex full autoneg on

Then to check what is the difference between the old non-working configuration and the configuration that works:

ethtool ens5f5 > ethernet_after.txt
diff ethernet_before.txt ethernet_after.txt

If it doesn’t work try other ways, e.g. looking for specific issue on your Ethernet driver:

lspci | grep Ethernet

or

lspci | grep ethernet

to check your driver.

If the issue reappears after reboot, to make the command to run on startup do:

sudo bash
crontab -e

And add:

@reboot /sbin/ethtool -s ens5f5 speed 100 duplex full autoneg on

Now reboot to check if changes takes effect

Failed to start Authorization manager (SOLVED)

If you get the “Failed to start Authorization manager” while booting a Linux OS (e.g. CentOS) most likely you’ve a SELinux misconfiguration.

The main issue is that if you’ve restarted the system, you cannot login since OpenSSH service is not running. If you’ve a virtual server, even the console shipped by your provider is stuck on the Authorization manager error.

The reason behind this in my case was a completely wrong value in SELINUXTYPE.

The example below is valid for a Digital Ocean’s droplet running a CentOS 7 but can be adapted to other providers and distro.

To fix it I had to:

1. Boot the OS with initramfs (also called Recovery Kernel). You can do this on the dashboard provided by your virtual server provider (e.g. on Digital Ocean)

2. Open the console from your provider’s dashboard

3. Mount the root filesystem, e.g.

mkdir /mnt
mount /dev/vfsa1 /mnt

Where /dev/vfsa1 is your root partition.

4. Move broken configuration and recreate selinux config file manually with:

mv /mnt/etc/selinux/config /mnt/etc/selinux/config.BROKEN
touch /mnt/etc/selinux/config
cat > /mnt/etc/selinux/config

Then type these two lines:

SELINUX=disabled
SELINUXTYPE=targeted

Go to a new blank line after, then type Ctrl+D to quit. Since initramfs hasn’t a text editor like vi or nano, the simple cat > filename do the trick.

5. From your dashboard, change the kernel version to what you’ve used previously and Power Cycle (reboot) the machine using the dashboard because a reboot from initramfs in this state will restart initramfs itself.

Open the provider’s console on boot to see your system working again. Wait the services to start and then connect with your SSH client to see again your files and get again the control of your server.

Be careful when configuring SELinux again and create a shapshot and/or backup after you’ve restored your server…and before changing SELinux again.

HTTPS: how to add TLS ciphers on nginx (update regularly)

HTTPS is a great improvement to a website security. However, HTTPS comes in different flavours and among these there are very weak ones.

Among protocols, SSL have to be avoided because it is not secure. Its successor, TLS, comes in different versions and supports different ciphers. To be short, the cipher is the encryption method/algorihms the website and the client use to talk each other.

The combination of protocols and ciphers available to implement HTTPS will limits the type of clients capable to access the website.

To be sure your website will not lose traffic, you have to balance the strongest ciphers available with the most compatible but still secure, dropping all weaker ciphers.

Check the strenght of your HTTPS implementation

If you’ve already implemented HTTPS on your website, first you’ve to check ist current security status of protocols and ciphers.

Check your hostname on Qualys SSL Labs pasting the HTTPS protected domain on the Test your server section. It’s a fast method with a very detailed output for public websites.

The report will give your hostname a rank, a detailed list of issues, browser support, and the complete list of supported ciphers. Among these ciphers, you can get some ciphers highlighted in yellow. You have to get the rid of these no matter what.

The list of ciphers actually differs from a typical cipher declaration on nginx because nginx can use the OpenSSL naming and Qualys uses IANA naming.

Here’s an helpful conversion table by Mozilla where you can convert IANA to OpenSSL and the other way round. Take note of the weak ciphers but wait before start to cut your cipher declaration on nginx.

You’ve to check how many visitors you’ll lose after the cut first.

Get the website statistics

Using Google Analytics or similar services and software, go to the Audience > Technology > Browser to get a list of your visitors’ browsers. Select a timespan like the last year or less.

You can add Browser version or OS version as secondary dimension to match the list of supported browsers from SSL Labs. You’ll get something similar:

analytics-technology-browsers

Well, someone is still using Internet Explorer 9.0 in 2018.

Since Internet Explorer running on old Windows versions (like XP) is one of the most troublesome combination, check how many visitors use this legacy software.

On Google Analytics type on the search box “Internet Explorer” and you’ll get the browser usage of this legacy browser. Select OS version as secondary dimension to get a list of OSes using IE.

Compare this list with the report from SSL Labs and with the conversion table from Mozilla cited above and count the number of visitors you want to cut off from your website in the sake of security.

Cut the weak ciphers

Trimming down the ciphers declaration on nginx conf you’ll get something like this:

ssl_ciphers ‘ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!3DES’;

Each cipher is separated by a ‘:’ and at the end some elements (typically using OpenSSL naming) are forbidden with a ‘!’.

Here’s the context:

server {
        # the port your site will be served on
        listen      443 ssl;
        # the domain name it will serve for
        # substitute your machine's IP address or FQDN
        server_name example.com www.example.com;
        ssl_certificate /path/to/fullchain.pem;
        ssl_certificate_key /path/to/privkey.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_dhparam /etc/ssl/certs/dhparam.pem;
        # cfr. ........................................
        ssl_ciphers ** PASTE CIPHERS HERE **;
        ssl_session_timeout 1d;
        ssl_session_cache shared:SSL:50m;
        ssl_stapling on;
        ssl_stapling_verify on;
        add_header Strict-Transport-Security max-age=15768000;
        charset     utf-8;
        # This is for Let's Encrypt
        location ^~ /.well-known {
                alias /path/to/.well-known;
                allow all;
        }

        # max upload size
        client_max_body_size 75M;   # adjust to taste

        location /webpath  {
                alias /path/to/web;
        }
}

Change the conf file, reload nginx (on CentOS 7 systemctl reload nginx) and then re-run the SSL Labs test.

The Qualys’ tool will show you the new incompatibility with legacy browsers in the Handshake Simulation section:

tls-handshake

Modern protocols and ciphers implemented using the above declaration on nginx cut off IE 8 on XP and IE 6, the report explain.

According to the technology used by visitors of the analyzed website, few visits are sacrificed for better security for both visitors and host.

Tune these settings according to your needs, keep monitoring the tecnology used by site visitors and dropping legacy system progressively, with Modern compatibility as a (not so) long term objective.

Certbot: auto-renew LetsEncrypt certificate on cron

Certificates renewal can be difficult to automate leading to errors that will mark the website as “Insecure”.

Before continue, ensure you haven’t the /etc/cron.d/certbot file launching:

less /etc/cron.d/certbot

If a crontab appear, you already have an automatic renew enabled via a certbot plugin like nginx or apache (the preferred method) and you shouldn’t do nothing.

The following will apply to standalone method, a small webserver provided by Let’s Encrypt that will listen the 80 port, without any existing auto-renew cron script. It’s very useful in cases where a plugin for your webserver is not available, like haproxy.

In this case, here’s how to automate certificate renewal with nginx as webserver:

sudo nano /etc/cron.d/certbot-custom

And then add to the crontab these lines pressing A to edit:

37 02 * * * root certbot -q renew --pre-hook="systemctl stop nginx" --post-hook="systemctl start nginx"

The root before certbot is the user who will run the command.

Every day at 02.37 the certificate will be asked for renewal in quiet mode to avoid to spam on root mailbox, especially useful if you use external SMTP server for system mails on Linux.

After you’ve typed these lines, type:

:wq

To write and quit. The /etc/cron.d/certbot-custom file will be easily accessible and maintainable compared to crontab -e, but it is a valid method too. If you use crontab -e, do it as root and do not add root after the last *.

To exactly mimic the certbot cron plugin, transform the line on /etc/cron.d/certbot-custom file to:

37 02 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew --pre-hook="systemctl stop nginx" --post-hook="systemctl start nginx"

Before run, all paths will be checked, and a random wait will be added. If your webserver requires additional commands, create a script to be executed in –post-hook.

Other Linux distributions

If you’re using a different Linux OS you can locate certbot-auto using the following command:

whereis certbot

And then you can use it on the crontab.

You’ve also to use the alternative of systemctl for your system to refresh the certificates on the webserver.

If you’ve to do get your first certificate, here you can get more information about how to install free Let’s Encrypt certificates on nginx.

When auto-renew certificates

Since certificates lasts for about three months, you can tell crontab to run this every week instead every day accordingly.

Adding the 0 on the 5th position will tell crontab to run the command every Sunday at 2 AM (2nd position) and 37 minutes (1st position).

If you find this syntax difficult you can use crontab.guru to easily generate the crontab.

Updated on august 16th, 2019