Give flatpack a chance

Talking about package manager on Linux, flatpak gained attention recently. Installation is easy, this is about installing on Ubuntu.

If you’ve some trouble installing an application using the package manager shipped with your distro, you can give it a try, since it’s available on 22 distro by now.

In the above case, torbrowser packet is broken on a Ubuntu 18.04, as many users noted out on the package page. Installing the package via flatpak, everything run smoothly in minutes.

Simply:

flatpak search PACKAGENAME

Then:

flatpak install PACKAGENAME

Command line output is very complete, but if you want to use a GUI, on Ubuntu gnome-software (Ubuntu Software) will list the flatpak packages too, just read the package info on bottom of the description page and search packages as usual.

Full backup of PostgreSQL and restore on docker

In this howto I will show how to backup all PostgreSQL databases, roles and permission on a server and restore them on another server on a docker.

1. Backup databases on origin server

To backup all PostgreSQL databases automatically, preserving roles and permission, you can create this cron.d script (this example is from a CentOS):

# Dump postgresql database automatically
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
40 22 * * 0 postgres /bin/bash -c 'pg_dumpall -Upostgres | gzip > "/var/backup/myexport.sql.gz"'

In order:

  1. Run on at 22:40 on Sunday
  2. Run as postgres user (cron is executed by root, and root can impersonate it without password)
  3. Use the bash interpreter (not sh)
  4. Run pg_dumpall to dump all database
  5. Pipe the output to gzip to compress the SQL into a /var/backup/myexport.sql.gz file

2. Transfer on another server

Transfer the dump using ssh using something like this:

rsync -rltvz --no-o --no-g myuser@originserver:/var/backup/dumpall.sql.gz /opt/backup/myproj/data/

Use .ssh/config to store connection info.

3. Restore on a clean docker container

The following bash script will create a docker container, populating it with

set -e
echo "Remove identical container, keep data on directory"
NOW=`date +%Y-%m-%d`
# create an id based on timestamp plus MAC address 
UNIQUID=`uuidgen -t`
NAME="postgresql-myproj-$NOW-$UNIQUID"
echo "Get postgres docker image"
docker pull postgres:9.4-alpine
echo "Generate 1 password with 48 char length"
PPASS=`pwgen 48 1`
mkdir -p /opt/docker_data/$NAME
echo "Save psql password on /opt/docker_data/$NAME.pwd"
echo $PPASS > "/opt/docker_data/$NAME/psql.pwd"
echo "extract database"
mkdir -p /opt/docker_data/$NAME/psqldata
mkdir -p /opt/docker_data/$NAME/share
gunzip -c /opt/backup/myproj/data/dumpall.sql.gz > /opt/docker_data/$NAME/share/restore.out
echo "Run a clean docker"
docker run --name $NAME -e POSTGRES_PASSWORD=$PPASS -d -p 44432:5432 -v /opt/docker_data/$NAME/psqldata:/var/lib/postgresql/data -v /opt/docker_data/$NAME/share:/extshare --restart always postgres:9.4-alpine
sleep 10
echo "Restore from /extshare/restore.out using user postgres (-upostgres) the database postgres (all dbs)"
docker exec -upostgres $NAME psql -f /extshare/restore.out postgres
echo "Clear the restore.out file"
rm /opt/docker_data/$NAME/share/restore.out

In order, this script:

  1. Download a postgres:9.4-alpine image (choose your version)
  2. Generate a random container name postgresql-myproj-YYYY-MM-DD-random-id based on MAC address and timestamp
  3. Generate a random password and save on a file
  4. Generate a directory structure on host system to keep postgres file and dump outside docker
  5. Create a new postgres container exposed to host on port 44432
  6. Save postgres files on /opt/docker_data/$NAME/psqldata on host
  7. Expose the dump file directory on /extshare on guest
  8. Restore the dump using role postgres
  9. Delete the dump

Resulting directory structure on host will be:

.
└── postgresql-myproj-2019-28-12-*******
    β”œβ”€β”€ psqldata [error opening dir]
    β”œβ”€β”€ psql.pwd
    └── share
        └── restore.out

Then to connect just use the same user and password of the database on origin server.

Permissions

Usually on /opt/docker_data/$NAME/psqldata/pg_hba.conf, you’ve to add a line like this:

host all all 172.17.0.1/32 md5

giving to host (reachable by 172.17.0.1 inside docker) full access to database. But the default image ship a handy, permissive entry:

host all all all md5

So you can connect without any step to the database.

Connection

If you connect to the destination server with ssh, if you cannot access the port remember to forward the PostgreSQL on .config like:

Host destinationsrv
    Hostname destinationaddr
    User mydestservuser
    IdentityFile /home/myuser/.ssh/id_rsa_destination_server
    LocalForward 44432 127.0.0.1:44432

Remember that if you haven’t a firewall, docker container will start with a permissive rule like:

0.0.0.0:44432->5432/tcp

So will be exposed and reachable using password.

Sort of CDN to serve client-side libraries via an auto-pull git repo on tmpfs

This configuration will allow to install on a Debian-based system a fast server for client libraries. Key technologies used are:

  • tmpfs to serve files from volatile memory
  • git / mercurial from github / bitbucket to get files from a public or private repository
  • systemd units to mount tmpfs and sync
  • nginx to serve files to user

On this first step you’ll create a service to reserve some RAM for static files, pulling them from a private or public repo.

Mount tmpfs with systemd

To serve files directly from RAM, you have to mount a tmpfs directory. You can do it on fstab:

/etc/fstab

tmpfs /mnt/cdn tmpfs rw,nodev,nosuid,size=300M 0 0

Or with a systemd unit:

/etc/systemd/system/mnt-cdn.mount

[Unit]
Description=Mount empty CDN directory on volatile memory

[Mount]
What=tmpfs
Where=/mnt/cdn
Type=tmpfs
Options=defaults,noatime,size=300M

[Install]
WantedBy=multi-user.target
  • noatime will disable last access on contained files, reducing write on disk
  • size will reserve 300MB for /mnt/cdn partition on RAM (increase as needed)
  • WantedBy=multi-user.target mount the partition on runlevel 3 (multi-user mode with networking)

Create two units on a local path like /usr/local/share/systemd then create a symlinks on /etc/systemd/system or create directly them on /etc/systemd/system. You can also directly create them on /usr/local/share/systemd.

Create the pull service

When the /mnt/cdn is successfully loaded, pull static files from your repository.

/etc/systemd/system/cdn-pull.service

[Unit]
Description=Pull on CDN directory.
After=network-online.target

[Service]
User=youruserhere
Group=youruserhere
ExecStart=/usr/local/share/systemd/cdn-pull.sh

[Install]
WantedBy=mnt-cdn.mount
  • Clone the git repository with a user on system using a key with an alias
  • Change youruserhere to the user who cloned the repository
  • Add to /root/.ssh/config and toΒ  /root/.ssh/my_private_key the private key to do the pull

Meaning:

  • WantedBy=mnt-cdn.mount copy the files to RAM only after the /mnt/cdn is created
  • After=network-online.targetΒ pull the repository only when the network is ready

On pull, all files will be written by root as youruserhere:youruserhere.

After the pull, to reduce RAM occupation, this script doesn’t download directly to RAM .git directory but copy them with rsync excluding them:

/usr/local/share/systemd/cdn-pull.sh

#!/bin/bash
# stop on first error
set -e
cd /srv/cdn-all
git pull
exec rsync -a --exclude=.git --exclude=.gitignore /srv/cdn-all/* /mnt/cdn/

Get systemd to know about the mount and service

To reload systemd units, you have to

systemctl daemon-reload

Then do the mount via the systemd unit:

systemctl start mnt-cdn.mount

Enable on boot

Since the cdn-pull.service is tied to mnt-cdn.mount, both have to be enabled to run:

systemctl enable mnt-cdn.mount
systemctl enable cdn-pull.service
  1. When the system is ready create the tmpfs on /mnt/cdn/
  2. After tmpfs is successfully created by the unit, the file will be automatically synced through cdn-pull.service.

Mount will auto-start sync

Start only the mnt-cdn.mount:

systemctl start mnt-cdn.mount

And then ask for info about both services:

systemctl status mnt-cdn.mount
systemctl status cdn-pull.service
  • mnt-cdn.mount have to be active (mounted)
  • cdn-pull.service should be active (script is running) or inactive (sync is completed). In both cases, it’s ok.

With this set-up, when you restart the mnt-cdn.mount files will be automatically pulled and synced to RAM when system starts and when you start or restart mnt-cdn.mount service.

Next you can serve these files on nginx and the final step could be to auto-detect push to update files automagically.

See also

Create a Windows 10 recovery disk on Linux

In this howto there are the steps to follow when a Windows 10 OS is not bootable anymore and you haven’t a recovery disk. This is a typical case after a new OS will be installed on Dual boot or boot partition was altered.

    1. Download Windows 10 iso:
      1. Download the official Windows 10 image
    2. Prepare USB to be bootable:
      1. Open GParted with
        gparted /dev/DEVICE-TO-ERASE
        
      2. Select the USB drive
      3. Device > New partition table
      4. Select GPT
      5. Apply: this will delete any data on the USB
      6. Create a new NTFS partition then Apply (do not use FAT32 since some files can be greater than 4GB)
      7. Close GParted
    3. Write files:
      1. Unplug and plug USB
      2. Copy all Windows files to the empty USB drive using 7zip with:
        7z x -y -o/media/user/path-to-USB/ Win10_1809Oct_Italian_x64.iso
      3. If something goes wrong during copy, you can mount the ISO image then rsync the source with the USB drive (the trailing slash is important):
        cd path/to/usb/drive
        rsync -avzp /media/myuser/CCCOMA_X64FRE_IT-IT_DV91/ .
        
      4. umount
    4. Add boot flag
      1. Open the
      2. Select the new partition then
      3. Select Partition > Manage flags
      4. Select boot flag (esp will be auto-selected)
  1. Use windows tools
    1. Follow this howto by MS to recover MBR, restore BCD or similar actions

You can follow these steps to write on a USB a recovery ISO from windows the same way.

Disable password authentication on sshd

To disallow password authentication on ssh, adduser –disabled-password will not disable openSSH password.

To disable the password authentication, you have to put these values on /etc/ssh/sshd_config to:

PasswordAuthentication no
UsePAM no
PermitRootLogin no

Then you’ve to:

systemctl restart sshd

to apply changes.

Connection will not be reset so before logout try to login on a different terminal to check you can login.

Actually PermitRootLogin disable the root login for any method, but it’s an useful addition. Remember to add at least one user to the sudo group or you will not be able to operate as super-user without using su – root.

To check if password auth is disabled:

ssh -o PreferredAuthentications=password USER@HOST

Exprected output is:

USER@HOST: Permission denied (publickey).

Partition a new disk on linux using fdisk, lsblk and mkfs

First, you’ve to create a new partition.

You can list all available storage device with:

lsblk

If your disk is new, the new device will appear empty (without children on the tree).

Then:

fdisk /dev/sdc

Press m to show the manual.

To create a partition larger than 2TB, you’ve to use a GPT partition (g) then create a new extended partition (n) then with (p) it will show you how the partition will like before you write (w) them.

Then, lsblk will show the device with the new partition, e.g.:

sdc 8:32 0 3.5T 0 disk 
└─sdc1 8:33 0 3.5T 0 part

Then format the new partition /dev/sdc1 with the specified filesystem (e.g. ext4):

mkfs -t ext4 /dev/sdc1

If you haven’t take not of the UUID shown by mkfs after format, use blkid command to list the UUID of the device, so if device name change the fstab is still valid.

And add to /etc/fstab (put the last 0 to 1 to check filesystem on startup):

UUID=xxxxxxx-xxx-xxxx-xxx-xxxx /mnt/mydata ext4 defaults 0 0

To get the UUID later:

sudo blkid /dev/sdc1

Create the mount directory with:

mkdir /mnt/mydata

Then mount the new partition with:

mount /mnt/mydata

Get number of files or directory using tree

Tree is an useful linux command to display a tree representation a full directory structure or a part of it.

On a Debian based distro like Ubuntu install:

sudo apt-get install tree

The last line of tree print a line like this:

346 directories, 174 files

If you’re changing files and directories and you want a real-time update of files and directories number, you can use watch.

watch -n 20 'tree | tail -n 1'

Tree will print the tree, tail will extract the last line, then watch will refresh the result every 20 seconds.