Autorestart nodejs app with supervisor on error and changes

This howto will show how to restart automatically a nodejs app on crash or on file changes using supervisor and nodemon.

Autorestart on changes

To install nodemon to autorestart app when you change app files:

npm install -g nodemon

To test it’s working, use nodemon like node, passing all parameters you would pass to node:

nodemon app.js --myoptionalparameter MYVALUE;

Autorestart on errors

To install supervisor on a Debian-based system to restart app on crashes:

sudo apt-get install supervisor

Then create a supervisor-app-run.sh wrapper script on a custom location to monitor:

#!/bin/bash
cd /path/to/my/app;
exec node app.js --myoptionalparameter MYVALUE;
  • exec is very important since it gave to supervisor the control of the process
  • if you specify nodemon instead of node , the app will not autorestart on crashes but only on changes. On production, only node should be used while on development nodemon is fine to track errors.

Now set up the config file for supervisor creating a new file on /etc/supervisor/conf.d/myapp.conf with:

[program:myapp]
directory=/path/to/my/app
command=bash /path/to/my/supervisor-app-run.sh
priority=10                ; the relative start priority (default 999)
autostart=true              ; start at supervisord start (default: true)
autorestart=true            ; retstart at unexpected quit (default: true)
; startsecs=-1                ; number of secs prog must stay running (def. 10)
; startretries=3              ; max # of serial start failures (default 3)
exitcodes=0,2               ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT             ; signal used to kill process (default TERM)
; stopwaitsecs=10             ; max num secs to wait before SIGKILL (default 10)
user=USER_TO_RUN_APP_HERE                   ; setuid to this UNIX account to run the program
log_stdout=true             ; if true, log program stdout (default true)
log_stderr=true             ; if true, log program stderr (def false)
logfile_maxbytes=10MB        ; max # logfile bytes b4 rotation (default 50MB)
logfile_backups=10          ; # of logfile backups (default 10)
  • change USER_TO_RUN_APP_HERE to a system user who can access to app files and directory

Now you have to reread to apply changes without restarting all other services:

sudo supervisorctl reread

So in case of errors you got something like:

ERROR: CANT_REREAD: Invalid user name fakeuserhere in section ‘program:myapp’ (file: ‘/etc/supervisor/conf.d/myapp.conf’)

On success:

myapp: available

If you’ve changed the app configuration, you have to:

sudo supervisorctl update

To apply, then restart the specific app.

sudo supervisorctl restart myapp

Keep an eye on supervisor processes with:

sudo supervisorctl status

Results:

myapp                            RUNNING   pid ****, uptime 0:00:59
anotherapp                       RUNNING   pid ****, uptime 0:29:33

Control the processes

Since exec was specified in the wrapper script before, supervisor can stop the node app on demand with:

sudo supervisorctl stop myapp

Then supervisorctl status will display something like this:

myapp                             STOPPED   Apr 27 22:53 AM
anotherapp                        RUNNING   pid ****, uptime 0:28:16

To run again:

sudo supervisorctl start myapp
  • When you will restart the service with systemctl restart supervisor, all /supervisor/conf.d/ files will be read again, then if they are set to autostart they will even if you’ve stopped them.
  • If your node.js app has more than one file to run (e.g. a couple of servers) you can copy and append the [program:myapp] configuration on the same file changing this second block to something like [program:myapptwo] specifying a new wrapper script

Using multiple deploy keys on github using .ssh/config

You can use multiple deploy keys for Github created with ssh-keygen following with these steps.

You have to add to your ~/.ssh/config

Host github_deploy_key_1
    Hostname github.com
    User git
    IdentityFile ~/.ssh/github_deploy_key_1_rsa

Host github_deploy_key_2
    Hostname github.com
    User git
    IdentityFile ~/.ssh/github_deploy_key_2_rsa

If you haven’t added your github name on git:

git config --global user.name "yourgithubname"
git config --global user.email "youremail@example.com"

Then clone your repository specifying your custom host, adapting what github suggest to you on repo page:

git clone git@github_deploy_key_1:yourgithubname/your-repo.git

If you have enabled push permissions you can use this deploy key even to update the repository.

In this way you can keep a server clean from your github passepartout and add only the keys it needs.

Create a Windows 10 recovery disk on Linux

In this howto there are the steps to follow when a Windows 10 OS is not bootable anymore and you haven’t a recovery disk. This is a typical case after a new OS will be installed on Dual boot or boot partition was altered.

  1. Download Windows 10 iso:
    1. Download the official Windows 10 image
  2. Prepare USB to be bootable:
    1. Open GParted with
gparted /dev/DEVICE-TO-ERASE
  • Select the USB drive
  • Device > New partition table
  • Select GPT
  • Apply: this will delete any data on the USB
  • Create a new NTFS partition then Apply (do not use FAT32 since some files can be greater than 4GB)
  • Close GParted
  • Write files:
    1. Unplug and plug USB
    2. Copy all Windows files to the empty USB drive using 7zip with:
      7z x -y -o/media/user/path-to-USB/ Win10_1809Oct_Italian_x64.iso
    3. If something goes wrong during copy, you can mount the ISO image then rsync the source with the USB drive (the trailing slash is important):
  • cd path/to/usb/drive
    rsync -avzp /media/myuser/CCCOMA_X64FRE_IT-IT_DV91/ .
    
  • umount
  • Add boot flag
    1. Open GParted selecting the device just written
    2. Select the new partition then
    3. Select Partition > Manage flags
    4. Select boot flag (esp will be auto-selected)
    1. Use windows tools
      1. Follow this howto by MS to recover MBR, restore BCD or similar actions

    You can follow these steps to write on a USB a recovery ISO from windows the same way.

    Disable password authentication on sshd

    To disallow password authentication on ssh, adduser –disabled-password will not disable openSSH password.

    To disable the password authentication, you have to put these values on /etc/ssh/sshd_config to:

    PasswordAuthentication no
    UsePAM no
    PermitRootLogin no

    Then you’ve to:

    systemctl restart sshd

    to apply changes.

    Connection will not be reset so before logout try to login on a different terminal to check you can login.

    Actually PermitRootLogin disable the root login for any method, but it’s an useful addition. Remember to add at least one user to the sudo group or you will not be able to operate as super-user without using su – root.

    To check if password auth is disabled:

    ssh -o PreferredAuthentications=password USER@HOST

    Exprected output is:

    USER@HOST: Permission denied (publickey).

    Partition a new disk on linux using fdisk, lsblk and mkfs

    First, you’ve to create a new partition.

    You can list all available storage device with:

    lsblk

    If your disk is new, the new device will appear empty (without children on the tree).

    Then:

    fdisk /dev/sdc

    Press m to show the manual.

    To create a partition larger than 2TB, you’ve to use a GPT partition (g) then create a new extended partition (n) then with (p) it will show you how the partition will like before you write (w) them.

    Then, lsblk will show the device with the new partition, e.g.:

    sdc 8:32 0 3.5T 0 disk 
    └─sdc1 8:33 0 3.5T 0 part

    Then format the new partition /dev/sdc1 with the specified filesystem (e.g. ext4):

    mkfs -t ext4 /dev/sdc1

    If you haven’t take not of the UUID shown by mkfs after format, use blkid command to list the UUID of the device, so if device name change the fstab is still valid.

    And add to /etc/fstab (put the last 0 to 1 to check filesystem on startup):

    UUID=xxxxxxx-xxx-xxxx-xxx-xxxx /mnt/mydata ext4 defaults 0 0

    To get the UUID later:

    sudo blkid /dev/sdc1

    Create the mount directory with:

    mkdir /mnt/mydata

    Then mount the new partition with:

    mount /mnt/mydata

    Get number of files or directory using tree

    Tree is an useful linux command to display a tree representation a full directory structure or a part of it.

    On a Debian based distro like Ubuntu install:

    sudo apt-get install tree
    

    The last line of tree print a line like this:

    346 directories, 174 files

    If you’re changing files and directories and you want a real-time update of files and directories number, you can use watch.

    watch -n 20 'tree | tail -n 1'
    

    Tree will print the tree, tail will extract the last line, then watch will refresh the result every 20 seconds.

    Nginx configuration for Django

    Django is a powerful framework for building websites. To run a production website, usually an application server is used. So nginx will do two basic things:

    • Serve your Django application from the application server port to the web port (Reverse Proxy)
    • Serve static and media files

    The application server used in this example is gunicorn, the application server chosen by Instagram of the earlier days, but it can be anything running on port 9999. Change port number as required in the example.

    The following nginx conf was adapted from this, with some additions and it contains:

    • a commented non www to www website redirect
    • gzip for javascript, json, css and proxy routes
    • media files with etag (1 year)
    • static files with etag (1 minute)
    • an host-based favicon distributor (reusable as is)
    • a commented basic auth to make a website private
    • reverse proxy to gunicorn
    • a simple block for a common type of malicious activity

    It works fine with Django 1 and 2.

    # Howto: https://chirale.org/2018/08/23/nginx-configuration-for-django/
    # uncomment for redirect
    # server {
    #    # redirect WITH www from example.com and example.net
    #    listen 80;
    #    server_name example.com example.net;
    #    return 301 http://www.example.com$request_uri;
    # }
    
    server {
        listen	80;
        # the domain name it will serve for
        server_name www.example.com;
        charset     utf-8;
    
        # max upload size
        client_max_body_size 75M;
    
        # enable gzip for proxy requests
        gzip on;
        gzip_proxied any;
        gzip_vary on;
        gzip_http_version 1.1;
        gzip_types application/javascript application/json text/css text/xml;
        gzip_comp_level 4;
    
        # @see http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html#configure-nginx-for-your-site
    
        # Django media
        location /media  {
            etag on;
            expires 365d;
            alias /path/to/media_root;  # your Django project's media files - amend as required
        }
    
        location /static {
            etag on;
            expires 1m;
            alias /path/to/static_root; # your Django project's static files - amend as required
        }
    
        location /favicon.ico {
            # all favicons inside /path/to/favicons/ this directory
            # notation: www.example.com.ico
           alias /path/to/favicons/$host.ico;
        }
    
        location / {
            # an HTTP header important enough to have its own Wikipedia entry:
            #   http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    
            # enable this if and only if you use HTTPS, this helps Rack
            # set the proper protocol for doing redirects:
            # proxy_set_header X-Forwarded-Proto https;
    
            # pass the Host: header from the client right along so redirects
            # can be set properly within the Rack application
            proxy_set_header Host $http_host;
    
            # we don't want nginx trying to do something clever with
            # redirects, we set the Host: header above already.
            proxy_redirect off;
    
            # set "proxy_buffering off" *only* for Rainbows! when doing
            # Comet/long-poll stuff.  It's also safe to set if you're
            # using only serving fast clients with Unicorn + nginx.
            # Otherwise you _want_ nginx to buffer responses to slow
            # clients, really.
            # proxy_buffering off;
    
            # Uncomment for maintenance
            ### auth_basic "Insert password here";
            ### auth_basic_user_file /path/to/.htpasswd;
    
            proxy_connect_timeout       30000;
            proxy_send_timeout          30000;
            proxy_read_timeout          30000;
            send_timeout                30000;
    
            # @see https://eng.eelcowesemann.nl/linux-unix-android/nginx/nginx-blocking/ and seositecheckup.com
            if ($http_user_agent ~ "libwww-perl") {
              return 403;
              break;
            }
    
            # Try to serve static files from nginx, no point in making an
            # *application* server like Unicorn/Rainbows! serve static files.
            if (!-f $request_filename) {
                proxy_pass http://localhost:9999;
                break;
            }
        }
    }
    
    

    Run nginx -t to check and then systemctl reload nginx to apply.

    This is a http version, to configure the website for https follow this howto.