Autorestart nodejs app with supervisor on error and changes

This howto will show how to restart automatically a nodejs app on crash or on file changes using supervisor and nodemon.

Autorestart on changes

To install nodemon to autorestart app when you change app files:

npm install -g nodemon

To test it’s working, use nodemon like node, passing all parameters you would pass to node:

nodemon app.js --myoptionalparameter MYVALUE;

Autorestart on errors

To install supervisor on a Debian-based system to restart app on crashes:

sudo apt-get install supervisor

Then create a supervisor-app-run.sh wrapper script on a custom location to monitor:

#!/bin/bash
cd /path/to/my/app;
exec node app.js --myoptionalparameter MYVALUE;
  • exec is very important since it gave to supervisor the control of the process
  • if you specify nodemon instead of node , the app will not autorestart on crashes but only on changes. On production, only node should be used while on development nodemon is fine to track errors.

Now set up the config file for supervisor creating a new file on /etc/supervisor/conf.d/myapp.conf with:

[program:myapp]
directory=/path/to/my/app
command=bash /path/to/my/supervisor-app-run.sh
priority=10                ; the relative start priority (default 999)
autostart=true              ; start at supervisord start (default: true)
autorestart=true            ; retstart at unexpected quit (default: true)
; startsecs=-1                ; number of secs prog must stay running (def. 10)
; startretries=3              ; max # of serial start failures (default 3)
exitcodes=0,2               ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT             ; signal used to kill process (default TERM)
; stopwaitsecs=10             ; max num secs to wait before SIGKILL (default 10)
user=USER_TO_RUN_APP_HERE                   ; setuid to this UNIX account to run the program
log_stdout=true             ; if true, log program stdout (default true)
log_stderr=true             ; if true, log program stderr (def false)
logfile_maxbytes=10MB        ; max # logfile bytes b4 rotation (default 50MB)
logfile_backups=10          ; # of logfile backups (default 10)
  • change USER_TO_RUN_APP_HERE to a system user who can access to app files and directory

Now you have to reread to apply changes without restarting all other services:

sudo supervisorctl reread

So in case of errors you got something like:

ERROR: CANT_REREAD: Invalid user name fakeuserhere in section ‘program:myapp’ (file: ‘/etc/supervisor/conf.d/myapp.conf’)

On success:

myapp: available

Keep an eye on supervisor processes with:

sudo supervisorctl status

Results:

myapp                            RUNNING   pid ****, uptime 0:00:59
anotherapp                       RUNNING   pid ****, uptime 0:29:33

Control the processes

Since exec was specified in the wrapper script before, supervisor can stop the node app on demand with:

sudo supervisorctl stop myapp

Then supervisorctl status will display something like this:

myapp                             STOPPED   Apr 27 22:53 AM
anotherapp                        RUNNING   pid ****, uptime 0:28:16

To run again:

sudo supervisorctl start myapp
  • When you will restart the service with systemctl restart supervisor, all /supervisor/conf.d/ files will be read again, then if they are set to autostart they will even if you’ve stopped them.
  • If you’ve specified node instead of nodemon on the wrapper script, you should use supervisorctl restart myapp to apply changes
  • If your node.js app has more than one file to run (e.g. a couple of servers) you can copy and append the [program:myapp] configuration on the same file changing this second block to something like [program:myapptwo] specifying a new wrapper script
Advertisements

Autolaunch a command and restart it on quit on CentOS

I’ve a django-admin command running as a server thanks to gevent. I want this server to run on boot and autorestart on quit.  StackOverflow give me a hint: use Supervisor.

On a Centos 5 distro:

# find supervisor for your distro...
yum search supervisor
# ...and install it
yum install supervisor.noarch
nano /etc/supervisord.conf

At the end of the file, add a new program:

[program:myfunnydjangocommand]
command=/usr/bin/env python26 /usr/local/etc/django-apps/foo/manage.py tcpapi 4114
priority=999                ; the relative start priority (default 999)
autostart=true              ; start at supervisord start (default: true)
autorestart=true            ; retstart at unexpected quit (default: true)
; startsecs=-1                ; number of secs prog must stay running (def. 10)
; startretries=3              ; max # of serial start failures (default 3)
exitcodes=0,2               ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT             ; signal used to kill process (default TERM)
; stopwaitsecs=10             ; max num secs to wait before SIGKILL (default 10)
; user=root                   ; setuid to this UNIX account to run the program
log_stdout=true             ; if true, log program stdout (default true)
log_stderr=true             ; if true, log program stderr (def false)
logfile=/var/log/myfunnydjangocommand.log    ; child log path, use NONE for none; default AUTO
logfile_maxbytes=1MB        ; max # logfile bytes b4 rotation (default 50MB)
logfile_backups=10          ; # of logfile backups (default 10)

Then, start supervisord.

service supervisord start

Take a look to supervisord log file:

less +G /var/log/supervisor/supervisord.log

You’ll see something like this:

2013-06-07 11:54:16,559 CRIT Supervisor running as root (no user in config file)
2013-06-07 11:54:16,576 INFO /var/tmp/supervisor.sock:Medusa (V1.1.1.1) started at Fri Jun  7 11:54:16 2013
        Hostname: <unix domain socket>
        Port:/var/tmp/supervisor.sock
2013-06-07 11:54:16,645 CRIT Running without any HTTP authentication checking
2013-06-07 11:54:16,654 INFO daemonizing the process
2013-06-07 11:54:16,657 INFO supervisord started with pid 19316
2013-06-07 11:54:16,666 INFO spawned: 'myfunnydjangocommand' with pid 19318
2013-06-07 11:54:17,670 INFO success: myfunnydjangocommand entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Read documentation about the configuration options but keep in mind your Supervisor version. I don’t use supervisorctl because of this bug, if you get an error simply go with service supervisord… but if you have a newer version this should be already fixed.

Note: myfunnydjangocommand.log doesn’t contain anything useful in my experience but maybe it’s related how I write the output since I’ve written it to use interactively, outputting lines directly to the user. I’ll update this post if I find how to solve this issue.