ram

Sort of CDN to serve client-side libraries via an auto-pull git repo on tmpfs

/ chirale / Leave a comment

This configuration will allow to install on a Debian-based system a fast server for client libraries. Key technologies used are:

  • tmpfs to serve files from volatile memory
  • git / mercurial from github / bitbucket to get files from a public or private repository
  • systemd units to mount tmpfs and sync
  • nginx to serve files to user

On this first step you’ll create a service to reserve some RAM for static files, pulling them from a private or public repo.

Mount tmpfs with systemd

To serve files directly from RAM, you have to mount a tmpfs directory. You can do it on fstab:

/etc/fstab

tmpfs /mnt/cdn tmpfs rw,nodev,nosuid,size=300M 0 0

Or with a systemd unit:

/etc/systemd/system/mnt-cdn.mount

[Unit]
Description=Mount empty CDN directory on volatile memory

[Mount]
What=tmpfs
Where=/mnt/cdn
Type=tmpfs
Options=defaults,noatime,size=300M

[Install]
WantedBy=multi-user.target
  • noatime will disable last access on contained files, reducing write on disk
  • size will reserve 300MB for /mnt/cdn partition on RAM (increase as needed)
  • WantedBy=multi-user.target mount the partition on runlevel 3 (multi-user mode with networking)

Create two units on a local path like /usr/local/share/systemd then create a symlinks on /etc/systemd/system or create directly them on /etc/systemd/system. You can also directly create them on /usr/local/share/systemd.

Create the pull service

When the /mnt/cdn is successfully loaded, pull static files from your repository.

/etc/systemd/system/cdn-pull.service

[Unit]
Description=Pull on CDN directory.
After=network-online.target

[Service]
User=youruserhere
Group=youruserhere
ExecStart=/usr/local/share/systemd/cdn-pull.sh

[Install]
WantedBy=mnt-cdn.mount
  • Clone the git repository with a user on system using a key with an alias
  • Change youruserhere to the user who cloned the repository
  • Add to /root/.ssh/config and to  /root/.ssh/my_private_key the private key to do the pull

Meaning:

  • WantedBy=mnt-cdn.mount copy the files to RAM only after the /mnt/cdn is created
  • After=network-online.target pull the repository only when the network is ready

On pull, all files will be written by root as youruserhere:youruserhere.

After the pull, to reduce RAM occupation, this script doesn’t download directly to RAM .git directory but copy them with rsync excluding them:

/usr/local/share/systemd/cdn-pull.sh

#!/bin/bash
# stop on first error
set -e
cd /srv/cdn-all
git pull
exec rsync -a --exclude=.git --exclude=.gitignore /srv/cdn-all/* /mnt/cdn/

Get systemd to know about the mount and service

To reload systemd units, you have to

systemctl daemon-reload

Then do the mount via the systemd unit:

systemctl start mnt-cdn.mount

Enable on boot

Since the cdn-pull.service is tied to mnt-cdn.mount, both have to be enabled to run:

systemctl enable mnt-cdn.mount
systemctl enable cdn-pull.service
  1. When the system is ready create the tmpfs on /mnt/cdn/
  2. After tmpfs is successfully created by the unit, the file will be automatically synced through cdn-pull.service.

Mount will auto-start sync

Start only the mnt-cdn.mount:

systemctl start mnt-cdn.mount

And then ask for info about both services:

systemctl status mnt-cdn.mount
systemctl status cdn-pull.service
  • mnt-cdn.mount have to be active (mounted)
  • cdn-pull.service should be active (script is running) or inactive (sync is completed). In both cases, it’s ok.

With this set-up, when you restart the mnt-cdn.mount files will be automatically pulled and synced to RAM when system starts and when you start or restart mnt-cdn.mount service.

Next you can serve these files on nginx and the final step could be to auto-detect push to update files automagically.

See also