Sort of CDN to serve client-side libraries via an auto-pull git repo on tmpfs

This configuration will allow to install on a Debian-based system a fast server for client libraries. Key technologies used are:

  • tmpfs to serve files from volatile memory
  • git / mercurial from github / bitbucket to get files from a public or private repository
  • systemd units to mount tmpfs and sync
  • nginx to serve files to user

On this first step you’ll create a service to reserve some RAM for static files, pulling them from a private or public repo.

Mount tmpfs with systemd

To serve files directly from RAM, you have to mount a tmpfs directory. You can do it on fstab:

/etc/fstab

tmpfs /mnt/cdn tmpfs rw,nodev,nosuid,size=300M 0 0

Or with a systemd unit:

/etc/systemd/system/mnt-cdn.mount

[Unit]
Description=Mount empty CDN directory on volatile memory

[Mount]
What=tmpfs
Where=/mnt/cdn
Type=tmpfs
Options=defaults,noatime,size=300M

[Install]
WantedBy=multi-user.target
  • noatime will disable last access on contained files, reducing write on disk
  • size will reserve 300MB for /mnt/cdn partition on RAM (increase as needed)
  • WantedBy=multi-user.target mount the partition on runlevel 3 (multi-user mode with networking)

Create two units on a local path like /usr/local/share/systemd then create a symlinks on /etc/systemd/system or create directly them on /etc/systemd/system. You can also directly create them on /usr/local/share/systemd.

Create the pull service

When the /mnt/cdn is successfully loaded, pull static files from your repository.

/etc/systemd/system/cdn-pull.service

[Unit]
Description=Pull on CDN directory.
After=network-online.target

[Service]
User=youruserhere
Group=youruserhere
ExecStart=/usr/local/share/systemd/cdn-pull.sh

[Install]
WantedBy=mnt-cdn.mount
  • Clone the git repository with a user on system using a key with an alias
  • Change youruserhere to the user who cloned the repository
  • Add to /root/.ssh/config and to  /root/.ssh/my_private_key the private key to do the pull

Meaning:

  • WantedBy=mnt-cdn.mount copy the files to RAM only after the /mnt/cdn is created
  • After=network-online.target pull the repository only when the network is ready

On pull, all files will be written by root as youruserhere:youruserhere.

After the pull, to reduce RAM occupation, this script doesn’t download directly to RAM .git directory but copy them with rsync excluding them:

/usr/local/share/systemd/cdn-pull.sh

#!/bin/bash
# stop on first error
set -e
cd /srv/cdn-all
git pull
exec rsync -a --exclude=.git --exclude=.gitignore /srv/cdn-all/* /mnt/cdn/

Get systemd to know about the mount and service

To reload systemd units, you have to

systemctl daemon-reload

Then do the mount via the systemd unit:

systemctl start mnt-cdn.mount

Enable on boot

Since the cdn-pull.service is tied to mnt-cdn.mount, both have to be enabled to run:

systemctl enable mnt-cdn.mount
systemctl enable cdn-pull.service
  1. When the system is ready create the tmpfs on /mnt/cdn/
  2. After tmpfs is successfully created by the unit, the file will be automatically synced through cdn-pull.service.

Mount will auto-start sync

Start only the mnt-cdn.mount:

systemctl start mnt-cdn.mount

And then ask for info about both services:

systemctl status mnt-cdn.mount
systemctl status cdn-pull.service
  • mnt-cdn.mount have to be active (mounted)
  • cdn-pull.service should be active (script is running) or inactive (sync is completed). In both cases, it’s ok.

With this set-up, when you restart the mnt-cdn.mount files will be automatically pulled and synced to RAM when system starts and when you start or restart mnt-cdn.mount service.

Next you can serve these files on nginx and the final step could be to auto-detect push to update files automagically.

See also

Advertisements

Delete git files from public GitHub history

To delete git files uploaded accidentally to GitHub (or any other public repository) do these steps:

  1. Download https://rtyley.github.io/bfg-repo-cleaner/ as suggested by GitHub
  2. git clone –mirror GIT_REPOSITORY_URL
  3. cd path/to/cloned/repository
  4. Download BFG
  5. java -jar /path/to/download/dir/bfg-VERSION.jar –delete-files filename.ext
  6. Run the command specified by BFG (usually git reflog expire –expire=now –all && git gc –prune=now –aggressive)
  7. git push

If you get an error on pull, probably you haven’t cloned the repository as step 2.

Browsing the public history, any reference to the filename.ext file disappear.

Read more about BFG and the –mirror option on this discussion.