Clear Varnish cache via PHP: a Drupal 7 proof of concept

Using Varnish as reverse proxy or proxy is an useful approach to reduce the load of webservers like Apache.

In Drupal 7 I’ve to clear the varnish cache of a specific domain when Drupal caches are globally cleared. Drupal has the right hook invoked when cache are cleared:

function clearcachevarnish_flush_caches() {
  $filename = '/var/www/varnishdomains2cleardir/varnishdomains2clear';
  // each domain on a separate line: append to the end of the file
  $myfile = fopen($filename, "a");
  $h = $_SERVER['HTTP_HOST'];
  $txt = $h . "\n";
  fwrite($myfile, $txt);
  fclose($myfile);
  drupal_set_message('Varnish cache queued to be cleared. Please wait 1 minute before checking.');
  // no cache table should be cleared
  return array();
}

Now this piece of code simply adds the current domain to a ASCII text file on /var/www/varnishdomains2cleardir/varnishdomains2clear.

Preparing the file to the write

On CentOS you have to add /var/www/varnishdomains2cleardir to the httpd-writable directories list using:

mkdir /var/www/varnishdomains2cleardir;
chcon -v --type=httpd_sys_content_t /var/www/varnishdomains2cleardir;
chown myuser:mygroup /var/www/varnishdomains2cleardir;
chmod -R 777 /var/www/varnishdomains2cleardir;
touch /var/www/varnishdomains2cleardir/varnishdomains2clear;

Now the empty file is ready to be written by your hook_flush_caches() implementation. Now enable the clearvarnishcache module and clear the cache to write the current domain name to the file.

The clear varnish cache script

To clear the varnish cache you usually have to be logged as root using the command varnishadm. Here a script that will read the domains file written above, clear the varnish cache for that domain and then remove the domains lines.

#!/bin/bash
callinguser=`whoami`
if [ "root" != "$callinguser" ]
then
 echo "Only root can run this command."
 exit 1
fi
cd /path/to/clear/cache/command/

date=`date +%Y-%m-%d_%H:%M:%S`

# check lock
# prevent the script from being run more than once
if [ -f /tmp/clearcachevarnish-lock ]; then
echo "Script clearcachevarnish is already running. You can rm /tmp/clearcachevarnish-lock to break the lock manually."
exit 1
fi
touch /tmp/clearcachevarnish-lock
dominidapulire=`less /var/www/varnishdomains2cleardir/varnishdomains2clear`
while [[ ! -z $dominidapulire ]]
do
 dominio=$(echo "$dominidapulire" | sed -n '$p')
 echo $dominio
 dominidapulire=$(echo "$dominidapulire" | sed '$d')
 if [ "" != "$dominio" ]
 then
 varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret ban req.http.host == "$dominio"
 echo "varnish cleared on $dominio"
 fi
done
# remove all domains lines
truncate --size 0 /var/www/varnishdomains2cleardir/varnishdomains2clear

# remove lock
rm /tmp/clearcachevarnish-lock

Make¬†this script as executable .sh file using chmod a+x on it. If you run the bash script, varnish cache for files on the domains list will be cleared. It’s not so useful when using the Drupal UI so we should schedule this task periodically, e.g. every minute.

Scheduling the varnish clear cache

Here the crontab entry for execute the script every minute:


* * * * * root /path/to/clear/cache/command/clearcachevarnish.sh

The steps

  1. User clear Drupal cache
  2. hook_flush_caches() is invoked: the domains list file is written
  3. clear varnish cache script is launched by root every minute
  4. for each domain in the list, varnish cache is cleared

This is the end of this proof of concept. The code wasn’t tested against attacks so please comment if you have any suggestion to improve it. I’m not very fond of the idea of a php script writing something read by a bash script but this is the less problematic solution I found for this case.

Advertisements

Web fonts and dynamic height calculation issues on jQuery

Recently we’ve nice fonts on web pages like Google Fonts and other web fonts. Take this case, you have to set two divs to the same height. One (div.funny) has some text with Google Fonts, the other is empty.

On Chrome console you type something like:

jQuery(".very", ".myview").height(function () {
  jQuery(this).height(jQuery(this).parent(".myview").find('.funny').height());
});

Div.very and div.funny are now at the same height.

Now if you try to do the same on jquery document ready you got elements with different height. Why?

Because the calculation happens on document ready but before fonts are loaded. The solution is to wrap the code on $(window).load().

$(window).load(function () {
  $(".very", ".myview").height(function () {
    $(this).height($(this).parent(".myview").find('.funny').height());
  });
});

Now .very and .funny are at the same height.

See also:
Calculate Container’s Height After The Font File Loads

Web fonts and dynamic height calculation issues on jQuery

Recently we’ve nice fonts on web pages like Google Fonts and other web fonts. Take this case, you have to set two divs to the same height. One (div.funny) has some text with Google Fonts, the other is empty.

On Chrome console you type something like:

jQuery(".very", ".myview").height(function () {
  jQuery(this).height(jQuery(this).parent(".myview").find('.funny').height());
});

Div.very and div.funny are now at the same height.

Now if you try to do the same on jquery document ready you got elements with different height. Why?

Because the calculation happens on document ready but before fonts are loaded. The solution is to wrap the code on $(window).load().

$(window).load(function () {
  $(".very", ".myview").height(function () {
    $(this).height($(this).parent(".myview").find('.funny').height());
  });
});

Now .very and .funny are at the same height.

See also:
Calculate Container’s Height After The Font File Loads

Create nice unicode PDF using Python

Today I started one of the less motivating activities in Python 2.x: encoding.

In Python 3 unicode will be everywhere, but as of the 2.6 version I’ve on one of the server I have to endure.

Objective: get data from a UTF-8 encoded json and print a nice PDF.

Tools: json, urllib2, fpdf, cgi

What you need:
pyfpdf: https://code.google.com/p/pyfpdf/downloads/list

  • Download¬†fpdf-1.7.hg.zip¬†or more recent
  • Unzip, enter the directory and python setup.py install
  • locate fpdf
  • cd¬†/usr/lib/python2.6/site-packages/fpdf (or the directory name you got with locate)
  • Download unicode fonts for fpdf
  • Unzip and copy the fonts folder in the fpdf directory

Now you have a working FPDF with unicode support and unicode fonts. Start to write your script, I assume you’re using python 2.6, if not change python2.6 to your python version (e.g. 2.7) or remove version number in the heading (just python). As now FPDF works with Python 2.5 to 2.7.

Here I write a simple cgi-bin script, so you have to put it in the /var/www/cgi-bin directory (CentOS) or in /usr/lib/cgi-bin (Debian).

#!/usr/bin/env python2.6
#-*- coding: utf-8 -*-
from fpdf import FPDF
import json
import urllib2
import os
import cgi
import sys
# set system encoding to unicode
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

Now get some arguments from url. These will be used to compile a query to a external json service.

# e.g. http://example.com/cgi-bin/myscript.py?lang=en&sid=2
sid = arguments.getlist('sid')[0]
lang = arguments.getlist('lang')[0]
# compile a request to get a particular element from an external json
dataurl = "http://example.com/external-json-source?lang=%s&sid=%s" % (lang, sid)
# load json from dataurl and convert into python elements
data = json.load(urllib2.urlopen(dataurl))
# the json has a user attribute: the user attribute has name and surname attributes as strings
user = data['user']
# title is a simple string
title = data['title']

Now you have to load the json from the external source. Json must be encoded in UTF-8:

lato_lungo = 297
lato_corto = 210
pdf = FPDF('L','mm','A4')
# add unicode font
pdf.add_font('DejaVu','','DejaVuSansCondensed.ttf',uni=True)
pdf.add_page()
pdf.cell(w=lato_lungo,h=9,txt=title,border=0,ln=1,align='L',fill=0)
pdf.set_font('DejaVu','',12)
# paragraphs rendered as MultiCell
# @see https://code.google.com/p/pyfpdf/wiki/MultiCell
# print key: values for each user['data'] dictionary attributes
for val in user.iteritems():
    pdf.multi_cell(w=0,h=5,txt="%s: %s" % val)
# finally print pdf
print pdf.output(dest='S')

Now:

  1. Open your browser and visit http://example.com/cgi-bin/myscript.py?lang=en&sid=2
  2. The external source http://example.com/external-json-source?lang=en&sid=2 is grabbed and converted into a python data structure. Both source and destination encoding are unicode utf-8.
  3. Data from external source are used to create the pdf.

You can use as many fonts as you have in the fpdf/font directory, just add those using pdf.add_font().

https://code.google.com/p/pyfpdf/downloads/list

Django development on Virtualbox: step by step setup

I had a bad morning trying to repair my Cygwin installation from a virtualenv mess. It’s time to get a Debian and install it on a Virtualbox for my new django project!

  • Windows: host
  • Debian: guest

Choosing the distro: what I want

  • Python 2.6
  • Django 1.4
  • Apache + Mysql

I’m a Debian fan from years so I go to the Debian website and download Wheezy netinst¬†iso (32 bit, since I’m on a 32 bit OS and I want to use more core): wheezy met all the requirements above.

I already have a Virtualbox, so what I do is to add a new virtual disk and to add the new Wheezy netinst iso on CD/DVD images. Then I create a new Debian machine (32 bit) with two cores. I choose the iso image to be mounted on startup so the Debian setup process will start on boot.

As network device, I choose the Bridge option, so I can access the machine later from my windows host.

Installing the system

When you turn your machine on, many choices will be prompted to you. I install the webserver (apache) from the list, removed SQL server and print server and then leave desktop selected and the other default values. After some minutes Debian is installed and I can log in with the credential I have specified during installation.

Use WORKGROUP as network name if you’re running a Windows host when asked.

Install django packages

Under the Application menu, find the Debian package management tools¬†to install what you want. As the requirements I’ve listeded above I search and install those packages:

  • python-django (1.4.1-2)
  • libapache2-mod-uwsgi
  • libapache2-mod-wsgi
  • mysql-server
  • samba

Later you can install more useful packages like virtualenv and phpmyadmin.

After you’ve installed those packages, you can do some test.¬†Open a shell (Accessories > Terminal) and then type these commands:

What version of python I’m running?

$ python

Python 2.7.3rc2 (default, Apr 22 2012, 22:35:38)
[GCC 4.6.3] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>>

So I’ve python 2.7. Good!

>>> import django
>>> django.VERSION
(1, 4, 1, ‘final’, 0)

And I’ve Django 1.4.1.

Share your code to the Windows network (workgroup)

Now I want to read the code from one machine to another. I choose Samba server to read and write files from the virtual machine to windows and back. It will be useful since I’ve a complete Eclipse + pydev¬†IDE on windows and I love work with it.

I open a Root terminal and type:

# ifconfig

If you choosed the Bridge network interface on installation, you will got something like this:

eth0 Link encap:Ethernet HWaddr ??????
inet addr:192.168.0.104 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: ???????????/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:49280 errors:0 dropped:0 overruns:0 frame:0
TX packets:19777 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:45400047 (43.2 MiB) TX bytes:1887849 (1.8 MiB)
Interrupt:19 Base address:0xd020

The address in bold (192.168.0.104) is the local network address of my virtual machine. If I just type this address in the Chrome browser it’s running on Windows (host) I got the “It works!” from Apache on the virtual machine. If you can’t see nothing, left click on the network icon on the bottom of your virtualbox windows > click on the menu voice and then choose the Bridge option. Then redo the ifconfig as above.

Samba tuning

Create a directory to store your django code (inside current user home folder). Open the terminal as normal user:

$ cd
$ mkdir my-django-code

Then share this folder with samba. To do this, let’s create a new user without a password:

adduser guest –home=/home/public –shell=/bin/false –disabled-password

Then add these lines to /etc/samba/smb.conf on “## Authentication ##” section:

security = share
guest_account = guest
invalid_users = root

obey pam restrictions = yes

And then after the [cdrom] commented text:

[my-django-code]
comment=Django-code
read only = no
locking = no
path = /home/myuser/my-django-code
guest ok = yes
force user = myuser

Where myuser is my (normal) user name. The lines above tell something like this to samba:

  • Let a guest user access without a password
  • …to the path¬†/home/myuser/my-django-code
  • …”masquerading” like myuser

The “masquerade” thing is all about having the right to write files created from myuser from the guest user on the host.

When i browse my Workgroup on windows, I found the machine name I choose during installation and inside I found the my-django-code directory. I try to read and write files from the host (Windows) and from the guest (Debian) and it’s all ok.

Django, finally!

If you’re starting to develop on django, so this howto for beginners will help you a lot. Since I’ve installed the python-django package from Debian, to start a project is simple as typing this:

$ cd
$ cd my-django-code
$ django-admin startproject django_unchained
$ cd django_unchained
$ python manage.py runserver 192.168.0.104:8000

Where 192.168.0.104 is the virtual machine local network address from above and 8000 the port of the django testing webserver.

I type:

http://192.168.0.104:8000

on Chrome (host: Windows) and I get the hello page from Django. Perfect!

Then, I can just follow the django howto to do the right things during the creation of my new app django_unchained!

You can also explore the must-have list of tools and sites for Python developers.

Joomla Webform alternative: JForms

I don’t use Joomla, but sometimes a friend or a colleague of mine ask me some tips. My answer could be: in Drupal it will take 5 minutes.

Since I’m very open-minded, today instead of replying in that awkward way I try to found an alternative to a very useful module for Drupal called Webform. I’ve found a limited but working equivalent for Joomla called JForms.

Tested on:

Installation:

  1. Install the extension using URL installation and paste the extension link (zip file from link above).
  2. Create a new form (see the visual howto below) with at least one field, one button, a form title, and a Thank you page (eg. index.php)
  3. Save the form.
  4. Create a new menu item from Menu > Main Menu > JForms > Standard forms  and fill the Select Form field.

Now that you’ve a form, you can use it from your site (click on menu item written created on (4) ).

If you have to link two forms together (eg. Customers -> Company), you have to use the DBList field.

  1. Create the Firm form
  2. Create the Customers form
  3. On Customers form add a DBList field

DB List field must have these values:

Table name #__jforms_9c7aa
Field key id
Field value hcf94
Field sort hcf94
  • Table name is the table name (mydbprefix_jforms_9c7aa) with the placeholder #__ instead of mydbprefix_.
  • Field key is id. The form select value will be set to an univocal id instead of the raw value.
  • Field value: value as displayed by the user.
  • Field sort: sort the list by this field.

HTML result:

<select style="width: 70px;" id="h4443_14" name="h4443[]">
<option selected="selected" value="1">Barilla</option>
<option value="2">Parmalat</option>
</select>

When creating a new customer, you’ll see a dropdown with a list of companies from the Company table and the two forms are connected.

See also: