Dancer:

stupidtwitterstats.com pulls out some interesting stats about people you follow on twitter. I use Net::Twitter::Lite to talk to Twitter. I wrote a small class to analyze the data I get from Twitter and that keeps my route clean. I use Template Toolkit for, well, templating. There is a ‘lite’ version of Template Toolkit which comes as default with Dancer, but since I’ve been a TT user (dare I add a power user) for a while now, I went with the real thing.

I initially ran my app using Perl as:

perl bin/app.pl

I tried enabling “auto reloading” of my module so that any changes to it are immediately availble without restarting the app but for some reasons I couldn’t get it to work consistently on MacOS. A quick note to the Dancer mailing list revealed an alternate solution – using Plack with the ‘shotgun’ loader. The latter reloads your entire app for each request – a bit like CGI. If you are using modules that tend to have a long start-up time (like Moose), you can also tell plack to not load them every time:

plackup -L Shotgun -p 3000 bin/app.pl

To prevent certain modules from being reloaded:

plackup -MMoose -MDBIx::Class -L Shotgun bin/app.pl

This post from the 2009 plack advent calendars has more details.

Deploying on EC2:

The biggest problem I had starting with EC2 was documentation. Amazon overwhelms you with a lot of TLAs and their circuitous documentation makes going in circles seem like walking in a straight line (as the diameter of the circle tends to infinity this is indeed how it will feel, but I digress). That said, I came across their getting started guide which, alongwith the new web-based management console, made things a breeze.

The next big hurdle was picking the right distro of Linux to deploy on. I have more experience running Debian/Ubuntu in production than any other distro. Canonical’s 10.04 LTS Server was my first choice. While setting it up was a breeze, logging into it for the first time informed me that I had some pending updates. Installing those updates led me to a point in Grub configuration where the machine just froze. I didn’t take things any further.

To keep things simple I decided to go with the default Amazon 64-bit Linux instance. Now I would’ve loved to get hold of their custom Linux build just to replicate the production environment on my machine but Amazon doesn’t give it out. It looks like it is a Fedora derivative (there are fingerprints all over the place – e.g. in the welcome page you get when you install niginx) so one could run Fedora and get quite close to the Amazon provided Linux instance.

The Amazon Linux image comes with Perl 5.10.1. My first step was to install the ‘Development Tools’ bundle so that I could build things from source if needed.

sudo yum groupinstall 'Development Tools'

I then installed CPAN Minus (App::cpanminus), which is my preferred tool for installing things off CPAN.

sudo cpan App::cpanminus

I then used it to get Dancer:

sudo cpanm dancer

followed by other CPAN dependencies my app had.

At this stage I opened port 80 through the AWS EC2 console and tested my app to make sure it was running fine and was accessible over the internet using the temporary Amazon supplied domain name. I then got an Elastic IP and tied it to my running machine instance. I also went to my domain registrar (Dreamhost) and pointed my domain’s A record to this IP.

My next step was to install the Starman web-server under which my application is deployed.

sudo cpanm starman

I ran my app again – this time under Starman and checked it over the internet to make sure that everything was fine so far:

sudo /usr/local/bin/plackup -s Starman -p 80 -E deployment --workers=10 \
 -a /home/apps/TwitterToys/bin/app.pl

Satisfied, I moved on to the next big step – installing and configuring nginx. While ideally I would’ve loved to install the latest 0.8.x branch of nginx, it wasn’t available out of the box on Amazon’s Linux image. Indeed, even the most recent Linux distros (Ubuntu 10.04 Server or Debian Lenny/Squeeze) seem to give it a miss. While nginx compiles from source on most distros without problems, keeping it updated, patched and running can be daunting. So I settled for the default 0.7.67 install via yum:

sudo yum install nginx

I use nginx to do all the HTTP related stuff like gzipping content, serving static files, adding the right expires header and so on. It also acts as a caching proxy in my setup. Dancer running under Starman/Plack does everything else.

The following lines setup gzip response compression and a caching zone:

http {
    ..
    ..
    ..
    gzip on;
    gzip_min_length 1024;

    proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=twitter:8m
max_size=64m inactive=60m;
    proxy_temp_path /tmp;
    proxy_cache_key "$scheme://$host$request_uri";
    proxy_cache_valid 200 60m;
	..
	..
	..

I then setup the server to proxy the requests to my running Dancer instance:

  server {
        server_name stupidtwitterstats.com;
        listen       80;

        location / {
            proxy_cache twitter;

            #bypass cache for this path so that I can check my API usage
            set $do_not_cache 0;

            if ( $request_uri ~ "^/xyz$" ) {
                 set $do_not_cache 1;
            }
            proxy_no_cache $do_not_cache;
            proxy_pass http://127.0.0.1:5001;
            proxy_redirect http://127.0.0.1:5001/ http://$host/;
            expires 1h;
        }

I also setup another location block within the same server block to let nginx serve all the images directly:

        location /images/ {
            alias /home/apps/TwitterToys/public/images/;
            expires 30d;
            access_log off;
        }

I fired up Starman again – this time on port 5001 and bound to 127.0.0.1 and tested nginx from the internet to make sure that everything was working fine. I did run into a problem with serving static content. A look at the error log (/var/log/nginx/error.log) showed that nginx worker process was running into a permission issue reading the files:

2011/01/02 07:01:39 [error] 3781#0: *1 open()
"/home/apps/TwitterToys/public/images/logo.png" failed
(13: Permission denied), client: 122.167.81.253, server: _,
request: "GET /images/logo.png?x=1 HTTP/1.1", host:

I gave ‘others’ read and execute permissions on the /home/apps/ folder to make sure that nginx worker process could get in and read the files and the ’13: Permission denied’ errors went away.

sudo chmod -R o+rx /home/apps

This brought me to the last big task – configuring my Dancer application to run as a daemon so that it runs in the background and comes up when the OS boots the next time. I chose Daemontools for this. Unfortunately Daemontools were not available on Amazon’s Linux image via yum (they are available on Ubuntu 10.04 via the default repositories using apt-get), so I decided to roll pull the source and build. Here, I ran into another wall – the compilation would stop with some vague reference to errno.h. After some tense moments of frantically searching the internet, I found that I had to modify error.h in the src/ directory of the daemontools source distribution to something like this:

/* extern int errno; */
#include <errno.h>

The compilation and subsequent installation went fine. I restarted my EC2 instance to make sure that ‘svscan’ came up after the reboot. Things were much simpler this point on. All I had to do was create a folder for my daemon (I called it TwitterToys) under /service and place a shell script called ‘run’ with execute bit set:

#!/bin/sh

export PERL5LIB='/home/apps/TwitterToys/lib'
exec 2>&1 \
/usr/local/bin/plackup -s Starman -l 127.0.0.1:5001 -E deployment \
--workers=10 -a /home/apps/TwitterToys/bin/app.pl

Within moments my new and shiny daemon came up. I did another reboot to make sure that it indeed does come up as expected. By this time my domain changes had replicated to my ISP and pointing my browser http://stupidtwitterstats.com brought up my site.

My last steps before announcing it to the world were:

1. to add a CNAME record for ‘www’ pointing to stupidtwitterstats.com so that people who prefix URLs with www do make it to my site.

2. to add the following server block to my nginx configuration so that www.stupidtwitterstats.com redirects to stupidtwtterstats.com (via)

    server {
        server_name www.stupidtwitterstats.com;
	rewrite ^(.*) http://stupidtwitterstats.com$1 permanent;
    }

There you have it! A site running on Perl and Dancer from start to finish.

1. Login as root or su.
2. Make sure you have installed the kernel sources and build tools.

apt-get install build-essential

Then run:

apt-get install linux-headers

This will probably give you a message saying: “Package linux-headers is a virtual package provided by:”, followed by a list of available kernel versions. Choose your version by looking it up:

cat /proc/version

or

uname -a

e.g. in my case I chose:

apt-get install linux-headers-2.6-2-amd64

3. Next choose Virtual Machine -> Install VMWare Tools option from the menu (this assumes VMWare Fusion running on Mac but there should be a similar option for VMWare Workstation on Linux and Windows as well).

Now mount the VMWare tools virtual CD:


mount /cdrom
cd /cdrom


Copy VMWare tool source code to /tmp and extract the files (the actual filename will depend on your VMWare version [which the VMWare installer would have reported in case of a mismatch]):

cp VMWareTools-7.9.6-173382.tar.gz /tmp
cd /tmp
tar -xzvf VMWareTools-7.9.6-173382.tar.gz


At this stage you should have a folder called vmware-tools-distrib inside your /tmp folder. Visit this folder and run vmware-install.pl, then follow the on-screen instructions.

cd /tmp/vmware-tools-distrib
./vmware-install.pl


Now at one stage the VMWare tool installer complained about my installed gcc (4.3) version being different from the gcc version that was used to compile my kernel (4.1). Press Ctrl+C to abort the installer at this stage.

Turns out that gcc is merely a symbolic link in /usr/bin folder. Chances are that you’ll have 2 versions of gcc on your system (e.g. Lenny comes with /usr/bin/gcc-4.3 and /usr/bin/gcc-4.1). Just make the symlink point to the version of gcc that was used to compile your kernel:

cd /usr/bin
rm gcc
ln -s /usr/bin/gcc-4.1 gcc


Run vmware-install.pl again and this time things should go through. Restart your VM to finish the installation. You should now be able to see folders shared from the host machine under /mnt/hgfs.

Most of these steps should also apply to Ubuntu.

First issue the following command:

sudo apt-get install libapreq2

This installs 2 things – the libapreq2 shared library, and an Apache module – mod_apreq2.

Next we install the Perl bindings – Apache2::Request – which we use in our handler code.

sudo apt-get install libapache2-request-perl

At this stage if you restart Apache, it will load your Perl handler without any complaints. However if you visit a handler that uses Apache2::Request, it’ll error out with the following entry in error.log:


/usr/sbin/apache2: symbol lookup error: /usr/lib/perl5/auto/APR/Request/Apache2/Apache2.so: undefined symbol: apreq_handle_apache2


This is because unlike our mod_perl installation, apt-get doesn’t enable mod_apreq2 after installing it. We enable it manually by creating a symbolic link to /etc/apache2/mods-available/apreq.load under /etc/apache2/mods-enabled/:


sudo bash
cd /etc/apache2/mods-enabled
ln -s ../mods-available/apreq.load
apache2ctl restart


Updated: You can also run a2enmod to enable an Apache module and use a2dismod to disable it.

Visit your handler url again and this time it should work withour errors. Here is the modified handler from the mod_perl installation post that now uses Apache2::Request to enable you to test if your installation is working correctly:

package Hello;
use strict;

use Apache2::RequestRec ();
use Apache2::RequestIO ();

use Apache2::Const -compile => qw(OK);

use Apache2::Request;

sub handler {
    my $r = shift;

    $r->content_type('text/plain');

    my $req = Apache2::Request->new($r);
    my $name = $req->param("name");
    $name = $name ? $name : "World";

    print "Hello $name, the time here is " . localtime() . "\n";

    return Apache2::Const::OK;
}

1;

Getting started

At one stage during the installation of Ubuntu server, you’ll be asked what components you want installed. Pick LAMP at the very least. After booting up for the first time (and logging in), fire up the following commands:

sudo bash #fire up a root shell so that we don't have to sudo every command
apt-get update
apt-get dist-update
reboot

#install things that could come in handy later
sudo bash
apt-get emacs #skip this if you prefer vi - it's already there
apt-get install linux-headers-server build-essential


At this stage you’ll have the latest kernel running. I find the default 80×24 display a little too restrictive. We’ll fix that by editing /boot/grub/menu.lst. Open the file in emacs or whatever editor you like, and scroll down to the end to a bunch of options that look like title, uuid, kernel, initrd. Append vga=0x31A to the end of the first kernel statement. e.g. in my case

kernel /boot/vmlinuz-2.6.27-9-server root=UUID=d9f9cc35-d880-494d-8cd3-92da418a438b ro quiet splash


became

kernel /boot/vmlinuz-2.6.27-9-server root=UUID=d9f9cc35-d880-494d-8cd3-92da418a438b ro quiet splash vga=0x31A


Reboot.

vga=0x31A gives me a resolution of 1280×1024 and 64k colors. Here are other options that you can play with:

#  FRAMEBUFFER RESOLUTION SETTINGS
#     +-------------------------------------------------+
#          | 640x480    800x600    1024x768   1280x1024
#      ----+--------------------------------------------
#      256 | 0x301=769  0x303=771  0x305=773   0x307=775
#      32K | 0x310=784  0x313=787  0x316=790   0x319=793
#      64K | 0x311=785  0x314=788  0x317=791   0x31A=794
#      16M | 0x312=786  0x315=789  0x318=792   0x31B=795
#     +-------------------------------------------------+

Installing mod_perl

At this stage we already have Apache and Perl installed. If you do:
tail /var/log/apache2/error.log, you’ll see that out of the box, you only get support for PHP.

[Sun Dec 14 12:04:05 2008] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4 with Suhosin-Patch configured -- resuming normal operations


Here is how you add mod_perl support:


sudo bash
apt-get install libapache2-mod-perl2

#restart apache so that it loads mod_perl
apache2ctl restart

#if all went well, you'll see something to the effect of (emphasis mine):
[Sun Dec 14 12:19:17 2008] [notice] Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4 with Suhosin-Patch mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations


Testing our mod_perl installation

Let’s write a simple mod_perl response handler to make sure our installation was successful. Create Hello.pm in your home directory – which is /home/deepakg/ on my machine:

package Hello;
use strict;

use Apache2::RequestRec ();
use Apache2::RequestIO ();

use Apache2::Const -compile => qw(OK);

sub handler {
    my $r = shift;

    $r->content_type('text/plain');
    print "Hello World, the time here is " . localtime() . "\n";

    return Apache2::Const::OK;
}

1;

Then to make sure that we didn’t make any typos:

perl -c Hello.pm
Hello.pm syntax OK


Next, open /etc/apache2/apache2.conf and type the following right at the end:

PerlRequire /home/deepakg/Hello.pm
<Location /time>
   SetHandler perl-script
   PerlResponseHandler Hello
</Location>

Restart apache and check the Apache error log to make sure that it started without any issues:

sudo apache2ctl restart
tail /var/log/apache2/error.log


Install lynx, so that you can check your handy work:

sudo apt-get install lynx

# and once it is installed
lynx http://localhost/time


If everything is working then you’ll be greeted with something like this:

Hello World, the time here is Sat Jan 10 15:25:51 2009

Of course, the actual date and time will vary on your system .

Miscellaneous

If the time shown by the script above looks awkward, your time zone might not have been configured correctly. Configure the time zone to where you are:

sudo dpkg-reconfigure tzdata


And then may be tweak the clock by hand if needed:

sudo date MMDDhhmm #MM - month, DD - date, hh - hour (24 format), mm - minute