Painless roll-outs with FCGI::Engine

2009-02-24 − 🏷 catalyst 🏷 iusethis 🏷 perl

Our software site i use this is built on the MVC Framework Catalyst. We currently run it using the Russian web server Nginx and standalone fastcgi servers. I am using the Moose based CGI::Engine distribution by Stevan Little to start the servers. This module makes it really easy to manage your applications. You just create a YML config file like this:

---
- name: "iusethis-osx.server"
  nproc: 4
  scriptname: "/www/iusethis-osx/script/iusethis_fastcgi.pl"
  pidfile: "/var/run/iusethis-osx.pid"
  socket: "/var/run/iusethis-osx.sock" 

with an entry for each server you want to run. (Note that the paths has been changed to protect the innocent.) Then you just create a simple perl script (See the FCGI::Engine::Manager SYNOPSIS for a sample), and you can easily start, stop and check the status for each application individually or every application in your config. If you have a system v style init, you can just stick the script in /etc/init.d/ and it will behave just like any of your other startup scripts.

There is one annoying detail. Each time you roll out code, you have to restart your fastcgi processes. Since Catalyst takes some time to initialize, the application is down, and end users gets 500 Internal Server Error responses, unless you have a load balancer in front and take the node out of the cluster before upgrading. It does not have to work like this. Since the fcgi workers use a non-exclusive lock on the socket, you can start a new set of processes before you kill the old ones. this way, no requests are lost.

I really wanted this feature, so I have spent some time today hacking on FCGI::Engine. Stevan accepted my patches, and released version 0.06, which supports this restart mode, via a new 'graceful' method added to FCGI::Engine::Manager + some bug fixes. Nginx already support On the fly upgrades, which means there is no need for us to drop a user connection when rolling out new code again.