Production Deployment Using Apache, FastCGI and mod_rewrite

If you are using dreamhost or running Python 2.3 you should follow these instructions instead of the ones here. In particular you might experience problems with flup and might be better with the fcgi.py file linked to in the instructions.

This should be considered a work in progress, feel free to add/extend as necessary.

Introduction

There are quite a few different ways to deploy a Pylons application, using CGI, paster serve, mod_proxy or others. Those of you familiar with Rails will be used to the idea of deploying production applications using Apache, FastCGI and mod_rewrite. This is also a very effective way of deploying Pylons applications for production use and can also be used with standard Apache Shared Web Hosts which supports FastCGI and mod_rewrite.

As an example What Should I Read Next is written in Pylons, deployed on a standard shared Apache hosting account using the technique described here and has nearly 500,000 users and serves tens of thousands of requests a day.

Getting Started

First you will need to ensure both mod_rewrite and mod_fastcgi are installed in whichever way is appropriate for your operating system. You then typically have to tweak the apache init.d script to start FastCGI correctly but refer to the documentation specific to your OS for details.

Once mod_fastcgi and mod_rewrite are installed you need to ensure they are loaded. In your httpd.conf file ensure the following lines are present:

1
2
3
4
5
6
7

LoadModule fastcgi_module     modules/mod_fastcgi.so
LoadModule rewrite_module     modules/mod_rewrite.so
    
<IfModule mod_fastcgi.c>
    FastCgiIpcDir /tmp/fcgi_ipc/
    AddHandler fastcgi-script .fcgi
</IfModule>

The first lines load the Apache modules, the later ones setup mod_fastcgi to use the directory /tmp/fcgi_ipc/ to store information in. You will need to create /tmp/fcgi_ipc/ if it doesn't already exist and also ensure it has the correct permissions so that the Apache process can write and read files there. This probably requires using a combination of chown, chgrp and chmod to set the group to apache or www depending on the name of the Apache group on your setup and ensuring Apache has permission to write to that directory.

The other doc http://pylonshq.com/docs/0.9.1/webserver_config.html
uses other FCGI directives; briefly:

1
2
3
4
5
6
7
8
9

LoadModule fastcgi_module     modules/mod_fastcgi.so

<IfModule mod_fastcgi.c>
      FastCgiIpcDir		/tmp/fcgi_ipc/
      FastCgiExternalServer	/tmp/myapp.fcgi -host 0.0.0.0:5000
</IfModule>

ScriptAliasMatch ^(/.*)$	/tmp/myapp.fcgi$1
AddHandler			fastcgi-script .fcgi

Note that this "/tmp/" path can be anything but MUST exist else Apache complains that it can't find it. The "myapp.fcgi" is just a placeholder.

By now we should have everything we need to setup mod_fastcgi and mod_rewrite so lets specify our virtual host:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

<VirtualHost 192.168.1.10:80>
       ServerAdmin james@example.com
       ServerName example.com
       ServerAlias www.example.com
       DocumentRoot /var/www/example.com/htdocs
       ErrorLog /var/www/example.com/log/error.log
       CustomLog /var/www/example.com/log/access.log common

       <Directory /var/www/example.com/htdocs>
          Options FollowSymLinks
          AllowOverride all
          Order allow,deny
          Allow from all
       </Directory>

</VirtualHost>

In the example above we have setup a fairly standard virtual host on 192.168.1.10 (you will need to replace this with your IP). We have setup error logging so that we can fix any problems that crop up and we have setup AllowOverride all so that we can specify mod_rewrite rules and FastCGI script extensions on a per-directory basis using .htaccess files. Of course you could just specify everything we are going to put into .htaccess files later in the section above but people using shared Apache hosting accounts will have the above already configured and will only have the option of using .htaccess files.

Setting up Python

Whilst you can use the version of Python provided by your web host it is often very useful to setup a virtual python install to your home directory. This enables you to install extra modules and software without root access to your own virtual Python install without affecting the main Python installation of the server but whilst retaining all of the installed modules.

Read the Virtual Python documentation to understand how it works then download the install script and setup your virtual Python install as described in the documentation.

Now that you have a virtual python installation you will probably want to install Pylons. Follow the instructions at "http://pylonshq.com/install" but ensure that you use your virtual Python at ~/bin/python in your home directory rather than the main python command on the server:

~/bin/python ez_setup.py -f http://pylonshq.com/download/ "Pylons==0.8"

Pointer to old docs

This will install easy_install, Pylons and all required modules.

Hint: If you have an old version of setuptools installed you will need to run the following command first:

~/bin/easy_install -U setuptools

or try:

~/bin/easy_install -U -D setuptools

Hint: ~ is a unix shortcut for the home directory of the current user. You could always type out /home/james or similar for your home directory instead if you prefer.

Installing Your Application

Now that you have a fully working virtual Python installation with Pylons you have two choices of how to setup your application. If you have relased an egg of your application you most likely will just want to install it as follows:

~/bin/easy_install your_app.egg

Then each time you make a change you can just install the latest version of the software. This is how the PylonsHQ website works.

Alternatively, if you are regularly making tweaks it is sometimes handy to upload your development copy of the code and run the following command to setup the code for development, much like you would on your local machine:

~/bin/python setup.py develop

Either way you now have your application installed.

Create Config Files

You now need to create config files for your application. Have a look at the documentation here or base your production.ini file on the development.ini file you have been using taking care to set debug to false otherwise malicious users could execute any arbitrary code on your server.

Pointer to old docs

Setting up the Dispatch Files

We will create a CGI dispatch and a FastCGI dispatch. The CGI dispatch is very useful for checking everything is working correctly before you use the production FastCGI dispatch.

dispatch.cgi looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

#!/home/james/bin/python
# Load the WSGI application from the config file
from paste.deploy import loadapp
wsgi_app = loadapp('config:/var/www/example.com/production.ini')
import os, sys
def run_with_cgi(application):
    environ = dict(os.environ.items())
    environ['wsgi.input']        = sys.stdin
    environ['wsgi.errors']       = sys.stderr
    environ['wsgi.version']      = (1,0)
    environ['wsgi.multithread']  = False
    environ['wsgi.multiprocess'] = True
    environ['wsgi.run_once']    = True
    if environ.get('HTTPS','off') in ('on','1'):
        environ['wsgi.url_scheme'] = 'https'
    else:
        environ['wsgi.url_scheme'] = 'http'
    headers_set = []
    headers_sent = []
    def write(data):
        if not headers_set:
             raise AssertionError("write() before start_response()")
        elif not headers_sent:
             # Before the first output, send the stored headers
             status, response_headers = headers_sent[:] = headers_set
             sys.stdout.write('Status: %s\r\n' % status)
             for header in response_headers:
                 sys.stdout.write('%s: %s\r\n' % header)
             sys.stdout.write('\r\n')
        sys.stdout.write(data)
        sys.stdout.flush()
    def start_response(status,response_headers,exc_info=None):
        if exc_info:
            try:
                if headers_sent:
                    # Re-raise original exception if headers sent
                    raise exc_info[0], exc_info[1], exc_info[2]
            finally:
                exc_info = None     # avoid dangling circular ref
        elif headers_set:
            raise AssertionError("Headers already set!")
        headers_set[:] = [status,response_headers]
        return write
    result = application(environ, start_response)
    try:
        for data in result:
            if data:    # don't send headers until body appears
                write(data)
        if not headers_sent:
            write('')   # send headers now if body was empty
    finally:
        if hasattr(result,'close'):
            result.close()
# Deploy it using FastCGI
if __name__ == '__main__':
    run_with_cgi(wsgi_app).run()

dispatch.fcgi looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/home/james/bin/python

# Load the WSGI application from the config file
from paste.deploy import loadapp
wsgi_app = loadapp('config:/var/www/example.com/production.ini')

# Deploy it using FastCGI
if __name__ == '__main__':
    from flup.server.fcgi import WSGIServer
    WSGIServer(wsgi_app).run()

Place both files in your htdocs directory but be sure to replace /var/www/example.com/production.ini with the path to your configuration file and #!/home/james/bin/python with the path to your virtual Python install.

Hint: You cannot use ~/bin/python as this time as the user executing the script will be the Apache webserver and no bin/python file exists in Apache's home directory, only yours.

Hint: Don't call your dispatch.cgi file dispatch.py otherwise if your Python application tries to import a dispatch module it will import the file above by mistake.

For FastCGI deployment you will also need to install flup:

~/bin/easy_install -U flup

Finally make both your files executable:

chmod 755 dispatch.cgi
chmod 755 dispatch.fcgi

Note: in some FastCGI environments, such as Dreamhost, flup might have some problems starting up correctly. In that case, your alternative is using the simpler fcgi.py script instead. See here for more detailed instructions.

Configuring Apache to use the Dispatch files

Next you need to tell Apache to execute .cgi and .fcgi files. Make a file called .htaccess in your htdocs directory and add the following:

1
2
3

Options +ExecCGI
AddHandler fastcgi-script .fcgi
AddHandler cgi-script .cgi

It is much easier to debug your application in CGI mode rather than FastCGI mode so visit your version of "http://www.example.com/dispatch.cgi/" and check to see if the root page of your application is visible. If there is a problem have a look at the error logs to find out what went wrong:

tail /var/www/example.com/logs/error.log

Things to check if you have a problem are that the executable path to your virtual python install is correct and that you can execute your script from a command prompt:

cd htdocs
./dispatch.cgi

Check the file is executable (chmod 755 dispatch.cgi) and that the path to your config file is correct. Also ensure that your application does indeed produce output if you visit / and that you shouldn't have typed a URL like "http://www.example.com/dispatch.cgi/my_controller" instead.

If everything worked try visiting "http://www.example.com/dispatch.fcgi/"

If you see the correct page, great. Otherwise try typing the following to see if the FastCGI processes have been spawned:

ps aux | grep dispatch.cgi

If they haven't been spawned you need to check you FastCGI setup. If they have check the Apache error log. Once the processes have been spawned you will need to kill them before you make changes to your code otherwise Apache will continue serving the existing applications with the old code.

Correcting the KeepAlive Problem

Once you have a working version of your application using FastCGI there is a chance you may experience a strange problem where the script runs but the browser doesn't stop waiting for it for say 30-45 seconds. This is a known issue with mod_fastcgi and graceful restarts but doesn't seem to be getting fixed very fast.

What is happening is that the script is being kept alive by the KeepAlive HTTP header which in turn is being set because of the KeepAlive directive in httpd.conf. The solution is to add a simple piece of middleware to your Pylons application to add the HTTP header Connection: close.

In config/middleware.py add CloseConnection middleware right at the end after ErrorDocuments:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# @@@ Display error documents for 401, 403, 404 status codes (if debug is False also intercepts 500) @@@
app = ErrorDocuments(app, global_conf, mapper=error_mapper, **app_conf)

class CloseConnection:
    def __init__(self, app):
        self.app = app
        
    def __call__(self, environ, start_response):
        def close_connection_start_response(status, headers, exc_info=None):
            final_headers = []
            for header in headers:
                if header[0].lower() != 'connection':
                    final_headers.append(header)
            final_headers.append(('connection','close'))
            return start_response(status, final_headers, exc_info)
        return self.app(environ, close_connection_start_response)

app = CloseConnection(app)

return app

You will need to kill all your FastCGI processes so they are reloaded but then the problem should be resolved.

Using mod_rewrite to simplify URLS

By now you should have a working Pylons application that is blisteringly fast. The only problem is that you don't want dispatch.fcgi as part of your URL. Add the following to the .htaccess file we created earlier:

1
2
3

RewriteEngine On
RewriteRule   ^$  /dispatch.fcgi/
RewriteRule   ^(\[-_a-zA-Z0-9/\.]+)$  /dispatch.fcgi/$1

Now you should be able to visit "http://www.example.com/" and get the same output as "http://www.example.com/dispatch.fcgi/" also check that controller names are correctly routed and tweak the mod_rewrite directives as necessary. You may also want to add directives so that files already in the htdocs directory can still be served by Apache. The example above routes everything to your Pylons application.

Fixing Broken Routes

You will probably have noticed that all the URLs generated by Routes now point to the version of the URL with dispatch.fcgi in. This is because that is the real URL. You can add the following code to lib/base.py in the _before_() method to correct this:

1
2
3
4
5
6
7
8

def __before__(self, action, **kw):
    env = {}
    for k,v in request.environ.items():
        env[k]=v
    env\['SCRIPT_NAME'] = ''
    import routes
    config = routes.request_config()
    config.environ = env

This will trick routes into thinking the dispatch.fcgi script doesn't exist so your URLs will be generated correctly.

Restarting FastCGI Processes

To restart FastCGI you can use something like this:

ps -efl | grep /home/james/bin/python | \
          grep -v grep | awk '{print $4}' | xargs kill

replacing /home/james/bin/python with something unique to all the FastCGI processes you want to kill. In this case the virtual python executable.

If kill doesn't do the trick have a look at sial.org's shell how-to for alternatives.

Comments

Feel free to add any comments or drop an email to james at pythonweb.org.