| Name | Creating a Debian package from your Pylons project |
|---|---|
| Space | Pylons CookBook |
| Section | Deployment |
| Page | Creating a Debian package from your Pylons project |
| Version | 1.0 |
| Status | Draft |
| Reviewed | False |
| Author(s) | Christoph Haas |
 | Consider using easy_install There is some controversy on how to install and maintain Python modules on Debian systems. Naturally on Debian you install software using apt-get because it allows you to install dependent packages automatically and remove them again later. Python provides another way of installing modules called easy_install that intends to provide similar features and conflicts with Debian's package management. While Debian developers prefer the Debian way Python developers seems to prefer using easy_install in a sandbox (that is described at James Gardner's article about sandboxing Pylons installations). This article deals with creating Pylons application the Debian way. |
Eggs versus Debian packages
When you want to deploy your Pylons project you typically create an egg file from it that other people can install. This is made easy through the power of easy_install. It installs all necessary files and even downloads packages it needs to run automatically. On Debian systems this approach is problematic though. Debian GNU/Linux is a Linux distribution that has a good software package management system. The package management typically knows about every file in your file system. You can install packages and remove packages without leaving cruft. A package installs other packages it needs to run just like easy_install but has more sophisticated features. The package management cannot work correctly if you install software without telling it. So using easy_install to install Python modules on Debian will quickly lead to chaos on your system. There is a Debian package called "python-pylons". Imagine what happens if you you easy_install to download and install the Pylons package. You would get two installations of Pylons on your system that conflict. You cannot even remove the easy_install'ed software properly because there is not yet such a feature. In the worst case you will need to reinstall your system soon.
A quick comparison...
Debian packages
Can be installed from the central Debian repositories- "stable" branch: Debian developers provide security patches automatically
- "unstable" (daily) branch: Debian developers provide up-to-date packages (well, mostly)
- Can be removed properly
- Can download and install dependent packages
- installation is system-wide so security patches will be applied for all applications automatically
installation is system-wide so all applications need to use the installed version of a package (different applications must not depend on different versions of the packages being installed)- Not all Python packages are up-to-date (or even present) in Debian
easy_install / CheeseShop / eggs
- Installs Python packages the way they are meant to be installed
- You get up-to-date packages
- Dependent packages are downloaded and installed automatically
- You cannot deinstall a package
- Debian's package management has no notion of per-user installed Python packages (think: security patches)
Conclusion
As you see there is no perfect solution for this dilemma. But whatever you do you must not mix apt-get installed packages with easy_install installed packages. So your options are:
- Install only Debian packages and don't ever use easy_install. Not even to install into /usr/local. This is the Debian way. You will then be limited to packages that already in Debian and need to create your own Debian packages for Python software that has not yet been debianized (although that is simpler than it may sound).
- Ignore the Debian package management system and install all Python packages through easy_install. This is nearly impossible because a lot of system tools require Python and certain Python packages so you cannot avoid using the apt-get system for some packages. This is not really an option.
- Run Python in a controlled environment (called a "sandbox") so that packages installed through easy_install don't conflict with what apt-get installed. You cannot update a sandbox easily though and every application needs its own sandbox. And your applications need to use the Python installation in the sandbox.
So besides the sandboxing approach I told you at the top there is a way to turn your Pylon project into a proper Debian package that you can install and remove on any Debian system.
Debianize a Pylons project
Packaging crash course
(The complete documentation about creating Debian packages is located at http://www.debian.org/doc/debian-policy/.)
To create a Debian package you can call the debuild tool. It looks up certain information in a subdirectory called "debian/" and creates a Debian package with a ".deb" suffix. To get the necessary tools to create a Debian package please install the _devscripts_ package: *aptitude install devscripts*
These files are important in the debian/ directory:
- debian/control
Contains information about dependencies on other Debian packages, the name of the package, the description, the name of the maintainer and to which section it belongs. - debian/copyright
A text file describing the licensing terms of your project. - debian/changelog
Everytime a new Debian package of some software is release the changelog file gets a new entry documenting the changes the package maintainer made. The newest entry in the changelog also sets the version number of the package. - debian/docs
A text file containing a list of documentary files that get installed into /usr/share/doc/$PACKAGE/ later. - debian/compat
A text file containing just one number. This is the version number of the debhelper tools that are used to build the package. Because most Debian packages are built in a similar way the debhelpers are shell tools for everyday tasks like installing init.d scripts, installing documentation or moving files around. Currently we are at version '5'. - debian/pycompat
Python modules in Debian are built in special ways. This version number described the compatibility version which is currently '2'. - debian/$PACKAGE.init
An init.d script that is copied to /etc/init.d/$PACKAGE during installation and that is used to start/stop the Pylons web server. - debian/rules
A program/shell script/Makefile that is run with different arguments to build the actual Debian package. It unpacks the egg, adds certain control information and in the end creates - among other files - a *.deb file.
If everything works you can run "debuild" and create the Debian package. I recommend the options "-uc -us" for most users which skips PGP-signing the package. It is a mandatory process for official Debian packages though so that only registered Debian developers may upload packages into the distribution.
Automatically adding the debian/ directory
To aid in adding these control files I have written a Python script that checks a few marginal conditions (trying to help you fix these issues) and creates a Debian package. If all goes well you can run "debuild -uc -us" afterwards and get your Debian package. The script is available at http://workaround.org/pylons but you should also find a copy attached to this wiki page.
Details on the installed package:
- the ini file that contains the setting that the system administrator will want to customize is at /etc/default/$PROJECT-production.ini
- as the /usr/share/doc/$PROJECT/README.Debian states after customizing the ini file you must run "paster setup-app /etc/default/$PROJECT-production.ini"
- the init.d script is copied to /etc/init.d/$PROJECT and will start the project automatically during system startup
- the "data" directory points to /var/cache/pylons-$PROJECT (this is where cached pages and session information is stored). This directory is owned by the "www-data" user.
- the log file is written to /var/log/$PROJECT.log (there is currently no logrotate cleaning up the log - this has still to be done)
Note: while the package is probably suitable for installing it in a production environment it is not suited to be uploaded directly into the Debian distribution. Mainly because it is a so called _native_ package (there is no orig.tar.gz and a patch file but all the project is contained in one .deb file). Consider digging deeper into Debian package maintenance if you like that or contain the debian-mentors mailing list on lists.debian.org.
Kudos
- Piotr Ozarowski (maintainer of several Pylons-related Debian packages)
- James Gardner (who documented how to sandbox a Pylons application which also works well on Debian)
- other people on the pylons-discuss mailing list adding valuable opinions and snippets
Comments (9)
Mar 17, 2007
Jon Rosebaugh says:
Why does the list for Eggs say "You cannot deinstall a package"? It's easy to de...Why does the list for Eggs say "You cannot deinstall a package"? It's easy to deinstall a package; type 'easy_install -m SomePackage' and then delete the egg.
Apr 02, 2007
Christoph Haas says:
I don't really consider the "-m" switch a clean way. Quote from http://peak.tele...I don't really consider the "-m" switch a clean way. Quote from http://peak.telecommunity.com/DevCenter/EasyInstall#uninstalling-packages:
---------
If you want to delete the currently installed version of a package (or all versions of a package), you should first run:
easy_install -m PackageName
This will ensure that Python doesn't continue to search for a package you're planning to remove. After you've done this, you can safely delete the .egg files or directories, along with any scripts you wish to remove.
---------
I don't want to search for egg files and directories manually. And when I last tested that I had a lot of clutter in my /usr/local/* that didn't get purged. On Debian when you "purge" something there should be no trace of it any more.
Apr 16, 2007
Jon Rosebaugh says:
Maybe it's not a clean way. But it is a way to deinstall a package. The list say...Maybe it's not a clean way. But it is a way to deinstall a package. The list says "You cannot deinstall a package". The list, therefore, is incorrect.
Oct 03
Rich Pixley says:
-m doesn't remove packages. It simply prepares packages to be removed....-m doesn't remove packages. It simply prepares packages to be removed. You still need something else, perhaps just "ls" and "rm" to actually find them and remove them.
And you still have a problem with any modules which were depending in that module.
"rm -rf /" will also remove packages. :). But most people would probably find that to be an inferior process.
Apr 29, 2008
Peter says:
Thanks for taking the time to jot all this down, saved me time which I'm gratefu...Thanks for taking the time to jot all this down, saved me time which I'm grateful to use elsewhere.
Kudos
Feb 04, 2009
Kevin Baker says:
Here is my logrotate script. I just dropped it into /etc/logrotate.d/ Note my ...Here is my logrotate script.
I just dropped it into /etc/logrotate.d/
Note my pylons startup script is at "/etc/init.d/pylons" and my pid file for startup is in "/var/run/paster.pid"
/var/log/pylons/pylons.log
Feb 04, 2009
Kevin Baker says:
It looks like the form hosed my script... trying again. /var/log/pylons/pylons....It looks like the form hosed my script... trying again.
/var/log/pylons/pylons.log { daily missingok rotate 360 compress delaycompress notifempty create 640 root adm sharedscripts postrotate if [ -f /var/run/paster.pid]; then /etc/init.d/pylons restart > /dev/null fi endscript }Oct 03
Rich Pixley says:
Thank you for this. I keep thinking that if setuptools can produce rpms, then i...Thank you for this.
I keep thinking that if setuptools can produce rpms, then it should be capable of producing debs. That would also induce the deb production process into python modules that weren't specifically set up to be debs, which would be extremely useful.
Oct 04
Rich Pixley says:
Also, for the comparison, debian's apt format can list installed packages and ve...Also, for the comparison, debian's apt format can list installed packages and versions. I haven't yet found a way to do that through the easy_install interface. (Yes, you can root around inside the format, but if I do that programmatically, then my code will break when the format changes.)
Too, debian can and does support multiple versions of the same package - they just don't support arbitrary combinations. It's common for them to support different major versions of things like libc, java[56], gcc, firefox, etc.
And... if we're going to include virtualenv and friends as a possible solution to the package removal problem for python, then we also need to look at things like virtualbox, (http://www.virtualbox.org), qemu, (http://www.qemu.org), and vmware, (http://vware.org), which are each capable of producing a recursive instance of a debian/ubuntu system running on a debian/ubuntu system. This also essentially allows for per-user installed debian packages. Heck, even chroot or scratchbox can be used to produce recursive environments for per-user installed debians.
I see only three features in favor of easy_install. They overlap. And I don't consider them to be sufficiently compelling.
1. easy_install packages are operating system independent. You can build one on linux and install it on macosx, freebsd, eg. Debians can't do that and for an interpreted system, that's very useful. (Would not be particularly useful for, say,
2. In part because of #1, the definitive place to look for additional python modules is pypi. While debian has an equivalent, it's not useful for freebsd nor macosx.
3. distutils/setuptools does seem to be ubiquitous. Not only is there an obvious place to look, (pypi), but pretty much all python packages I've seen include a setup.py. I think that argues in favor of extending distutils or setuptools to produce debians.