Saturday, August 30, 2008

Building a Debian GNU/Linux package

I recently built packages for the 0.9.3 release of Realeyes. These include both source and Debian GNU/Linux packages. For reasons that I have forgotten, I built the Debian packages before I started working on the source packages, but I'm glad that I did. Going through that process made me add several things that I would have overlooked, especially man pages.

I actually built an entire distribution as part of the process of creating my packages. It does not meet the requirements for re-distribution, but is very handy for laying down a fresh install with exactly what is needed to run Realeyes. I will provide the steps for that in another post.

The source is comparatively easy, once all of the files are collected, a tar file is created of the directory. The trick here is to verify that the C code will compile. I have read a lot of comments about how straight-forward the standard configure/make installation procedure is. But from a developer's perspective, there are several issues, mainly having to do with autoconf and automake. I don't have enough space here to discuss these, and I really don't have any wisdom to impart if I did, because I have only learned as much as I needed for my own packages.

Debian packaging is somewhat more complicated. A package that is included in a Debian distribution must go through fairly rigorous tests. There are several scripts for checking correct packaging procedures, including lintian and linda. These verify such conditions as:
  • executables are built correctly

  • man pages exist for every executable file

  • naming conventions are followed

  • Debian documentation is formatted correctly
I have seen a few HOWTOs on building a Debian package, and there is a huge amount of information on the Debian site. But I still had to piece together a working plan for myself, so I thought it would be worth sharing it. There may be better ways to do some of it, but I have written scripts that get me through the process with only a little manual effort. And even when there is an automated process, it is still good to know what is happening under the hood. So without further ado, I offer my experience.

Building a Debian Package


I. Read enough of the manuals to get a sense how Debian packages are built, and then keep links to them for reference:

II. Create a working directory for the package

  • Get a package to use as a model, using the following commands to extract the package files and the Debian metadata files:

    apt-get -d -y --reinstall install package_name
    dpkg -x package_name
    cd package_dir
    dpkg -e ../package_name

  • Create a working directory, and under it make the directories to be installed for the package, even if there are no files to be saved in them. These may include:

    working_dir/etc/package_name
    working_dir/etc/init.d
    working_dir/usr/sbin
    working_dir/usr/share/package_name
    working_dir/usr/share/doc/package_name
    working_dir/usr/share/man
    working_dir/var/log/package_name

  • Make the Debian control directory with its files (as needed)

    working_dir/DEBIAN

    • control: This contains the description of the package, including dependencies, architecture, and the description used by aptitude or synaptic -- use the model to create this for the first time

    • conffiles: This contains any configuration files installed with the package -- I put mine in /etc/realeyes

    • preinst: This is a shell script that runs before the package is installed, if it exists

    • postinst: This is a shell script that runs after the package is installed, if it exists -- I use it to create user IDs

    • prerm: This is a shell script that runs before the package is de-installed, if it exists

    • postrm: This is a shell script that runs after the package is de-installed, if it exists

  • Populate the directories with the application files: Use the model to help understand what goes where

III. Use the maintainer tools to verify package acceptability
  • Build the package:
    cd working_dir
    dpkg-deb --build package_dir package_name
  • lintian/linda: Check for package discrepancies. Note that lintian and linda are not in the standard package and must be installed separately. Lintian uses Perl and linda use Python, so there may be several dependencies installed with them.
    lintian -i package_name > package_name.lintian
    linda package_name > package_name.linda
  • Fix all the problems, here are some helpful hints from my experience:

    • Man pages: txt2man is a program that takes ascii text and converts it to a man page. It works for simple pages, but the resulting groff file may have to be edited manually in some cases. Use 'gzip -9' to compress man pages.

    • Compiled programs: Compiled programs must be stripped. Use the command:
      install -s
    • Identify all non-executable files in system directories (ie. /etc/package_name) in the package's DEBIAN/conffiles.

    • Lintian provides the section in the Debian Policy manual that describes the requirement that was flagged.

  • Sign the package: Create a GPG key for the package and sign each package with the key
    gpg --gen-key
    dpkg-sig -s builder pkg.deb
    NOTES:

    • The public keyring is in $HOME/.gnupg/pubring.gpg

    • There should be a lot of entropy on the system to help the random number generator, (grep -R abc /usr/* seems to work well)

    • Issue the command 'cat /proc/sys/kernel/random/entropy_avail' to find out how much entropy is currently available, it should be at least above 1,500
At this point, the package can be installed using the dpkg command. However, if there are dependencies that must be installed, dpkg will issue a warning about them, but does not handle their installation. So if you want to go to the next level, here is what you have to do.

IV. Repository directories

A Debian repository has a relatively simple directory structure to maintain the packages and metadata about them. An installation ISO is basically a repository tree with just the stable packages. A custom repository tree can be added to the apt sources.list to be accessed just like officially maintained packages, with aptitude or synaptic.

In the top repository directory, the following are mandatory:
  • md5sum.txt: The list of all files in the tree with their md5 checksums

    To create the md5sum.txt file for my mini-distro, I wrote a script that ran in the top ISO directory. It did a recursive ls, ran md5sum on all regular files, and wrote the output to the md5sum.txt file. I keep that as a template and only update the files that change.

  • pool: The subdirectory where packages are kept. Under the pool directory, there are a few pre-defined directories where the different categories of packages are kept. Anyone who has edited a sources.list file has seen most of these:

    • main: Technically, these are packages that meet the Debian Free Software Guidelines (DFSG), but I think of them as the officially maintained packages

    • contrib: Contributed packages are DFSG, but depend on packages that are not -- I use this for my own packages, even though I don't have any non-DFSG dependencies

    • non-free: These are non-DFSG packages

    The structure of each of these is the same. The package directories are in subdirectories named with the first letter of the package name. The exception is libraries, which are under directories named libletter, which is the prefix of the library package name. Below these subdirectories are the directories with the actual package files.

  • dists: The dists directory contains the metadata about packages. There is a directory for the distribution, in this case, etch. In an ISO, there are also the directories, frozen, stable, testing, and unstable, which are links to the distribution directory. In a repository, these may have their own files. But for my purposes, I only include the distribution subdirectory.

    Under the distribution subdirectory are the following:

    • Release: This file describes the packages, including the architecture, the components, and contains md5sums for the package metadata files -- the file information also includes the file size, and since there are only a few of these, I created the original by hand

    • main: This directory contains the metadata about the main packages

    • contrib: This directory contains the metadata about the contrib packages

    The structure of main and contrib is the same, and again, I only use contrib. The architecture directories are below contrib, and in my case, there is only binary-i386. In the architecture directory there are three files:

    • Packages: This uses information from the DEBIAN/control file and adds such things as the full path of the package file

    • Packages.gz

    • Release: This contains metadata about the contrib directory

  • I also put a few optional files in the top level directory. These include a copy of the GPL (I use version 3), installation instructions, and the public key for the signed packages. The installation instructions explain how to add the public key so that aptitude and synaptic can validate the packages.

V. Add the packages
  • Copy the packages to the appropriate pool directory. In my case, this means copying them to:

      iso_dir/pool/contrib/r/realeyescomponent

  • Create an override file for the packages. This consists of a line for each packages with the following information:

      package priority section

    In my case it looks like this:
    realeyesDB   optional  net
    realeyesDBD optional net
    realeyesGUI optional net
    realeyesIDS optional net
    The man page for dpkg-scanpackages (the next command) has a description of each field and says that the override file for official packages is in the indices directory on Debian mirrors.

  • Build the metadata:
    dpkg-scanpackages \
    pool/contrib/ override.etch.contrib > \
    dists/etch/contrib/binary-i386/Packages
    cd dists/etch/contrib/binary-i386
    gzip -c -9 Packages > Packages.gz
    cd ../..
    md5sum contrib/binary-i386/Packages* > md5.tmp
    ls -l contrib/binary-i386/Packages* >> md5.tmp
  • The file, md5.tmp, is edited to put the file size after the md5 checksum and before the file name, and the file listing lines are deleted. Then the file, Release, is edited to read in the file, md5.tmp, at the end and the duplicate lines are deleted.
    gpg --sign -ba -o Release.gpg Release
    cd ../..
    cp -a ~/.gnupg/pubring.gpg RE_pubring.gpg
    md5sum ./INSTALL* > md5sum.txt
    md5sum ./GPL* >> md5sum.txt
    md5sum ./dists/etch/Release >> md5sum.txt
    md5sum ./dists/etch/contrib/binary-i386/* >> md5sum.txt
    md5sum ./pool/contrib/r/*/* >> md5sum.txt

VI. Installing the package

The instructions for installing this package are:
  • Copy the debian packages to a directory that will be used for the initial installation and future updates, such as, /var/tmp/realeyes. Untar the packages:
    tar xvzf realeyes_debian.tar.gz
  • Change to the top level packages directory and add the public key file, RE_pubring.gpg, to the Debian trusted sources with the command:
    apt-key add RE_pubring.gpg
  • Edit the file /etc/apt/sources.list to add the line:
    deb file://install_dir/realeyes/ etch contrib
  • Update the package lists with one of the following methods:

    • On the command line enter:
      apt-get update
    • In aptitude, select Actions -> Update package list

  • Install the package using aptitude or synaptic
So there you have it. I hope it shortcuts the learning curve a little.

Later . . . Jim

No comments: