Introduction to the arch patch queue manager

Colin Walters

Copyright 2003 Colin Walters. Permission is granted to copy, distribute and/or modify this document under the terms of the GPL.

Portions Copyright (c) 2004 Robert Collins. Permission is granted to copy, distribute and/or modify this document under the terms of the GPL.


Table of Contents

Getting arch-pqm
What problem does it solve?
How it works
Handling conflicts
Running tests
Security
Setup
Requirements
Compilation and installation
Using arch-pqm
Defining the upstream archive
Submitting a merge request via a shared filesystem
Submitting a merge request via GNUPG-signed email
Processing merge requests

Getting arch-pqm

Source tarballs are available here.

arch-pqm is also maintained in arch.

'official' arch-pqm archive location

Archive

walters@verbum.org--2003

Location

http://arch.verbum.org/arch

Revision

arch-pqm--main--version

Robert Collins' arch-pqm archive location

Archive

rbtcollins@hotmail.com--2003

Location

http://people.initd.org/robertc/arch/public

Revision

arch-pqm--mainline--version

What problem does it solve?

The idea is simple. You have a project with a number of developers. With a revision control system like CVS, it's obvious that the project code will be kept in a single repository, which all the developers use. You really don't have much of a choice.

But arch is fully distributed. You want to take advantage of those features, allowing your developers to commit while disconnected (say while they are travelling with a laptop), easily create their own temporary branches without affecting the main repository, and more. To accomplish these things, each developer needs to have their own arch archive.

This then raises a question - where is the project? One solution is to pick a specific developer to perform the task of merging in the other developer's code. That developer's archive becomes the canonical one for the project.

However, there is a better way. The main idea of the tla patch queue manager is to have a special archive which is managed entirely by the patch queue software.

How it works

You create the distinguished archive to be managed by the patch queue manager. Then, every developer branches off of it. When a developer has reached a milestone and wants to sync up the main tree with their archive, they submit a merge request.

Essentially, this just leverages tla's "star-merge" functionality to perform the actual merge.

Handling conflicts

One question that arises at this point - what happens if there's a conflict? If two people submit merge requests, the second could conflict with the first. Obviously the patch queue manager is not an AI; it can't automatically resolve all conflicts. So the simple solution is to just to inform the submitter of the second patch of this fact, and reject the merge request. They can then sync up their tree with the main branch, and resubmit the merge request.

Running tests

However, textual conflicts aren't the only kind of conflict; you could also have _semantic_ conflicts, where the patches don't touch the exact same portion of code, but taken together they break it. One way to help solve this problem is to run a testsuite before every commit to the main archive. The tla patch queue manager lets you do this with a precommit hook. You can actually do pretty much anything you want inside this hook.

Security

You have two choices for submitting merge requests; one option is to send them via email. arch-pqm supports GPG-signed messages for security.

If it can be arranged for developers to all have write access to the same filesystem (via NFS/SFS, over sftp, etc), then you can have them simply drop merge requests into a special queue directory.

Setup

Requirements

  1. Python 2.3 (or Python 2.2 and the python-logging module)

  2. GNUPG (optional, but highly recommended)

Compilation and installation

  1. Use the normal configure, make, make install commands to install the software.

  2. Next, choose a directory for arch-pqm to use to store the queued requests, as well as its log file and temporary working directories. From now on, we will assume you have chosen /usr/src/arch-pqm/; adjust the path to suit yourself.

  3. Set the id of the archive, using the following command:

    $ HOME=/usr/src/arch-pqm/ tla my-id 'Example <example-dev@example.com>'
    	  

    Note that the id you choose should not be specific to one of your developers; instead choose a generic name that closely corresponds to the product. A good choice would be the name of the development mailing list.

  4. Next, create the archive which will hold the software, and set up the various categories you want:

    $ HOME=/usr/src/arch-pqm/ tla make-archive example-dev@example.com--2003 /usr/src/example
    $ HOME=/usr/src/arch-pqm/ tla my-default-archive example-dev@example.com--2003
    $ HOME=/usr/src/arch-pqm/ tla archive-setup hello-world--mainline--1.0
    $ HOME=/usr/src/arch-pqm/ tla archive-setup libhello--mainline--3.4
    	  
  5. At this point, you have two choices. If you have an existing arch repository with your code, you will likely want to make the new archive have a tag from your previous one, so you don't lose any history.

    $ HOME=/usr/src/arch-pqm/ tla tag jane@example.com--2003/hello-world--mainline--1.0 hello-world--mainline--1.0
    	  

    Alternatively, you could use the normal tla import command to import an existing project from source.

  6. Now, you need to register the other developer's archives:

    $ HOME=/usr/src/arch-pqm/ tla register-archive hacker1@example.com--2003 http://devserv.example.com/archives/hacker1
    $ HOME=/usr/src/arch-pqm/ tla register-archive james@example.com--2003 http://branch.example.co.uk/archives/james
    	  
  7. Next, you need to create a ~/.arch-pqm.conf configuration file. This lists the location of the queue, the path to the GNUPG keyring, whether or not to verify signatures, etc. In addition, the arch-pqm.conf file also specifies which archives are valid to merge into. In the future, it will also allow specifying a keyring per-archive, and some other nice things.

    Please see the example file sample-arch-pqm.conf as a starting point. This documents all the options to arch-pqm.

  8. Finally, have each developer register the central archive, and create branches from it.

Using arch-pqm

Defining the upstream archive

The commands in this section will be easier if we record in a little file which archive is the "upstream". It is recommended you do something like this:

$ echo 'example-dev@example.com'/$(tla parse-package-name --non-arch $(tla tree-version)) > {arch}/+upstream

Submitting a merge request via a shared filesystem

If you can provide all of the developers with write access to the same filesystem, then you can simply have them place merge request files in the queue directory, using a small shell script such as this:

#!/bin/sh (echo "From: $(tla my-id)"; echo "Subject: $1"; echo; echo
star-merge "$(tla tree-version)" "$(cat {arch}/+upstream)") >
/usr/src/arch-pqm/patch.$(date +%s)
    

Save that as tla-submit-merge. Then you would use this script by saying:

$ tla-submit-merge 'fix lots of bugs'

In the future, we would like to have better support integrated in tla for this kind of thing.

Submitting a merge request via GNUPG-signed email

If your site is widely distributed, or giving shell accounts to all developers doesn't work well, you can also use GNUPG-signed email. Here's how you could modify the script above:

#!/bin/sh
echo star-merge "$(tla tree-version)" "$(cat {arch}/+upstream)" | gpg --clearsign | mail -s "$1" "$2"

You might use the above script by saying:

$ tla-submit-merge 'fix bugs' pqm@example.com 	

In order to process requests sent by email, you will likely want to use procmail. If you have an account dedicated to processing merge requests (a good idea), you could use this sample .procmailrc entry:

:0:
| arch-pqm --read
    

If you don't have a dedicated account, you will have to pick a specific Subject: or other header to use to differentiate merge requests from your regular email.

Processing merge requests

The key command is:

$ arch-pqm --run
      

That simply iterates through the pending merge requests in order (by mtime), and processes them. A good way to use this would be to run it via a cron job.