As just announced today, Project-Builder.org 0.11.1 has now been officialy published. All the latest news are detailed in the announce.

Now I’l probably release soon a 0.11.2 to fix some small annoying bugs that my colleagues using it reported in between, and in order to use the best version availble for the training I’m organizing next week (presentation ° Lab). So expect hopefully a 0.11.2 over the week-end !

So now that I have your attention with this interesting Cloud buzzword, I can develop the idea i’m adding to project-builder.org at the moment.

Currently project-builder.org supports building in Virtual Machines (VM) or Virtual Environments (VE) aka chroot, which are all managed by the machine running the pb command. But with the expansion of the project I’m working on in our joint HP/Intel collaboration, we want to be able to support Continuous Packaging also on machine for which we can’t have a VM or a VE hosted, such as an Itanium HP-UX one or a Sparc Solaris one.

Of course, it’s always possble to log on one of these systems, install pb, give access to the VCS/CMS repository and voila, you’ve shiny new packages. But you (as I) want more no ?

So I’m realizing a new series of patches so that pb can launch build operations on a Cloud of machines, as long as you can connext to them through SSH. In fact the process is very similar to what pb currently does to build in a VM, except that this time, it can be a remote machine. I found it much easier to support than I thought in the code, proof that the design is not too bad and allows for easy improvements.

Also I made recently some good code cleanup after my stay at the FOSDEM where I attended a lot of perl sessions, that gave me energy to do that … during the 4 and half train trip I had to go back home.

And today, it took me less than 20 minutes and no cloud in the horizon to prepare 2 new VMs in order to support Debian 6.0 for future project releases. Tooling stuff is really helpful.

So expect a new version of pb RSN, in order to provide that additional support.

As of version 0.17, gitpkg ships with a hook called quilt-patches-deb-export-hook. This can be used to export patches from git at the time of creating the source package.

This is controlled by a file debian/source/git-patches. Each line contains a range suitable for passing to git-format-patch(1). The variables UPSTREAM_VERSION and DEB_VERSION are replaced with values taken from debian/changelog. Note that $UPSTREAM_VERSION is the first part of $DEB_VERSION

An example is

 upstream/$UPSTREAM_VERSION..patches/$DEB_VERSION
 upstream/$UPSTREAM_VERSION..embedded-libs/$DEB_VERSION

This tells gitpkg to export the given two ranges of commits to debian/patches while generating the source package. Each commit becomes a patch in debian/patches, with names generated from the commit messages. In this example, we get 5 patches from the two ranges.

 0001-expand-pattern-in-no-java-rule.patch
 0002-fix-dd_free_global_constants.patch
 0003-Backported-patch-for-CPlusPlus-name-mangling-guesser.patch
 0004-Use-system-copy-of-nauty-in-apps-graph.patch
 0005-Comment-out-jreality-installation.patch

Thanks to the wonders of 3.0 (quilt) packages, these are applied when the source package is unpacked.

Caveats.

• Current lintian complains bitterly about debian/source/git-patches. This should be fixed with the next upload.

• It’s a bit dangerous if you checkout such package from git, don’t read any of the documentation, and build with debuild or something similar, since you won’t get the patches applied. There is a proposed check that catches most of such booboos. You could also cause the build to fail if the same error is detected; this a matter of personal taste I guess.

So, finally Project-Builder.org 0.10.1 has been published.

A lot has been done in that version, so look at the ChangeLogs files for details. In particular, that version of pb is not backward compatible with the previous one, thus the change of version (from 0.9 to 0.10). Which also means that it is highly recommended to update your VMs/VEs with that new version of pb with the setupvm|ve command.

Another incompatibility is with the cms2* commands which now really take the content of the CMS/VCS not the sandbox status. In order to get the previous behaviour, please use the sbx2* commands instead. man pb is probably helpful In particular, if you build from files (via http/ftp) and not a VCS, you’ll have to use the new sbx commands to have your package build correctly, as there is no CMS in that case.

This version now supports parallelism during the sbx|cms2build, sbx|cms|build2vm|ve phases which can drastically reduce build time for packages. For this to work, as explained earlier, you need the perl module Parallel::ForkManager installed on your system. The VMs|VEs may benefit from it but it is not required, and only useful if you have lots of cores on your host, to enable multi-cpus VM|VEs.

A small bug has been found post-release for those of you using the announce command. What ? nobody ranted ? Well, as I’m probably the only one using that feature, sounds obvious

I’ll now also work more with my FOSSology colleagues to help them produce packages for their just announced 1.3.0 version with pb.

More interesting is what will come next: As part of our joint work with Intel in our Solution Center, Project-Builder.org is part of a new solution stack we’re developing with their collaboration around RISC/Unix to IA/Linux migration. Continuous packaging was identified as a key aspect, and helping our customers support multiple platforms in parallel is indeed very useful. So it was agreed, also following an internal presentation and some surveys made during it, that our next supported platform had to be HP-UX.

So I’m working with the help of Josh Zhao on adding this, and this is expected for 0.10.2. And another colleague Nicolas Doualot is helping setting up a demo environment to showcase our Continuous Packaging approach.

Signing packages is also high on my list, but is still a problem on RPM based distro. Hopefully I’ll be able to come up with a solution after some FOSDEM discussions !

Next version of project-builder.org won’t be 0.9.11 as expected ! This is due first to the incompatibilities introduced with 0.9.10. a pb 0.9.10 cannot work in a VE|VM with the current devel version of pb, and the VE|VM will have to be updated with setupve|vm.

Also on top of that quite a new set of features have been added since last version, in particular the parallelism of tasks. After the work on sbx|cms2build, I’ve done now build2pkg and also build2ve. It implied more modifications in order to have each script file generated during the preparation phase unique so that each VE could get it correctly. I’ve tested this night with 2 VEs to build pb packages, and it is now working quite well, and that’s where having a multi-core, multi-thread machine such as min base on a Core i7 Xeon helps a lot aand makes a difference.

The last remaining part is to also add that feature to VMs. I’m very near from it, I just need to handle the fact that communication based on SSH with each VM has to use a different port, whereas now it’s the same for a given project. Hopefully this will be done over the week-end, which will allow publication of the new stable version of pb in sequence.

This feature is also useful for my colleagues working on the FOSSology project to generate packages for their 1.3.0 version on their side.

Looking around to improve some of the performances of project-builder.org, I found an interesting perl module: Parallel-ForkManager.

Its man page shows how you can quickly parallelize a loop in order to benefit from the numerous cores that each system today has. It took me just a couple of minutes to transform project-builder.org, so that the generation of all build files becomes a parallel operation, thus reducing dramatically the time it was taking – at least on my Xeon X5680 @ 3.33GHz !!

Very easy as you can see in the changeset, and very efficient. And if you don’t have the module, you can still work in serial mode, but once you tested it, you don’t want to go back

Next step, is to also apply that feature to the build in itself on multiple VMs or VEs, but that will require a bit more work, as the loop is not so easily usable as of now. More on that later, once the code is re-architectured.

I recently decided to try maintaining a Debian package (bibutils) without committing any patches to Git. One of the disadvantages of this approach is that the patches for upstream are not nicely sorted out in ./debian/patches. I decided to write a little tool to sort out which commits should be sent to upstream. I’m not too happy about the length of it, or the name “git-classify”, but I’m posting in case someone has some suggestions. Or maybe somebody finds this useful.

#!/usr/bin/perl

use strict;

my $upstreamonly=0;

if ($ARGV[0] eq "-u"){
  $upstreamonly=1;
  shift (@ARGV);
}

open(GIT,"git log -z --format=\"%n%x00%H\" --name-only  @ARGV|");

# throw away blank line at the beginning.
$_=<GIT>;

my $sha="";
LINE: while(<GIT>){

  chomp();

  next LINE if (m/^\s*$/);

  if (m/^\x0([0-9a-fA-F]+)/){
    $sha=$1;
  } else {
    my $debian=0;
    my $upstream=0;

    foreach my $word  ( split("\x00",$_) ) {
      if  ($word=~m@^debian/@) {
        $debian++;
      } elsif (length($word)>0)  {
        $upstream++;
      }
    }

    if (!$upstreamonly){
      print "$sha\t";
      print "MIXED" if ($upstream>0  && $debian>0);
      print "upstream" if ($upstream>0  && $debian==0);
      print "debian" if ($upstream==0  && $debian>0);
      print "\n";
    } else {
      print "$sha\n" if ($upstream>0  && $debian==0);
    }

  }
}

=pod

=head1 Name
git-classify  - Classify commits as upstream, debian, or MIXED

=head1 Synopsis

=over

=item B<git classify> [I<-u>] [I<arguments for git-log>]

=back

=head1 Description

Classify a range of commits (specified as for git-log) as I<upstream>
(touching only files outside ./debian), I<debian> (touching files only
inside ./debian) or I<MIXED>. Presumably these last kind are to be
discouraged.

=head2 Options

=over

=item B<-u> output only the SHA1 hashes of upstream commits (as
      defined above).

=back

=head1 Examples

Generate all likely patches to send upstream

     git classify -u $SHA..HEAD | xargs -L1 git format-patch -1

racket (previously known as plt-scheme) is an interpreter/JIT-compiler/development environment with about 6 years of subversion history in a converted git repo. Debian packaging has been done in subversion, with only the contents of ./debian in version control. I wanted to merge these into a single git repository.

The first step is to create a repo and fetch the relevant history.

TMPDIR=/var/tmp
export TMPDIR
ME=`readlink -f $0`
AUTHORS=`dirname $ME`/authors

mkdir racket && cd racket && git init
git remote add racket git://git.racket-lang.org/plt
git fetch --tags racket
git config  merge.renameLimit 10000
git svn init  --stdlayout svn://svn.debian.org/svn/pkg-plt-scheme/plt-scheme/ 
git svn fetch -A$AUTHORS
git branch debian

A couple points to note:

• At some point there were huge numbers of renames when then the project renamed itself, hense the setting for merge.renameLimit

• Note the use of an authors file to make sure the author names and emails are reasonable in the imported history.

• git svn creates a branch master, which we will eventually forcibly overwrite; we stash that branch as debian for later use.

Now a couple complications arose about upstream’s git repo.

1. Upstream releases seperate source tarballs for unix, mac, and windows. Each of these is constructed by deleting a large number of files from version control, and occasionally some last minute fiddling with README files and so on.

2. The history of the release tags is not completely linear. For example,

rocinante:~/projects/racket  (git-svn)-[master]-% git diff --shortstat v4.2.4 `git merge-base v4.2.4 v5.0`
 48 files changed, 242 insertions(+), 393 deletions(-)

rocinante:~/projects/racket  (git-svn)-[master]-% git diff --shortstat v4.2.1 `git merge-base v4.2.1 v4.2.4`
 76 files changed, 642 insertions(+), 1485 deletions(-)

The combination made my straight forward attempt at constructing a history synched with release tarballs generate many conflicts. I ended up importing each tarball on a temporary branch, and the merges went smoother. Note also the use of “git merge -s recursive -X theirs” to resolve conflicts in favour of the new upstream version.

The repetitive bits of the merge are collected as shell functions.

import_tgz() { 
    if [ -f $1 ]; then 
        git clean -fxd; 
        git ls-files -z | xargs -0 rm -f; 
        tar --strip-components=1 -zxvf $1 ; 
        git add -A; 
        git commit -m'Importing '`basename $1`;
    else
        echo "missing tarball $1"; 
    fi; 
}

do_merge() {
    version=$1
    git checkout -b v$version-tarball v$version
    import_tgz ../plt-scheme_$version.orig.tar.gz
    git checkout upstream 
    git merge -s recursive -X theirs v$version-tarball
}

post_merge() {
    version=$1
    git tag -f upstream/$version
    pristine-tar commit ../plt-scheme_$version.orig.tar.gz
    git branch -d v$version-tarball
}

The entire merge script is here. A typical step looks like

do_merge 5.0
git rm collects/tests/stepper/automatic-tests.ss
git add `git status -s | egrep ^UA | cut -f2 -d' '`
git checkout v5.0-tarball doc/release-notes/teachpack/HISTORY.txt
git rm readme.txt
git add  collects/tests/web-server/info.rkt
git commit -m'Resolve conflicts from new upstream version 5.0'
post_merge 5.0

Finally, we have the comparatively easy task of merging the upstream and Debian branches. In one or two places git was confused by all of the copying and renaming of files and I had to manually fix things up with git rm.

cd racket || /bin/true
set -e

git checkout debian
git tag -f packaging/4.0.1-2 `git svn find-rev r98`
git tag -f packaging/4.2.1-1 `git svn find-rev r113`
git tag -f packaging/4.2.4-2 `git svn find-rev r126`

git branch -f  master upstream/4.0.1
git checkout master
git merge packaging/4.0.1-2
git tag -f debian/4.0.1-2

git merge upstream/4.2.1
git merge packaging/4.2.1-1
git tag -f debian/4.2.1-1

git merge upstream/4.2.4
git merge packaging/4.2.4-2
git rm collects/tests/stxclass/more-tests.ss && git commit -m'fix false rename detection'
git tag -f debian/4.2.4-2

git merge -s recursive -X theirs upstream/5.0
git rm collects/tests/web-server/info.rkt
git commit -m 'Merge upstream 5.0'

English: I have split my old blog in 3 different blogs, redirections are in place when it was possible to have a meaningful mapping. This does not appear to be the case for the feed you subscribed to. Please update it manually, here are the new sites:

• Free software blog in English: Raphaël’s Free Life (feed)
• Free software blog in French: Destination libre (feed)
• Other personal ramblings: Ouaza.com (feed)

Français: j’ai découpé mon ancien blog en 3, des redirections sont en place lorsqu’il y avait une correspondance évidentes. Cela ne semble pas être le cas pour le flux auquel vous êtes abonné. Merci de mettre à jour manuellement, voici les nouveaux sites :

• Blog sur le logiciel libre en Anglais : Raphaël’s Free Life (feed)
• Blog sur le logiciel libre en Français : Destination libre (feed)
• Autres sujets et blog privé : Ouaza.com (feed)

This year, I’ve been very happy to be selected for the first time in 10 years for conferences at 2 majors events in Europe, and on one of my preferred subject: packaging. After Fosdem in February, tomorrow, I’ll present project-builder.org during LinuxTag in Berlin.

It will be the first talk covering the newly announced 0.9.10 version, which adds Ubuntu 10.04 support, and also a more robust version of rpmbootstrap. Hopefully making these presentations will help gathering other inputs and users so that the tools become useful for more FLOSS projects.

So would be happy to meet with you to discuss around packaging, disaster recovery, cloning, HP, Linux or anything related to Open Source … and early music as well

I decided to freeze the current state of project-builder.org to deliver it under the 0.9.9 name.

It’s more to give access externaly to the current state of the art of this project, rather than it represents a final point of features in fact. A certain number of small bugs have been fixed, so it’s useful at least for that. Also I introduced with this version a new tool (and package) called rpmbootstrap, whose role is to create a chroot of a RPM based distro. I’ve been able to create CentOS and Fedora chroot up to now. Still working on it for OpenSUSE and Mandriva. It’s largely based on ideas of rinse, and debootstrap, but tightly coupled with pb and reusing some of its interesting features around distribution detection.

And at the same time, support of debootstrap has also been added so it’s possible to easily build now with chroots for most distro.

And I have passed a lot of time trying to improve documentation. Pod doc was already quite good for functions. Now in addition, we have the Lab for easy setup, and a complete documentation of the pb.conf possibilities, which will allow to work now on the possibility to use tool to manipulate them (Target is to use Config-Model). And a website has been setup. Not as complete as I’d like it to be, but a start

And I’ll be presenting the tool, and re-deliver the Lab during the upcoming HP Technology Summit in Las Vegas in June.

Hope to meet with some of you there, and also next week in Valencia in Spain for the Red Hat EMEA Partner Summit

Design document for a patch management system on a DVCS

# patch branches without an explicit dependency depend on the root of the
# patchset tree
# A Root can be given as either a fix commit (seen here), a branch or a tag.
# A fixed commit or tag is useful to maintain a patchset against an older
# upstream version
ROOT: 6a8589de32d490806ab86432a3181370e65953ca
# A tag as a dependency
#ROOT: upstream/0.1.2
# A branch as a dependency
#ROOT: upstream

# A regular patch with it's name and last commit
BRANCH: debian/use-debian-jars-in-build-xml 4bab542c261ff1a1ae87151c3536f19ef02d7937

# two other regular patches
BRANCH: upstream-jira/HDFS-1234 a8e4af13106582ca1bfbbcaeb0537f73faf46d87
BRANCH: upstream-jira/MAP-REDUCE-007 e3426bcbcb2537478f851edcf6eb04b34f6c7106

# This patch depends on the above two patches
# The sha1 below the dependency patches references a merge commit of the two
# dependencies
BRANCH: upstream-jira/HDFS-008 517851aa829d77e09bc5e59985fed1b0aa339cc6
DEPENDENCIES:
  upstream-jira/HDFS-1234
  upstream-jira/MAP-REDUCE-007
    cc294f2e4773c4ff71efb83648a0e16835fca841

# A patch branch that belongs to the patch branch, but won't get exported (yet)
BRANCH: upstream-jira/HDFS-9999 74257905azgsa4689bc5e59985fed1b0aa339cc6
BRANCH-FLAGS: noexport

The upcoming version of project-builder.org will provide a new package called rpmbootstrap. Easy to guess what it does no ?!

Well for those of you Debian/Ubuntu/… fans, who already routinely use debootstrap, it will be no surprise. rpmbootstrap aims to become the debootstrap of rpm world. So it will allow you to create chroot (VE in pb terminology) for supported rpm based distro, in which you’ll be able to do what you have to do. And for pb, what it has to do is building packages.

So the first good news, is that pb in 0.9.9 will support both debootstrap and rpmbootstrap out of the box (on top of rinse and mock). The second good news, is that rpmbootstrap will be delivered jointly with it, trying to follow rpm distros evolution and providing an easy to build VE to build packages in them. Of course, you’ll be able to make a snapshot of that VE to always start over from a known state.

I was initialy tempted to continue to make patches to rinse. But I soon realized that the version I had was a bit far from what rinse provided, without benefiting from the integration I wanted for pb. Also, I couldn’t easily benefit from all the low level libs I now have in perl to help me deal with distro and conf files very efficiently.

I already had the centralized post-install script written for rinse, that I could integrate easily in rpmbootstrap. And writing just that new script took me some 8 hours. Was very easy with the libs ! And now I have a fully integrated solution to cover the whole build lifecycle. I tested it successfully with fedora 12, and I now need to check with other older Fedora distro, and add support for urpmi for mandriva and zipper for opensuse. As well as some tests with RHEL and SLES. But the whole infrastructure is there, just a couple of conf files entries are needed and a bit of code around.

Now, once I solve the package signage on rpm side (using a gpg-agent from perl), I’ll be able to publish version 1.0.

Linux.conf.au 2010 has come to an end and I am looking back at an intense week of conferencing. A big shout out to the organisers for their excellent work. I think LCA (as well as DebConf) just keeps getting better every year. This does not at all discredit previous organisers, because they were the best at their times and then passed on the wisdom and experience to help make it even better in the following year.

The week started off with the DistroSummit, which Fabio and I organised. Slides are forthcoming, as I failed to get them off the speakers right after their talks — it’s interesting how stress levels and adrenaline can cause one to forget the most obvious things. This is where experience comes in. I’ll be there again next year, I hope, to do things better.

The theme of the day was cross-distro collaboration, and we started the day a little bit on the Debian-side with Lucas Nussbaum telling us about quality assurance in Debian, alongside an overview of available resources. We hoped to give people from other distros pointers, and solicit feedback that would enable us to tie quality assurance closer together.

Next up was Bdale Garbee who talked about the status of the Linux Standard Base. While I am really interested in such standardisation efforts, I realised during his talks that I had considerable difficulties paying attention because as organiser of the conference, I had all sorts of other things occupying my thoughts.

I proceeded to tell the audience — the room was mostly filled throughout the day with an estimated 40–50 folks, and I’d say about half of them stayed throughout, while the other half came in and left the room between talks. I could not get the projector to work with my laptop after the upgrade to Kernel Mode Setting, and thus used the whiteboard to give a brief introduction to vcs-pkg.org, talk about the current state of affairs, summarise the trends in discussions around patch management and collaboration, give an outlook of what’s up next, and solicit some discussion.

Sadly, just like during Bdale’s talk, I found myself worrying over the organisation of the day, rather than actually taking in most of the discussion. Fortunately, others have written about the most important points, so I defer to them.

Michael Homer then told us about GoboLinux’s Aliens system, which is a way to integrate domain-specific packages with distro-specific package maintenance — e.g. how to get APT to handle CPAN directly, or how to let YUM manage Python packages. The ensuing discussion was interesting, and we carried it over to the next slot, because Scott, the next speaker, was stuck in traffic. To summarise briefly: scripting languages have a lot of NIH-style solutions because it works for them, but these are a nightmare to distro packagers. One symptom of the status quo is that complex software packages like Zimbra are forced to distribute all required components in their installation packages, which make distro packaging, quality assurance, and security support even harder. I don’t think we found a solution, other than the need for further standardisation (like the LSB), but the road seems to be a long and windy one.

Laszlo Peter introduced the audience to SourceJuicer, a new build system used by OpenSolaris. The idea is that contributors submit packages via a web interface, kicking off a workflow incorporating discussion and vetting, and only after changes have been signed-off are packages forwarded to auto-builders and eventually end up in the package repository. This is very similar to upload ideas I’ve had a while ago, which I’ve started to (finally) implement. Unfortunately, SourceJuicer seems very specific to OpenSolaris, as well as non-modular, so that I probably won’t be able to reuse e.g. the web interface on top of a Debian-specific package builder.

After the break, Dustin Kirkland stepped up to tell us about his user experience of Launchpad. Unfortunately, I found his talk a bit too enthusiastic. Launchpad undoubtedly has some very cool features and ideas, but it’s just one of the available solutions.

The dicussion of Launchpad also dominated the next talk, in which Lucas Nussbaum told us about the Debian-Ubuntu relationship. While his presentation showed that the relationship was improving (Matt Zimmerman made the point that there are rather many relationships, rather than one relationship), I was a bit disturbed by the comments of Launchpad developers in the room, ranging from “Debian is declining anyway” to “Just use Launchpad if you want to collaborate with others and not go down”. There was a slight aura of arrogance in their comments which tainted my experience of the otherwise constructive discussions of the day.

Overall I had a great time. Debian and Ubuntu made up the vast majority of attendants, with only a handful of representatives from other distros present. I wonder why that would be. One reason might be that around 70% of LCA attendants declared themselves Debian or Ubuntu users, and so there weren’t many other distros around. Another might be that I still haven’t spread the word enough. Let’s hope to do better next year!

Thanks to all the speakers. We may have organised the day, but you made it happen and interesting!

Slides and recordings of the talks will be linked from the archived website when they become available (yes, the archive page does not exist yet either).

Update: Jelmer informed me that the people who spoke up against Debian during and after the Launchpad talk were not officially affiliated with Launchpad. It’s a shame that this negatively reflected upon Launchpad for some of the attendees (not just myself).

For the first time this year, I’ll attend the FOSDEM in Brussels. Some FLOSS friends already explained to me it was a must attend event in EMEA, so this I tried to propose some conference material, and my proposal around Project-Builder.org was retained

I’ll only arrive on the Saturday morning, but will leave on Monday, combining a customer visit with it. So if any of the reader is around and want to discuss about Continuous Packaging or Disaster Recovery with Mondo Rescue, or anything else, that would be with great pleasure !

And there will be lots of interesting talks there as well, such as the one from Dominique Dumont on Config-Model, Guillaume Rousse on Youri or Jeff Johnson on RPM, just to name a few in the same room.

I hope to meet you there as

Here’s stuff that I’d like to do this year, more or less by decreasing order of importance:

• translate my Debian book into English and get it published;
• finish the cleanup of the perl API in dpkg-dev in order to create libdpkg-perl;
• create dpkg-buildflags to export default build flags that packages should use (and get rid of the code setting those environment variables in dpkg-buildpackage), needed to properly fix #489771;
• ensure the new source formats continue to gain acceptance by improving whatever is needed;
• design a generic vcs-buildpackage infrastructure to be integrated in dpkg-dev. This design will probably happen through a DEP (Debian Enhancement Proposal) to ensure we have had proper discussion before someone gets to the implementation;
• continue fixing dpkg bugs faster than they are reported;
• enhance our infrastructure to ease interaction between contributors and to have a better view of how each package is maintained (see my last blog entry on this topic);
• update the developers-reference where needed and fix some of the numerous wishlist bugs;
• rewrite in C the last perl scripts provided by the dpkg binary package (update-alternatives/mksplit mainly, for dpkg-divert there’s a preliminary patch available already) so that it’s easier to build a minimal system without perl-base;
• integrate the 3-way merge tool for Debian changelogs in dpkg-dev;

All of this probably doesn’t fit in my free time (being a father since last month does not help increasing my free time ), so if you’re interested in seeing one or more of those projects completed, and if you know some person/company that could sponsor them, get in touch with me!

Bzr build-deb is very nice, but it can be very tricky to get started. I recently did a fresh debianisation of a project that is in bzr upstream, and I thought I’d record the recipe to make it work (at least until the various bugs making it hard re fixed).

Assuming that the upstream uses bzr, it goes like this:

1. Start with a branch that is close to the code you want to Debianise. E.g. if the release was off trunk, 3 commits back: bzr branch trunk -r -3 debian
2. Debianise as normal: put the tarball with the right name in the parent dir,  add a debian directory and fiddle until you build a package you’re happy with. Don’t commit while doing this.
3. Build a source package- debuild -S, or bzr builddeb -S
4. Revert your changes – bzr revert.
5. Import the dsc – bzr import-dsc ../*.dsc
6. Now, you may find that some dot files, such as .bzrignore have been discarded inappropriately (there is a bug open on this). If that happened, keep going. Otherwise, you’re done: you can now use merge-upstream on future upstream releases, and debcommit etc.
7. bzr uncommit
8. bzr revert .bzrignore (and any other files that you want to get back)
9. debcommit
10. All done, see point  6 for details.

Hope-this-helps

Software comes in many shapes and styles. One of the problems the author of software faces is distributing it to their users.

As distributors we should not discourage upstreams that wish to generate binary packages themselves, rather we should cooperate with them, and ideally they will end up maintaining their stable release packages in our distributions. Currently the Debian and Ubuntu communities have a tendancy to actively discourage this by objecting when an upstream software author includes a debian/ directory in their shipped code.  I don’t know if Redhat or Suse have similar concerns, but for the dpkg toolchain, the presence of an upstream debian directory can cause toolchain issues.

In this blog post, I hope to make a case that we should consider the toolchain issues bugs rather than just-the-way-it-is, or even features.

To start at the beginning, consider the difficulty of installing software: the harder it is to install a piece of software, the more important having it has to be for a user to jump through hoops to install it.

Thus projects which care about users will make it easy to install – and there is a spectrum of ease. At one end,

checkout from version control, install various build dependencies like autoconf, gcc and so on

through to

download and run this installer

Now, where some software authors get lucky, is when someone else makes it easy to install their software, they make binary packages, so that users can simply do

apt-get install product

Now some platforms like MacOSX and Microsoft Windows really do need an installer, but in the Unix world we generally have packaging systems that can track interdependencies between libraries, download needed dependencies automatically, perform uninstalls and so on. Binary packaging in a Linux distribution has numerous benefits including better management of security updates (because a binary package can sensibly use shared libraries that are not part of the LSB).

So given the above, its no surprise to me to see the following sort of discussion on #ubuntu-motu:

1. upstream> Hi, I want to package product.
2. developer> Hi, you should start by reading the packaging guide
3. (upstream is understandably daunted – the packaging guide is a substantial amount of information, but only a small fraction is needed to package any one product.)

or (less usefully)

1. upstream> Hi, I want to package product.
2. developer> If you want to contribute, you should start with existing bugs
3. upstream> But I want to package product.

Another conversation, which I think is very closely related is

1. developer> Argh, product has a debian dir, why do they do this to me?!

The reasons for this should be pretty obvious at this point:

• Folk want to make their product easy to install and are not themselves DD’s, DM’s or MOTU’s.
• So they package it privately – such as in a PPA, or their own archive.
• When they package it, they naturally put the packaging rules in their source tree.

Now, why should we encourage this, rather than ask the upstream to delete their debian directory?

Because it lets us, distributors, share the packaging effort with the upstream.

Upstreams that are making packages will likely be doing this for betas, or even daily builds. As such they will find issues related to new binaries, libraries and so on well in advance of their actual release. And if we are building on their efforts, rather than discarding them, we can spend less time repeating what they did and more packaging other things.

We can also encourage the upstream to become a maintainer in the distro and do their own uploads: many upstreams will come to this on their own, but by working with them as they take their early steps we can make this more likely and an easier path.