618838 - jetpack hg bridge is broken

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect)

Description

[Brian Warner [:warner :bwarner]]

When we branched 1.0b1 (specifically when the first commit on this branch was
pushed into github), the git-to-hg bridge that I run on my personal linux box
stopped working. The result is that http://hg.mozilla.org/labs/jetpack-sdk is
out of date: the last commit it sees is on 01-Dec-2010, and contains the
patch that changes the version number to "1.0b1".

The problem is fairly deep. In the hg world, published branches are expressed
as immutable "branch" properties on each revision. In the git world, branches
are merely mutable pointers to a revision. When converting from hg to git,
branches are fairly easy to deal with. But when converting the other way,
they are not: individual git revisions do not naturally have a "branch", so
far from clear what to stuff into hg.

The problem that occurred on 01-Dec is that the git-to-hg bridge (which uses
the http://hg-git.github.com/ mercurial plugin) copied the 1.0b1-branch patch
into hg without any special branch markers: it is now the current 'tip'.
Shortly thereafter, when the git master branch was updated to the new trunk
revision (which changed the version number to '1.0b2pre'), the bridge tried
to copy that revision into hg, again without any special branch markers. This
creates two 'heads' in hg, which it doesn't really like. It likes pushing
multiple heads into a remote repository even less.

The current situation is that the hg repo on my linux box has two heads, and
the hg repo on hg.m.o has just one (but it is the 1.0b1-branch head, not the
trunk head). I've disabled the cron job that normally runs the bridge script,
since it was just spewing error messages into my email every 5 minutes.

I'm not sure how to fix this. I think the first step is to modify (or perhaps
merely configure) the git-to-hg bridge to create hg revisions with branch=
attributes when bridging git revisions reached through a non-"master" branch
reference. The second step is probably to manually remove the 1.0b1-branch
revision from hg.m.o . Then the third step will be to allow the bridge to
resume. If that all works. we should wind up with an hg.m.o repo with
release-branch revisions labeled "branch=1.0b1", and trunk revisions that
lack branch indicators.

Comment 1

[Brian Warner [:warner :bwarner]]

Ah, I just had an idea while talking with Lloyd. We could use an intermediate git tree, to which we only ever pull 'master' (never pulling any other branches). We run the bridge against that tree: so no other branches will be touched, and we'll never create a multiple head in the hg repo.

(what I learned by reading through the hg_git code is that it has some internal function arguments to limit the git-to-hg directory to a subset of the refs, but there's nothing in the CLI that exposes this. I also learned that hg_git will put a "branch:" attribute on an hg commit if-and-only-if that commit came from hg originally, i.e. the hg-to-git direction saw a branch attribute and inserted a --HG-- block into the git comments, and then later the git-to-hg direction encountered that same commit. So not so useful).

We'll still need to back out the leftover head in hg.m.o: I'm not sure how to accomplish that, or if there's something we can *add* to hg.m.o (a new child commit with a different branch attribute?) that would fix the multiple-heads situation.

Comment 2

[Brian Warner [:warner :bwarner]]

hm, to do this correctly will require using a 'git pull' that doesn't fetch/follow tags, since otherwise the 1.0b1 tag will pull in the troublesome branch.

Comment 3

[Brian Warner [:warner :bwarner]]

ok, I think I finally have a script that will do this correctly:

* tree1 is built with: 'git init --bare; git fetch --no-tags github master:master'
* tree2 is built with: 'hg init; hg pull git+ssh://localhost/path/to/tree1'
* then inside tree2 we can do 'hg push hg.m.o'

The --no-tags and explicit 'master:master' refspec means that tree1 will only have commits that are reachable from the 'master' reference: no release branches, no random feature branches, and no tags. Preventing tags is important because the 1.0b1 release tag is not reachable from 'master' (it lives on its own branch), and would result in multiple heads.

The tree2 'hg pull' from localhost (which requires a bit of ssh key management) is necessary because the hg_git plugin can't actually handle pulling from git trees directly from the local filesystem (e.g. 'hg pull ../tree1'). The initial pull succeeds, but any subsequent pulls fail. If we didn't need --no-tags and the 'master:master' limitation, we could pull directly from github. But we do, so we can't.

One downside of this approach is that we don't get any release tags in the HG repo. The repo has an .hgtags file, leftover from the days when jetpack lived in HG and the bridge went hg->git, but for some reason 'hg tags' doesn't recognize it, and of course the --no-tags argument means that hg_git isn't creating new hg tags for any of the git tags.

The resulting HG repo is really only useful for pulling current HEAD.

The next steps will be to get IT to create a new HG repo on hg.mozilla.org, named "addon-sdk", and to delete the old repo (named "jetpack-sdk"). Then I'll run my scripts to get it populated as described above, and re-establish the cronjob to update it every 10 minutes.

Comment 4

[Brian Warner [:warner :bwarner]]

Ok, Bug 623787 is about creating the new hg.m.o repo and deleting the old one

Comment 5

[Brian Warner [:warner :bwarner]]

ok, I've updated the scripts, and they appear to be pushing to the new http://hg.mozilla.org/projects/addon-sdk repo correctly. We'll find out for sure after the next commit.

The old repo is still in place. Bug 623787 suggested we could put a comment on it to advise users of its obsoleteness, but I don't see how to do that. It's probably a good idea to add such an advisory, because folks searching for "jetpack" will likely find the old repo first, and then wonder why it hasn't been updated since December 2010.