Last Comment Bug 470570 - (bz-bzr) Switch away from CVS to Bazaar (bzr)
(bz-bzr)
: Switch away from CVS to Bazaar (bzr)
Status: RESOLVED FIXED
:
Product: Bugzilla
Classification: Server Software
Component: bugzilla.org (show other bugs)
: unspecified
: All All
: P1 enhancement with 1 vote (vote)
: ---
Assigned To: Max Kanat-Alexander
: default-qa
Mentors:
http://bzr.bugzilla.org/
Depends on: 517131
Blocks:
  Show dependency treegraph
 
Reported: 2008-12-20 10:52 PST by Max Kanat-Alexander
Modified: 2010-02-01 16:17 PST (History)
16 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Max Kanat-Alexander 2008-12-20 10:52:28 PST
(Please note: This bug is not the place to discuss the merits of this proposal, I will start a thread on developers@ for that.)

So, I'd like the Bugzilla Project to switch to using some modern VCS system, and get away from CVS.

The modern VCS feature that I most-frequently want is atomic directory and file renames. Basically, this would mean that we could rename a file during a patch, but the diff would still only show the changes that we made to the file.

There are also a LOT of other advantages to using a modern VCS. People can make local branches (great for customizations and major projects), merging in changes to customized releases is much easier, server-side administration is way easier, you can check out through HTTP (handy if you're behind a firewall), there are way more useful commands (compare "bzr status" to "cvs status")--the list just goes on and on.

So, when talking about modern VCS systems, there are really only three serious choices--git, mercurial, and bzr.

git is very powerful, but enormously cryptic (even though it's less cryptic than it used to be). It also has a heavy focus on Linux development with the Windows support being not as good.

So then we want to compare Mercurial and Bazaar. Mercurial is fine, but not quite as featureful as bzr, I believe, and bzr is developed a lot faster--bzr has more new major releases more frequently than any other piece of software I use. Also, Everything Solved (my company) and Mozilla have been using bzr for Bugzilla work for a long time now, and we have a lot of experience with bzr.

The only real reason to go with Mercurial would be to re-use the current Mozilla infrastructure around it. However, I think we'd actually rather control the list of committers ourselves, though we'd still want them to sign the Mozilla Committers Agreement and all, and have the Foundation deal with the legal aspects. Also, I probably have more experience running a bzr server than anybody at Mozilla.

The move would preseve our entire history--that is, every commit to every branch would be available in bzr.

I talked to justdave about the move, and he's fine with it, as long as we set up a bzr->cvs mirror, so that people who checked out of CVS, or who are still using a tarball from one of our releases, can "cvs up" to get updates.

So, as near as I can tell, the process for going forward would be:

1. Set up a basic bzr.bugzilla.org as a mirror of CVS.
2. Install loggerhead (a bzr repository browser) on bzr.bugzilla.org.
3. Investigate a bzr->cvs mirror, and prepare to set it up.
4. Update bonsai links on www.bugzilla.org to point to loggerhead instead 
   of bonsai.
5. Close down CVS to commits.
6. Fix the bzr import (it has a few tiny errors that need to be fixed
   once we start using it as our main repo, and these errors can't be
   fixed as long as it's a mirror of CVS, due to the way the importer works).
7. Tag all the old releases in bzr using "bzr tag" (also something I
   I think we can't do as long as we're a mirror of CVS).
Comment 1 Max Kanat-Alexander 2008-12-20 10:58:24 PST
The discussion thread is here, for anybody who wants to follow along at home:

http://groups.google.com/group/mozilla.dev.apps.bugzilla/browse_thread/thread/8f1d9255845b3794#
Comment 2 Max Kanat-Alexander 2008-12-20 12:11:47 PST
Here's a few notes on how I'd want to configure the bzr server:

* It should reject changes unless the user has specified some valid
  "bzr whoami". In fact, I'd really like to make sure that the committer
  is who they say they are (for the purpose of data consistency), so we
  might want to require signing commits (though that's a bit obnoxious...).

  Requiring signatures is done with "check_signature = require" in the
  branch.conf for the branch.

* It should send an email to the commits list. I believe there are plugins
  or possibly even built-in features to do this.

* Should possibly set "append_revisions_only" on the main branches so that
  nobody runs "uncommit" or anything on the server by default?

* Make sure that all the branches in the server repository are unbound.

* We may want to use the 1.6 or 1.6.1-rich-root repository format, to get
  better performance. I believe people accessing the repo over bzr://
  could still use an earlier version of bzr, but anybody using http://
  or sftp:// would have to have 1.6.1 or later (which is pretty new
  right now). There's also the 1.9 and 1.9-rich-root formats, but those
  are definitely too new.
Comment 3 Max Kanat-Alexander 2008-12-20 12:16:21 PST
Also, we should set bugzilla_url or bugzilla_bmo_url in the branch.conf of each of our branches, so that people can do "bzr commit --fixes bmo:123456" when committing.
Comment 4 Max Kanat-Alexander 2008-12-20 12:23:13 PST
The layout will be:

bzr.bugzilla.org
  qa
    3.2
    trunk
  bugzilla
    trunk
    3.2
    3.0
    2.22
    ...
    stable

Where "stable" is a symlink to the current stable branch. (This is the only way we can maintain something like the Bugzilla_Stable tag we have now.)

We may also have a "testopia" directory if Greg wants to move to our bzr. Alternately, we may have an "extensions" directory with a "testopia" subdirectory, and offer hosting to anybody who wants to write an extension.

Branches will be tagged for each release, with the version numbers. So, for example, 3.0 would have tags like: 3.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4.

Stable branches will also have a "stable" tag that points to the latest stable release.
Comment 5 Max Kanat-Alexander 2008-12-20 13:01:59 PST
Okay, here is a bzr extension (or script) that can sync back to CVS just like we need:

https://edge.launchpad.net/bzrcvsserve
Comment 6 Vitaly Fedrushkov 2008-12-20 15:02:07 PST
8. Update documentation and wiki about keeping Bugzilla up-to-date with bzr
instead of CVS (and how to migrate and merge local changes)
Comment 7 Max Kanat-Alexander 2008-12-20 15:12:30 PST
Oh, right! We also have to fix the /download/ page on bugzilla.org.
Comment 8 Gervase Markham [:gerv] 2008-12-20 15:18:54 PST
Max: just to clarify, is Everything Solved offering to set up and run the bzr server and the bzr->cvs mirror (software and hardware), or is that something we'd ask MoCo IT to do?

Gerv
Comment 9 Max Kanat-Alexander 2008-12-20 15:33:02 PST
(In reply to comment #8)
> Max: just to clarify, is Everything Solved offering to set up and run the bzr
> server and the bzr->cvs mirror (software and hardware), or is that something
> we'd ask MoCo IT to do?

  I'm offering to have the Bugzilla Project run it, which would be me (at least for now), using a VM on cg-vmware01. I already have a bzr.bugzilla.org that just runs off landfill.
Comment 10 Frédéric Buclin 2008-12-21 04:41:31 PST
(In reply to comment #6)
> 8. Update documentation and wiki about keeping Bugzilla up-to-date with bzr
> instead of CVS (and how to migrate and merge local changes)

9. Will despot.mozilla.org still work with bzr? Or how can we manage who can commit patches? Will bzr still require a valid SSH key?
9'. In Despot, justdave is currently defined as project owner and mkanat and me as peers. Will we keep our "privileges", or are you going to become the unique "administrator"?
10. Will tinderbox.mozilla.org still work with bzr?
11. Note that PatchReader must be patched in order to read patches made with bzr correctly, and we don't own PatchReader. Or we have to fork it and include it in our own code (Bugzilla/Attachment/PatchReader/*.pm).
12. Can bonsai be fixed to support bzr? Maybe this includes we contribute to Bonsai to implement this? I can't live without Bonsai.
13. What happens the day you decide to leave the project (or if you are on vacation for several days/weeks)? Who will administrate/upgrade/maintain bzr?
Comment 11 Teemu Mannermaa (:wicked) 2008-12-21 05:46:50 PST
14. Remember to update Patch Database Administration website and scripts in /usr/local/bin/ of landfill server to work with bzr instead of cvs. These make Bugzilla development on landfill so much easier that they should just keep working.
Comment 12 Max Kanat-Alexander 2008-12-21 09:28:17 PST
(In reply to comment #10)
> 9. Will despot.mozilla.org still work with bzr?

  We could possibly make it do so, but it won't be as important since we'll have our own server, and we won't need to partition off "Bugzilla".

> Or how can we manage who can commit patches?

  Easiest way is with Unix group security, otherwise there are some pretty advanced authentication systems and hooks in bzr that we can use.

> Will bzr still require a valid SSH key?

  Probably, yeah. It all depends on how we set it up. It supports lots of methods of accessing the repo and various forms of authentication.

> 9'. In Despot, justdave is currently defined as project owner and mkanat and me
> as peers. Will we keep our "privileges", or are you going to become the unique
> "administrator"?

  We could probably work something out. It really depends on what you want to do.

> 10. Will tinderbox.mozilla.org still work with bzr?

  Yeah, of course. Also, there will be a read-only CVS mirror, which is all it needs, so we don't even have to update it, really.

> 11. Note that PatchReader must be patched in order to read patches made with
> bzr correctly, and we don't own PatchReader. Or we have to fork it and include
> it in our own code (Bugzilla/Attachment/PatchReader/*.pm).

  Yeah, true. Thankfully bmo has this patch.

> 12. Can bonsai be fixed to support bzr? Maybe this includes we contribute to
> Bonsai to implement this? I can't live without Bonsai.

  Loggerhead: http://bzr.bugzilla.org/

  As I mentioned to wicked on the list, it also has some undocumented features, so if you want to do something that you can't figure out how to do, ask me in IRC or ask #bzr on FreeNode.

> 13. What happens the day you decide to leave the project (or if you are on
> vacation for several days/weeks)? Who will administrate/upgrade/maintain bzr?

  Whoever has access to the VM. This is the same question as "Who will administer tinderbox.bugzilla.lan if mkanat disappears?"
Comment 13 Frédéric Buclin 2008-12-21 09:49:26 PST
(In reply to comment #12)
> > 9'. In Despot, justdave is currently defined as project owner and mkanat and me
> > as peers. Will we keep our "privileges", or are you going to become the unique
> > "administrator"?
> 
>   We could probably work something out. It really depends on what you want to
> do.

I want to be able to exclude everbody besides justdave, you and me (i.e. owner and peers) from committing patches on some specific branch(es) while we are in the release process, or when a branch is locked to security fixes only.
Comment 14 Max Kanat-Alexander 2008-12-21 10:09:37 PST
(In reply to comment #13)
> I want to be able to exclude everbody besides justdave, you and me (i.e. owner
> and peers) from committing patches on some specific branch(es) while we are in
> the release process, or when a branch is locked to security fixes only.

  Yeah, that should be possible. I definitely want other people to be able to do that. I don't really *want* to do all the administration of the bzr server myself forever--I'd love somebody else to take it over at some point. I just want to do the initial setup of stuff since I have the most experience with it.
Comment 15 timeless 2008-12-25 08:39:03 PST
fwiw, i've repeatedly checked loggerhead, and its support for reasonable urls is nonexistent, which means i can't integrate mxr with bzr.

i have the beginnings of hg support in a bonsai...
Comment 16 Max Kanat-Alexander 2008-12-26 11:59:24 PST
(In reply to comment #15)
> fwiw, i've repeatedly checked loggerhead, and its support for reasonable urls
> is nonexistent

  http://bzr.bugzilla.org/bugzilla/trunk/annotate/6363/Bugzilla/Constants.pm

  There's a bug opened against loggerhead to make it use those URLs by default instead of the cryptic file_id URLs. (That URL above works now, though. You can also replace "6363" with "head:" to get the latest revision.)
Comment 17 Martin Albisetti 2009-01-06 19:54:40 PST
(In reply to comment #16)
>   There's a bug opened against loggerhead to make it use those URLs by default
> instead of the cryptic file_id URLs. (That URL above works now, though. You can
> also replace "6363" with "head:" to get the latest revision.)

It's already the default in the latest release (1.10), and I've posted a branch that generates a view with a range of revisions (https://bugs.launchpad.net/loggerhead/+bug/310342).
Comment 18 Max Kanat-Alexander 2009-09-16 19:16:58 PDT
Okay, we took a vote on the developers list, and (including two emails sent to me privately) overwhelmingly in favor of bzr. So it looks like we'll be moving to bzr.

The first step is to get Mozilla to set up a bzr server.

This requires:

* An rpm installation of the latest bzr available (as they all get faster, each version, and the faster the better).

* A checkout and installation of the current trunk of loggerhead:
    bzr co lp:loggerhead
    cd loggerhead
    python setup.py install
    Copy and install my modified loggerheadd initscript from landfill (it lives in /etc/init.d)
    Copy and install my Apache configuration for loggerhead (in the bzr.bugzilla.org VirtualHost).

* Set up a bzr server--possibly through xinetd if that's actually the most efficient (I just have it set up that way right now on bzr.bugzilla.org because it was the easiest configuration--it might actually be easier to set up a persistent server, we should look into it.) This enables read-only bzr:// access.

* Decide on our permissions scheme. This is a good idea to do before creating the repository, because it allows us to set some +s groups or owners on the repository and then have everything "just work".

* Init a repository. The easiest place is /var/www/html/bzr.bugzilla.org or wherever Apache normally stores html files. This allows easy direct http:// access to the bzr files.

According to the folks in #bzr, the best repository format to have right now would be 1.9-rich-root. Eventually 2a will surpass that, but it's only supported as of 1.16, and we don't want to make it too difficult for people to check out if they have older versions. 1.9 is a year old at this point, so it should be OK. Perhaps by next year or a year later we can convert to 2a.

So the command to init a repo would be:

bzr init-repo --no-trees --1.9-rich-root /var/www/html/bzr.bugzilla.org/

Every time you want to have different commit permissions, though, you will need a separate repository within the repository. For example, on bzr.bugzilla.org, the /selenium/ directory is actually its own repository, because it's owned by lpsolit:qateam, as opposed to mkanat:bugzilla for the other one.

(If you ever have to convert some branches in a shared repository to having their own repo, the thing to do is the convert them all the standalone branches, then create a new repository below their directory level, then reconfigure them all to use the shared repository.)

* Pull or branch all the bzr.bugzilla.org branches into the current repository.

* Make sure that the branches are configured correctly. For example, they should all have the proper nickname set, and then all the stuff in comment 2.
Comment 19 Gervase Markham [:gerv] 2009-09-17 03:40:07 PDT
Just to check: we are preserving CVS history, right?

Gerv
Comment 20 Max Kanat-Alexander 2009-09-17 03:42:01 PDT
(In reply to comment #19)
> Just to check: we are preserving CVS history, right?

  Absolutely and completely, for every branch.
Comment 21 Max Kanat-Alexander 2010-01-31 14:36:12 PST
Okay, the bzr server is now fully ready for our use.

I'm going to be doing our pending releases, hopefully today, and then shortly after that we'll be moving to bzr.
Comment 22 Max Kanat-Alexander 2010-02-01 16:17:40 PST
Okay, I had to sync the bzr server data with the CVS data. I did this for 3.0, 3.2, 3.4, and trunk. I also added a .bzrignore file to those branches, and removed the .cvsignore files from bzr.

I've left the 2.22, 2.20, 2.18, 2.16, and 2.14 branches untouched. I think it doesn't matter if they have a .bzrignore file, since we won't be doing any committing or development against them, they're just there to preserve the history of the project within bzr.

The bzr-to-cvs script is functioning, and once I'm sure it's fully syncing everything to CVS automatically, I'll be announcing this to the developers list. :-)

So basically, we're done.

Note You need to log in before you can comment on or make changes to this bug.