376315 - Set up mercurial mirror of CVS

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Benjamin Smedberg]

Attachment: Import from CVS to HG, rev. 1

The script I am going to attach will create and maintain a mercurial mirror of the CVS repository for mozilla 2. It should be set up on a HG repository which is publicly available but only writable by this script (there will be no public checkins).

To initialize the repository, run

cvs2hg-import.py --inirepo <dir>

To do an import, run
cvs2hg-import.py --cvsroot=<cvsroot> <dir>

The import should be performed frequently, Ideally, it would be performed whenever there are unimported checkins in CVS, but it could probably be run just as well on a cron job or in a permanent loop. The script takes about 30 mins on my machine.

I'd like jst or vlad to review the script, before we set it up for use.

Comment 1

[Benjamin Smedberg]

vlad/jst, can one of you look this over?

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

It looks fine to me; for my personal use I tend to just do a client.mk update in a "clean" tree, though it's only bandwidth (and this will be running local to cvs anyway, I assume) that's the issue.

Comment 3

[Aravind Gottipati [:aravind]]

Tried the CVS import, failed with this message.

cvs checkout: Updating mozilla/xulrunner/stub
U mozilla/xulrunner/stub/Makefile.in
U mozilla/xulrunner/stub/nsXULStub.cpp
cvs checkout: Updating mozilla/xulrunner/util
Moving /tmp/tmpXvYOvtcvsimport/mozilla/parser to /repo/hg/mozilla/parser
Traceback (most recent call last):
  File "../cvs2hg-import.py", line 225, in ?
    ImportMozillaCVS(args[0], options.hg, options.cvsroot)
  File "../cvs2hg-import.py", line 178, in ImportMozillaCVS
    rename(source, dest)
OSError: [Errno 18] Invalid cross-device link
[hg@dm-svn01 mozilla]$                                  

Comment 4

[Benjamin Smedberg]

Aravind, hrm, this was tested on a system where /tmp and the repo were on the same filesystem. If I add a flag to specify a tmpdir that's on the same filesystem, will that be sufficient?

Comment 5

[Benjamin Smedberg]

Attachment: Import from CVS to HG, rev. 1.1

Revised script takes an optional --tempdir parameter.

Comment 6

[Aravind Gottipati [:aravind]]

Attachment: cvs-trunk to hg import script.

Fixed the calls to hg.

Comment 7

[Aravind Gottipati [:aravind]]

Imported the current CVS trunk code in mercurial.

You can browse this at http://hg.mozilla.org/

I am keeping this bug open, since I still have to cron this, check security perms etc on the repo.

Comment 8

[Aravind Gottipati [:aravind]]

How often do you need the repo updated?

Comment 9

[Brendan Eich [:brendan]]

We were talking about nightly, even with a CVS lock down to speed up the process and guarantee a coherent snapshot. We need the ability to pick and choose when we merge from the 1.9 trunk mirror branch into the Mozilla 2 integration branch, and we may wish to do this two or more days in a row, as well as for any given day. Especially if there's a nightly hook-clearing protocol with QA in the a.m. and sheriffing, again.

/be

Comment 10

[Benjamin Smedberg]

As I mentioned in private email, I don't see any way to do the mirror->integration branch merge automatically, because it will probably involve regular merging.

Personally, I would prefer that the mirroring happen as often as possible, if CVS server load will allow it. The finer the granularity of commits, the easier it is to merge when you have conflicts.

If coherence is likely to be an issue, we should check out by date.

Comment 11

[Brendan Eich [:brendan]]

Dbaron was in on the sync once per day idea. IIRC server load and coherence were both considerations. Checking out by date doesn't guarantee coherence with CVS locking, alas.

/be

Comment 12

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'd be quite happy with "as often as possible", since it would make it possible for those of us still working on 1.9 to use it rather than all keep our own copies.  Once per day probably wouldn't be sufficient for that.

Comment 13

[Reed Loden [:reed]]

Attachment: Import from CVS to HG, rev. 2.1

It might help if toolkit gets imported! :)

Aravind, can you use this version instead? I just added "toolkit" to the list of things to import.

Comment 14

[Benjamin Smedberg]

Yeah whoops! r=me

Aravind, we need to check this script into VCS somewhere. Perhaps somewhere in mozilla/tools in CVS?

Comment 15

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Do we really want to exclude non-Firefox apps from this mirror?

Comment 16

[Aravind Gottipati [:aravind]]

Okay, will modify the script to get toolkit as well. I have set this up in cron to sync every 6 hours.  Let me know if you want it done more frequently.

I will check this script into mozilla/tools.  Can this bug be closed now?

Comment 17

[Benjamin Smedberg]

Sure!

Comment 18

[Brendan Eich [:brendan]]

(In reply to comment #15)
> Do we really want to exclude non-Firefox apps from this mirror?

I think so; it's not a general purpose mirror. It's for Mozilla 2.

See http://groups.google.com/group/mozilla.dev.platform/browse_frm/thread/7673df08a4e1a499/a5300dc31385d601#a5300dc31385d601

You're right, it could be a larger mirror, but as a matter of policy I think it should not be. A separate hg mirror of all of cvs-mirror could be set up, with different access control and no development model as we want for Mozilla 2. File another bug?

/be

Comment 19

[J. Paul Reed [:preed]]

Talked to bsmedberg and aravind about this, and there are a couple of things we wanted to have as artifacts of the cvs import:

-- Complete log (stdin/stderr) from the import
-- Import as of a well-known epoch date (the 1.9a3 tag is a good one; but we should actually tag CVS with the directories that were imported)
-- Import from cvs.m.o, not cvs-mirror.m.o

Reopening to tweak the script and re-run it.

Comment 20

[J. Paul Reed [:preed]]

Attachment: Import CVS to Hg, rev 2.2

Talked with Benjamin and Aravind about making these changes; this patch:

-- For the initial import, pull the CVS tree from a (known) tag; right now, this tag doesn't exist, but it will shortly.
-- Remove all the quiet options to hg/cvs; we want the output from these automated processes
-- Point at cvs.m.o, not cvs-mirror.m.o (implies that whomever is running this script has access to the cltbld key and can use it for this purpose)
-- Check in each "module" (which it really isn't that, but...) separately
-- Pull by date for imports.
-- Include the checkin "module", the branch it came from, and the pull date in the Hg checkin message.

Comment 21

[Benjamin Smedberg]

Comment on attachment 261307 [details] [diff] [review]
Import CVS to Hg, rev 2.2

I suppose this is ok. I'm not quite sure why you bothered with the importModules hash.

And it looks like you eat exceptions that happen during import, which means that the script will exit with a zero code. Is this what you intended?

Comment 22

[J. Paul Reed [:preed]]

Attachment: Import CVS to Hg, rev 2.2, take ii

(In reply to comment #21)
> (From update of attachment 261307 [details] [diff] [review])
> I suppose this is ok. I'm not quite sure why you bothered with the
> importModules hash.

I did this so that the commits into the Hg repository show up as separate commits. The reason I did *that* is so that when we change [branch] tags that we pull from, the commit messages reflect that. I suppose we could do it as one commit (which I'd prefer), and then build up a commit message that has all the details.

What do you think?

> And it looks like you eat exceptions that happen during import, which means
> that the script will exit with a zero code. Is this what you intended?

Nope; good catch; fixed.

Admittedly, my python is a bit rusty. :-/

Comment 23

[J. Paul Reed [:preed]]

Attachment: Import CVS to Hg, rev 2.2, take III

Benjamin pointed out that doing separate commits for these modules would cause NSPR/NSS to be entirely removed in those commits.

We still want the per module log messages to provide the most information possible, so take 3:

-- Fix a bug where mozilla wouldn't get the right tag
-- Move back to one Hg commit, but building up a message with specific information about each module
-- Make sure rmtree runs in the finally, since we re-raise the exception.
-- Use UTC for CVS pulls, like Tinderbox

Comment 24

[J. Paul Reed [:preed]]

Attachment: bz2'd log file of the application of the HG_REPO_INITIAL_IMPORT tag in CVS

This is a bz2 compressed log file (with set -e/set -v) from the run I did to tag the CVS repo with the HG_REPO_INITIAL_IMPORT tag, which is a clone of the MOZILLA_1_9a3_RELEASE tag, but only applied to the areas that were actually checked into Hg.

Comment 25

[Benjamin Smedberg]

2.2 take III looks correct

Comment 26

[J. Paul Reed [:preed]]

(In reply to comment #25)
> 2.2 take III looks correct

Great!

I'll get together with Aravind, and we'll run through it.

Comment 27

[J. Paul Reed [:preed]]

Attachment: bz2'd log file of the hg repo initialization using version 1.6 of cvs2hg-import.py

Comment 28

[Aravind Gottipati [:aravind]]

Can this be resolved now?  I think jst is actually syncing stuff back to the hg repo.

Comment 29

[Johnny Stenback (:jst)]

Yes, this can IMO be considered done now. There's still work to be done for automation of the cvs-trunk-mirror updates etc, but we'll open new bugs for that work.