968958 - [meta] Migrate dev VCS tools and extensions to the new version-control-tools repository

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Gregory Szorc [:gps]]

So, we now have a shiny new intended-to-be-unified repository for holding ALL OF THE version control things. I blogged about this at http://gregoryszorc.com/blog/2014/02/05/new-repository-for-mozilla-version-control-tools/

The eventual goal is to move hghooks, pushlog, hgtemplates, etc into this repo. This bug will track that effort.

As I mentioned on the blog post, the initial thrust is towards unifying the user-level extensions and hooks. But the sooner we can get IT systems moved to the new repo, the better.

I understand the local disk migration is likely higher priority and I don't want to create fire drills for people. I and other engineers can assist with the migration to this new repo if we know how to help.

I'm largely ignorant of the IT processes involved, but I think the general process would look something like:

1) Make the version-control-tools repo part of the deployment process and available to all machines
2) Merge some projects into the version-control-tools repo
3) Update IT/system configs to reference new paths
4) Deploy
5) Rinse and repeat 2-4 until everything is running from version-control-tools

fubar: Could you please provide additional details to help us assess the effort required? Also, if you would like to start dumping IT stuff into this new repo, I highly encourage that! As mentioned in the blog post, I think it would be rad if developers could reproduce the IT setup so we can test things locally before pushing to stage.

Comment 1

[Ed Morley [:emorley]]

(For my own reference as much as anything else)

Repos already moved into https://hg.mozilla.org/hgcustom/version-control-tools
* https://hg.mozilla.org/users/gszorc_mozilla.com/hgext-gecko-dev/
* https://hg.mozilla.org/users/tmielczarek_mozilla.com/bzexport/

Further candidates...

Dev tools:
* https://hg.mozilla.org/users/robarnold_cmu.edu/qimportbz/
* https://bitbucket.org/sfink/qbackout
* https://bitbucket.org/sfink/mqext
* https://bitbucket.org/indygreg/python-mozautomation
* https://bitbucket.org/sfink/trychooser (newer fork of pbiggar's extension aiui)
* https://github.com/pbiggar/trychooser
(note for these last two, they also have to be kept in sync with the html version (https://hg.mozilla.org/build/tools/file/tip/trychooser) - which often fails to happen. I'd prefer if we could either move the html version into the version-control-tools repo, or else have a central config which it shares with the hg extension.

hg.m.o management:
* https://hg.mozilla.org/hgcustom/hgscripts/

hg.m.o custom hg / hgweb:
* https://hg.mozilla.org/hgcustom/hg_templates/
* https://hg.mozilla.org/hgcustom/pushlog/
* https://hg.mozilla.org/hgcustom/hghooks/

Automation utils/wrappers:
* https://hg.mozilla.org/build/mozharness/file/tip/mozharness/base/vcs/
* https://hg.mozilla.org/build/tools/file/tip/buildfarm/utils/hgtool.py (~dupe of mozharness copy, since not all jobs use mozharness yet)
* https://hg.mozilla.org/build/tools/file/tip/buildfarm/utils/gittool.py (ditto)
* https://hg.mozilla.org/build/mozharness/file/tip/configs/vcs_sync/
* https://hg.mozilla.org/automation/hg-broker/ (unclear if used)

Things that look like they should be depreciated:
* https://hg.mozilla.org/hgcustom/library_overrides/
* https://hg.mozilla.org/hgcustom/hgmirror/ (has recent activity, might just be for testing?)

Comment 2

[Ed Morley [:emorley]]

gps, how would you recommend importing repos? Bug 875323 comment 4 is how I've done it in the past, but looking at your import of mqext, it seems as though yours was rebased (there's no break in the graph)?

Some of the extensions don't have licence headers - are we ok to add those post-import, or would you rather before?

Comment 3

[Gregory Szorc [:gps]]

I used `hg convert` to import the full history into the new repo. This required a little bit of hacking.

I generated a "file map" for hg convert by first writing the following Python script (could be made into a Perl or shell one-liner - meh):

import sys
line = sys.stdin.readline()
words = set(line.split())
for word in sorted(words):
    print('rename "%s" "%s/%s"' % (word, sys.argv[1], word))

Let's call it remap.py.

Now, I generate a "file map" for a repo:

$ hg log --template "{join(files, ' ')} " | python remap.py hgext/bzexport > filemap

Then, I run `hg convert`:

$ hg convert --filemap filemap . path/to/version-control-tools

This should apply the changes on top of tip in version-control-tools. No extra "unrelated head" or merging should be required. Just a nice linear repo history.

You may have to run `hg convert` a few times before you get things right. If you do, watch out for the .hg/shamap files, which is how hg convert keeps track of which commits have been processed and which ones to ignore on subsequent commit. Even if you strip version-control-tools, subsequent converts will no-op unless the .hg/shamap file is pruned or removed. I just removed it, since we don't performing ongoing conversions.

I don't think it matters when we change the license header, as long as it is present. I'd prefer to not rewrite history, so let's just add it after the fact.

FWIW, if a license header isn't present, it defaults to copyright of the author, all rights reserved. But, Mercurial extensions are bound by the GPL. I /think/ GPL trumps all rights reserved in this case?

Comment 4

[Hal Wine [:hwine] use NI!]

If we can avoid changing sha's on any already inhouse repos, that would be ideal. Old bugs will refer to existing shas for deployment -- it would be nice not to loose that history. I think such deploy requests have only been for repos currently under:
  https://hg.mozilla.org/hgcustom/

Comment 5

[Ed Morley [:emorley]]

Thank you for that!

However, when I tried it, I still ended up with detached commits.

[/c/src/vc-tools-test]$ hg id
f1e9d09714f3 tip

[/c/src/vc-tools-test]$ hg heads
[/c/src/version-control-tools]$ hg heads
changeset:   234:f1e9d09714f3
tag:         tip
user:        Ed Morley <snip>
date:        Tue Feb 18 12:14:21 2014 +0000
summary:     Bug 973843 - On Windows use the user profile root rather than documents directory for the bzexport cache; r=sfink

[/c/src/qbackout]$ cat filemap
rename ".hgignore" "hgext/qbackout/.hgignore"
rename "README" "hgext/qbackout/README"
rename "__init__.py" "hgext/qbackout/__init__.py"

[/c/src/qbackout]$ hg convert --filemap filemap . ../vc-tools-test
scanning source...
sorting...
converting...
14 initial version
13 add examples, pointers to more help
12 It turns out that the default patch format *still* isn't git, so file adds/removes were ignored.
11 Backwards compatibility with hg 1.7.5
10 Implement --apply and --broken arguments
9 pair of stoopid bugs that resulted in not getting bug numbers included
8 cannot index sets, stupid
7 Fix failure when reporting error
6 pyflakes fixes
5 Automatically reuse original commit message when appropriate
4 Patch order is backwards for --apply
3 Bug 894894 - Preserve the original author when using --apply; r=sfink
2 Mercurial likes to blame extensions that don't declare what they've been tested with. r=divineright
1 Add .hgignore; r=sfink
0 Create a README file

[/c/src/vcs3]$ hg heads
changeset:   249:f247c75fe875
tag:         tip
user:        'Nigel Babu <snip>'
date:        Tue Oct 22 20:57:09 2013 +0530
summary:     Create a README file

changeset:   234:f1e9d09714f3
user:        Ed Morley <snip>
date:        Tue Feb 18 12:14:21 2014 +0000
summary:     Bug 973843 - On Windows use the user profile root rather than documents directory for the bzexport cache; r=sfink

Is there an advantage over using a file-by-file filemap as opposed to the one line "rename . hgext/repoName" ?

That said, I managed to get linear history by forcing a rebase with the revisions specified (otherwise it bails out), eg:

[/c/src/vc-tools-test]$ hg rebase -s 6fca64e25b56 -d f1e9d09714f3
saved backup bundle to c:\src\vc-tools-test\.hg\strip-backup\6fca64e25b56-backup.hg

(where the argument to 's' is the earliest ancester of the imported changesets, and 'd' is the old tip of the version-control-tools repo)

[/c/src/vc-tools-test]$ hg heads
changeset:   249:14b649b4f94a
tag:         tip
user:        'Nigel Babu <snip>'
date:        Tue Oct 22 20:57:09 2013 +0530
summary:     Create a README file

Comment 6

[Gregory Szorc [:gps]]

I /may/ have also used a manually edited splice map to force the incoming changesets onto the tip. But using rebase -s should be fine.

We can't preserve SHA-1 when we move things to the new repo.

The old repos should still exist for reference purposes and we could potentially publish the SHA-1 mappings. And, assuming commit messages are well-formed, we should be able to find things by looking for bugs in commit messages. I'm not too worried about losing SHA-1s.

Comment 7

[Gregory Szorc [:gps]]

I tried using "." as a file map source and it didn't work for some reason. I may have been using things wrong.

Comment 8

[Gregory Szorc [:gps]]

Oh, if you update extensions to notify about the new source location (like we did with bzexport), the approach is flawed because `mach mercurial-setup` may abort due to a pulled legacy extension aborting. See bug 970233. I may update the old locations to merely print a non-fatal warning.

Comment 9

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #1)
> Further candidates...
> 
> Dev tools:

Also:
https://github.com/mozilla/git-bz-moz
https://github.com/mozilla/moz-git-tools

(In reply to Gregory Szorc [:gps] from comment #8)
> Oh, if you update extensions to notify about the new source location (like
> we did with bzexport), the approach is flawed because `mach mercurial-setup`
> may abort due to a pulled legacy extension aborting. See bug 970233. I may
> update the old locations to merely print a non-fatal warning.

Thank you - have updated the WIPs I have.

Comment 10

[Sean Rich [:srich]]

Updating to enhancement per 5/29 triage session with hwine and bkero

Comment 11

[Ed Morley [:emorley]]

Comment 0 was about two things - migrating tools into the new repo & also getting IT to use the tools from their new home. Most of the work I've done so far is for user tools - so I think we should morph this bug into migrating those & file another uncrufty one for the IT side. As such, moving this to repos & hooks component.

Comment 12

[Gregory Szorc [:gps]]

Calling this one done, even though there is still code elsewhere related to version control foo.