1352478 - vcssync: cleanup and adjust to prevent gecko-dev update delays

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[:dhouse]

The vcssync for all of the gecko-dev branches was timing out for nearly a week. It self-healed eventually, but we can try some adjustments and cleanup to the configuration to prevent the problem from repeating.

Comment 1

[:dhouse]

Plan:
1. simple cleanup
1a. "Trimming old, known-retired branches like b2g* and esr <45"
1b. remove gitmo targets
2. look at "Maybe let's bump the gexport timeout for a one-time conversion"

Comment 2

[:dhouse]

Attachment: bug1352478_vcssync-remove-b2g-gitmo.patch

removing b2g*, B2G tags, <esr45, and gitmo targets

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 8853479 [details] [diff] [review]
bug1352478_vcssync-remove-b2g-gitmo.patch

Removing gitmo will definitely be an improvement, and removing old branches is good as long as gecko-dev doesn't need to be rebuilt from scratch.  (And if it does, we can re-add the old branches at that point.)  I don't think this will save on current gexport timing, but if we rebuild the conversion dir from scratch, it should be smaller.

Comment 4

[hwine]

Comment on attachment 8853479 [details] [diff] [review]
bug1352478_vcssync-remove-b2g-gitmo.patch

Review of attachment 8853479 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm -- all those deletions++ :)

Comment 5

[Gregory Szorc [:gps]]

I'd like to give a periodic reminder that the hg-git we run is out of date and is lacking numerous performance improvements. The latest version of hg-git + Mercurial could be over a magnitude faster.

Comment 6

[Aki Sasaki (not active)]

The main thing we want to avoid is updating hg-git and finding later that reconverting from scratch gives us different shas, forcing us to either rebase all developers on gecko-dev, or putting in a hardcoded "use hg-git version X until this point in the conversion, and then switch over to version Y".

IIRC I had to cherry pick hg-git fixes because of divergent shas, so unless that's been changed, new hg-git will give divergent shas out of the box.

It might be worth trying:
- setting up a new workdir or conversion machine
- updating the configs for that process to not push to github
- trying the new hg-git version
- verifying the shas match
    - if so, switching over (push to github from this new conversion work dir, after disabling github pushes from the old one)
    - if not, trying again with the previous hg-git version, or cherry picking the appropriate patches for a new forked version.

Comment 7

[Gregory Szorc [:gps]]

Yup. aki's comment is why we're still on an old hg-git.

At some point, we should probably teach mozharness to convert up to commits X with an old version of hg-git then switch to a modern version for all new commits.

Comment 8

[Aki Sasaki (not active)]

(In reply to Gregory Szorc [:gps] from comment #7)
> At some point, we should probably teach mozharness to convert up to commits
> X with an old version of hg-git then switch to a modern version for all new
> commits.

We might have initial_beagle.py to get us to one watershed (with saved tarball), vcs_sync.py with old hg-git to get us to the second watershed (with saved tarball), and vcs_sync.py with new hg-git on top of the second extracted tarball.  Restoring from the second tarball is probably easier, as long as we don't lose it.  We could also record the latest revision per branch, so in the future we could potentially re-convert to the second watershed, once we hg strip everything after those revisions.

We may want some testing around what happens when we mark that watershed revision X, revision Y is on one branch (converted with old hg-git) but not another, and then it's cherry picked onto another branch (converted with new hg-git).  If revision Y has something odd in its commit message or email, we'll probably get a different git sha for it.  I'm not sure if that's a concern.

This is not a short or simple task, but can help speed up the conversion process.

Comment 9

[:dhouse]

I'll plan to revisit this in a couple of months (or if we see a delay again).

Notes:
I did some local tests of the latest hg-git (changeset:1008:4fa1812d1f24) compared with the 0.4.0 hg-git (not the moz patched version), git-remote-hg (https://github.com/felipec/git/wiki/git-remote-hg), and hg-fast-export (http://repo.or.cz/w/fast-export.git + https://git-scm.com/docs/git-fast-import).

I tested the performance of incremental updates the latest hg-git, and I only tested the full conversion in the others. Luckily, the initial conversion is fast on my 2015mbp laptop and the slowest of the pack, git-remote-hg (single threaded python with util process running git-fast-import that ran a total of 10 hours), took 28 hours.

TODO:
1. I had found that for a full conversion the majority of our time in hg-git is spent in iterating over the nodes separately from performing the work on them (from my basic qcachegrind profiling). I need to confirm if that is true, and if it is necessary, and if the same thing happens for the updates.
2. I need to also review the logs for when gecko-dev was behind for a week to see if it shows what the delays were. I do not remember the logs showing the problem or progress in completing a task if it was incrementally completing a conversion. I remember only that we saw timeouts on many of the conversions.

Comment 10

[Gregory Szorc [:gps]]

git-remote-hg and hg-fast-export should be avoided. Use git-cinnabar as replacements.

hg-git and git-cinnabar are the only generic tools I trust for converting Mercurial and Git repos to and from each other.

The version of hg-git we're using is doing some really clownshoesy things performance wise. I think the latest version will be 10-50x faster.

Comment 11

[:dhouse]

Thanks Greg! I'll close this out since we did some cleanup that can help avoid update delays. And I'll look at git-cinnabar when I come back to this.