799719 - (vcs-sync) tracker to retire legacy vcs2vcs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Hal Wine [:hwine] use NI!]

Some of the dependent tasks block b2g work, but not all. Therefore higher priorities are only set on a task by task basis.

At the moment, vcs2vcs is operating successfully on 2 machines in the dmz, and supporting connections to external mirrors on bitbucket.org & github.com, in addition to internal mirrors on git.m.o & hg.m.o

This bug is to track all the work to get it into full production, including the TOI to more folks.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

After today's meetings, kicking this to aki as I think he's now taking point on this.

Comment 2

[Aki Sasaki (not active)]

For posterity, here's the current to do list.
I will be filing bugs for various chunks shortly.
This will likely change as I dig into the details (code/configs), but as a general plan:

X take a look at docs + code + configs
X git repos to mirror https://bugzilla.mozilla.org/show_bug.cgi?id=846821

X need keep_updated (hwine)
X need .hgrc, config (hwine)
_ starter m-c cvs zip or something (hwine)
_ need staging box (hwine)
_  preferably multiple production-style boxes
_ port keep_updated, etc to mozharness
_  track down needed config files and draw relationships
_  rewrite config files to be complete, rather than all over the place
X  most likely will need to take advantage of multi-mozharness-configs to avoid massive duplication
_  draw up workflow to cover all scenarios:
*   all in one repo
*   single repo
*   hg->git
*   hg->hg
*   git->git
*   git->hg
*   repos that are converted and need to appear in different places
*   repos that are converted and need to have different branches appear in different places
(  steps needed (not all required before initial rollout):
_   create hg->git repo from scratch (git->hg too?)
_    create hgrc + .git/config from scratch/config ?
_    tests to verify?
_   create cvs history from scratch
_    tests to verify?
_   prepend cvs history to repo
_    tests to verify?
_   add additional repo(s)
_   mark branch(es) for addition
_    this should be target-repo specific
_    tests to verify?
_   push to the various target repos, with alternate configs
_  tests to cover known failure scenarios
_  figure out solutions for known-but-not-currently-fixed failure scenarios  
_ figure out way to run continuously without screen (cron-based; daemon based?)
_ figure out how to start automatically on boot
_ figure out how to be completely config driven, so checking in a new config (and potentially having IT update something) will get picked up automatically for existing or new conversions
_  tag, branch, or stay on tip of mozharness?  If the latter, we *need* mozharness CI
_ reporting: keep+port or replace current solution?
_ when ready, convert repo(s) side by side with current production bash conversions, and compare output
_  verify hg reset repo
_  verify large merges e.g. git gc
_ roll out replacement in stages?
_  probably first gecko.git
_  l10n + b2g-blocking repos
_  lower priority repos
_ find and replace old non-cvs-based github repos
_  (hwine) need a list of non-cvs repos, or how to get them
_  need to determine whether we have the existing cvs-based repos converted already
_   if not, extra work here
_  notify any owner of non-cvs-repo-forks about impending reset
_  schedule + do reset
_ create + test beagle
_  compare with github/mozilla/mozilla-central
_  roll out

Comment 3

[Aki Sasaki (not active)]

A lot of the blocker bugs are legacy-process-specific.
I think I'm going to keep their intentions in mind, but file new bugs for the new processes, and close the old blocker bugs when those processes are no longer in production.

Comment 4

[Pete Moore [:pmoore][:pete]]

Next steps are getting the build/* repos working in the new system (Bug 866260), getting the new mapper live in Elastic Bean Stalk, then getting gecko-dev and l10n repos working in the new vcs2vcs system.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So over on bug 997743 I have noticed that closed named branches of a hg repository are still getting updated in the target repository (in our case github). This is somewhat confusing because it makes it very hard for users to see on which branches work actually happens. Have you thought about to not synchronize those closed named branches?

Comment 6

[Pete Moore [:pmoore][:pete]]

(In reply to Henrik Skupin (:whimboo) from comment #5)
> So over on bug 997743 I have noticed that closed named branches of a hg
> repository are still getting updated in the target repository (in our case
> github). This is somewhat confusing because it makes it very hard for users
> to see on which branches work actually happens. Have you thought about to
> not synchronize those closed named branches?

I think I spotted this once before, but I think it turned out that we weren't copying the closed branches, but when branches were closed, we weren't deleting them - so in effect they remained. So to the user it looks the same.

I'm not sure what the consensus will be on this one though, because I can imagine some people might wish to know what state closed branches are in. Do we need to open up this discussion, or does everyone think it is a no-brainer, and we should delete mirrored branches when they are closed on the source hg system?

Comment 7

[Aki Sasaki (not active)]

a) At the moment, inside of our current branch rules, we only have lists and regexes, not closed/open checks.
b) We are technically able to add those closed/open checks; however
c) I don't see the real problem here.  The closed branches should get updated at creation time, and then not populated again since there are no commits there.  So I don't see a strong benefit to putting time and energy into (b) right now.

Comment 8

[Hal Wine [:hwine] use NI!]

I agree with Aki. 

AND I'm not sure I understand the issue in comment 5 (bug 997743 comment 6)

No timestamps are modified in the mirroring or conversion process. So if a branch is really closed or otherwise inactive, there will be no recent activity showing. Typically, many web & cli views of repositories (both hg & git) present all branches/tags - but that's a UI decision, and doesn't reflect the contents of the repository (all those closed branches are still in the hg repo).

If, due to past practices, this is overly onerous for a given repository, then there may be a case made for "abandoning" the existing repository, and restarting with filtered view of history. That's completely outside the scope of the mirroring/conversion service, imo.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

When we can keep the history for open branches, and can get rid of all the closed ones by re-creating the repo, I'm fine.

A bigger problem is currently the mirroring of named branches, which does not happen at all! What I can see only the default branch is mirrored and that's all. 

https://github.com/mozilla/qa-mozmill-tests/tree/master (6 days)
https://github.com/mozilla/qa-mozmill-tests/tree/mozilla-aurora (2 months)
https://github.com/mozilla/qa-mozmill-tests/tree/mozilla-beta (2 months)

The 2 months is exactly this time when we got the mirroring setup.

So please make sure that all named branches are getting mirrored.

Comment 10

[Aki Sasaki (not active)]

Mozmill is in legacy vcs sync, not new vcs sync currently.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So I assume I have to file a new bug for getting the QA related repositories moved over?

Comment 12

[Hal Wine [:hwine] use NI!]

(In reply to Henrik Skupin (:whimboo) from bug 799719 comment #9)
> When we can keep the history for open branches, and can get rid of all the
> closed ones by re-creating the repo, I'm fine.

Please file a new bug for this -- you'll have to say how you want
everything modified.

Remember that "only mirror open branches" is not a use case we currently support. If you want that on the existing repository, please open an enhancement request.

> A bigger problem is currently the mirroring of named branches, which does
> not happen at all! What I can see only the default branch is mirrored and
> that's all. 

Opened bug 1035934 for this issue

(In reply to Henrik Skupin (:whimboo) from comment #11)
> So I assume I have to file a new bug for getting the QA related repositories
> moved over?

No, when the new system is ready, we will migrate all jobs off the legacy system and shut it down. :)

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #12)
> (In reply to Henrik Skupin (:whimboo) from bug 799719 comment #9)
> > When we can keep the history for open branches, and can get rid of all the
> > closed ones by re-creating the repo, I'm fine.
> 
> Please file a new bug for this -- you'll have to say how you want
> everything modified.

Done as bug 1036622.

Comment 14

[Hal Wine [:hwine] use NI!]

all done in another bug, closing