Bug 799719 (vcs-sync)

tracker to retire legacy vcs2vcs

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
5 years ago
21 days ago

People

(Reporter: hwine, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2305] )

(Reporter)

Description

5 years ago
Some of the dependent tasks block b2g work, but not all. Therefore higher priorities are only set on a task by task basis.

At the moment, vcs2vcs is operating successfully on 2 machines in the dmz, and supporting connections to external mirrors on bitbucket.org & github.com, in addition to internal mirrors on git.m.o & hg.m.o

This bug is to track all the work to get it into full production, including the TOI to more folks.
(Reporter)

Updated

5 years ago
Depends on: 799724
(Reporter)

Updated

5 years ago
Depends on: 799730
(Reporter)

Updated

5 years ago
Depends on: 808040
(Reporter)

Updated

5 years ago
Depends on: 823333
(Reporter)

Updated

4 years ago
Depends on: 828140
(Reporter)

Updated

4 years ago
Depends on: 813738
(Reporter)

Updated

4 years ago
Depends on: 829025
(Reporter)

Updated

4 years ago
Depends on: 808129
(Reporter)

Updated

4 years ago
Depends on: 833590
(Reporter)

Updated

4 years ago
Depends on: 839595
After today's meetings, kicking this to aki as I think he's now taking point on this.
Assignee: nobody → aki

Comment 2

4 years ago
For posterity, here's the current to do list.
I will be filing bugs for various chunks shortly.
This will likely change as I dig into the details (code/configs), but as a general plan:

X take a look at docs + code + configs
X git repos to mirror https://bugzilla.mozilla.org/show_bug.cgi?id=846821

X need keep_updated (hwine)
X need .hgrc, config (hwine)
_ starter m-c cvs zip or something (hwine)
_ need staging box (hwine)
_  preferably multiple production-style boxes
_ port keep_updated, etc to mozharness
_  track down needed config files and draw relationships
_  rewrite config files to be complete, rather than all over the place
X  most likely will need to take advantage of multi-mozharness-configs to avoid massive duplication
_  draw up workflow to cover all scenarios:
*   all in one repo
*   single repo
*   hg->git
*   hg->hg
*   git->git
*   git->hg
*   repos that are converted and need to appear in different places
*   repos that are converted and need to have different branches appear in different places
(  steps needed (not all required before initial rollout):
_   create hg->git repo from scratch (git->hg too?)
_    create hgrc + .git/config from scratch/config ?
_    tests to verify?
_   create cvs history from scratch
_    tests to verify?
_   prepend cvs history to repo
_    tests to verify?
_   add additional repo(s)
_   mark branch(es) for addition
_    this should be target-repo specific
_    tests to verify?
_   push to the various target repos, with alternate configs
_  tests to cover known failure scenarios
_  figure out solutions for known-but-not-currently-fixed failure scenarios  
_ figure out way to run continuously without screen (cron-based; daemon based?)
_ figure out how to start automatically on boot
_ figure out how to be completely config driven, so checking in a new config (and potentially having IT update something) will get picked up automatically for existing or new conversions
_  tag, branch, or stay on tip of mozharness?  If the latter, we *need* mozharness CI
_ reporting: keep+port or replace current solution?
_ when ready, convert repo(s) side by side with current production bash conversions, and compare output
_  verify hg reset repo
_  verify large merges e.g. git gc
_ roll out replacement in stages?
_  probably first gecko.git
_  l10n + b2g-blocking repos
_  lower priority repos
_ find and replace old non-cvs-based github repos
_  (hwine) need a list of non-cvs repos, or how to get them
_  need to determine whether we have the existing cvs-based repos converted already
_   if not, extra work here
_  notify any owner of non-cvs-repo-forks about impending reset
_  schedule + do reset
_ create + test beagle
_  compare with github/mozilla/mozilla-central
_  roll out

Updated

4 years ago
Depends on: 799845

Updated

4 years ago
Depends on: 847640

Updated

4 years ago
Depends on: 847643

Updated

4 years ago
Depends on: 779294

Updated

4 years ago
Depends on: 749071

Updated

4 years ago
Depends on: 845908

Updated

4 years ago
Depends on: 847663

Updated

4 years ago
Depends on: 847670

Updated

4 years ago
Depends on: 847727
(Reporter)

Updated

4 years ago
Depends on: 848418
(Reporter)

Updated

4 years ago
Alias: vcs-sync
(Reporter)

Updated

4 years ago
Depends on: 851605
(Reporter)

Updated

4 years ago
Depends on: 851609
(Reporter)

Updated

4 years ago
Depends on: 851805
(Reporter)

Updated

4 years ago
Depends on: 854437
(Reporter)

Updated

4 years ago
Depends on: 855567
(Reporter)

Updated

4 years ago
Depends on: 857421
(Reporter)

Updated

4 years ago
Depends on: 857696
(Reporter)

Updated

4 years ago
Depends on: 857881

Updated

4 years ago
Depends on: 860038
(Reporter)

Updated

4 years ago
Depends on: 862438
(Reporter)

Updated

4 years ago
Depends on: 866260
(Assignee)

Updated

4 years ago
Product: mozilla.org → Release Engineering
(Reporter)

Updated

4 years ago
Depends on: 910790

Updated

4 years ago
Blocks: 918055

Comment 3

4 years ago
A lot of the blocker bugs are legacy-process-specific.
I think I'm going to keep their intentions in mind, but file new bugs for the new processes, and close the old blocker bugs when those processes are no longer in production.

Updated

4 years ago
Depends on: 927707

Updated

4 years ago
Depends on: 929080

Updated

4 years ago
Depends on: 929093

Updated

4 years ago
Depends on: 929334

Updated

4 years ago
Depends on: 946019
(Reporter)

Updated

4 years ago
Depends on: 949731

Updated

3 years ago
Depends on: 962853

Updated

3 years ago
Depends on: 962863

Updated

3 years ago
Depends on: 927199

Updated

3 years ago
Depends on: 962865

Updated

3 years ago
Depends on: 962866

Updated

3 years ago
Depends on: 962867

Updated

3 years ago
Depends on: 962869

Updated

3 years ago
Depends on: 962870

Updated

3 years ago
Summary: tracker for vcs2vcs as a production application → tracker to retire legacy vcs2vcs

Updated

3 years ago
Assignee: aki → pmoore

Updated

3 years ago
Depends on: 971270
Next steps are getting the build/* repos working in the new system (Bug 866260), getting the new mapper live in Elastic Bean Stalk, then getting gecko-dev and l10n repos working in the new vcs2vcs system.

Updated

3 years ago
Depends on: 972615
(Reporter)

Updated

3 years ago
Depends on: 973015
So over on bug 997743 I have noticed that closed named branches of a hg repository are still getting updated in the target repository (in our case github). This is somewhat confusing because it makes it very hard for users to see on which branches work actually happens. Have you thought about to not synchronize those closed named branches?

Updated

3 years ago
No longer blocks: 918055
Status: NEW → ASSIGNED
(In reply to Henrik Skupin (:whimboo) from comment #5)
> So over on bug 997743 I have noticed that closed named branches of a hg
> repository are still getting updated in the target repository (in our case
> github). This is somewhat confusing because it makes it very hard for users
> to see on which branches work actually happens. Have you thought about to
> not synchronize those closed named branches?

I think I spotted this once before, but I think it turned out that we weren't copying the closed branches, but when branches were closed, we weren't deleting them - so in effect they remained. So to the user it looks the same.

I'm not sure what the consensus will be on this one though, because I can imagine some people might wish to know what state closed branches are in. Do we need to open up this discussion, or does everyone think it is a no-brainer, and we should delete mirrored branches when they are closed on the source hg system?
Flags: needinfo?(hwine)
Flags: needinfo?(hskupin)
Flags: needinfo?(aki)

Comment 7

3 years ago
a) At the moment, inside of our current branch rules, we only have lists and regexes, not closed/open checks.
b) We are technically able to add those closed/open checks; however
c) I don't see the real problem here.  The closed branches should get updated at creation time, and then not populated again since there are no commits there.  So I don't see a strong benefit to putting time and energy into (b) right now.
Flags: needinfo?(aki)
(Reporter)

Comment 8

3 years ago
I agree with Aki. 

AND I'm not sure I understand the issue in comment 5 (bug 997743 comment 6)

No timestamps are modified in the mirroring or conversion process. So if a branch is really closed or otherwise inactive, there will be no recent activity showing. Typically, many web & cli views of repositories (both hg & git) present all branches/tags - but that's a UI decision, and doesn't reflect the contents of the repository (all those closed branches are still in the hg repo).

If, due to past practices, this is overly onerous for a given repository, then there may be a case made for "abandoning" the existing repository, and restarting with filtered view of history. That's completely outside the scope of the mirroring/conversion service, imo.
Flags: needinfo?(hwine)
When we can keep the history for open branches, and can get rid of all the closed ones by re-creating the repo, I'm fine.

A bigger problem is currently the mirroring of named branches, which does not happen at all! What I can see only the default branch is mirrored and that's all. 

https://github.com/mozilla/qa-mozmill-tests/tree/master (6 days)
https://github.com/mozilla/qa-mozmill-tests/tree/mozilla-aurora (2 months)
https://github.com/mozilla/qa-mozmill-tests/tree/mozilla-beta (2 months)

The 2 months is exactly this time when we got the mirroring setup.

So please make sure that all named branches are getting mirrored.
Flags: needinfo?(hskupin)

Comment 10

3 years ago
Mozmill is in legacy vcs sync, not new vcs sync currently.
So I assume I have to file a new bug for getting the QA related repositories moved over?
(Reporter)

Comment 12

3 years ago
(In reply to Henrik Skupin (:whimboo) from bug 799719 comment #9)
> When we can keep the history for open branches, and can get rid of all the
> closed ones by re-creating the repo, I'm fine.

Please file a new bug for this -- you'll have to say how you want
everything modified.

Remember that "only mirror open branches" is not a use case we currently support. If you want that on the existing repository, please open an enhancement request.

> A bigger problem is currently the mirroring of named branches, which does
> not happen at all! What I can see only the default branch is mirrored and
> that's all. 

Opened bug 1035934 for this issue

(In reply to Henrik Skupin (:whimboo) from comment #11)
> So I assume I have to file a new bug for getting the QA related repositories
> moved over?

No, when the new system is ready, we will migrate all jobs off the legacy system and shut it down. :)
(In reply to Hal Wine [:hwine] (use needinfo) from comment #12)
> (In reply to Henrik Skupin (:whimboo) from bug 799719 comment #9)
> > When we can keep the history for open branches, and can get rid of all the
> > closed ones by re-creating the repo, I'm fine.
> 
> Please file a new bug for this -- you'll have to say how you want
> everything modified.

Done as bug 1036622.

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2301]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2301] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2305]
Assignee: pmoore → nobody
(Reporter)

Comment 14

8 months ago
all done in another bug, closing
Status: ASSIGNED → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
(Assignee)

Updated

21 days ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.