910040 - decommission gitmirror.m.o

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Because of bug#908211, we discovered that some systems were using the unsupported gitmirror.m.o instead of the newer, supported, git.m.o. 

At this point, all ATeam systems are no longer using gitmirror.m.o. RelEng systems were never using gitmirror.m.o. As far as we can tell, no-one is using gitmirror.m.o, and it is adding confusion, so the entire gitmirror.m.o machine should be deleted. We believe all repos on gitmirror.m.o are already also on git.m.o, so nothing unique is on gitmirror.m.o to be saved... of course, this should be verified.

Filing in RelEng:Hooks and Repos first, so we can out of an abundance of caution, first start by deleting all the repos we *know* are not needed? Once we have all known repos deleted, we can look for any remaining repos (and their owners) prior to deleting. 

Once all repos are deleted from gitmirror.m.o, we can reassign this bug over to IT for final deletion.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

laura: any repos you care about on gitmirror.m.o? If yes, can we get them moved to git.m.o?

Comment 2

[Shyam Mani [:fox2mike]]

CC'ing Jeremy since he set this up and Jake to make sure no webdev projects depends on this.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Web QA are using gitmirror.m.o for all of their testcase repositories. This has proven to be faster and more reliable than using github directly. I've had a look through the repositories on gitmirror.m.o and listed the ones that I believe are still in current use below.

mozilla/Addon-Tests
mozilla/Affiliates-Tests
mozilla/FlightDeck-selenium
mozilla/Garmr
mozilla/Socorro-Tests
mozilla/bidpom
mozilla/bouncer-tests
mozilla/gaia-ui-tests
mozilla/marketplace-tests
mozilla/mcom-tests
mozilla/mdn-tests
mozilla/mozillians-tests
mozilla/moztrap-tests
mozilla/mozwebqa-test-templates
mozilla/qmo-tests
mozilla/snippets-tests
mozilla/sumo-tests
mozilla/wiki-tests

If these fit the requirements to be moved to git.m.o then we should do that, otherwise we should revert back to using github directly before shutting down this mirror.

CC'ing Stephen Donner for any further comments from Web QA

Comment 4

[u429623]

(In reply to John O'Duinn [:joduinn] from comment #0)
> Because of bug#908211, we discovered that some systems were using the
> unsupported gitmirror.m.o instead of the newer, supported, git.m.o.

I would phrase this quite differently. gitmirror.m.o & git.m.o are 2 different approaches to handling git repositories used by internal systems at Mozilla.

git.m.o & gitmirror.m.o have very different service levels, both for initial setup and for ongoing operations. It is not clear to me that "one size fits all" for Mozilla.

For any repositories that are accessed from within the RelEng build farm, gitmirror.m.o is likely not robust enough for repositories already on git.m.o. That was the issue in bug 910040 and has been addressed.

For all other systems, it depends on operational needs as to which approach better serves. Each group should make that determination. Some service differences I'll highlight for the use case of mirroring a repository from github:

 +=================+=================+==============+
 | service         |   gitmirror.m.o | git.m.o      |
 +=================+=================+==============+
 | add repo        |   self-serve    | bug file,    |
 |                 |                 | ~2 day turn  |
 +-----------------+-----------------+--------------+
 | update latency  |   event driven  | polling      |
 | from github     |   (~1 min)      | (~15 min)    |
 +-----------------+-----------------+--------------+
 | monitored       |   no            | yes          |
 +-----------------+-----------------+--------------+
 | 24x7 page       |   no            | yes          |
 +=================+=================+==============+

> We believe all repos on gitmirror.m.o are already
> also on git.m.o, so nothing unique is on gitmirror.m.o to be saved... of
> course, this should be verified.

This is not true -- at the moment, only repositories required for operation of the RelEng build farm are on git.m.o (primarily those that support b2g builds and partner commitments).

As noted in comment 3, there are many other repositories on gitmirror. The counts are:
 98 served by gitmirror.m.o
 79 from Mozilla's github accounts
  1 also served by git.m.o (gaia)

This suggests there are many users not yet involved in the discussion.

Comment 5

[Kumar McMillan [:kumar]]

I checked ci.mozilla.org and I don't see any AMO/Marketplace builds that are still using gitmirror.m.o. Has anyone looked at the gitmirror.m.o access logs? I think some deployments still use it. oremj will know.

PS. If any git.m.o repos start to die should we force feed them? Here all week.

Comment 6

[Fred Wenzel [:wenzel]]

One thing that gitmirror tried to solve but fell short on due to the manual setup needed every time you create a repo is: Local mirroring of github repos in case github goes down or something evil happens. This is not the same as a backup, but I'd like to see it tied into a backup anyway.

gitmirror was meant to solve these problems (and largely did, for the repos opting in), but it's not clear to me how git.m.o is doing in comparison. In particular, how it git.m.o advancing the safety (from data loss and down time) of our currently 634 github/mozilla repositories?

Comment 7

[Les Orchard [:lorchard]]

FWIW, the repos on gitmirror I care about are:

* mozilla/kuma
* mozilla/kuma-lib
* mozilla/kumascript

Please excuse my random stumbling into this bug, but I didn't know there was a git.mozilla.org. Are there docs somewhere I can read to learn more?

IIRC, webdev set up gitmirror long ago as a local cache and an insurance policy for github. Since it works in reaction to web hooks, changes get mirrored nearly instantly. That's really nice for when github goes down, and you wish you had that commit someone pushed a few minutes ago.

That said, though, I honestly can't remember the last time I pulled code from gitmirror. We had a notion to deploy to MDN production from there, but have never made it a priority.

Comment 8

[Paul [:pmac] McLanahan]

Bedrock is in much the same situation as kuma it seems. We have gitmirror setup for mirroring and I like knowing it's there for when github goes down (which it sometimes does). I have designs on moving our deployment to use gitmirror.m.o instead of github directly, but have not yet done so. If git.m.o also supports webhook based mirroring of github repos then I'm happy to move over there, but if not then I think gitmirror has quite a bit of value and is very different from just simply hosting git repos.

Comment 9

[Osmose [:osmose, :mkelly]]

Reporting in for:

* mozilla/affiliates
* mozilla/captain
* mozilla/firefox-flicks
* mozilla/geodude
* mozilla/shove
* mozilla/snippets-service

Otherwise echoing the sentiments above. Webhook support is important enough that I think we should keep the mirror around if git.m.o is not meant to support that.

Comment 10

[Shyam Mani [:fox2mike]]

Speaking from a purely IT perspective, if we can port this webhook over to git.mozilla.org, that'd be great. We don't want to manage two different systems if we can avoid doing that.

Comment 11

[Jake Maul [:jakem]]

Clearing my own NEEDINFO... asked and answered, webdev uses it rather significantly, so it seems. I didn't even realize this system existed... all our deploys pull from github.com, not gitmirror.mozilla.org, and that's about the extent of our involvement in this particular realm.

At a glance I don't know of any reason this couldn't be ported over to git.mozilla.org, but I'm really not qualified to speak on that. Comment 4 and comment 8 give me pause. :bkero might have better insight...

Comment 12

[Michal Purzynski [:michal`] (use NEEDINFO)]

When we do remove this system, let's also remember about the flow from untrust zone (read: Internet) -> dmz. The flow is on the fw1.phx1.

Comment 13

[Jeremy Orem [:oremj]]

(In reply to Shyam Mani [:fox2mike] from comment #2)
> CC'ing Jeremy since he set this up and Jake to make sure no webdev projects
> depends on this.

I'm not using this for anything anymore.

Comment 14

[Ben Kero [:bkero]]

Having repos mirror from github and update on an event base is certainly something that could be done on git.mozilla.org.

The caveat is that a hook would need to be placed on the github repos to tell git.mozilla.org to pull. This likely already exists and is how gitmirror.mozilla.org is currently accomplishing this.

Self-service creation of repositories is also available, although self-service associating with github repositories would be more difficult to accomplish.

If moving these repositories over is something people are interested in, could I get a list of ones which are currently in use?

Comment 15

[Osmose [:osmose, :mkelly]]

Unless you meant "which ones are actively in use", the current list of repos mirrored on gitmirror can be viewed at http://gitmirror.mozilla.org/.

Comment 16

[Ben Kero [:bkero]]

I was assuming there are some (possibly a majority) that are still present on gitmirror.m.o, but which are not actively in use and shouldn't be moved over. I hope it's possible to obtain this list, otherwise we'll have a bunch of (essentially) dead repositories being moved over, cluttering an otherwise clean system.

Comment 17

[Paul [:pmac] McLanahan]

(In reply to Ben Kero [:bkero] from comment #14) 
> The caveat is that a hook would need to be placed on the github repos to
> tell git.mozilla.org to pull. This likely already exists and is how
> gitmirror.mozilla.org is currently accomplishing this.

Yes. People set up a web-hook in the github admin per repository, and github posts some data to the url when it receives a push. This works quite well.

> Self-service creation of repositories is also available, although
> self-service associating with github repositories would be more difficult to
> accomplish.

gitmirror.m.o already accomplishes this. It's why it's nice. All you need to do to have a mirror is paste a web-hook url in your project's settings. The process is documented on the intranet:

https://intranet.mozilla.org/Git

This will create the repo if it doesn't exist, and update it to the current state of the github repo. The code for doing this is here:

https://github.com/jbalogh/reflecto

> If moving these repositories over is something people are interested in,
> could I get a list of ones which are currently in use?

As long as the setup process for the new system is similar to the old people can just change their web-hook URLs and trigger the post. You won't need to move any over since people can setup the ones they need themselves. Since these are all just mirrors, there's no unique data on gitmirror.m.o, so no manual move of the repos is required. As long as the deprecation plan and new setup docs are widely publicized well before the sunset date, the pain when gitmirror.m.o is shut down should be minimal.

Comment 18

[Ben Kero [:bkero]]

I've read and grok the reflecto code. This should be easily movable to git.mozilla.org.

Would you prefer that I keep the mirrors on a separate apache vhost, and not interact with gitolite at all, or display the mirrored repositories in the same interface as the other repositories on the box?

I'll comment on the bug again when I've written puppet code to deploy this.

Comment 19

[Paul [:pmac] McLanahan]

(In reply to Ben Kero [:bkero] from comment #18)
> I've read and grok the reflecto code. This should be easily movable to
> git.mozilla.org.

Fantastic news! Thanks :bkero!

> Would you prefer that I keep the mirrors on a separate apache vhost, and not
> interact with gitolite at all, or display the mirrored repositories in the
> same interface as the other repositories on the box?

I've no strong opinion on this. Since they're mirrors it's probably safest to separate them, but I think the more active users of git.m.o (tee hee) should decide this one.

Comment 20

[Fred Wenzel [:wenzel]]

(In reply to Paul McLanahan [:pmac] from comment #19)
> (In reply to Ben Kero [:bkero] from comment #18)
> > I've read and grok the reflecto code. This should be easily movable to
> > git.mozilla.org.
> 
> Fantastic news! Thanks :bkero!

Not trying to scope creep, but will we be able to turn this into a git mirror for *all* repositories under gh/mozilla then? The current opt-in method leads to almost no repos actually being mirrored in-house.

Comment 21

[Ben Kero [:bkero]]

That really depends on if github can have an organization-wide repository post-commit hook, and how how much space all the github.com/mozilla repositories takes.

Comment 22

[u429623]

(In reply to Fred Wenzel [:wenzel] from comment #20)
> (In reply to Paul McLanahan [:pmac] from comment #19)
> > (In reply to Ben Kero [:bkero] from comment #18)
> > > I've read and grok the reflecto code. This should be easily movable to
> > > git.mozilla.org.
> > 
> > Fantastic news! Thanks :bkero!
> 
> Not trying to scope creep, but will we be able to turn this into a git
> mirror for *all* repositories under gh/mozilla then? The current opt-in
> method leads to almost no repos actually being mirrored in-house.

That's is a huge scope creep -- we wouldn't want to commit to this until we're sure that git.m.o can handle the load for b2g operations.

And, since there aren't any account wide settings like that, tooling would need to be written to "auto opt in" via polling the github API.

Is there a specific business need you're seeking to address? Or just "in house copies are good"? (This later could be met at a lower service level than git.m.o.)

Comment 23

[Greg Cox [:gcox]]

The gitmirror box is on expired hardware, and the decom appears stalled, so we're going to P2V it to get off the hardware.

I offer up Monday 21 July at ~0600 PDT for about 1 hour downtime, with "silence=consent" as an assumption.  If you want to NOGO that time, bug 1037435.

Comment 24

[Fred Wenzel [:wenzel]]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #22)
> Is there a specific business need you're seeking to address? Or just "in
> house copies are good"? (This later could be met at a lower service level
> than git.m.o.)

The business need is backup. That doesn't have to be an accessible git mirror (thus, to answer your question, no it does not need the same service level as git.m.o, at all).

Git repos in general, and github with its limited ACLs, are particularly susceptible to accidental or malicious damage. I hope it's not unreasonable for me to hope we can somehow back up our master copies of code repositories we rely on on a daily basis.

Comment 25

[Mike Hoye [:mhoye]]

There's a discussion ongoing in dev-planning about having list of the large and growing number of github-hosted projects somewhere that's somewhat automated, maintainable and discoverable.

Gitmirror.mozilla.org - with the corresponding "add your repo" intranet page - seems to meet a lot of people's needs there, and could be somewhat-trivially extended to meet the rest. Though our flagship projects should be owned and run in-house as noted above, I think that there's a lot of value to the notion of keeping gitmirror.mozilla.org alive as a discoverable, reasonably well-automated index of all the dozens of other "second-tier" projects that so many Mozillians keep on Github.

I'm CC'ing Ehsan and Ralph on this bug for having initiated the conversation and expressed interest in updating the code backing the intranet page, respectively.

Comment 26

[Gregory Szorc [:gps]]

For the longest time I've wanted to create a database holding info on all version control repositories related to Mozilla. This includes repos from hg.mozilla.org, git.mozilla.org, bitbucket, and github.

Said database could contain repository configuration as well (what extensions and hooks to use on each repo, etc). This would make repository management for hg.mozilla.org and git.mozilla.org more self-service and more robust. Our current solution is a mess of haphazard one-offs.

We'll likely need to build a database to hold "pushlog" data for the bundle-based Try. We could throw that in the database as well.

I would also like to periodically poll the GitHub API and have "Mozilla owned" repos periodically "backed up" to a Mozilla property. You could even extend this to having pull request and issue data synced down as well. This is what Facebook does. See https://www.youtube.com/watch?v=fzL6Zoy_ndk&list=PLb0IAmt7-GS1hdDcokpVp1MBk-IaeaSgP

I would also like to build a service that indexes in-flight code reviews and preemptively notifies people when multiple people are working on the same files or when their patch has been bit rotted by someone else.

I would like Bugzilla to be automatically updated when pushes occur. No more manually posting commit URLs. This service is easy to build if you can query a database for all changes since X.

The autoland service could also be built on top of this database.

I want to build this database. I want to create a web service that exposes all the information. I want to have an HTML interface to this data so we can do dashboards. Let's call it https://code.mozilla.org/. I think Mozilla needs this to move faster and to make better data-driven decisions.

I've already built something that does the repo management and indexing pieces. See http://gregoryszorc.com/blog/2014/01/21/aggregating-version-control-info-at-mozilla/. With a little feature work, it would be awesome.

Unfortunately, I have commitments for the next few months and don't have time to build this. But I'd love to see this vision become reality. I think Laura Thomson's group would own it. I've already said some things about how I think building a database for the hosted repository data her merits for just repository management purposes. It's not a huge transition to extend that database and web service to do the awesomeness I described above. Expose the data and awesome features will come.

Comment 27

[Ralph Giles (:rillian)]

I think it's fine to integrate external repo tracking and backup into git.mozilla.org or other, newer services. In the meantime, I think gitmirror is serving a useful function, and we should continue to maintain it.

Comment 28

[u429623]

For folks following along, we're going to start turning down services for repos already available on git.mozilla.org.

Those represent a duplication of effort! b2g ones occasionally cause excessive load on that machine causes loads and alerts which interrupt folks.

Comment 29

[u429623]

First notice of stopping some repositories posted to dev lists:

-------- Forwarded Message --------
Subject: 	FYI - gitmirror.m.o duplicate repositories will be removed Feb 16
Date: 	Wed, 04 Feb 2015 13:49:56 -0800
From: 	Hal Wine 
To: 	[e-mail], [e-mail]
CC: 	[e-mail], Developer Services 

tl;dr: if you reference git://gitmirror.mozilla.org/mozilla-b2g/gaia, 
please change to git://git.mozilla.org/releases/gaia.git

gitmirror.mozilla.org gets overloaded from time to time. To reduce load, 
we will be removing dead repositories and repositories already mirrored 
to git.mozilla.org. (see https://bugzil.la/910040)

For this first pass on Feb 16, that means:
 - mozilla-b2g/gaia will be removed, as it exists at 
git.mozilla.org/releases/gaia.git
 - mozilla/mozilla-central will be removed, as that repo was deprecated 
long ago
    https://github.com/mozilla/mozilla-central

--Hal

Comment 30

[Kendall Libby [:fubar] (he/him)]

mozilla-b2g/gaia and mozilla/mozilla-central have been removed. we forgot to come back to this on the 16th, but gitmirror's been whinging about swap today, so away they go.

Comment 31

[Greg Cox [:gcox]]

The web logs on the gitmirror box are a snoozefest.  I know there's a dependency bug open, but this seems unused at this point in time.  Someone with some authority here might want to consider just having it switched off and see if anyone notices.

Comment 32

[Fred Wenzel [:wenzel]]

Considering this bug has been open for 3 years, I'd suggest go for it ;)

Comment 33

[u429623]

If the logs have been a snooze fest, let's:
 - cab it for moc awareness
 - shut it down with no redirect (500 errors likely to be noisier)
 - be ready to turn it back on (temporarily) for at least a month
 - run a lottery -- winner gets to rm the image.

Comment 34

[Greg Cox [:gcox]]

Filed CAB, aiming for a takedown on 2016-10-10 but not destroying it for ~ 1 month.

Comment 35

[Greg Cox [:gcox]]

VM powered off around 2016-10-12 19:50 UTC.  Haven't exploded the trees or had anyone screaming so far.
Slated the full decom to begin in about a month.

Comment 36

[Greg Cox [:gcox]]

DNS cleanup:
63.245.214.190 <-> gitmirror.pub.phx1
10.8.74.43     <-> gitmirror.dmz.phx1
gitmirror.mozilla.org CNAME gitmirror.pub.phx1

Inventory and puppetdashboard cleaned.  no RHN entry.
Puppet manifest and module cleared out, 972e990737b5f6fd0fd8ef2370de0d19b219a2ce
Opened netops bug for cleanup there.  Will open a webops bug, too.

VM deleted, closing this out.