943132 - Determine why github mozilla/mozilla-central has been removed from service by github

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Hal Wine [:hwine] use NI!]

The github repository at:
    https://github.com/mozilla/mozilla-central/
and all of its github forks have been disabled.

The information received from [e-mail] is:
    Sorry about that - our ops and systems folks are taking a look now,
    but it looks like the size and particularly unique nature of
    mozilla/mozilla-central are causing fileserver issues for us at the
    moment. The team is looking at options to resolve this and we'll
    follow up as soon as we have more information for you.

There are alternatives for developers listed in the newsgroup at:
    https://groups.google.com/d/msg/mozilla.dev.planning/rkyKnCMQmYg/YykH5Ad1ZzEJ


This bug is for capturing the root cause as reported by github, and
ensuring that the production replacement repositories will not run into
the same issue.

Comment 1

[Kevin Grandon :kgrandon]

I reached out to them before I knew others had. Received this response so far, "Sorry about that - our ops and systems folks are taking a look now, but it looks like the repository is affecting fileserver performance at the moment. The checks we have in place have the repository disabled but the team is looking at different options to resolve this. We'll follow up when we have more information for you. "

Comment 2

[Ben Bucksch (:BenB)]

It may be that due to the dev.planning annoucement, more people were fetching the old repo, thus creating higher stress on the server.

Comment 3

[Gregory Szorc [:gps]]

GitHub has killed repos without notice before. Ehsan's hg-git mappings repo comes to mind. Granted, that repo was special in that it contained very large deltas to a single file and effectively constitutes a version control stress test.

Comment 4

[Octoploid]

https://github.com/mozilla/releases-mozilla-central.git seems to work fine.

Comment 5

[Hal Wine [:hwine] use NI!]

(In reply to Octoploid from comment #4)
> https://github.com/mozilla/releases-mozilla-central.git seems to work fine.

Please do not use that repo -- while stable, it does not contain the official shas and will be removed shortly.

The better repo is https://github.com/mozilla/gecko-dev for m-c content. See the dev.planning post from Ehsan for more details.

Comment 6

[Hal Wine [:hwine] use NI!]

Update: github continues to work on this. Last status from them:

On 2013-11-25 20:59 , Robert Sese (GitHub Staff) wrote:
> Hi Hal,
>
> Just wanted to follow up with you again and let you know that the team has a good looking potential fix in place that's being tested.

Comment 7

[Hal Wine [:hwine] use NI!]

Attachment: github_email, without email addresses

For posterity, attaching the original email notification to account admins. (email addresses removed)

Comment 8

[Hal Wine [:hwine] use NI!]

And the repository is back online again.

Preliminary information is that someone did a merge that caused the file system to thrash, affecting performance of the entire site.

Leaving this bug open until we have more details:
 - was the merge to mozilla/mozilla-central or to a fork
 - did they back out the merge or somehow allow it to complete

Comment 9

[John Schoenick [:johns]]

This sounds like someone pushed the old shas into the new shas in some repo, which would be a ~1.5million commit push.

Someone doing something github doesn't handle well on a fork taking down the original repository sounds very bad...

Comment 10

[(no longer active)]

It's also interesting to see why the said merge affected only mozilla-central and not gecko-dev...

Comment 11

[(no longer active)]

(In reply to John Schoenick [:johns] from comment #9)
> This sounds like someone pushed the old shas into the new shas in some repo,
> which would be a ~1.5million commit push.
> 
> Someone doing something github doesn't handle well on a fork taking down the
> original repository sounds very bad...

Hmm, yeah if someone did this in one of their forks, then this sounds like a huge problem, and it will surely affect gecko-dev sooner or later :(

Comment 12

[(no longer active)]

Also, thanks Hal for working with them to resolve this issue!

Comment 13

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Can we give them some SHAs from the two repoos to blacklist somehow in the relevant repos/forks?  (If that makes sense; in other words -- take a hash from repo A and blacklist it from pushing in B and all forks of B, and vice versa.)

Comment 14

[Ben Bucksch (:BenB)]

More generally, I'd block pushes that are over a certain size, both in number of commits and in file size. That would also cover people who accidentially commit build artifacts, commit videos etc. (I've seen it all, happen in reality, even in small teams.)
I'd recommend that git.mozilla.org does that, too.

Comment 15

[(no longer active)]

(In reply to comment #13)
> Can we give them some SHAs from the two repoos to blacklist somehow in the
> relevant repos/forks?  (If that makes sense; in other words -- take a hash from
> repo A and blacklist it from pushing in B and all forks of B, and vice versa.)

I have explained the situation to one of the Github staff that I have been in touch with and have asked them what we can do to make sure this doesn't happen in the future...

Comment 16

[Gregory Szorc [:gps]]

It sure would be nice if GitHub offered support for pre-push/commit hooks. I won't hold my breath on us getting the control we need out of hooks however (it's a security and scaling problem for GitHub and all other repository hosting service providers).

Comment 17

[Hal Wine [:hwine] use NI!]

I believe we're done here - we had a root cause (a fork was "abused" with excessive commits, and that blocked the root repo as well). Github made changes to ensure any future occurrence only blocks the affected fork repo, and not the rest of the fork-network. And we've moved from this repo to using mozilla/gecko-dev & mozilla/gecko-projects on github.

If anyone feels different, please open a new bug or change the summary on this one.