990311 - git.mozilla.org is broken | error: The requested URL returned error: 500 while accessing https://git.mozilla.org/external/caf/platform/system/security/info/refs

Status: RESOLVED DUPLICATE of bug 985864

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

git.mozilla.org is having issues which result in all b2g builds failing: 
https://tbpl.mozilla.org/php/getParsedLog.php?id=37031776&tree=Mozilla-Central#error0

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Trees closed at 5:12PM Pacific time.

Comment 2

[Hal Wine [:hwine] use NI!]

Issue is OOM on git.m.o due to high load coming in through releng firewall. Netops helping uncover source.

Comment 3

[Hal Wine [:hwine] use NI!]

Event appears to be over. Waiting for jobs to finish before trees will be reopened. RFO will happen in bug 990344

Comment 4

[Kevin Grandon :kgrandon]

Adding myself to the CC here. Hoping this is resolved, but if it's not we should close gaia as well probably.

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Started happening again. Trees closed at 2014-04-01T19:08:03

Comment 6

[Kendall Libby [:fubar] (he/him)]

git1 was wedged good. all memory *and* swap used, so it was thrashing itself to death. killed a half dozen or so of the largest memory consumers and it self corrected in fairly short order.

normally we catch it before swap gets used up, but at that point there's not much to do except shoot things in the head. I think ben tends to try and SIGSTOP a bunch and slowly unfreeze 'em when things start to recover, in "normal" cases.

Comment 7

[Hal Wine [:hwine] use NI!]

(In reply to Wes Kocher (:KWierso) from comment #5)
> Started happening again. Trees closed at 2014-04-01T19:08:03

reopened at 2014-04-01T19:54:05 (PT)

:catlee landed http://hg.mozilla.org/build/mozharness/rev/a4dcb8b815a9 which hopefully will prevent a "pile on" if this occurs again.

Comment 8

[Hal Wine [:hwine] use NI!]

some folks are using the bug from the 3/20 event, which is a dupe at this point. Please put all new information of problem occurrences in bug 985864