1500411 - (bmo-errors-2018-10-19) Intermittent 500 server errors on BMO all around

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect, P1)

Description

[Ludovic Hirlimann [:Usul]]

I got reports from both Pike and Aryx that they encountered issues submitting bug changes to bugzilla.

13:29:11 CET <Pike> bugzilla just puked a rainbow on me. "something went very wrong" this was on bug 1317336 around 13:28 CET

Aryx had the issue 5/6 minutes before on either bug 1500031 or 1500037 (he wasn't sure).

At 14:05 Aryx reproduced:
<Aryx> Usul: just got it again for https://bugzilla.mozilla.org/show_bug.cgi?id=1500312
<Aryx> edited bug summary and tried to submit

Comment 1

[:Gijs (out for now; he/him)]

I had this a few minutes ago on bug 1487703, and now tried 3 times to 'follow' this bug using the 1-click button at the top of the bug, and all of them came back "Unexpected Error".

Comment 2

[Ludovic Hirlimann [:Usul]]

<Gijs> yeah, multiple people reporting this now.
<Gijs> see #fx-team in the last 5 minutes

had the issue when trying to add this comment.

Comment 3

[Boris Zbarsky [:bzbarsky]]

I just got this 5 times in a row when trying to check the "block reviews" checkbox.  The 6th time worked.

And got it twice when trying to change a reviewer on a bug; third time worked.

Comment 4

[Jorg K (CEST = GMT+2)]

For me, BMO is unusable right now. Can't update or file bugs.

Comment 5

[Jorg K (CEST = GMT+2)]

Comment #4 got posted on the 3rd attempt :-(

Comment 6

[Florian Quèze [:florian]]

It took me 5 attempts to submit bug 1498188 comment 1, and then 9 attempts to set the "wontfix" flag in the same bug.

Comment 7

[Victor Porof [:vporof][:vp]]

I tried 5 times to upload a patch to bug 1498740. Gave up.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I cannot review, give feedback, edit attachments, needinfo others, or edit my profile.

Comment 10

[Sylvestre Ledru [:Sylvestre]]

Folks, I don't think the "me too" comments are bringing new information ;)

Comment 12

[:Gijs (out for now; he/him)]

bugmail seems to also be broken? I see no new bugmail since 13:38 (british summer time, 30 minutes prior to me writing this comment), despite all the comments here (for which I should have had bugmail).

Comment 13

[Jorg K (CEST = GMT+2)]

BTW, "Preview" doesn't work either, or is this another bug?

Comment 14

[:Gijs (out for now; he/him)]

(In reply to :Gijs (he/him) from comment #12)
> bugmail seems to also be broken? I see no new bugmail since 13:38 (british
> summer time, 30 minutes prior to me writing this comment), despite all the
> comments here (for which I should have had bugmail).

... except apparently the first time I commented, I didn't get CC'd. So maybe the lack of bugmail is simply a symptom of people struggling to change anything. Apologies for the noise.

Comment 15

[Kohei Yoshino]

(In reply to Jorg K (GMT+2) from comment #13)
> BTW, "Preview" doesn't work either, or is this another bug?

Looks like the same; the server returns 500 Internal Server Error.

Comment 16

[Dylan Hardison [:dylan] (he/him)]

Hey! For the moment the error should be resolved.

Preliminary theory is that while making the Phabricator/bugzilla sync daemon more resilient in the face of MySQL errors, the web heads are now less resilient to MySQL errors. This manifested today because (until we upgrade mysql) the MySQL servers will crash every Friday.

Comment 17

[Dylan Hardison [:dylan] (he/him)]

Just a note, you didn't receive emails from my last update because my account was set to silent mode, from handling admin tasks yesterday. Sorry for any worry that may have caused.

Comment 18

[Dylan Hardison [:dylan] (he/him)]

Okay, so the preliminary diagnosis was not entirely correct.

The root cause was MySQL crashing, but the error that was triggering the 500 errors was that the /tmp directory became filled up. We're fixing that in bug 1500547. There is some room for improvement in how we handle and report database errors, so that change will go in bug 1500547. This bug will be resolved when both those (and anything else we find and link to this bug).