1315947 - Balrog admin is down

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (frontend), defect)

Description

[Rail Aliiev [:rail]]

Aurora nighlies fail to submit data to balrog, reporting 

02:50:51     INFO -  ConnectionError: ('Connection aborted.', BadStatusLine("''",))

I tried to login and got Secure Connection Failed in Firefox.


https://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-08-00-40-19-mozilla-aurora/mozilla-aurora-linux64-nightly-bm74-build1-build8.txt.gz

Comment 1

[[e-mail] (:bhearsum)]

I can't get as far as a auth prompt in Firefox (it times out), but curl quickly gives me a 401:
$ curl -IL https://aus4-admin.mozilla.org
HTTP/1.1 401 Unauthorized
Date: Tue, 08 Nov 2016 13:58:42 GMT
Content-Type: text/html
Content-Length: 188
Connection: keep-alive
WWW-Authenticate: Basic realm="Please log in to Mozilla LDAP"

...but if I authenticate and try a request, I get an empty reply.

Curiously, Datadog doesn't show anything abnormal compared to the same time yesterday.

I'm paging CloudOps.

Comment 2

[[e-mail] (:bhearsum)]

It's fixed!

Comment 3

[Benson Wong [:mostlygeek]]

Dug into this a bit more. 

It appears that nginx, specifically nginx-auth-ldap is segfaulting.
If this happens again I will build a new version of openresty with an updated ldap module.

Comment 4

[Intermittent Failures Robot]

33 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 32
* mozilla-central: 1

Platform breakdown:
* linux32: 10
* osx-10-10: 8
* linux64: 8
* android-4-2-x86: 3
* windowsxp: 1
* windows8-64: 1
* osx-10-7: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1315947&startday=2016-11-08&endday=2016-11-08&tree=all

Comment 5

[Carsten Book [:Tomcat]]

seems not fixed -> https://treeherder.mozilla.org/logviewer.html#?job_id=5552870&repo=mozilla-central

Comment 6

[Carsten Book [:Tomcat]]

also aurora https://treeherder.mozilla.org/logviewer.html#?job_id=4097445&repo=mozilla-aurora

Comment 7

[Rail Aliiev [:rail]]

I paged cloudops

Comment 8

[Benson Wong [:mostlygeek]]

Rolling out a new version of openresty with an updated nginx-auth-ldap module today. Hopefully this prevents some of the segfaults happening.

Comment 9

[Amy Rich [:arr] [:arich]]

Does this need a postmortem?

Comment 10

[Benson Wong [:mostlygeek]]

The deploy yesterday did not cause any alerts. 
Will monitor for a few more days to make sure things are stable / ok.

Comment 11

[Intermittent Failures Robot]

31 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 30
* mozilla-central: 1

Platform breakdown:
* linux32: 9
* osx-10-10: 8
* linux64: 7
* android-4-2-x86: 3
* windowsxp: 1
* windows8-64: 1
* osx-10-7: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1315947&startday=2016-11-07&endday=2016-11-13&tree=all

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Benson Wong [:mostlygeek] from comment #10)
> The deploy yesterday did not cause any alerts. 
> Will monitor for a few more days to make sure things are stable / ok.

Looks like this fix is holding.