Status

Release Engineering
Balrog: Frontend
--
blocker
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: rail, Assigned: mostlygeek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

a year ago
Aurora nighlies fail to submit data to balrog, reporting 

02:50:51     INFO -  ConnectionError: ('Connection aborted.', BadStatusLine("''",))

I tried to login and got Secure Connection Failed in Firefox.


https://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-08-00-40-19-mozilla-aurora/mozilla-aurora-linux64-nightly-bm74-build1-build8.txt.gz
I can't get as far as a auth prompt in Firefox (it times out), but curl quickly gives me a 401:
$ curl -IL https://aus4-admin.mozilla.org
HTTP/1.1 401 Unauthorized
Date: Tue, 08 Nov 2016 13:58:42 GMT
Content-Type: text/html
Content-Length: 188
Connection: keep-alive
WWW-Authenticate: Basic realm="Please log in to Mozilla LDAP"

...but if I authenticate and try a request, I get an empty reply.

Curiously, Datadog doesn't show anything abnormal compared to the same time yesterday.

I'm paging CloudOps.
It's fixed!
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
(Assignee)

Comment 3

a year ago
Dug into this a bit more. 

It appears that nginx, specifically nginx-auth-ldap is segfaulting.
If this happens again I will build a new version of openresty with an updated ldap module.

Comment 4

a year ago
33 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 32
* mozilla-central: 1

Platform breakdown:
* linux32: 10
* osx-10-10: 8
* linux64: 8
* android-4-2-x86: 3
* windowsxp: 1
* windows8-64: 1
* osx-10-7: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1315947&startday=2016-11-08&endday=2016-11-08&tree=all
seems not fixed -> https://treeherder.mozilla.org/logviewer.html#?job_id=5552870&repo=mozilla-central
Status: RESOLVED → REOPENED
Flags: needinfo?(rail)
Resolution: FIXED → ---
also aurora https://treeherder.mozilla.org/logviewer.html#?job_id=4097445&repo=mozilla-aurora
(Reporter)

Comment 7

a year ago
I paged cloudops
Flags: needinfo?(rail)
Assignee: nobody → bwong
(Assignee)

Comment 8

a year ago
Rolling out a new version of openresty with an updated nginx-auth-ldap module today. Hopefully this prevents some of the segfaults happening.
Does this need a postmortem?
(Assignee)

Comment 10

a year ago
The deploy yesterday did not cause any alerts. 
Will monitor for a few more days to make sure things are stable / ok.
31 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 30
* mozilla-central: 1

Platform breakdown:
* linux32: 9
* osx-10-10: 8
* linux64: 7
* android-4-2-x86: 3
* windowsxp: 1
* windows8-64: 1
* osx-10-7: 1
* android-4-0-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1315947&startday=2016-11-07&endday=2016-11-13&tree=all
(In reply to Benson Wong [:mostlygeek] from comment #10)
> The deploy yesterday did not cause any alerts. 
> Will monitor for a few more days to make sure things are stable / ok.

Looks like this fix is holding.
Status: REOPENED → RESOLVED
Last Resolved: a year agoa year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.