Status

--
blocker
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rail, Assigned: mostlygeek)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
Aurora nighlies fail to submit data to balrog, reporting 

02:50:51     INFO -  ConnectionError: ('Connection aborted.', BadStatusLine("''",))

I tried to login and got Secure Connection Failed in Firefox.


https://archive.mozilla.org/pub/firefox/nightly/2016/11/2016-11-08-00-40-19-mozilla-aurora/mozilla-aurora-linux64-nightly-bm74-build1-build8.txt.gz
I can't get as far as a auth prompt in Firefox (it times out), but curl quickly gives me a 401:
$ curl -IL https://aus4-admin.mozilla.org
HTTP/1.1 401 Unauthorized
Date: Tue, 08 Nov 2016 13:58:42 GMT
Content-Type: text/html
Content-Length: 188
Connection: keep-alive
WWW-Authenticate: Basic realm="Please log in to Mozilla LDAP"

...but if I authenticate and try a request, I get an empty reply.

Curiously, Datadog doesn't show anything abnormal compared to the same time yesterday.

I'm paging CloudOps.
It's fixed!
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Comment 3

2 years ago
Dug into this a bit more. 

It appears that nginx, specifically nginx-auth-ldap is segfaulting.
If this happens again I will build a new version of openresty with an updated ldap module.
Comment hidden (Intermittent Failures Robot)
seems not fixed -> https://treeherder.mozilla.org/logviewer.html#?job_id=5552870&repo=mozilla-central
Status: RESOLVED → REOPENED
Flags: needinfo?(rail)
Resolution: FIXED → ---
(Reporter)

Comment 7

2 years ago
I paged cloudops
Flags: needinfo?(rail)
Assignee: nobody → bwong
(Assignee)

Comment 8

2 years ago
Rolling out a new version of openresty with an updated nginx-auth-ldap module today. Hopefully this prevents some of the segfaults happening.
Does this need a postmortem?
(Assignee)

Comment 10

2 years ago
The deploy yesterday did not cause any alerts. 
Will monitor for a few more days to make sure things are stable / ok.
Comment hidden (Intermittent Failures Robot)
(In reply to Benson Wong [:mostlygeek] from comment #10)
> The deploy yesterday did not cause any alerts. 
> Will monitor for a few more days to make sure things are stable / ok.

Looks like this fix is holding.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.