Closed
Bug 1093065
Opened 12 years ago
Closed 12 years ago
[basket] every few requests is returning a 503
Categories
(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)
Infrastructure & Operations Graveyard
WebOps: Product Delivery
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 1092804
People
(Reporter: pmac, Assigned: cliang)
References
()
Details
(Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1795] )
I get a 503 every 2 or 3 requests, and New Relic has been throwing alerts for around 24 hours. It must be one of the servers, but I've not yet figured out which or why.
I'm trying:
https://basket.mozilla.org/news/
In Fx and just hitting refresh. I've seen it only a a couple times via curl:
$ curl -I https://basket.mozilla.org/news/
HTTP/1.1 503 Service Temporarily Unavailable
Server: Apache
X-Backend-Server: generic1.webapp.phx1.mozilla.com
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Mon, 03 Nov 2014 14:10:43 GMT
Connection: close
X-Cache-Info: not cacheable; response code not cacheable
$ curl -I https://basket.mozilla.org/news/
HTTP/1.1 200 OK
Server: Apache
Vary: Cookie, Accept-Encoding
X-Backend-Server: generic3.webapp.phx1.mozilla.com
Content-Type: text/html; charset=utf-8
Date: Mon, 03 Nov 2014 14:10:47 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Cache-Info: caching
This appears to only happen via HTTPS.
| Reporter | ||
Updated•12 years ago
|
Severity: critical → blocker
Comment 1•12 years ago
|
||
Taking this bug to stop it from paging MOC. Trying to reach someone at webops to work on this.
Updated•12 years ago
|
Assignee: server-ops-webops → rwatson
Comment 2•12 years ago
|
||
As per your message, it appears bad requests are coming from generic1.webapp.phx1.
I have drained that node and removing it from the pool serving content. Hopefully webops can get to this soon.
| Assignee | ||
Updated•12 years ago
|
Assignee: rwatson → cliang
| Reporter | ||
Comment 3•12 years ago
|
||
In my testing I wasn't always getting bad responses from generic1.webapp.phx1, only when using HTTPS. And according to my latest test (below) it's not just generic1 in phx.
$ curl -I https://basket.mozilla.org/news/
HTTP/1.1 503 Service Temporarily Unavailable
Server: Apache
X-Backend-Server: generic2.webapp.phx1.mozilla.com
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
Date: Mon, 03 Nov 2014 15:22:09 GMT
Connection: close
X-Cache-Info: not cacheable; response code not cacheable
Comment 4•12 years ago
|
||
[Mon Nov 03 13:00:56 2014] [error] [client 10.8.81.215] (11)Resource temporarily unavailable: mod_wsgi (pid=13022): Unable to connect to WSGI daemon process 'basket-ssl' on '/var/run/wsgi.9377.19.5.sock'.
I've kicked the machines with leftover problems from bug 1092804 yesterday, let us know if you see any further problems.
I'm concerned that no monitoring picked this up :/
| Reporter | ||
Comment 5•12 years ago
|
||
I share your concern :pir. But the good news is that the errors seem to have stopped flowing and requests to basket are reliable again. Thanks!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Updated•12 years ago
|
Resolution: FIXED → DUPLICATE
Updated•10 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•