Closed Bug 1093065 Opened 12 years ago Closed 12 years ago

[basket] every few requests is returning a 503

Categories

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

task
Not set
blocker

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1092804

People

(Reporter: pmac, Assigned: cliang)

References

()

Details

(Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1795] )

I get a 503 every 2 or 3 requests, and New Relic has been throwing alerts for around 24 hours. It must be one of the servers, but I've not yet figured out which or why. I'm trying: https://basket.mozilla.org/news/ In Fx and just hitting refresh. I've seen it only a a couple times via curl: $ curl -I https://basket.mozilla.org/news/ HTTP/1.1 503 Service Temporarily Unavailable Server: Apache X-Backend-Server: generic1.webapp.phx1.mozilla.com Vary: Accept-Encoding Content-Type: text/html; charset=iso-8859-1 Date: Mon, 03 Nov 2014 14:10:43 GMT Connection: close X-Cache-Info: not cacheable; response code not cacheable $ curl -I https://basket.mozilla.org/news/ HTTP/1.1 200 OK Server: Apache Vary: Cookie, Accept-Encoding X-Backend-Server: generic3.webapp.phx1.mozilla.com Content-Type: text/html; charset=utf-8 Date: Mon, 03 Nov 2014 14:10:47 GMT Transfer-Encoding: chunked Connection: Keep-Alive X-Cache-Info: caching This appears to only happen via HTTPS.
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1795]
Severity: critical → blocker
Taking this bug to stop it from paging MOC. Trying to reach someone at webops to work on this.
Assignee: server-ops-webops → rwatson
As per your message, it appears bad requests are coming from generic1.webapp.phx1. I have drained that node and removing it from the pool serving content. Hopefully webops can get to this soon.
Assignee: rwatson → cliang
In my testing I wasn't always getting bad responses from generic1.webapp.phx1, only when using HTTPS. And according to my latest test (below) it's not just generic1 in phx. $ curl -I https://basket.mozilla.org/news/ HTTP/1.1 503 Service Temporarily Unavailable Server: Apache X-Backend-Server: generic2.webapp.phx1.mozilla.com Vary: Accept-Encoding Content-Type: text/html; charset=iso-8859-1 Date: Mon, 03 Nov 2014 15:22:09 GMT Connection: close X-Cache-Info: not cacheable; response code not cacheable
[Mon Nov 03 13:00:56 2014] [error] [client 10.8.81.215] (11)Resource temporarily unavailable: mod_wsgi (pid=13022): Unable to connect to WSGI daemon process 'basket-ssl' on '/var/run/wsgi.9377.19.5.sock'. I've kicked the machines with leftover problems from bug 1092804 yesterday, let us know if you see any further problems. I'm concerned that no monitoring picked this up :/
I share your concern :pir. But the good news is that the errors seem to have stopped flowing and requests to basket are reliable again. Thanks!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Resolution: FIXED → DUPLICATE
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.