1154584 - [basket] DOWNTIME. Error rate very high in New Relic.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Paul [:pmac] McLanahan]

Can't get MOC to do anything via IRC, so I'm coming here. New Relic for basket.mozilla.org is showing a lot of errors related to an inability to contact the DB starting at 6:30pm Pacific. I can contact that DB if I SSH in, so I'm guessing Apache just needs a kick. No idea what went wrong yet, but it looks like all DB related requests are 500ing. 

https://rpm.newrelic.com/accounts/263620/applications/2646232/traced_errors

Comment 1

[Shyam Mani [:fox2mike]]

atoll and pmac diagnosed this and I kicked httpd on the generic webheads. This seems to have fixed the issue.

Comment 2

[Paul [:pmac] McLanahan]

This appears to have happened again this morning between ~9:45AM and 10:30AM Eastern time. :cyliang believes it was resolved on its own this time by an apache segfault and restart. Let's see if we can track the root of this so we can avoid it in future.

Comment 3

[[github robot]]

Commit pushed to master at https://github.com/mozilla/basket

https://github.com/mozilla/basket/commit/b4c375cf1f1a62920f06dc6ce258e471d97f7974
Bug 1154584: Disable ratelimit for email subscribe.

We think it might be causing problems. Disabling temporarily
to see if our theory is correct.

Comment 4

[Shyam Mani [:fox2mike]]

This has been stable.