Basket server throwing 500 errors in production since 2015-08-18T21:10 UTC

RESOLVED FIXED

Status

--
critical
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: jrgm, Unassigned)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1578] )

(Reporter)

Description

3 years ago
Happened to notice, but basket is throwing regular 500 requests back at FxA content-server requests in production starting at 2015-08-18T21:10 UTC

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1578]
Can we get the content of one of those 500s (headers, body)?
Looks like mine is a dupe of this (bug 1195750). I added a screenshot from New Relic to my bug :atoll.
Duplicate of this bug: 1195750
Basically a lot of "OperationalError: Can't initialize character set utf8 (path: /usr/share/mysql/charsets/)"

Comment 5

3 years ago
I *think* the errors are no longer occurring but I do not particularly know why this occurred.

I first verified that there have been no major changes to either the version of configuration of MySQL or OS patches, etc. on the database server.   

After noting that  almost all of the errors seem to have originated from just one of the webheads, I restarted the basket processes on that server to see if any other servers were having issues.  After 15 minutes, I haven't seen any errors.
The New Relic graphs are looking good. I'll keep an eye on it today. Thanks C!
(Reporter)

Comment 7

3 years ago
Errors stopped from my vantage point about 9am PDT.

Comment 8

3 years ago
(For posterity's sake: I issued an 'apachectl graceful' on the misbehaving web server.)

Comment 9

3 years ago
Closing this basket 500 errors bug for now.  If this crops up again, please cross-reference this bug: we'll probably end up taking the errant node out of the load balancer pool and do a proper analysis.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.