Happened to notice, but basket is throwing regular 500 requests back at FxA content-server requests in production starting at 2015-08-18T21:10 UTC
Can we get the content of one of those 500s (headers, body)?
Looks like mine is a dupe of this (bug 1195750). I added a screenshot from New Relic to my bug :atoll.
Basically a lot of "OperationalError: Can't initialize character set utf8 (path: /usr/share/mysql/charsets/)"
I *think* the errors are no longer occurring but I do not particularly know why this occurred. I first verified that there have been no major changes to either the version of configuration of MySQL or OS patches, etc. on the database server. After noting that almost all of the errors seem to have originated from just one of the webheads, I restarted the basket processes on that server to see if any other servers were having issues. After 15 minutes, I haven't seen any errors.
The New Relic graphs are looking good. I'll keep an eye on it today. Thanks C!
Errors stopped from my vantage point about 9am PDT.
(For posterity's sake: I issued an 'apachectl graceful' on the misbehaving web server.)
Closing this basket 500 errors bug for now. If this crops up again, please cross-reference this bug: we'll probably end up taking the errant node out of the load balancer pool and do a proper analysis.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.