bugzilla.mozilla.org will be intermittently unavailable on Saturday, March 24th, from 16:00 until 20:00 UTC.
While testing Fx4b1 today we've noticed that the site is pretty slow. You can get to it by clicking on the new Feedback button. In addition, it also seems like the happy face link takes longer to load than the frowny face one: http://grab.by/5iyz I don't know under what to file it, but it seems like a blocker when users can't get to the feedback site.
Seems like there's a problem with Sphinx?
What is Sphinx? If this is an IT bug, let's move components and file as blocker.
Aakash (got a hold of him via SMS) also suspects Sphinx
Can I get a link to the nagios monitoring for pm-sphinx-01|02 (assuming that it's the same sphinx cluster). Stage search is working fine. Chizu, if you can open up the mysql ports on sphinx01 like you did for AMO, that'd be immensely helpful. -d
If it's not on the same hardware and on a VM or something instead, that'd be why. Shyam -- where's it hosted?
Okay, it looks like the FF4 feedback thingy is hitting input fairly hard and the DB setup of the app isn't configured in a master/slave config and that in turn is caused the master which has the DB to become overloaded...and is causing issues for every DB hosted on the c01 DB cluster. Dave is working on a fix for the app that'll support master/slave DB configs and I'll work with him to get the fix staged and then pushed to production.
(In reply to comment #5) > If it's not on the same hardware and on a VM or something instead, that'd be > why. Shyam -- where's it hosted? Where's what hosted? Sphinx? That's on dedicated hardware, 2 machines. I don't think Sphinx is the issue here. Comment #6 has more details.
Ok - hadn't read 6 yet! Plan sounds good.
We're in the process of pushing it live.
Take 1 - push to live resulted in the reads and writes still going to master. We're trying a take 2 now.
Alright, so we have the reads going to the slave successfully and the master is holding up in much better shape. There seems to be an issue with trends as such though, the queries take too long to execute and are holding up the slave as well.... Fred is looking into it.
Alright, the pages having/using the query have been updated to not hit the query for now. We've pushed these changes out and everything seems much happier than before. Calling this fixed, Fred will figure out the query later.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.