577170 - input.mozilla.com seems slow

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[juan becerra [:juanb]]

While testing Fx4b1 today we've noticed that the site is pretty slow. You can get to it by clicking on the new Feedback button. In addition, it also seems like the happy face link takes longer to load than the frowny face one: http://grab.by/5iyz

I don't know under what to file it, but it seems like a blocker when users can't get to the feedback site.

Comment 1

[Fred Wenzel [:wenzel]]

Seems like there's a problem with Sphinx?

Comment 2

[Mike Beltzner [:beltzner, not reading bugmail]]

What is Sphinx? If this is an IT bug, let's move components and file as blocker.

Comment 3

[Mike Beltzner [:beltzner, not reading bugmail]]

Aakash (got a hold of him via SMS) also suspects Sphinx

Comment 4

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Can I get a link to the nagios monitoring for pm-sphinx-01|02 (assuming that it's the same sphinx cluster).

Stage search is working fine.

Chizu, if you can open up the mysql ports on sphinx01 like you did for AMO, that'd be immensely helpful.

-d

Comment 5

[Michael Morgan [:morgamic]]

If it's not on the same hardware and on a VM or something instead, that'd be why.  Shyam -- where's it hosted?

Comment 6

[Shyam Mani [:fox2mike]]

Okay, it looks like the FF4 feedback thingy is hitting input fairly hard and the DB setup of the app isn't configured in a master/slave config and that in turn is caused the master which has the DB to become overloaded...and is causing issues for every DB hosted on the c01 DB cluster.

Dave is working on a fix for the app that'll support master/slave DB configs and I'll work with him to get the fix staged and then pushed to production.

Comment 7

[Shyam Mani [:fox2mike]]

(In reply to comment #5)
> If it's not on the same hardware and on a VM or something instead, that'd be
> why.  Shyam -- where's it hosted?

Where's what hosted? Sphinx? That's on dedicated hardware, 2 machines. I don't think Sphinx is the issue here. Comment #6 has more details.

Comment 8

[Michael Morgan [:morgamic]]

Ok - hadn't read 6 yet!  Plan sounds good.

Comment 9

[Fred Wenzel [:wenzel]]

We're in the process of pushing it live.

Comment 10

[Aakash Desai [:aakashd]]

Take 1 - push to live resulted in the reads and writes still going to master. We're trying a take 2 now.

Comment 11

[Shyam Mani [:fox2mike]]

Alright, so we have the reads going to the slave successfully and the master is holding up in much better shape. 

There seems to be an issue with trends as such though, the queries take too long to execute and are holding up the slave as well....

Fred is looking into it.

Comment 12

[Shyam Mani [:fox2mike]]

Alright, the pages having/using the query have been updated to not hit the query for now. We've pushed these changes out and everything seems much happier than before.

Calling this fixed, Fred will figure out the query later.