671444 - AMO Discovery Pane very slow sometimes

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Justin Scott [:fligtar]]

Both times I've opened the Discovery Pane in the Add-ons Manager this week it has take a ridiculous amount of time to load -- 10 seconds or more. I've confirmed with others that it's been slow lately as well, and this Gomez report confirms that load times occasionally double: http://cl.ly/2O070v1J3r1J2h3q362J

There haven't been any code changes lately and I'm wondering if this is a CDN issue.

Comment 1

[Jeff Balogh (:jbalogh)]

We only get about 2 requests/second hitting Python[1], so I don't have great numbers to suggest it's a backend problem. We have pretty consistent 2-second responses on these pages[2], which is awful, but we expect it to be served by zeus just about all the time.

The URL getting hit is something like https://services.addons.mozilla.org/en-US/firefox/discovery/pane/8.0a1/Darwin.

[1] https://graphite-sjc.mozilla.org/render/?width=586&height=308&_salt=1310601769.155&areaMode=first&target=stats.timers.amo.view.discovery.views.pane.GET.count&from=-7days
[2] https://graphite-sjc.mozilla.org/render/?width=1000&height=525&_salt=1310601640.929&target=stats.timers.amo.view.discovery.views.pane.GET.mean&target=stats.timers.amo.view.discovery.views.pane.GET.lower&target=stats.timers.amo.view.discovery.views.pane.GET.upper_90&areaMode=first

Comment 2

[Jeremy Orem [:oremj]]

Gomez says, "Current Average DNS Lookup time exceeds 90 ms!". Not sure if DNS is becoming very slow at times.

Comment 3

[Justin Scott [:fligtar]]

This doesn't seem limited to just the disco pane. Several times in the past week, including about 30 seconds before this comment, I've typed addons.mozilla.org in Firefox and it's taken more than 10 seconds to respond.

Comment 4

[Jeremy Orem [:oremj]]

I think we are just overloaded with traffic and don't have the option of adding additional servers right now. Let's look at this again once addons has been moved to PHX.

Comment 5

[Justin Scott [:fligtar]]

When is that supposed to happen?

Comment 6

[Jeremy Orem [:oremj]]

Almost have dev and stage up. Prod should be pretty quick after that. Plan is to have prod ready by middle of next week and then hopefully switch it the week after.

Comment 7

[Jeremy Orem [:oremj]]

I also noticed, while in the office, that addons often took much longer to load than at home.

Comment 8

[Jeremy Orem [:oremj]]

Moved to PHX. Let's reopen if this is still a problem.