723591 - QMO is painfully slow to load

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I'm trying to add some content to QMO and it's taking forever to load.

Comment 1

[Al Billings [:abillings - ex-MoCo]]

Works for me fine in Oakland.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Still an issue here in the UK. It's currently unusable.

Comment 3

[Shyam Mani [:fox2mike]]

Works fine from Singapore too. Can you run a traceroute to quality.mozilla.org? Maybe some connection issues on your end? Does it eventually load or not at all?

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It does eventually load. It seems a little faster to respond now, but still noticeably slow. Here's my traceroute results:

traceroute quality.mozilla.org
traceroute to quality.mozilla.org (63.245.209.47), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  1.919 ms  1.647 ms  1.653 ms
 2  losubs.subs.dsl13.wh-man.zen.net.uk (62.3.83.20)  58.042 ms  48.379 ms  45.568 ms
 3  ge-2-1-0-232.cr2.wh-man.zen.net.uk (62.3.80.233)  47.176 ms  47.367 ms  47.286 ms
 4  ge-3-0-0-0.cr1.kp-leeds.zen.net.uk (62.3.80.58)  51.334 ms  64.914 ms  48.825 ms
 5  ge-3-1-0-0.cr2.kp-leeds.zen.net.uk (62.3.80.74)  48.414 ms  49.376 ms  48.818 ms
 6  so-0-0-0-0.cr1.ha-sthp.zen.net.uk (62.3.80.82)  50.096 ms  54.419 ms  51.261 ms
 7  so-0-0-0-0.cr1.tlx-nyc.zen.net.uk (62.3.80.86)  131.228 ms  128.483 ms  129.129 ms
 8  ny-iix.above.net (198.32.160.22)  130.111 ms  130.362 ms  129.028 ms
 9  xe-1-1-0.cr1.lga5.us.above.net (64.125.26.161)  131.402 ms  128.906 ms  131.626 ms
10  xe-3-2-0.cr1.ord2.us.above.net (64.125.24.34)  155.180 ms  166.539 ms  156.284 ms
11  xe-1-2-0.cr1.sjc2.us.above.net (64.125.26.229)  209.698 ms  208.882 ms  211.350 ms
12  xe-4-3-0.er1.sjc2.us.above.net (64.125.28.53)  209.095 ms  208.344 ms  209.152 ms
13  64.125.189.26.t00539-02.above.net (64.125.189.26)  210.982 ms  209.006 ms  209.144 ms
14  te-8-2.core2.sjc.mozilla.net (63.245.208.98)  208.946 ms  209.849 ms  209.617 ms
15  dyna-qmo.nslb.sj.mozilla.com (63.245.209.47)  208.420 ms  208.987 ms *

Comment 5

[Jake Maul [:jakem]]

One thing I can do to potentially speed this up is to set this up to have a caching node in Amsterdam.

However, in order for this to work, the server needs to start sending some effective cache headers (Expires or Cache-Control). At present it does nothing, meaning a hit to the homepage requires 34 requests, with or without a primed cache. Most of them are responded to with a "304 Not Modified", but it's still 34 round-trips to the server.

I would recommend starting out by looking at one of the common Wordpress cache plugins... I suspect (but do not know for certain) that these will set the proper headers to make this happen, in a nice clean Wordpress-y way. if nothing else they'll certainly speed up page delivery in other ways.

WP Super Cache is kinda the defacto standard: http://wordpress.org/extend/plugins/wp-super-cache/

Also popular is the W3 Total Cache: http://wordpress.org/extend/plugins/w3-total-cache/


They also both have "CDN Support" which means we can probably throw the static content up on Akamai or something, without too much extra effort.

Comment 6

[Shyam Mani [:fox2mike]]

Also, this is a VM. A one cpu, 500 MB RAM VM.

This will move to the new SCL3 datacenter in a month or so. I'm not sure we should keep qmo on a VM, we should file bugs to integrate it with the rest of the generic and make that happen :) That should surely help some.

Comment 7

[Al Billings [:abillings - ex-MoCo]]

Adding Craig, who does dev work on QMO, and Raymond, who has taken over QMO ownership from me.

I'm not in the QA org anymore.

Comment 8

[raymond [:retornam] (inactive)]

(In reply to Shyam Mani [:fox2mike] from comment #6)
> Also, this is a VM. A one cpu, 500 MB RAM VM.
> 
> This will move to the new SCL3 datacenter in a month or so. I'm not sure we
> should keep qmo on a VM, we should file bugs to integrate it with the rest
> of the generic and make that happen :) That should surely help some.

Which component should I file the bugs under

Comment 9

[Jake Maul [:jakem]]

If you want to have it migrated to the generic cluster, I would open a bug in Server Ops: Web Operations. We would move it to the 'genericrhel6' cluster in PHX1, most likely, unless it takes a long time to get going in which case it might go to SCL3.

If you want a VM upgrade, and want to migrate to SCL3 later on (still in your own VM), you probably want a bug in Server Operations to have the VM beefed up. I don't know how much capacity is available at the moment. FWIW, I think more CPU won't help but that double the RAM (increase to 1GB) might be useful.

Comment 10

[Shyam Mani [:fox2mike]]

CC'ing Dustin for comment #6 and comment #9 (just FYI, to keep you in the loop)

Comment 11

[raymond [:retornam] (inactive)]

(In reply to Shyam Mani [:fox2mike] from comment #10)
> CC'ing Dustin for comment #6 and comment #9 (just FYI, to keep you in the
> loop)

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=725174 to upgrade RAM

Comment 12

[Shyam Mani [:fox2mike]]

Jason/Corey,

Any objections to moving this off this single VM to the generic cluster in phx1? If we do that, this VM can die.

Comment 13

[Corey Shields [:cshields]]

This site needs to be moved to phx1...  Not sure how it missed the list.

Can we WONTFIX 725174 and focus on migrating it instead?

Comment 14

[Jason Thomas [:jason]]

(In reply to Shyam Mani [:fox2mike] from comment #12)

> Any objections to moving this off this single VM to the generic cluster in
> phx1? If we do that, this VM can die.

Generic cluster currently hosts the -dev and -stage instances (bug 641627), so it would make sense to get prod over there as well.

Comment 15

[Jake Maul [:jakem]]

Can we handle a migration to genericrhel6 cluster in PHX1 in a separate bug? This one could be dependent on that, but there's more to performance than just that migration.

If we want to make this faster in Europe, we need to look at setting up WP Super Cache, followed potentially by using the AMS1 caching node and/or a CDN. Craig / Raymond, thoughts on this? The Infrasec review is Bug 720434.

I believe IT doesn't currently have admin access to this blog... that's probably a good place to start in getting this stuff set up for you.

Comment 16

[Shyam Mani [:fox2mike]]

Jake,

The reason I bought this up here is I'm sure Dave access a lot of other Mozilla sites which aren't proxied via AMS and doesn't find it as slow as QMO. Dave? 

If that holds true, a chuck of this issue will go away with better hardware.

Comment 17

[Dave Hunt [:davehunt] [he/him] ⌚BST]

QMO is the only site that I have noticed to be so slow, yes.

Comment 18

[Jake Maul [:jakem]]

Opened a new bug that is specifically about migrating this to the PHX1 genericrhel6 cluster... this bug is 17 (now 18) comments long about a variety of things, so a separate dedicated bug is warranted IMO.

This bug should now be effectively just a tracking bug for performance improvements to QMO.

Comments on the hardware migration belong in Bug 726733.

Comments on the stopgap VM RAM upgrade belong in Bug 725174. This is moot once the hardware migration is completed.

Comments on the "caching in AMS1" idea belong in Bug 726739. This is blocked by both the WP Super Cache review and the hardware migration.

Comment 19

[Brandon Burton [:solarce]]

QMO has been migrated to our generic cluster in PHX and that cluster recently received a big hardware upgraded.

Feel free to re-open if you feel performance needs further work.

Comment 20

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It does seem much more responsive to me now. Thanks!