772468 - MDN: developer-new performance

Status: VERIFIED WORKSFORME

(developer.mozilla.org Graveyard :: Editing, defect, P1)

Description

[Luke Crouch [:groovecoder]]

Running multiple web-heads and memcache caused serious performance issues.

Working here: https://etherpad.mozilla.org/developer-new-is-slow

Comment 1

[Les Orchard [:lorchard]]

Do we have additional developments or actions needed here?

Comment 2

[Jake Maul [:jakem]]

There are at present no actions related to this that are waiting on IT (as far as I know). I don't know for sure that the problem is "resolved", however.

Comment 4

[Jake Maul [:jakem]]

Performance has been consistently good, after the changes in bug 770271#c9. Good enough that today we re-enabled a second web node this morning, and it's *still* been good all day.

Comment 5

[raymond [:retornam] (inactive)]

I haven't noticed any performance issues lately. I'll update bug with any issues I find

Comment 6

[Luke Crouch [:groovecoder]]

Jake, can we run another stress test on the 3 web-heads and put the results here. If they are comparable or better than the single web-head, let's close this.

Comment 7

[Jake Maul [:jakem]]

I've done quite a bit of stress-testing work on this. One thing I determined is that httperf (my tool of choice for the previous testing) does not play nicely with Zeus caching. The headers are correct (it works), but the statistical output is useless... it counts most of the attempts as failures. Disabling the cache fixes this, and in such a config it sustains approximately 3x the hit rate as a single node being hit directly (no Zeus). httperf is rather old and not in active development, so I'm completely willing to chalk this up as a bug in the tool. I like it because it can be nicely scaled to multiple nodes.

Switching to Siege instead of httperf, and I was able to roughly replicate the same performance level without Zeus caching... and it worked *with* caching as well. The performance in that config is (as one would expect) much higher. I suspect it would be higher still, but I reached the limit of what I could feasibly output with a single siege client node. Siege has no convenient way to use several client machines at once, so this is the best I can do without a lot more effort.

Testing against https://developer-new.mozilla.org/en-US/learn:

1 node, directly: 200-250 hits/sec
3 nodes, through Zeus w/ no cache: 720 hits/sec
3 nodes, through Zeus w/ cache: ~4200 hits/sec  <-- client-side testing limitation

This is a pretty synthetic test though- that page was previously chosen because it has minimal MySQL calls and no Kumascript calls... it's about as simple as a Kuma page will ever be.

We could do another test against a more common page, but I'm not sure it's really necessary.


Adding a blocker for getting Graphite to work. I thought that would "just work", but it appears not to.

Comment 8

[Les Orchard [:lorchard]]

Things seem snappy. Going to close this and call it done. 

We're still futzing around with statsd, but specific follow-up bugs should be opened from here out. This one seems too general for action items. (ie. "Make it go, make it go fast.")

Comment 9

[John Karahalis [:openjck]]

Agreed. Thanks, Les.