704595 - need to determine if new web server for mobile talos testing is required

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Mike Taylor [:bear]]

right now we have 3 systems behind a load balancer to act as the talos test web server and with the increase of tegras we may need to increase this.

I say "may" because we need to measure their load and web response and first we should find out if we are measuring that.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

It'd be great to have requests-per-second data here, so we can replace these with something both more powerful than and more stable than r2 minis.

Comment 2

[Mike Taylor [:bear]]

are these even mini's - I keep thinking they are and just now the spirit of aki reached out from his holiday and smacked me in the head:

uname -a:

Linux bm-remote-talos-webhost-01.build.mozilla.org 2.6.32.10-90.fc12.i686.PAE #1 SMP Tue Mar 23 10:04:28 UTC 2010 i686 i686 i386 GNU/Linux

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I've seen them - they're non-unibody minis sitting on top of the load-balancer in 3.MDF.

https://inventory.mozilla.org/systems/show/1712/
etc.

Comment 4

[Justin Dow [:jabba]]

https://bugzilla.mozilla.org/show_bug.cgi?id=550357 is where they were set up originally. There is a Netscaler sitting in front of them that is only used for this task, and you could probably gain a lot of performance by tuning it and you can probably get all your stats from it as well.

https://inventory.mozilla.org/systems/show/3529/ is the Netscaler.

Comment 5

[Aki Sasaki (not active)]

IIRC they're rev1 minis.

Comment 6

[Amy Rich [:arr] [:arich]]

I took a look at the dashboard for the netscaler for about 20 min, and it says we're spreading the following over the three nodes:

* typically 0 HTTP requests/sec to 10/sec (never above 10) while I was watching
* all GET requests, no POST or others
* for current connections, I saw it mostly stay around 5/8 with a burst of 20/25 at one point.
* no errors

Comment 7

[Mike Taylor [:bear]]

That looks like a very bored load balancer :) -- if it still shows those levels tomorrow after the pre/post m-c merge of mobile then I think we can close this as not required.

thanks!