333553 - set up replacement linux perf testing tinderboxes

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

gtk1 and probably gcc 2.95 support will becoming obsolete pretty soon, so we need some replacements for tinderboxes. The affected tinderboxes that I know of are:

balsa (Firefox) - running leak tests
btek (SeaMonkey) - running Tp
luna (SeaMonkey) - running basically every perf test

The difficulty is that the replacements need to be gtk2/gcc3, and need to give stable numbers.  My naiive attempt can be seen on http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox-Cairo with the linux-perf box; I made no attempt to disable daemons and other non-essential services there.

Comment 1

[Boris Zbarsky [:bzbarsky]]

Is it desirable (and possible) to set up parallel cairo and non-cairo boxes so that we can tell apart perf impact from cairo work from perf impact from layout changes?  I think this was brought up before in the context of cairo being expected to be slower than gfx for now (essentially giving a perf regression that might hide other things when cairo gets switched on by default).

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Yes, that was the goal with the boxes that I set up -- one of them has "Ref" after it, which is the reference non-cairo build.  In retrospect that should've said NoCairo...

Comment 3

[J. Paul Reed [:preed]]

Vlad: is this taken care of by rhelmer's test-only Tinderboxen work?

Comment 4

[J. Paul Reed [:preed]]

Over to rhelmer.

Comment 5

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to comment #3)
> Vlad: is this taken care of by rhelmer's test-only Tinderboxen work?

I don't really know; I don't think btek is producing any useful numbers for us right now (because of its config).  However, even given that, we need stable numbers and the test-only tinderboxes aren't producing stable numbers.  If you compare:

http://build-graphs.mozilla.org/graph/query.cgi?testname=pageload&tbox=btek&autoscale=1&days=7&avg=1
http://build-graphs.mozilla.org/graph/query.cgi?testname=pageload&units=ms&tbox=Fx-Trunk-linux-test1&autoscale=1&days=7&avg=1

The btek graph has a variance of <1%, whereas the Fx-Trunk-linux-test1 graph seems to be in the 3% range, if you ignore the random spikes.  Tp2 is in the same boat.  1-2% changes are a big deal; we need to have sub-1% precision on whatever performance numbers that we have.

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to comment #5)
> (In reply to comment #3)
> > Vlad: is this taken care of by rhelmer's test-only Tinderboxen work?
> 
> The btek graph has a variance of <1%, whereas the Fx-Trunk-linux-test1 graph
> seems to be in the 3% range, if you ignore the random spikes.  Tp2 is in the
> same boat.  1-2% changes are a big deal; we need to have sub-1% precision on
> whatever performance numbers that we have.

One theory for this discrepancy is that btek is running the 2.2 Linux kernel, while bl-bldlnx01 is 2.6 (FC4). Since we've tried to minimize userspace process activity on both, this may have to do with the scheduling characteristics.

I am going to try some different scheduling options; for example changing the IO elevator to deadline (default is CFQ) as of now.

Comment 7

[Robert Helmer [:rhelmer]]

This machine is set up and reporting to Mozilla1.8 and Firefox trees; if there is more we can do to get more reliable numbers please open a separate bug.

Comment 8

[Boris Zbarsky [:bzbarsky]]

The original bug mentioned leak tests.  We're still running those on balsa, and we need to stop doing that to achieve the goal this bug was filed on.

Do you want a separate bug on that too?

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to comment #8)
> The original bug mentioned leak tests.  We're still running those on balsa, and
> we need to stop doing that to achieve the goal this bug was filed on.
> 
> Do you want a separate bug on that too?

Yes please. We should be able to allocate this on VMs and not need to use a standalone machine like we do for wall-clock-sensitive tests.

Comment 10

[Boris Zbarsky [:bzbarsky]]

Filed bug 360206.