631528 - Scale size of test pools by time to complete a full set of tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

According to:
http://hg.mozilla.org/build/buildbot-configs/file/production/mozilla-tests/production_config.py#l1
* Fedora - 51 slaves
* Fedora64 - 53 slaves
* XP - 50 slaves
* Win7 - 49 slaves*
* Win7x64 - 48 slaves
* Leopard - 51 slaves
* Snow Leopard - 53 slaves

win7's pool capacity is 7.5% smaller compared to fed64 and snow leopard in optimal conditions.

* 2 slaves will be added back on bug 630309

Comment 1

[Armen [:armenzg]]

I can help in a couple of weeks to determine what a more proportionate distribution could be.

Passing to joduinn to prioritization.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

A full strength Win7 (51) vs Fed64 (53) is not a significant difference in the number of machines (4%) IMO, if we wanted to make all the platforms the same. Based on your results though, the issue is that there is significant variation in the time to do tests and Win7 is particularly slow.

So I've re-summarized the bug a bit. Seems to me that we should consider changing the number of slaves for each platform (N) so that
    Time to complete tests = (CPU hours for a full set of tests) / N   
is similar for all platforms. Blocked on any Win7 fixes we know are in the works of course.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Oops, misread your * to mean 49 was going up to 51, rather being 49 best case. You're over-counting by one by though, there are 52 Fed64 vs 48 Win7 (check with len() in python!) and that's not great.

Still think we should make the calculation in comment #2 for all platforms to figure out how many to move to Win7. Might as well balance them all out in one go.

Comment 4

[Armen [:armenzg]]

joduinn, mind if I take it and re-assign it back to you when I am done? I will get a report on how much CPU we use per complete set of tests.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> joduinn, mind if I take it and re-assign it back to you when I am done? I
> will get a report on how much CPU we use per complete set of tests.

Per meeting with Armen this morning. 

1) Now that the win64 builders are coming close to reality (bug#645024), we need to figure out how many win64 tester can be re-converted back to win64 testers without impacting the overall waittimes too much. Calculating relative sizes of different test pools is part of that.

2) Armen is already doing all the work here, so pushing the bug over to him.

Comment 6

[Armen [:armenzg]]

(In reply to comment #5)
> (In reply to comment #4)
> > joduinn, mind if I take it and re-assign it back to you when I am done? I
> > will get a report on how much CPU we use per complete set of tests.

The purpose of this bug finished its usefulness.
In bug 659118 we adjusted the pool's distribution and our wait times are more or less similar accross OSes.
We gave more capacity to XP and Win7 since they have higher CPU usage (see report below).

As per https://spreadsheets.google.com/spreadsheet/ccc?key=tNfPnuOZvLcbnItGP6gCYSQ&authkey=COzKr8wL#gid=0

* Win7     - 42689 secs
* Xp       - 40823 secs
* Leopard  - 37912 secs
* SnowLeop - 37025 secs
* Fedora   - 35869 secs
* Fedora64 - 32919 secs

There are some improvements that will be worked on bug 661585.

Recreating the Win7 x64 pool is still a point of debate on what we want to do and will file a bug at due time.

Comment 7

[Chris Cooper [:coop] (he/him)]

(In reply to comment #6)
 
> Recreating the Win7 x64 pool is still a point of debate on what we want to
> do and will file a bug at due time.

I filed bug 667024 to get Win7 x64 testers re-imaged once we have builders to support them.