578558 - try tests/perf should be run at lower priority on mini farm

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

When try and mozilla-central are busy we often end up with pending builds for both branches. We don't allow try to merge unit and perf requests but mozilla-central will if we meet buildbot's criteria for a backlog. The result is reduced test coverage on m-c when requests for several revisions get collapsed to the last one, and a very sparse looking TBPL. Meanwhile the try backlog clears slowly.

I haven't looked at the buildbot 0.7.10 code but I would like to set talos-master02 and friends to chose a m-c build over a try one whenever there is a slave shortage.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

catlee suggested enhancing 
 http://hg.mozilla.org/build/buildbot-configs/file/cb9660188d97/talos-r3/master-common.cfg#l43

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Prioritize branches over try

Works in staging for this scenario:
* try sendchange
* try job on starts on slave
* m-c sendchange
* 2nd try job starts on slave (try sendchange is older)
* reconfig with this patch
* m-c job starts on slave (prioritizeBuilders doing its new thing)
* continues to do m-c jobs

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 457486 [details] [diff] [review]
Prioritize branches over try

Should we add this to mozilla/builder_config.py as well, to futureproof that?

Comment 4

[Chris AtLee [:catlee]]

Just one thought before going live with this: do we want tryserver jobs to always be at the bottom of the pile?  Could we instead give tryserver jobs a time penalty, e.g. leave it as priority 1, but add 30 minutes to the request time.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Can we do that in a way that doesn't distort a wait time analysis ?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

From IRC, Chris is suggesting a prioritizeBuilders function that looks something like this:
+        elif builder.builder_status.category == 'tryserver':
+            return 1, builder.getOldestRequestTime() + 30*60

I'm not sure what the effect of this would be when there are multiple try changes pending, so that the penalty has effectively expired for the oldest ones. And that approach doesn't really address the basic problem as I see it - that m-c tests which impact on many developers will be merged in favor of try tests which impact a single dev. It kinda works around it in a unpredictable way.

Chris also suggested a simulator to explore what would happen in various regimes. The nerd in me would love this but unfortunately we don't have easy access to a list of sendchanges (eg from scheduler db) for a some busy day, so it's somewhat laborious to reconstruct a validation test.

Overall I think the value of complete test-sets for m-c outweighs any waits people may experience on try, given the relatively large disruption when figuring out which changeset regressed tests on m-c. That's particularly true on freeze days.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 457486 [details] [diff] [review]
Prioritize branches over try

Landing this in the hope of getting some better test coverage on m-c on freeze night. http://hg.mozilla.org/build/buildbot-configs/rev/d60b702f8151

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Reconfig complete on talos-master02 at 21:40; still going on test-master01.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

test-master01 complete at 22:14.

Chris, please reopen if you feel strongly enough that we shouldn't do this.