If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

try tests/perf should be run at lower priority on mini farm

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: nthomas, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildmasters][tryserver][q3-goal?])

Attachments

(1 attachment)

(Assignee)

Description

7 years ago
When try and mozilla-central are busy we often end up with pending builds for both branches. We don't allow try to merge unit and perf requests but mozilla-central will if we meet buildbot's criteria for a backlog. The result is reduced test coverage on m-c when requests for several revisions get collapsed to the last one, and a very sparse looking TBPL. Meanwhile the try backlog clears slowly.

I haven't looked at the buildbot 0.7.10 code but I would like to set talos-master02 and friends to chose a m-c build over a try one whenever there is a slave shortage.
Priority: -- → P2
Whiteboard: [buildmasters] → [buildmasters][tryserver]
Priority: P2 → P4
Whiteboard: [buildmasters][tryserver] → [buildmasters][tryserver][q3-goal?]
(Assignee)

Comment 1

7 years ago
catlee suggested enhancing 
 http://hg.mozilla.org/build/buildbot-configs/file/cb9660188d97/talos-r3/master-common.cfg#l43
Assignee: nobody → nrthomas
Priority: P4 → P2
(Assignee)

Comment 2

7 years ago
Created attachment 457486 [details] [diff] [review]
Prioritize branches over try

Works in staging for this scenario:
* try sendchange
* try job on starts on slave
* m-c sendchange
* 2nd try job starts on slave (try sendchange is older)
* reconfig with this patch
* m-c job starts on slave (prioritizeBuilders doing its new thing)
* continues to do m-c jobs
Attachment #457486 - Flags: review?(bhearsum)
Comment on attachment 457486 [details] [diff] [review]
Prioritize branches over try

Should we add this to mozilla/builder_config.py as well, to futureproof that?
Attachment #457486 - Flags: review?(bhearsum) → review+
Just one thought before going live with this: do we want tryserver jobs to always be at the bottom of the pile?  Could we instead give tryserver jobs a time penalty, e.g. leave it as priority 1, but add 30 minutes to the request time.
(Assignee)

Comment 5

7 years ago
Can we do that in a way that doesn't distort a wait time analysis ?
(Assignee)

Comment 6

7 years ago
From IRC, Chris is suggesting a prioritizeBuilders function that looks something like this:
+        elif builder.builder_status.category == 'tryserver':
+            return 1, builder.getOldestRequestTime() + 30*60

I'm not sure what the effect of this would be when there are multiple try changes pending, so that the penalty has effectively expired for the oldest ones. And that approach doesn't really address the basic problem as I see it - that m-c tests which impact on many developers will be merged in favor of try tests which impact a single dev. It kinda works around it in a unpredictable way.

Chris also suggested a simulator to explore what would happen in various regimes. The nerd in me would love this but unfortunately we don't have easy access to a list of sendchanges (eg from scheduler db) for a some busy day, so it's somewhat laborious to reconstruct a validation test.

Overall I think the value of complete test-sets for m-c outweighs any waits people may experience on try, given the relatively large disruption when figuring out which changeset regressed tests on m-c. That's particularly true on freeze days.
(Assignee)

Comment 7

7 years ago
Comment on attachment 457486 [details] [diff] [review]
Prioritize branches over try

Landing this in the hope of getting some better test coverage on m-c on freeze night. http://hg.mozilla.org/build/buildbot-configs/rev/d60b702f8151
Attachment #457486 - Flags: checked-in+
(Assignee)

Comment 8

7 years ago
Reconfig complete on talos-master02 at 21:40; still going on test-master01.
(Assignee)

Comment 9

7 years ago
test-master01 complete at 22:14.

Chris, please reopen if you feel strongly enough that we shouldn't do this.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.