997692 - Try Backlog of 10.8 Tests

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

Seems we have a huge backlog of tests on 10.8 on try since serveral hours. Would be great if someone could check into this whats wrong

Comment 1

[Armen [:armenzg]]

It is the natural state of affairs.
There are not enough machines for that specific pool.

There's only one out of action (bottom of page):
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slavetype.html?class=test&type=talos-mtnlion-r5

Comment 2

[Phil Ringnalda (:philor)]

There's my initial shot at it, in the dependency tree.

What we have now is no longer the natural state of affairs, which was a peak at 10-12 hours and catching up overnight most nights, but instead a new natural state of ranging between 10-12 hours overnight and up to 20 hours during the day, a change which seems to have happened around the time that we added cpp and jittest, got horribly slow at running browser-chrome, and as a result wound up splitting browser-chrome into three bcs and one (for opt) to three (for debug) dts, getting faster end-to-end at the price of five times the overhead per push.

There are also social changes we need, primarily persuading people to kill their try runs as soon as it becomes obvious they are broken and persuading people (mainly but not exclusively b2g developers) to not do -p all -u all when their push clearly only affects a single product, but those aren't something I can file and cc someone who will fix them.

Comment 3

[Armen [:armenzg]]

Any effort to improve this will take too much time to make a significant improvement.
Replacing this pool with the always-upcoming 10.9 test pool is the right way.