772458 - Try is extremely backed up

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Wait times for tests are closing in on 24 hrs.

Seems that Android and Windows are more backed up than the other platforms, but that's just anecdotal.

Comment 1

[Mounir Lamouri (:mounir)]

Do we have any idea of what is happening? Do people push more to try those days than before? Do we simply need more slaves?

It seems to be a critical issue for engineering: pushing to try takes a so ridiculous amount of time that it will whether reduce productivity or people will just push to m-i without waiting for full results.

Comment 2

[Ed Morley [:emorley]]

(In reply to Mounir Lamouri (:mounir) from comment #1)
> or people
> will just push to m-i without waiting for full results.

Which has already happened several times this week; with the ensuing layers of bustage made worse by high infra load on non-try trees and the coalescing that brings :-(

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Mounir Lamouri (:mounir) from comment #1)
> Do we have any idea of what is happening? Do people push more to try those
> days than before? Do we simply need more slaves?
> 
> It seems to be a critical issue for engineering: pushing to try takes a so
> ridiculous amount of time that it will whether reduce productivity or people
> will just push to m-i without waiting for full results.

One of the issues with test pool capacity is that *all* current tests run on Mac minis of various vintages. This is due to a historical notion that we wanted to be able to compare test results between different platforms/OSes on the same hardware. Apple's hardware rev cycle is aggressive, so we simply can't buy any more of these older rev minis any more. The existing pool capacity is static, modulo attrition via hardware failure. We can create some extra capacity on one platform only at the expense of another, e.g. stopping tests on 10.5 (bug 773120).

We no longer think these inter-platform comparisons are meaningful. Releng is extremely resource-constrained at present for setting up new hardware. We have an effort underway to refresh our test pool to newer non-Mac hardware (for non-Mac OSes), but this is blocked by getting test coverage setup for Win8 (bug 731280) and Mountain Lion (10.8) (bug 731278), platforms where we currently have no coverage at all.

Comment 4

[Ed Morley [:emorley]]

We've just had 38 consecutive pushes (81 changesets) of bustage on inbound, since Try results are taking so long to come back, that people are pushing regardless. 

Are there any other quick wins that we can do here? eg: disabling platforms/tests on twigs that could do without them; or re-balancing the try vs non-try buildpool?

Looking at http://build.mozilla.org/builds/pending/pending.html shows that the Try linux compile pending count is always an order of magnitude higher than the others. Can we spare some more non-try linux builds for Try? The graph for non-try would imply there is capacity going unused that could be switched over perhaps?

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

bug#750285, bug#777037 track disabling a bunch of android builds/unittest/talos jobs which will help reduce android load. This is an interim solution while we wait for additional tegras to come online.

Comment 6

[Patrick McManus [:mcmanus]]

If this bug and its dependents were resolved fix, what would the expected turn around time for try be?

try used to a tremendously useful development tool. Now its pretty much just a pain.

Comment 7

[Jared Wein [:jaws] (please needinfo? me)]

We can turn off tests for the UX branch (https://tbpl.mozilla.org/?tree=UX). I've been maintaining the branch for the past N months (doing daily merges between m-c and ux).

Devs sending their patch to UX branch can just run it through try server first and in total that will save some build resources since we have vastly more merges between m-c to ux then we do have checkins to ux.

It should also be noted that ux is a dead-end branch, which doesn't feed anywhere but is used for functional testing of new ux features.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Now that we're running android/b2g/nativefennec builds over on AWS, we're freeing up cycles on our linux32/linux64 machines. 

bug#784891 tracks converting a bunch of existing linux32/linux64/win32 physical ix builders into win64 builders. This will improve wait times for windows builds in both the production build pool and try build pool.

Comment 9

[Ed Morley [:emorley]]

(In reply to Jared Wein [:jaws] from comment #7)
> We can turn off tests for the UX branch (https://tbpl.mozilla.org/?tree=UX).

Done in bug 779419.

Comment 10

[[e-mail] (:bhearsum)]

This isn't an acute issue, and thus not a buildduty concern.

Comment 11

[Chris AtLee [:catlee]]

How are try turn around times these days?

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

:khuey: per dev.tree-management, we've been hitting consistently great wait times on builds and tests, across the board including Try. This is thanks to moving more jobs to AWS, reshuffling existing hardware inhouse, and turning off broken builds/tests. 

Any objections to closing this as FIXED?

Comment 13

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Yeah I think we're doing pretty well these days.  Someone else can file a new bug if they have current issues.

Good job folks.