385972 - WINNT 5.2 fx-win32-tbox Dep Nightly has gotten much slower

Status: RESOLVED DUPLICATE of bug 386074

(Release Engineering :: General, defect, P1)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

WINNT 5.2 fx-win32-tbox Dep Nightly on the Firefox tinderbox used to (as of a week ago) cycle every 30 minutes or so; now it cycles every 90 minutes or so.  This makes it very hard to see when performance regressions happened, since it slows down the cycles, and it slows down the feedback loop when backing out checkins to see if they were the cause of performance regressions.  (Regressions such as bug 385957, which is currently keeping the tree closed.)

Comment 1

[Phil Ringnalda (:philor)]

Guessing how long a build with a checkin will take that box has always been a mug's game (I've filed at least one invalid bug claiming it was stuck, and been tempted several other times), but something (which might well correspond with something on the VM migration page on the intranet that I can't see) certainly happened late last week: looking at no-checkin builds shortly after midnight, it was taking just over 30 minutes 6/21, 45 minutes 6/22, and 60 minutes since 6/23.

Comment 2

[J. Paul Reed [:preed]]

I'll take this and re-assign to me, for now, although Coop will probably get the fun.

The two time drops probably correlate with the addition of the fxnewref-linux-tbox VM on the same machine (bm-vmware09) and... maybe some other change.

There's one last round of "VM migration" stuff, which was related to re-distributing these VMs in useful ways, so we had an optimal balance across the farm.

Comment 3

[Adam Guthrie]

Four hour cycle times between tests is pretty unbearable. It makes it almost impossible to catch perf. regressions when they happen, not to mention figure out what check-ins caused them. This needs to get fixed ASAP.

Comment 4

[Chris Cooper [:coop] (he/him)]

We'll try to move this to another host in the short-term.

Comment 5

[J. Paul Reed [:preed]]

We moved the VM to a totally unloaded VM host as a first step; seeing how that goes.

Coop's working with IT on this; re-assigning to him.

Comment 6

[Chris Cooper [:coop] (he/him)]

The nightly clobber for this VM on an otherwise-empty VM host took 2 hours 30 minutes. 

Although the final email didn't make it through, the build did get uploaded and is available in latest-trunk/.