WINNT 5.2 fx-win32-tbox Dep Nightly on the Firefox tinderbox used to (as of a week ago) cycle every 30 minutes or so; now it cycles every 90 minutes or so. This makes it very hard to see when performance regressions happened, since it slows down the cycles, and it slows down the feedback loop when backing out checkins to see if they were the cause of performance regressions. (Regressions such as bug 385957, which is currently keeping the tree closed.)
Guessing how long a build with a checkin will take that box has always been a mug's game (I've filed at least one invalid bug claiming it was stuck, and been tempted several other times), but something (which might well correspond with something on the VM migration page on the intranet that I can't see) certainly happened late last week: looking at no-checkin builds shortly after midnight, it was taking just over 30 minutes 6/21, 45 minutes 6/22, and 60 minutes since 6/23.
Assignee: server-ops → reed
OS: Linux → All
Hardware: PC → All
I'll take this and re-assign to me, for now, although Coop will probably get the fun. The two time drops probably correlate with the addition of the fxnewref-linux-tbox VM on the same machine (bm-vmware09) and... maybe some other change. There's one last round of "VM migration" stuff, which was related to re-distributing these VMs in useful ways, so we had an optimal balance across the farm.
Assignee: reed → preed
Component: Server Operations: Tinderbox Maintenance → Build & Release
Priority: -- → P1
Four hour cycle times between tests is pretty unbearable. It makes it almost impossible to catch perf. regressions when they happen, not to mention figure out what check-ins caused them. This needs to get fixed ASAP.
Severity: major → blocker
We'll try to move this to another host in the short-term.
Depends on: 386074
We moved the VM to a totally unloaded VM host as a first step; seeing how that goes. Coop's working with IT on this; re-assigning to him.
Assignee: preed → ccooper
The nightly clobber for this VM on an otherwise-empty VM host took 2 hours 30 minutes. Although the final email didn't make it through, the build did get uploaded and is available in latest-trunk/.
Status: NEW → RESOLVED
Last Resolved: 11 years ago
No longer depends on: 386074
Resolution: --- → DUPLICATE
Duplicate of bug: 386074
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.