Windows OPT builds on the tinderbox and nightlies have been timing out for a while ('command timed out: 10800 seconds without output'). What is causing this? Is the timeout just too low, or is there something that is taking a lot longer to compile than on trunk? Would disabling PGO (as is the case on m-c) help?
Over to RelEng
Assignee: server-ops-releng → nobody
Component: Server Operations: RelEng → Release Engineering
QA Contact: zandr → release
Here's an example: (http://tinderbox.mozilla.org/showlog.cgi?log=Jaegermonkey/1307380082.1307395383.27413.gz) OBJDIR=obj-firefox python obj-firefox/_profile/pgo/profileserver.py args: ['e:\\builds\\moz2_slave\\jm-w32\\build\\obj-firefox\\dist\\firefox\\firefox.exe', '-no-remote', '-profile', 'e:\\builds\\moz2_slave\\jm-w32\\build\\obj-firefox\\_profile\\pgo\\pgoprofile/', 'http://localhost:8888/index.html'] INFO | automation.py | Application pid: 1888 command timed out: 10800 seconds without output program finished with exit code 1 Looks to me like the builds are hanging on launch, which is likely a code issue since we don't hit it on other branches. I've started another build on 334428e1d5aa and will see if there's anything to be seen via VNC.
Created attachment 537683 [details] Screenshot of hung firefox.exe Sitting there chewing up one core's worth of CPU time; memory usage is static. Since it's not publishing the bits to ftp.m.o do you want a copy of the objdir for debugging ?
Created attachment 537688 [details] Stack trace for rev 334428e1d5aa From following https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg objdir coming up ...
Ah, this stack may actually have been fallout from orange on that changeset which showed up on other platforms. How can I get the objdir for other changesets, e.g. 1b5429edb553 when it comes up (this fixes the problem in ExpandInlineFrames).
If we get a hang on 1b5429edb553 (or whatever) then I can rebuild and catch it in the act. Do you still want the objdir from 334428e1d5aa ?
No, I think waiting and seeing what 1b5429edb553 does should be fine.
Attachment #537688 - Attachment description: Stack trace → Stack trace for rev 334428e1d5aa
Created attachment 537701 [details] Stack trace on rev 1b5429edb553 Hung again on 1b5429edb553. Do you have a windows box you could be debugging this on ?
Yeah, I have a Windows VM which should work (though I haven't debugged Windows opt problems for a while).
Assigning to Brian since he's debugging. If you need further assistance after debugging, please ask in the bug and/or unassign yourself.
Assignee: nobody → bhackett1024
I still need the objdir from 1b5429edb553. FWIW, this isn't as critical now because disabling PGO in the jaegermonkey branch (via a merge from m-c) stopped the hang, but it would still be good to know where PGO is (presumably) miscompiling.
The clobber set by yourself at 2011-06-07 07:11:47 PDT removed that objdir.
OS: Mac OS X → Windows Server 2003
Product: mozilla.org → Core
QA Contact: release → general
Version: other → Trunk
Brian, I don't know if you saw, but the hung stack in comment 9 seems to be pcQuadratic called from ExpandInlineFrames.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.