662283 - Windows OPT timeouts in Jaegermonkey branch

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine, defect)

Description

[Brian Hackett [Laid off!]]

Windows OPT builds on the tinderbox and nightlies have been timing out for a while ('command timed out: 10800 seconds without output').  What is causing this?  Is the timeout just too low, or is there something that is taking a lot longer to compile than on trunk?  Would disabling PGO (as is the case on m-c) help?

Comment 1

[Zandr Milewski [:zandr]]

Over to RelEng

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Here's an example:
(http://tinderbox.mozilla.org/showlog.cgi?log=Jaegermonkey/1307380082.1307395383.27413.gz)

OBJDIR=obj-firefox python obj-firefox/_profile/pgo/profileserver.py
args: ['e:\\builds\\moz2_slave\\jm-w32\\build\\obj-firefox\\dist\\firefox\\firefox.exe', '-no-remote', '-profile', 'e:\\builds\\moz2_slave\\jm-w32\\build\\obj-firefox\\_profile\\pgo\\pgoprofile/', 'http://localhost:8888/index.html']
INFO | automation.py | Application pid: 1888

command timed out: 10800 seconds without output
program finished with exit code 1

Looks to me like the builds are hanging on launch, which is likely a code issue since we don't hit it on other branches. I've started another build on 334428e1d5aa and will see if there's anything to be seen via VNC.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Screenshot of hung firefox.exe

Sitting there chewing up one core's worth of CPU time; memory usage is static. Since it's not publishing the bits to ftp.m.o do you want a copy of the objdir for debugging ?

Comment 4

[Brian Hackett [Laid off!]]

Sure.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Stack trace for rev 334428e1d5aa

From following https://developer.mozilla.org/en/How_to_get_a_stacktrace_with_WinDbg

objdir coming up ...

Comment 6

[Brian Hackett [Laid off!]]

Ah, this stack may actually have been fallout from orange on that changeset which showed up on other platforms.  How can I get the objdir for other changesets, e.g. 1b5429edb553 when it comes up (this fixes the problem in ExpandInlineFrames).

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

If we get a hang on 1b5429edb553 (or whatever) then I can rebuild and catch it in the act. Do you still want the objdir from 334428e1d5aa ?

Comment 8

[Brian Hackett [Laid off!]]

No, I think waiting and seeing what 1b5429edb553 does should be fine.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Stack trace on rev 1b5429edb553

Hung again on 1b5429edb553. Do you have a windows box you could be debugging this on ?

Comment 10

[Brian Hackett [Laid off!]]

Yeah, I have a Windows VM which should work (though I haven't debugged Windows opt problems for a while).

Comment 11

[Aki Sasaki (not active)]

Assigning to Brian since he's debugging.

If you need further assistance after debugging, please ask in the bug and/or unassign yourself.

Comment 12

[Brian Hackett [Laid off!]]

I still need the objdir from 1b5429edb553.  FWIW, this isn't as critical now because disabling PGO in the jaegermonkey branch (via a merge from m-c) stopped the hang, but it would still be good to know where PGO is (presumably) miscompiling.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

The clobber set by yourself at 2011-06-07 07:11:47 PDT removed that objdir.

Comment 14

[Luke Wagner [:luke]]

Brian, I don't know if you saw, but the hung stack in comment 9 seems to be pcQuadratic called from ExpandInlineFrames.