959158 - Jit-test tests\asm.js\testZOOB.js fails on Windows 8 test machines

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Dan Minor [:dminor]]

This test fails on Cedar for Windows 8 (both opt and debug):

Appears to timeout:
09:58:49  WARNING -  TEST-UNEXPECTED-FAIL | tests\jit-test\jit-test\tests\asm.js\testZOOB.js |
09:58:49     INFO -  INFO exit-status     : -1
09:58:49     INFO -  INFO timed-out       : True
09:58:49     INFO -  INFO stdout          >

Comment 1

[Luke Wagner [:luke]]

Always or intermittently

Comment 2

[Dan Minor [:dminor]]

I'm looking at Cedar which has a low volume of commits, but it appears to have failed consistently since this merge from m-c: https://tbpl.mozilla.org/?tree=Cedar&rev=fee74abdc1e3 which was on Wed. Jan 8.

Comment 3

[Jason Orendorff [:jorendorff]]

Probably a dup of a bug long since closed. Could be anything.

Cc-ing aki, who has Cedar booked indefinitely, according to https://wiki.mozilla.org/ReleaseEngineering/DisposableProjectBranches .

I'm in favor of just closing this.

Comment 4

[Aki Sasaki (not active)]

Cedar is the new build/test baking area for #ateam and #releng.
Currently windows builds are busted: https://bugzilla.mozilla.org/show_bug.cgi?id=940690#c61

It's difficult to know whether the bustage is due to a change that #ateam or #releng made on Cedar, or an actual test bustage.  This will slow down rollout of new tests and features to m-c, though.

Comment 5

[Jason Orendorff [:jorendorff]]

Yeah. Terrence straightened me out about Cedar. This is real and weird, likely a result of the test environment being a little different.

Comment 6

[Terrence Cole [:terrence]]

Does this test run to completion if given free reign? If so, how long does it take? And how does that time compare to its runtime on other platforms?

Comment 7

[Dan Minor [:dminor]]

Frustratingly this doesn't seem to reproduce on my loaner test machine, but does seem to fail consistently in the wild. I'll have to see what I can do using try.

Comment 8

[Dan Minor [:dminor]]

I have a try run here: https://tbpl.mozilla.org/?tree=Try&rev=da83119b547b&showall=1

It appears to fail consistently on opt and intermittently on debug. Is there any useful information in the logs comparing the debug runs where it passes to the debug runs where it fails?

Comment 9

[Luke Wagner [:luke]]

Attachment: disable-caching-in-zoob

Ah, all the individual asm.js compilations (of which there are hundreds) seem to take a really long time (300ms apiece).  On debug OSX they all take <1ms.  Since the compilation time doesn't vary much between debug and optimize, I expect this slowness is from the disk IO resulting from caching.  This patch disables caching in testZOOB (which doesn't care about caching).  Does this fix the timeouts?

Comment 10

[Dan Minor [:dminor]]

(In reply to Luke Wagner [:luke] from comment #9)
> Created attachment 8367950 [details] [diff] [review]
> disable-caching-in-zoob
> 
> Ah, all the individual asm.js compilations (of which there are hundreds)
> seem to take a really long time (300ms apiece).  On debug OSX they all take
> <1ms.  Since the compilation time doesn't vary much between debug and
> optimize, I expect this slowness is from the disk IO resulting from caching.
> This patch disables caching in testZOOB (which doesn't care about caching). 
> Does this fix the timeouts?

The initial results seem positive [1]. I'll retrigger the jobs a few times to see if any intermittents show up.

I also did a try run with the timeout effectively disabled [2]. This also seems to initially work, without a huge impact on overall running time. I've retriggered the jobs there as well. I believe an overall testsuite running time of one hour is enforced at the mozharness level, so this may be solveable by tweaking the timeout value without much risk.

[1] https://tbpl.mozilla.org/?tree=Try&rev=3e0daaab684d&showall=1
[2] https://tbpl.mozilla.org/?tree=Try&rev=e8c1d7a5d1fc&showall=1

Comment 11

[Luke Wagner [:luke]]

The patch doesn't lose any testing potency, so if it solves the problem, we should land that to avoid wasting time, I think.

Comment 12

[Dan Minor [:dminor]]

(In reply to Luke Wagner [:luke] from comment #11)
> The patch doesn't lose any testing potency, so if it solves the problem, we
> should land that to avoid wasting time, I think.

Sounds good to me. Playing with the timeout means we might end up "fixing" this several times as we run on different test machines.

Comment 13

[Dan Minor [:dminor]]

Comment on attachment 8367950 [details] [diff] [review]
disable-caching-in-zoob

Thanks for the patch! I'd be happy to get this landed.

Comment 14

[Terrence Cole [:terrence]]

Comment on attachment 8367950 [details] [diff] [review]
disable-caching-in-zoob

Forwarding review to Luke.

Comment 15

[Dan Minor [:dminor]]

Thanks, pushed to https://hg.mozilla.org/integration/mozilla-inbound/rev/ae57851a4308

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ae57851a4308