1354273 - Intermittent tests\jit-test\jit-test\tests\debug\bug-1248162.js | Assertion failure: [unhandlable oom] Failed to allocate slots while tenuring., at z:/build/build/src/js/src/jscntxt.cpp:1504 (code -2147483645, args "--no-baseline --no-ion")

Status: RESOLVED FIXED

(Core :: JavaScript Engine: JIT, defect)

Description

[Treeherder Bug Filer]

Filed by: wkocher [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=89174418&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/Nu8_jPmJT1mKU7ZuhyTm_g/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

These failures begin with the landing of bug 1333003:

https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=windows%20vm%20debug%20tc%28jit%29&fromchange=7632e3ba826252942ead773e3f70bcec0168f88e&tochange=060bca004d791980ebe510654efb0cab0760f1dc

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

And with fail, I mean permafail on Windows 7 VM debug tc(Jit).

Comment 3

[Intermittent Failures Robot]

14 failures in 867 pushes (0.016 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 6
* mozilla-central: 6
* graphics: 1
* autoland: 1

Platform breakdown:
* windows7-32-vm: 14

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354273&startday=2017-04-03&endday=2017-04-09&tree=all

Comment 4

[Ting-Yu Chou [:ting] (away)]

Similar to bug 1354306, there's a patch https://hg.mozilla.org/mozilla-central/rev/9ab7778f16d9 in bug 1333003 to fix Jit test which did *not* run any test case on taskcluster. The patch just makes the tests to be run, which I don't think is the root cause of the intermittent failure.

Comment 5

[Geoff Brown [:gbrown]]

In this bug and in bug 1354306, we now have frequent failures in jit tests on Windows 7 VM debug, but only on tc. These tests run for Windows 7 VM debug on buildbot without any sign of failure. 

Failures began with :ting's push because his changes started running those tests on tc(jit).

We'll need to fix or disable the affected tests soon.

:h4writer -- Can you find someone to investigate the failures?

:jmaher -- Would you expect differences between Windows 7 VM tc and non-tc environments? Who should we loop in?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

:grenade- we have an example of a specific tc only failure here- can you help provide information about the differences in the OS for buildbot VM vs taskcluster VM, likewise any operating system differences?

Comment 7

[Intermittent Failures Robot]

35 failures in 894 pushes (0.039 failures/push) were associated with this bug in the last 7 days. 

This is the #48 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 22
* mozilla-central: 6
* autoland: 6
* graphics: 1

Platform breakdown:
* windows7-32-vm: 35

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354273&startday=2017-04-10&endday=2017-04-16&tree=all

Comment 8

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Main differences are:

- tc win 7 tests run under GenericWorker user account, which is dynamically created on instance first boot. if something needs
  to exist in the user profile (like registry settings or file-system objects) we must also create those at first boot. under
  buildbot, tests run under the cltbld account which is created by GPO on top of an MDT image where some profile objects were
  historically, manually created and undocumented (or documentation lost). its entirely possible we've omitted some aspect of
  the user profile on tc test instances.

- buildbot test slaves use an older version of python (2.7.3, i believe). tc uses 2.7.11 we used to see problems caused by this
  difference, but i thought most had been resolved (or maybe bb instances were upgraded, i'm out of the loop there)

- pmoore did a lot of work to understand and fix problems around python subprocess creation, which was problematic on tc due to
  our use of go code to parent the build processes. i know very little about how that all works, but it used to be a big source 
  of pain in test output differences

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

:naveed, can you help find someone to look at this bug?  between this bug and bug 1354076, we have 54 failures in the last week.

Comment 10

[Jon Coppeard (:jonco)]

(In reply to Rob Thijssen (:grenade - CEST) from comment #8)
This looks like OOM.  Are the memory configuration differences between builtbot and taskcluster?  

For reference this test takes ~520MiB max RSS on linux64 (I don't have access to Windows to test on right now).

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

odd  that we get OOM only in taskcluster, maybe there are settings at the OS level or user shell that cause this.

Comment 12

[Jon Coppeard (:jonco)]

Attachment: bug1354273-oomtest-gc

I looked at why this test was taking so much memory.  Ironically enough it's an OOM test.  When we do something like |oomTest(newGlobal)| we partially create over a thousand globals which takes up a lot of memory.  This doesn't trip any of our GC triggers however.  I think this is because they're based on things like total memory used, not number of compartments.

We should investigate why this is, but for now let's just force a GC if our OOM testing creates too many compartments.

This patch reduces the memory needed by the test to ~40MiB.  My patches in bug 1352430 make this test permafail, but pass with this patch applied.

Comment 13

[Jon Coppeard (:jonco)]

I filed bug 1358435 for the followup investigation.

Comment 14

[Rob Thijssen [:grenade (EET/UTC+0300)]]

win 7 instances are limited to 3gb ram because of the 32 bit os limitation. but this is the same between bb and tc. i'm not aware of any other memory configuration or difference.

Comment 15

[Steve Fink [:sfink] [:s:]]

Comment on attachment 8860322 [details] [diff] [review]
bug1354273-oomtest-gc

Review of attachment 8860322 [details] [diff] [review]:
-----------------------------------------------------------------

Seems fine as a stopgap, though it also makes me wonder if we *should* persistently account for some things like this and use them as additional GC triggers. I wouldn't expect this to happen much at all with normal sites, but could happen with buggy JS that creates too many iframes or something, and could easily be a malicious denial of service attack. But that last is not all that convincing; I'm sure there are plenty of ways to DOS the web platform.

Comment 16

[Intermittent Failures Robot]

33 failures in 817 pushes (0.04 failures/push) were associated with this bug in the last 7 days. 

This is the #29 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 19
* mozilla-central: 6
* autoland: 6
* graphics: 2

Platform breakdown:
* windows7-32-vm: 33

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354273&startday=2017-04-17&endday=2017-04-23&tree=all

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6f40efd60fcb
GC during OOMTest if we create too many compartments r=sfink

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/6f40efd60fcb