1445922 - Intermittent z:\build\build\src\third_party\aom\aom_dsp\simd\v64_intrinsics_c.h(754) : fatal error C1002: compiler is out of heap space in pass 2

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ebalazs [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=168148242&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/XapiraiXQWm1Ix7jX29VJw/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Intermittent Failures Robot]

11 failures in 814 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 7
* autoland: 4

Platform breakdown:
* windows2012-32: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1445922&startday=2018-03-12&endday=2018-03-18&tree=all

Comment 2

[(Away)]

From bug 1312238 comment 38

> 12:22:13     INFO - 
> z:\build\build\src\third_party\aom\aom_dsp\simd\v64_intrinsics_c.h(719) :
> fatal error C1002: compiler is out of heap space in pass 2
> 
> Since this is in AOM code, I suspect it is due to the large function sizes
> seen in bug 1412889. It is fixed upstream but our efforts to update (bug
> 1445683) have hit roadblocks.
> 
> If this is blocking you, we can probably just disable PGO in the affected
> code (AOM won't be hit in a profile anyway).

Comment 3

[(Away)]

Oops, I mean bug 1412238 comment 38

Comment 4

[(Away)]

Alternatively we could cherry pick https://aomedia-review.googlesource.com/c/aom/+/39401

and https://chromium-review.googlesource.com/c/webm/libvpx/+/841103 for good measure

Comment 5

[Intermittent Failures Robot]

36 failures in 822 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #50 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 23
* mozilla-inbound: 12
* try: 1

Platform breakdown:
* windows2012-32: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445922&startday=2018-03-20&endday=2018-03-26&tree=trunk

Comment 6

[Intermittent Failures Robot]

28 failures in 718 pushes (0.039 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 15
* mozilla-inbound: 13

Platform breakdown:
* windows2012-32: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445922&startday=2018-03-26&endday=2018-04-01&tree=trunk

Comment 7

[Noemi Erli[:noemi_erli]]

There are 30 failures in the past 7 days, all occurrences happened on windows2012-32 pgo.
Recent log failure: 
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=171998012&lineNumber=37516

Relevant part of the log: 
02:04:15     INFO -  z:\build\build\src\third_party\aom\aom_dsp\simd\v64_intrinsics_c.h(754) : fatal error C1002: compiler is out of heap space in pass 2
02:04:15     INFO -  z:\build\build\src\third_party\aom\aom_dsp\simd\v256_intrinsics_c.h(101) : fatal error C1002: compiler is out of heap space in pass 2
02:04:15     INFO -  z:\build\build\src\third_party\aom\aom_dsp\simd\v64_intrinsics_c.h(719) : fatal error C1002: compiler is out of heap space in pass 2
02:04:15     INFO -  z:\build\build\src\third_party\aom\av1\common\cdef_block_simd.h(252) : fatal error C1002: compiler is out of heap space in pass 2
02:04:15     INFO -  z:\build\build\src\third_party\aom\aom_dsp\simd\v64_intrinsics_c.h(271) : fatal error C1002: compiler is out of heap space in pass 2
02:04:15     INFO -  LINK : fatal error LNK1257: code generation failed
02:04:15     INFO -  z:\build\build\src\third_party\aom\aom_dsp\simd\v128_intrinsics_c.h(83) : fatal error C1002: compiler is out of heap space in pass 2
02:04:15     INFO -  z:/build/build/src/config/rules.mk:679: recipe for target 'xul.dll' failed
02:04:15     INFO -  mozmake.EXE[5]: *** [xul.dll] Error 1257
02:04:15     INFO -  mozmake.EXE[5]: Leaving directory 'z:/build/build/src/obj-firefox/toolkit/library'
02:04:15     INFO -  z:/build/build/src/config/recurse.mk:73: recipe for target 'toolkit/library/target' failed
02:04:15     INFO -  mozmake.EXE[4]: *** [toolkit/library/target] Error 2
02:04:15     INFO -  z:/build/build/src/config/recurse.mk:32: recipe for target 'compile' failed
02:04:15     INFO -  mozmake.EXE[3]: *** [compile] Error 2
02:04:15     INFO -  z:/build/build/src/config/rules.mk:418: recipe for target 'default' failed
02:04:15     INFO -  mozmake.EXE[2]: *** [default] Error 2
02:04:15     INFO -  Makefile:237: recipe for target 'profiledbuild' failed
02:04:15     INFO -  mozmake.EXE[1]: *** [profiledbuild] Error 2
02:04:15     INFO -  client.mk:168: recipe for target 'build' failed
02:04:15     INFO -  mozmake.EXE: *** [build] Error 2
02:04:15     INFO -  125 compiler warnings present.
02:04:15    ERROR - Return code: 2
02:04:15  WARNING - setting return code to 2
02:04:15    FATAL - 'mach build' did not run successfully. Please check log for errors.
02:04:15    FATAL - Running post_fatal callback...
02:04:15    FATAL - Exiting -1
02:04:15     INFO - [mozharness: 2018-04-05 02:04:15.668000Z] Finished build step (failed)
02:04:15     INFO - Running post-run listener: _summarize
02:04:15     INFO - [mozharness: 2018-04-05 02:04:15.668000Z] FxDesktopBuild summary:
02:04:15     INFO - Running post-run listener: copy_logs_to_upload_dir
02:04:15     INFO - Copying logs to upload dir...
02:04:15     INFO - mkdir: z:\build\build\upload\logs
[taskcluster:error]    Exit Code: 4294967295
[taskcluster:error]    User Time: 0s
[taskcluster:error]  Kernel Time: 15.625ms
[taskcluster:error]    Wall Time: 1h4m35.327078s
[taskcluster:error]       Result: FAILED
[taskcluster 2018-04-05T02:04:15.883Z] === Task Finished ===
[taskcluster 2018-04-05T02:04:15.883Z] Task Duration: 1h11m23.565004s
[taskcluster:error] Uploading error artifact public/build from file public/build with message "Could not read directory 'Z:\\task_1522887787\\public\\build'", reason "file-missing-on-worker" and expiry 2019-04-05T00:51:41.733Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'Z:\task_1522887787\public\build'
[taskcluster 2018-04-05T02:04:16.735Z] Uploading artifact public/logs/certified.log from file generic-worker\certified.log with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-04-05T00:51:41.733Z
[taskcluster 2018-04-05T02:04:18.260Z] Uploading artifact public/chainOfTrust.json.asc from file generic-worker\chainOfTrust.json.asc with content encoding "gzip", mime type "text/plain; charset=utf-8" and expiry 2019-04-05T00:51:41.733Z
[taskcluster 2018-04-05T02:04:18.977Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/GnRhnrFCSQ6NrgY2l5eB8w/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2019-04-05T00:51:41.733Z
[taskcluster:error] Task not successful due to following exception(s):
[taskcluster:error] Exception 1)
[taskcluster:error] exit status 4294967295
[taskcluster:error] Exception 2)
[taskcluster:error] file-missing-on-worker: Could not read directory 'Z:\task_1522887787\public\build'
[taskcluster:error] 
:drno Can you please take a look here?

Comment 8

[(Away)]

(In reply to David Major [:dmajor] from comment #4)
> Alternatively we could cherry pick
> https://aomedia-review.googlesource.com/c/aom/+/39401
> 
> and https://chromium-review.googlesource.com/c/webm/libvpx/+/841103 for good measure

Drno, what do you think?

Comment 9

[Intermittent Failures Robot]

36 failures in 765 pushes (0.047 failures/push) were associated with this bug in the last 7 days. 

This is the #49 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 18
* mozilla-inbound: 17
* try: 1

Platform breakdown:
* windows2012-32: 36

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445922&startday=2018-04-02&endday=2018-04-08&tree=trunk

Comment 10

[Nils Ohlmeier [:drno]]

(In reply to David Major [:dmajor] from comment #8)
> (In reply to David Major [:dmajor] from comment #4)
> > Alternatively we could cherry pick
> > https://aomedia-review.googlesource.com/c/aom/+/39401
> > 
> > and https://chromium-review.googlesource.com/c/webm/libvpx/+/841103 for good measure
> 
> Drno, what do you think?

So the alternatives are:
- either disable PGO on win32 for this code
- wait for the libaom update to land in bug 1445683
- or cherry pick build fixes from upstream

Since bug 1445683 might still be a little bit out I guess either disable PGO or cherry pick. I don't have a preference for either. David feel free to chose either one.

Comment 11

[(Away)]

I tried cherry-picking those two fixes and my try push still hit this intermittent.

Code outside of aom is failing too: 
15:24:55     INFO -  z:\build\build\src\js\src\vm\interpreter.cpp(4339) : fatal error C1002: compiler is out of heap space in pass 2

I'm starting to think this is less about particular functions and more about xul.dll simply growing larger by the day.

Can we add more memory to the win32 pgo builders?

Comment 12

[Intermittent Failures Robot]

15 failures in 150 pushes (0.1 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 12
* autoland: 3

Platform breakdown:
* windows2012-32: 13
* windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445922&startday=2018-04-10&endday=2018-04-10&tree=trunk

Comment 13

[Phil Ringnalda (:philor)]

(In reply to David Major [:dmajor] from comment #11)
> Can we add more memory to the win32 pgo builders?

If what I looked up was up-to-date, we're building Windows on c4.4xlarge AWS instances, so the next step up is c4.8xlarge, at slightly more than twice the price because screw you. I don't know how much we're spending on Windows builds, but every number I've ever heard about our AWS spend has turned another chunk of my hair white, so I doubt that would sound like a good investment.

OTOH, because my sheriff colleagues are madmen and madwomen, very often the response to hitting this intermittent is to retrigger 5 builds, thus triggering 5 sets of tests, so it might be a close calculation to determine which would be cheaper, based on how frequently this hits, how frequently we do that, how much we could reduce the frequency of over-retriggering, and how much it would cost us in merged-around bustage to say "please stop retriggering PGO builds more than once" and wind up getting fewer retriggers of permaorange tests as a result.

Or, you know, we could just disable PGO for code that won't wind up being optimized anyway, and wind up spending *less* and failing less.

Comment 14

[Phil Ringnalda (:philor)]

As a little measure of frequency, of the last 12 Win32 PGO builds to finish on mozilla-inbound, 11 failed this way in AOM code, and 1 hit an infra failure.

Comment 15

[Phil Ringnalda (:philor)]

Oh, nevermind that frequency measure, that's just because

(In reply to David Major [:dmajor] from comment #11)
> I tried cherry-picking those two fixes and my try push still hit this
> intermittent.

Actually, they make this very nearly permanent.

Comment 16

[(Away)]

(In reply to Phil Ringnalda (:philor) from comment #13)
Ok, ok -- I wasn't aware of the configuration and pricing situation.

Comment 17

[(Away)]

Attachment: Bug 1445922: Disable PGO for Win32 libaom due to compiler OOMs.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c4e17dd68065
Disable PGO for Win32 libaom due to compiler OOMs. r=froydnj

Comment 19

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/c4e17dd68065

Comment 20

[Intermittent Failures Robot]

23 failures in 792 pushes (0.029 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 15
* autoland: 8

Platform breakdown:
* windows2012-32: 21
* windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1445922&startday=2018-04-09&endday=2018-04-15&tree=trunk

Comment 21

[Ionuț Goldan [:igoldan]]

This improved build times on Windows 7.

== Change summary for alert #12663 (as of Wed, 11 Apr 2018 21:37:14 GMT) ==

Improvements:

 11%  build times windows2012-32 pgo taskcluster-c4.4xlarge     4,682.35 -> 4,173.92

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=12663