Closed Bug 1542292 Opened 7 months ago Closed 6 months ago

Bump 64-bit code quota to 2gb

Categories

(Core :: Javascript: WebAssembly, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: luke, Assigned: luke)

Details

Attachments

(1 file)

It's currently 1gb and we see large wasm apps running into this limit in their debug builds (especially due to tiering creating 2x eager machine code). With the enormous 64-bit virtual address space, there isn't a significant loss in randomization, and we could just as well do 4, 8 or 16gb. There are two relevant limits, both of which could be worked around if need be:

  1. x64 calls/jumps only have a 2gb region, which we in theory don't depend on yet, but might accidentally
  2. RUNTIME_FUNCTION's begin/end offsets are 32-bit DWORDs (which we use for our breakpad hack)

We could work around 1 by only allowing wasm to be allocated above 2gb. We could work around 2 by registering the executable code in 4gb regions (or not doing the breakpad hack and registering individual functions (bug 1530552)).

But for now, I'd like to do the one-liner to bump to 2gb.

Type: defect → enhancement
Assignee: nobody → luke
Pushed by lwagner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/55b2b70ee036
Bump 64-bit executable quota to 2gb (r=jandem)

Backed out changeset 55b2b70ee036 (Bug 1542292) build bustage at builds/worker/workspace/build/src/js/src/jit/ProcessExecutableMemory.h:22:62. On a CLOSED TREE

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&selectedJob=238818034&revision=55b2b70ee03601598418a38be63294b287e9444e

Backout link: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&selectedJob=238818034&revision=ac6e176b391c4003109e0e3d286d021fbf891c22

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=238818034&repo=autoland&lineNumber=642

[task 2019-04-08T15:06:41.462Z] In file included from /builds/worker/workspace/build/src/js/src/jit/ExecutableAllocator.h:47:
[task 2019-04-08T15:06:41.462Z] /builds/worker/workspace/build/src/js/src/jit/ProcessExecutableMemory.h:22:62: error: overflow in expression; result is -2147483648 with type 'int' [-Werror,-Winteger-overflow]
[task 2019-04-08T15:06:41.462Z] static const size_t MaxCodeBytesPerProcess = 2 * 1024 * 1024 * 1024;
[task 2019-04-08T15:06:41.462Z] ^
[task 2019-04-08T15:06:41.470Z] Compiling proc-macro2 v0.4.27
[task 2019-04-08T15:06:41.519Z] Compiling libc v0.2.51
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/builtin/RegExp.cpp:7:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/builtin/RegExp.h:10:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/vm/RegExpObject.h:20:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/vm/ArrayObject.h:10:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/vm/NativeObject.h:22:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/vm/JSObject.h:18:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/vm/Shape.h:24:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/gc/FreeOp.h:12:
[task 2019-04-08T15:06:41.819Z] In file included from /builds/worker/workspace/build/src/js/src/jit/ExecutableAllocator.h:47:
[task 2019-04-08T15:06:41.819Z] /builds/worker/workspace/build/src/js/src/jit/ProcessExecutableMemory.h:22:62: error: overflow in expression; result is -2147483648 with type 'int' [-Werror,-Winteger-overflow]
[task 2019-04-08T15:06:41.819Z] static const size_t MaxCodeBytesPerProcess = 2 * 1024 * 1024 * 1024;
[task 2019-04-08T15:06:41.819Z] ^
[task 2019-04-08T15:06:41.837Z] In file included from /builds/worker/workspace/build/src/js/src/gc/StoreBuffer.cpp:7:
[task 2019-04-08T15:06:41.837Z] In file included from /builds/worker/workspace/build/src/js/src/gc/StoreBuffer-inl.h:15:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/gc/Heap-inl.h:13:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/gc/Zone.h:17:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/vm/Runtime.h:24:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/builtin/AtomicsObject.h:15:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/vm/JSObject.h:18:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/vm/Shape.h:24:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/gc/FreeOp.h:12:
[task 2019-04-08T15:06:41.838Z] In file included from /builds/worker/workspace/build/src/js/src/jit/ExecutableAllocator.h:47:
[task 2019-04-08T15:06:41.838Z] /builds/worker/workspace/build/src/js/src/jit/ProcessExecutableMemory.h:22:62: error: overflow in expression; result is -2147483648 with type 'int' [-Werror,-Winteger-overflow]
[task 2019-04-08T15:06:41.838Z] static const size_t MaxCodeBytesPerProcess = 2 * 1024 * 1024 * 1024;
[task 2019-04-08T15:06:41.838Z] ^
[task 2019-04-08T15:06:41.946Z] Compiling proc-macro2 v0.4.27
[task 2019-04-08T15:06:41.989Z] Compiling unicode-xid v0.1.0
[task 2019-04-08T15:06:42.094Z] Compiling syn v0.15.30
[task 2019-04-08T15:06:42.460Z] 1 error generated.
[task 2019-04-08T15:06:42.462Z] Compiling siphasher v0.2.1
[task 2019-04-08T15:06:42.466Z] /builds/worker/workspace/build/src/config/rules.mk:805: recipe for target 'BinASTParser.o' failed
[task 2019-04-08T15:06:42.466Z] make[3]: *** [BinASTParser.o] Error 1

Flags: needinfo?(luke)
Flags: needinfo?(luke)
Pushed by lwagner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9a6c6c9c8119
Bump 64-bit executable quota to 2gb (r=jandem)
Backout by csabou@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/22602954561c
Backed out changeset 9a6c6c9c8119 for causing build bustages. CLOSED TREE

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=busted&fromchange=9a6c6c9c81199563416ffe9795b76c382d736f41&tochange=2b272977a1e5267db194b3dc33fe4f240a226dd6&selectedJob=239058419

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239064907&repo=autoland&lineNumber=38673

Backout link: https://hg.mozilla.org/integration/autoland/rev/22602954561c86b58ab9fde7a019b361aba7080a

14:17:11 INFO - mozmake.EXE[1]: Leaving directory 'z:/task_1554815405/build/src/obj-firefox'
14:17:13 INFO - mozmake.EXE[1]: Entering directory 'z:/task_1554815405/build/src/obj-firefox'
14:17:13 INFO - JARLOG_FILE=jarlog/en-US.log z:/task_1554815405/build/src/obj-firefox/_virtualenvs/init/Scripts/python.exe z:/task_1554815405/build/src/build/pgo/profileserver.py
14:17:13 INFO - Firefox exited with code 1 during profile initialization
14:17:13 INFO - Firefox output (Z:\task_1554815405\public\build\profile-run-1.log):
14:17:13 INFO - Makefile:201: recipe for target 'profiledbuild' failed
14:17:13 INFO - mozmake.EXE[1]: *** [profiledbuild] Error 1
14:17:13 INFO - mozmake.EXE[1]: Leaving directory 'z:/task_1554815405/build/src/obj-firefox'
14:17:13 INFO - client.mk:125: recipe for target 'build' failed
14:17:13 INFO - mozmake.EXE: *** [build] Error 2
14:17:13 INFO - 238 compiler warnings present.
14:17:13 ERROR - Return code: 2
14:17:13 WARNING - setting return code to 2
14:17:13 FATAL - 'mach build -v' did not run successfully. Please check log for errors.
14:17:13 FATAL - Running post_fatal callback...
14:17:13 FATAL - Exiting -1
14:17:13 INFO - [mozharness: 2019-04-09 14:17:13.267000Z] Finished build step (failed)
14:17:13 INFO - Running post-run listener: _parse_build_tests_ccov
14:17:13 INFO - Running post-run listener: _shutdown_sccache
14:17:13 INFO - Running post-run listener: _summarize
14:17:13 ERROR - # TBPL FAILURE #
14:17:13 INFO - [mozharness: 2019-04-09 14:17:13.267000Z] FxDesktopBuild summary:
14:17:13 ERROR - # TBPL FAILURE #

Flags: needinfo?(luke)

Arg; I even try-servered this and assumed these failures were intermittent or perma-red try-only failures because they look like they're infra failures.

Flags: needinfo?(luke)
Component: JavaScript Engine: JIT → Javascript: WebAssembly
Priority: -- → P1

Ok, I finally found the actual cause, buried in the logs:

14:22:28     INFO - E   PID 3284 | Unable to load \\untrusted-startup-test-dll.dll; LoadLibraryW failed: 126Hit MOZ_CRASH(js::jit::InitProcessExecutableMemory() failed) at z:/build/build/src/xpcom/build/XPCOMInit.cpp:442

I made a local Win64 build which seems to work, so I guess we're simply failing to allocate (maybe it's a testing VM limitation). On a suspicion that the limitation was at 2gb, I did a try push for a little less than 2gb which seems to succeed; so I'll just do that. If we wanted to relax this later, we could perhaps allocate up to MaxCodeBytesPerProcess but fall back to a lesser allocation.

Oops, n/m, I had built 32-bit Windows. On 64-bit, I can repro and the failure comes from RtlInstallFunctionTableCallback(). In theory this takes a DWORD which the docs say is an uint32_t, but I guess there is an undocumented requirement that it's <= INT32_MAX. Oh well, at least now I have a better reason to record in a comment.

Attachment #9056149 - Attachment description: Bug 1542292 - Bump 64-bit executable quota to 2gb (r=jandem) → Bug 1542292 - Bump 64-bit executable quota to ~2gb (r=jandem)
Pushed by lwagner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7beb238a0ea0
Bump 64-bit executable quota to ~2gb (r=jandem)
Status: NEW → RESOLVED
Closed: 6 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
You need to log in before you can comment on or make changes to this bug.