Closed Bug 916564 Opened 9 years ago Closed 8 years ago

JS engine cannot execute large JS scripts with more than 8388608 lines.

Categories

(Core :: JavaScript Engine, defect)

x86_64
Windows 8
defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla29

People

(Reporter: jujjyl, Assigned: luke)

Details

Attachments

(3 files)

Attached file genlargescript.py
User Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36

Steps to reproduce:

To create a test case, run the attached .py script and pipe the output to a .js file, create a .html stub to run the .js file in a <script> tag in the browser.



Actual results:

If a .js script contains more than 2^23 = 8388608 lines, the JS engine aborts with an error 'script too large' and does not run the script.



Expected results:

The script should execute.

Testing other browsers, Chrome and Opera can run scripts with at least 100M lines, and IE can run scripts with 85-90M lines (has a limit somewhere between that range). Firefox aborts already at 8M lines.

This may seem a crazy bug report, i.e. 'who writes such large scripts!?' Compiling very large C++ applications with Emscripten can generate very large JavaScript files, and with asm.js, those scripts are hosted in a single .js module that contains the whole program. The optimized and minified versions of the scripts are of course much smaller than 8M lines, but this limitation is mainly occurring on debug builds, which are deliberately verbose+unoptimized and retain all symbols and newlines for readability.

We are working with a very large codebase that currently has 10M+ lines of .js code in the debug build, but I can imagine even larger codebases, e.g. a port of the Qt library, where debug builds could well be nearing the ~100M ballpark.
Assignee: nobody → general
Status: UNCONFIRMED → NEW
Component: Untriaged → JavaScript Engine
Ever confirmed: true
Product: Firefox → Core
This patch just moves coutnFinalSourceNotes to be next to FinishTakingSrcNotes.  No need to inline this bad boy.  The patch also removes DumpSrcNoteSizeHist() which probably hasn't been used in 15 years.
Assignee: general → luke
Status: NEW → ASSIGNED
Attachment #8361968 - Flags: review?(jorendorff)
Attached patch 4-byteSplinter Review
The limit we're hitting is SN_MAX_OFFSET, which is the max offset we can store in a source note.  The source note in question is SN_SETLINE which is trying to store the line number 100,000,000.  Source notes have a scheme where offsets are either stored as 7-bit integers in 1 byte or 23-bit integers in 3 bytes (depending on the value).  Thus, the simplest fix is just to inflate the latter case to be 31-bit integers in 4 bytes.  This was pretty easy to do; just rename SN_3BYTE_* to SN_4BYTE_* and the compiler points out all the places that need updating.
Attachment #8361969 - Flags: review?(jorendorff)
Attachment #8361968 - Flags: review?(jorendorff) → review+
Attachment #8361969 - Flags: review?(jorendorff) → review+
https://hg.mozilla.org/mozilla-central/rev/04505a54d175
https://hg.mozilla.org/mozilla-central/rev/db641fd01a48
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla29
You need to log in before you can comment on or make changes to this bug.