920765 - Reduce analysis memory usage

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Steve Fink [:sfink] [:s:]]

The analysis is failing on build slaves because it uses so much memory it starts swapping, and gets killed after an hour. (This is on a job that should take maybe 6 minutes or so.)

The main problem seems to be JS_Init(), which contains a loop for every entry in js.msg. It appears that sixgill creates a new "body" for every loop in a function, and sadly that body seems to contain a reference to every temporary anywhere in the function. In other words, the CFG for JS_Init contains n^2 references to temporaries. This requires about 1.25GB of RAM.

I don't want to spend the time now to fix this in sixgill. JS_Init is an order of magnitude larger than any other function in the browser.

Comment 1

[Steve Fink [:sfink] [:s:]]

Attachment: Reduce analysis memory usage

Tackle memory usage in a number of ways:

1. Split out the per-entry checks over js.msg into separate functions. (This makes JS_Init trivial in size.)

2. For the check containing a loop, declare the variables outside of the loop. That eliminates the n^2 temporary explosion.

Waldo, you don't need to review the following (since I'd normally land them without review anyway):

3. Remove the --jobs default value since it was overriding the value set in defaults.py, and resulting in more processes running at the same time (each using up memory.)

4. Free objects pulled out of .xdb files a little more eagerly (avoids keeping them alive while processing each function body.)

Comment 2

[Steve Fink [:sfink] [:s:]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/cfebceef3108

Comment 3

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/cfebceef3108