937402 - mach command to compile js shells

Status: RESOLVED DUPLICATE of bug 1210848

(Core :: JavaScript Engine, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

We have some python compilation scripts that we use to compile js shells for fuzzing purposes. What's lacking is mach integration, but the idea is to be able to use:

./mach build-js -b "-a 32 -c dbg --enable-more-deterministic"

to get a working 32-bit debug more-deterministic js shell.

jsfunfuzz fuzzing harness, autoBisect, JSBugMon all use these files. Bonus is that JS developers can use this with mach as well.

Setting s-s until this actually lands as they currently reside in the fuzzing repository. Attachments coming up.

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: Patch v0.1

These scripts have helped us compile a working js shell of various options as needed (32/64-bit, debug/opt, Win7/8/Mac 10.7/8/9/Ubuntu Linux 13.04, x86/ARM, deterministic or not, threadsafe or not, you get the idea).

These provide (via buildOptions.py) the following syntax:

-b "-a 32 -c opt --enable-more-deterministic --enable-threadsafe --enable-gcgenerational"

gives one a 32-bit opt more-deterministic threadsafe GGC build on whatever platform you're on, the common ones. (The first -b refers to the "build" that one requires)

The goal is to use mach with the proposed keyword "build-js" or "buildjs" or "jsbuild" to get a functional js shell with the required options.

Here's a code dump from fuzzing rev 0adfa55633c4. I'd like some feedback on the code quality, please.

Known issues:
- There is zero integration here into mach, this is just a starting point for feedback.
- It's written with Python 2.6/2.7 in mind, i.e. we don't use print_function yet but it should just be as simple as running 2to3. We removed 2.5 hacks a while ago, and 2.4 a lot longer ago.
- There is no working way (everything is untested) to get a build from this stage, other scripts currently call these files to get a working js shell. More work is needed to massage these into working order.
- There are a lot of hacks here, some old, some fairly recent. I'd prefer not to strip hacks out of this initial bug as a first step. (e.g. if GCC goes OOM, it seems to spawn a "internal compiler error: Killed (program cc1plus)" message, so if this is hit, retry once.)
- I'm sure there's more, since this is just a fairly straightforward code dump.

Comment 2

[Mike Hommey [:glandium]]

I don't see why that needs to be private.

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Mike Hommey [:glandium] from comment #2)
> I don't see why that needs to be private.

I just wanted to make sure I didn't put in stuff I didn't mean to. Will open this up since I've got these files isolated.

There's more:

- Even the directories they are in, js/build/*, is just suggested. I'd like ideas on what should go where.
- We currently use optparse. Due to some build machines (which run fuzzing in their spare time) still being on 2.6, we don't have argparse everywhere yet.
- There are some seemingly-unused helper functions for fuzzers, e.g. automatically get a stack on Windows using cdb, or grabbing crash logs on other platforms. They are used in other parts of the harness. Shall these remain here?
- Some files have MPL licences, some don't.

Comment 4

[Mike Hommey [:glandium]]

compileShell.py is kind of a horror show. It contains many things that have nothing to do there.

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: Patch v0.1b

v0.1b: Added missing hgCmds.py

Here's now a working version. When one types (using the long (e.g. --repoDir) options instead of short (-R) ones):

$ python compileShell.py --repoDir ~/trees/mozilla-central/ --build "--arch 32 --compileType opt --enable-more-deterministic --enable-exact-rooting --enable-gcgenerational"

in the js/build directory, a js shell is compiled and appears in ~/Desktop/autobisect-cache/js-opt-32-dm-er-ggc-darwin-16949049f03d/js-opt-32-dm-er-ggc-darwin-16949049f03d (it takes the current rev changeset hash in the tree specified by -R.)

Likewise, for a 64-bit debug more-deterministic GGC threadsafe shell:

$ python compileShell.py -R ~/trees/mozilla-central/ -b "-a 64 -c dbg --enable-more-deterministic --enable-exact-rooting --enable-gcgenerational --enable-threadsafe"

gives ~/Desktop/autobisect-cache/js-dbg-64-dm-ts-er-ggc-darwin-16949049f03d/js-dbg-64-dm-ts-er-ggc-darwin-16949049f03d

inspectShell will have verified that this is the shell compiled as required, using getBuildConfiguration.


(In reply to Mike Hommey [:glandium] from comment #4)
> compileShell.py is kind of a horror show. It contains many things that have
> nothing to do there.

I'm not proposing to add *all* the functions there, as I said about this code dump. And certainly many things can be refactored / written better. For all intents and purposes, someone could take these scripts and entirely rewrite a lean and mean alternative, if it moves us forward.

Comment 6

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Comment on attachment 830632 [details] [diff] [review]
Patch v0.1b

MattN tried running this and we hit some problems where I was hardcoding stuff specific to my computer. Eventually we did get this patch to compile js shells successfully on his Mac machine.

He gave me some useful feedback and I'll make some changes maybe tomorrow.

Comment 7

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: Patch v0.1c

Files now based on fuzzing rev 1b997d751635.

Ok, this should now work on Mac, possibly Linux:

python compileShell.py --build "--repoDir ~/trees/mozilla-central/ --arch 64 --compileType dbg --enable-more-deterministic --enable-exact-rooting --enable-gcgenerational"

should give a 64-bit debug more-deterministic GGC js shell, in ~/Desktop/autobisect-cache/ (this location should be changeable later).

I also set verbose to True in subprocesses.py to give output when the command is running, as a demonstration.

(this revision doesn't yet work on Windows, where it used to. Will fix it next, as well as removal of unused functions to tidy up the patch)

These scripts don't use mach to replace the current make build system. It sits on "top" of the current build system, but provides this interface to build required js shells.

Future refactorings could keep the interface but replace the backend, making the transition seamless to folks compiling the js shell.

Comment 8

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Tested on Waldo's machine, 64-bit debug and 32-bit opt GGC threadsafe shells successfully compiled but more inefficiencies found and more feedback:

- The patch shouldn't copy out the js sources. (Copying is preferred for preserving the state of the repo when fuzzing multiple revs, but shouldn't be needed for the purposes in this bug. It should be ripped out and placed in another part of the fuzzing repo)
- Patch assumes autoconf2.13 in Linux, but apparently this is only on Ubuntu and breaks on Fedora, which is autoconf-2.13.
- This shouldn't make use of temp folders to compile, it should use objdirs instead.
- The interface using --build "<some parameters" seems less than ideal. Perhaps push these to .mozconfig or .js-mozconfig?

Comment 9

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

This may become obsolete with bug 912359 and bug 950297.

Comment 10

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Duping forward, there is more useful information in bug 1210848.