721856 - Setup build+test infrastructure for Emscripten

Status: RESOLVED WONTFIX

(Release Engineering :: General, enhancement, P5)

Description

[(no longer active)]

The code for Emscripten lives on https://github.com/kripken/emscripten.  It is not based on mozilla-central and does not share code with any other projects.  In order to run Emscripten, the machine needs to have the following installed:

* node.js
* Spidermonkey shell
* The closure compiler
* clang 3.0

And also a ~/.emscripten file which points to the correct path names for the above.

Running the tests should involve something like this:

git clone git://github.com/kripken/emscripten.git
cd emscripten
git checkout incoming
./tests/runner.py > test.log

Alon can probably fill in the missing details here.

Comment 1

[Chris AtLee [:catlee]]

(In reply to Ehsan Akhgari [:ehsan] from comment #0)
> The code for Emscripten lives on https://github.com/kripken/emscripten.  It
> is not based on mozilla-central and does not share code with any other
> projects.  In order to run Emscripten, the machine needs to have the
> following installed:
> 
> * node.js

Where do I get this? Can/should this be retrieved as part of the build process?

> * Spidermonkey shell

Same here.

> * The closure compiler

Same here.

> * clang 3.0

We'll probably be deploying this shortly.

> And also a ~/.emscripten file which points to the correct path names for the
> above.

This needs to be local to the build directory, not a system-wide thing

> Running the tests should involve something like this:
> 
> git clone git://github.com/kripken/emscripten.git

We don't have git on the machines yet.

> cd emscripten
> git checkout incoming
> ./tests/runner.py > test.log
> 
> Alon can probably fill in the missing details here.

Comment 2

[Alon Zakai (:azakai)]

(In reply to Chris AtLee [:catlee] from comment #1)
> (In reply to Ehsan Akhgari [:ehsan] from comment #0)
> 
> > Running the tests should involve something like this:
> > 
> > git clone git://github.com/kripken/emscripten.git
> 
> We don't have git on the machines yet.
> 

I see that pdf.js also uses github, like Emscripten, and their build automation looks pretty good (I don't know how it works, but they have a tryserver-like bot that posts messages to pull requests, etc.). If Emscripten could have something similar that would be great.

> > The code for Emscripten lives on https://github.com/kripken/emscripten.  It
> > is not based on mozilla-central and does not share code with any other
> > projects.  In order to run Emscripten, the machine needs to have the
> > following installed:
> > 
> > * node.js
> 
> Where do I get this? Can/should this be retrieved as part of the build
> process?
> 

It's available at http://nodejs.org/

No need to constantly retrieve it, I don't think.

> > * Spidermonkey shell
> 
> Same here.
> 

This involves fetching mozilla-central, then in js/src doing

  ./configure --enable-optimize --disable-debug
  make

The shell is then generated as 'js'.

> > * The closure compiler
> 
> Same here.
> 

This is at http://code.google.com/closure/compiler/

Like node, it doesn't need to be fetched all the time, it's pretty stable.

> > * clang 3.0
> 
> We'll probably be deploying this shortly.
> 

Ah, nice, we'll be using that on Firefox soon? (OS X I assume?)

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) We typically do builds on separate machines from tests - this handles our cross-OS coverage story. Tweaking summary to match.

2) Does this code have to remain on the current remote git repo? Supporting git is something we are building out support for in our automation, but it doesnt exist today. Further, there are security concerns about our systems pulling in code+files from remote external sites. All of these could be avoided if we hosted the code on hg.m.o, and then pushed out updates to the external git repo.

2) Who would need commit access to this? Or to put another way, do we know that all the people who will commit to "emscripten" also have commit privs on mozilla-central, and so we can trust later m-c jobs run on these same machines?

4) There is another project looking to install node onto test machines, is this the same version of node that Emscripten is looking for? See bug#729392 for details.

Comment 4

[Alon Zakai (:azakai)]

(In reply to John O'Duinn [:joduinn] from comment #3)
> 2) Does this code have to remain on the current remote git repo?

I think so. We are getting a lot of contributions there, and github being familiar to everyone is a big part of that I think.

> 
> 2) Who would need commit access to this? Or to put another way, do we know
> that all the people who will commit to "emscripten" also have commit privs
> on mozilla-central, and so we can trust later m-c jobs run on these same
> machines?

Hmm, not sure. Does anyone know what pdf.js does? (They are also on github, have external contributors, and have automatic testing infrastructure.)

> 
> 4) There is another project looking to install node onto test machines, is
> this the same version of node that Emscripten is looking for? See bug#729392
> for details.

If I read that right, they will compile from source? That would be fine for Emscripten (a reasonably recent stable release would also be fine).

Comment 5

[Chris AtLee [:catlee]]

So we can't pull from github because we're planning on shutting off external network access this quarter.

Can you mirror this on hg for now?

Comment 6

[(no longer active)]

We can look into doing that, I think.  What do you require from the hg mirror?  Should it just contain the same stuff as the git repo?  If yes, I can look into using hg-git to do the mirroring.

Comment 7

[Chris AtLee [:catlee]]

Not sure I understand...why wouldn't it contain the same stuff as git? If you're looking for a subset, then I'd say mirror git's master branch to hg's default branch, and we'd watch that for changes.

Comment 8

[(no longer active)]

Oh, no particular reason.  Just wanted to make sure that you guys don't care about the repo contents.  :-)

Comment 9

[(no longer active)]

Should I file a bug for an hg repo to be created?

Comment 10

[Chris AtLee [:catlee]]

yup!

Comment 11

[(no longer active)]

To run the tests in parallel, one needs to do:

nosetests --processes=4 -v -s tests/runner.py

Where 4 is the number of processes to use.

Comment 12

[(no longer active)]

I filed bug 755833 to deploy the bookmarks extension on hg.m.o so that I can sync up the emscripten git repo to the hg mirror.

Comment 13

[[e-mail] (:bhearsum)]

Not sure if this is still useful or not, but this is probably something that can be self served with Taskcluster if it is.