1187958 - (node-support) [META] Add node buildsystem support

Status: NEW

(Firefox Build System :: General, task)

Description

[Selena Deckelmann :selenamarie :selena]

This is meant to be a tracker for adding npm and node support. 

Collecting notes here: https://etherpad.mozilla.org/npm-and-node-in-build-system

Comment 1

[Mike Hommey [:glandium]]

Motivation and goal would be better pasted here than referenced through an etherpad link.

Comment 2

[Gregory Szorc [:gps]]

Keep in mind build machines don't have access to the outside world. Also, we want builds and test results to be reproducible over time. We'll almost certainly need to run our own internal package mirror so all packages referenced by the tree are available for all of time. Or, we can vendor dependencies in mozilla-central like we do for Python. The latter is preferred, as it doesn't introduce an external dependency beyond the base `node` and `npm` executables and there is no potential for "drift" over time.

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

Here's a straw-man take on a user story here:

As a Firefox module owner or peer, I should be able to easily use key high-quality npm packages (such eslint, react-tools, various css linters, etc) in the build process.  

Motivation: most of the JavaScript and web tooling energy these days is happening in the node/npm ecosystem.  There's an opportunity to help take our build-tooling to the next level using well-maintained, industry-tested packages where those are already available.

This probably needs more fleshing out in terms of what the acceptance criteria would be, but it's a start.

Comment 4

[James Long (:jlongster)]

Any progress on this? This might become a critical part of building the Firefox developer tools. We are looking at processing our JS files on the fly, so we need to run packages on node to do this.

Comment 5

[James Long (:jlongster)]

(In reply to Gregory Szorc [:gps] from comment #2)
> Keep in mind build machines don't have access to the outside world. Also, we
> want builds and test results to be reproducible over time. We'll almost
> certainly need to run our own internal package mirror so all packages
> referenced by the tree are available for all of time. Or, we can vendor
> dependencies in mozilla-central like we do for Python. The latter is
> preferred, as it doesn't introduce an external dependency beyond the base
> `node` and `npm` executables and there is no potential for "drift" over time.

Vendoring deps seems far easier and we'd be fine with that. I don't think we plan on using too many build-time node dependencies, so it shouldn't be hard to get what we need vendored.

Comment 6

[Dan Mosedale (:dmosedale, :dmose)]

One thing that has moved forward slightly is that it's now part of the Ubuntu testing images used in TaskCluster (https://bugzilla.mozilla.org/show_bug.cgi?id=1273695#c4) (thanks Selena for the heads-up on this).

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

Notes from the old etherpad, mostly the stuff from Ted is still relevant:

From dmose: 
10:47 <dmose> the base requirement is probably being able to "npm install" something and then execute it with node
10:47 <dmose> eslint is probably a fine choice for that base
10:47 <dmose> since both Hello, Mobile, and maybe DevTools use it

on existance of npm/node on build machines: 

10:48 <dmose> because there are SPDY tests that run using it
10:49 <dmose> here's another request, which might well already be fulfilled by the existing stuff on the test machines: https://bugzilla.mozilla.org/show_bug.cgi?id=1083249
10:50 <dmose> npm depends on having git around, i believe
10:50 <dmose> there is a bug related to git
10:50 <dmose> https://bugzilla.mozilla.org/show_bug.cgi?id=590921
10:50 <dmose> for windows

From Ted:
07:57 <ted> 1) get it into mozillabuild for windows users
07:57 <ted> 2) make sure the package is easily installable on linux/mac, get it into the bootstrap scripts in-tree
07:57 <ted> 3) get it onto the build machines, maybe using tooltool
07:58 <ted> 4) decide if it's a hard requirement or if you can --disable-foo your way out of it, if it's the former that's going to put extra burden on tier 3 platforms (weird stuff like Solaris/AIX/whatever)
07:58 <ted> 5) add configure checks for it, error if it's missing

Comment 8

[Dan Mosedale (:dmosedale, :dmose)]

From Ted's list:

1) is done.
2) appears to done in service of making eslint work after './mach bootstrap' has been run

3-5 are still TBD.

Comment 9

[Mark Banner (:standard8)]

Note, there's a lot of work going on in bug 1401189 that aid or affect aspects affect this.

Also for 3), you basically just need to make sure you choose the right docker image for your task, or set one up with npm/node on. There's already a linux based taskcluster docker image called "lint" that has the necessary items.