Closed Bug 1443547 Opened 2 years ago Closed 2 years ago

Upgrade Node in lint Docker image to 8.9.4

Categories

(Firefox Build System :: Lint and Formatting, enhancement)

3 Branch
enhancement
Not set

Tracking

(firefox60 fixed)

RESOLVED FIXED
mozilla60
Tracking Status
firefox60 --- fixed

People

(Reporter: nalexander, Assigned: standard8)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

It's more or less just bumping versions in https://searchfox.org/mozilla-central/source/taskcluster/docker/recipes/install-node.sh#8 and verifying that the resulting tasks work.
I can take this on, shouldn't take much hopefully.
Assignee: nobody → standard8
Status: NEW → ASSIGNED
Joel, any idea why the desktop-test taskcluster image can't rebuild?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=228e8344cfc0ef674352da17210c10304e138f65&selectedJob=166303335
https://treeherder.mozilla.org/logviewer.html#?job_id=166303335&repo=try&lineNumber=1649925

[task 2018-03-06T22:52:06.476Z] gyp WARN EACCES user "worker" does not have permission to access the dev dir "/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/contextify/.node-gyp/8.9.4"
[task 2018-03-06T22:52:06.479Z] gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/contextify/.node-gyp"

Seems to be something to do with https://github.com/taskcluster/npm-cache, but I haven't worked out what...
Flags: needinfo?(jmaher)
let me redirect to someone who knows a bit more
Flags: needinfo?(jmaher)
Flags: needinfo?(gps)
Flags: needinfo?(dustin)
I'm not sure what's going on here. And so others don't have to load this massive log, here is where things start to go wrong:

[task 2018-03-06T21:15:11.654Z] Step 37 : RUN npm install -g taskcluster-npm-cache@1.1.14  && rm -rf ~/.npm
[task 2018-03-06T21:15:12.043Z]  ---> Running in 0e4be489edbb
[task 2018-03-06T21:15:14.226Z] ESC[91mnpmESC[0mESC[91m WARN deprecated line-numbers@0.2.0: Copy its ~20 LOC directly into your code instead.
[task 2018-03-06T21:15:14.483Z] ESC[0mESC[91mnpmESC[0mESC[91m WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
[task 2018-03-06T21:15:14.485Z] ESC[0mESC[91mnpmESC[0mESC[91m WARN deprecatedESC[0mESC[91m graceful-fs@2.0.3: please upgrade to graceful-fs 4 for compatibility with current and future versions of No
de.js
[task 2018-03-06T21:15:16.123Z] ESC[0mESC[91mnpmESC[0mESC[91m WARN deprecated graceful-fs@3.0.11: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
[task 2018-03-06T21:15:16.148Z] ESC[0mESC[91mnpm ESC[0mESC[91mWARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
[task 2018-03-06T21:15:16.518Z] ESC[0mESC[91mnpmESC[0mESC[91m WARN deprecated graceful-fs@1.2.3: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
[task 2018-03-06T21:15:16.586Z] ESC[0mESC[91mnpm ESC[0mESC[91mWARN deprecated native-or-bluebird@1.2.0: 'native-or-bluebird' is deprecated. Please use 'any-promise' instead.
[task 2018-03-06T21:15:17.223Z] ESC[0mESC[91mnpmESC[0mESC[91m WARN deprecated sockjs-client-node@0.1.1: deprecated in favor of the official sockjs-client
[task 2018-03-06T21:15:21.688Z] ESC[0m/usr/local/bin/taskcluster-npm-cache -> /usr/local/lib/node_modules/taskcluster-npm-cache/build/bin/taskcluster-npm-cache.js
[task 2018-03-06T21:15:21.688Z] /usr/local/bin/taskcluster-npm-cache-get -> /usr/local/lib/node_modules/taskcluster-npm-cache/build/bin/taskcluster-npm-cache-get.js
[task 2018-03-06T21:15:21.713Z]
[task 2018-03-06T21:15:21.713Z] > contextify@0.1.15 install /usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/contextify
[task 2018-03-06T21:15:21.713Z] > node-gyp rebuild
[task 2018-03-06T21:15:21.713Z]
[task 2018-03-06T21:15:22.075Z] ESC[91mgypESC[0mESC[91m ESC[0mESC[91mWARNESC[0mESC[91m ESC[0mESC[91mEACCESESC[0mESC[91m user "worker" does not have permission to access the dev dir "/builds/worker/.node-gyp/8.9.4"
[task 2018-03-06T21:15:22.075Z] gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/local/lib/node_modules/taskcluster-npm-cache/node_modules/contextify/.node-gyp"

AFAICT this loops forever until the task times out. The raw log is *only* 339,325,113 bytes spanning 1,649,927 lines. Eek.
Flags: needinfo?(gps)
Aha, updating to taskcluster-npm-cache 1.3.3 magically solves this: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3f6167d20d95d9855212805725507b27dcdc95f9&selectedJob=166522934

I'll get a full try build going in a bit. Leaving NI to Dustin in case there's other issues I need to be aware of.
Oy, I haven't heard of taskcluster-npm-cache before.  https://www.npmjs.com/package/taskcluster-npm-cache suggests it's pretty ancient.  I guess 1.1.14 was even more ancient!  The deprecation warnings are .. pretty exciting.

Based on the name, it sounds like this library was intended to help npm do some local caching?  Yarn does a good job with that, so maybe just switching from npm to yarn would allow you to avoid using taskcluster-npm-cache at all?
Flags: needinfo?(dustin)
If we're baking npm packages into Docker and not running `npm` at task time (we should be doing things this way because if we run `npm` at task time that will always be slower and less reliable than baking packages into Docker images), then why do we need a cache at all? As long as the Docker image can build mostly deterministically, we should be good.
TBH I don't know what taskcluster-npm-cache is used for, though the more I look around, the more this seems to be b2g related.

As far as I can tell, there's nothing using it within the tree:

https://searchfox.org/mozilla-central/search?q=npm-cache&case=false&regexp=false&path=

the only references I could find were in the b2g repos.

I also believe that the only tests currently running in mozilla-central automation that use node are the ESLint related ones.

Therefore, I think the best way forward is to try removing the installation of taskcluster-npm-cache from the Dockerfile, and see what happens.
Attachment #8956575 - Flags: review?(dustin)
Treeherder seems fine with the removal, so lets see what happens when we land it ;-)
(In reply to Mark Banner (:standard8) from comment #10)
> TBH I don't know what taskcluster-npm-cache is used for, though the more I
> look around, the more this seems to be b2g related.
> 
> As far as I can tell, there's nothing using it within the tree:
> 
> https://searchfox.org/mozilla-central/search?q=npm-
> cache&case=false&regexp=false&path=
> 
> the only references I could find were in the b2g repos.
> 
> I also believe that the only tests currently running in mozilla-central
> automation that use node are the ESLint related ones.
> 
> Therefore, I think the best way forward is to try removing the installation
> of taskcluster-npm-cache from the Dockerfile, and see what happens.

This is correct -- the TC npm cache stuff was b2g infra specific and hasn't been used in a long, long time.
Comment on attachment 8956575 [details]
Bug 1443547 - Upgrade Docker images to node 8.9.4 & npm 5.6.0.

https://reviewboard.mozilla.org/r/225486/#review232178
Attachment #8956575 - Flags: review?(dustin) → review+
Pushed by mbanner@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e0ea25f431fb
Upgrade Docker images to node 8.9.4 & npm 5.6.0. r=dustin
https://hg.mozilla.org/mozilla-central/rev/e0ea25f431fb
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Version: Version 3 → 3 Branch
You need to log in before you can comment on or make changes to this bug.