1150378 - Use --no-bin-links when running npm install in gaia-node-modules to support Windows

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[Julien Wajsberg [:julienw]]

"make node_modules" is broken on Windows because our tar contains symbolic links.

We should run npm install with --no-bin-links (source: [1]) to prevent this.

We need to fix gaia-node-modules and update the README file there.

We don't need to fix gaia's build because "npm install" has good defaults on Windows, the problem here is that we use a prebuilt node_modules that was built on a Unix system.

[1] http://askubuntu.com/a/269735

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia-node-modules] julienw:1150378-use-no-bin-links > mozilla-b2g:master

Comment 2

[Julien Wajsberg [:julienw]]

Comment on attachment 8587229 [details] [review]
[gaia-node-modules] julienw:1150378-use-no-bin-links > mozilla-b2g:master

I created this by running

  find -type d -name .bin -exec rm -rf {} \+

Then I checked we have no more links:

  find -type l

I tried removing node_modules completely and running "npm install --ignore-scripts --no-bin-links" but the result was a lot bigger than the current repository, so I chose a more chirurgical approach :)

Comment 3

[GitHub Autolander Bot]

Attachment: [gaia] julienw:1150378-use-julien-node-modules > mozilla-b2g:master

Comment 4

[Julien Wajsberg [:julienw]]

Comment on attachment 8587233 [details] [review]
[gaia] julienw:1150378-use-julien-node-modules > mozilla-b2g:master

Hey Alexandre, can you try running "make node_modules" with this branch on Windows ?

Thanks !

Comment 5

[Alexandre Poirot [:ochameau]]

Comment on attachment 8587233 [details] [review]
[gaia] julienw:1150378-use-julien-node-modules > mozilla-b2g:master

Please consider getting access to Windows...

Alex@ALEX-PC ~/Desktop/mozilla/gaia
$ make node_modules
# Running make without using a dependency ensures that we can run
# "make node_modules" with a custom NODE_MODULES_GIT_URL variable, and then
# run another target without specifying the variable
make modules.tar
/bin/bash: line 0: pushd: c:\Users\Alex\Desktop\mozilla\gaia\distribution: No su
ch file or directory
/bin/bash: line 0: popd: directory stack empty
make[1]: Entering directory `/c/Users/Alex/Desktop/mozilla/gaia'
Downloading latest node_modules package. This may take several minutes...
wget -c -nv https://github.com/julienw/gaia-node-modules/tarball/9af9abacfac000c
af6ddbc03daf2de242bdbd618 &&\
        mv 9af9abacfac000caf6ddbc03daf2de242bdbd618 "modules.tar"
2015-04-02 10:57:53 URL:https://codeload.github.com/julienw/gaia-node-modules/le
gacy.tar.gz/9af9abacfac000caf6ddbc03daf2de242bdbd618 [34308048] -> "9af9abacfac0
00caf6ddbc03daf2de242bdbd618" [1]
make[1]: Leaving directory `/c/Users/Alex/Desktop/mozilla/gaia'
tar --wildcards --strip-components 1 -x -m -f modules.tar "julienw-gaia-node-mod
ules-*/node_modules"
tar: node_modules/marionette-profile-builder/node_modules/mozilla-profile-builde
r/node_modules/packaged-webapp/node_modules/readdirp/examples/node_modules/event
-stream/node_modules/optimist/node_modules/wordwrap/example: Cannot mkdir: File
or path name too long
tar: Exiting with failure status due to previous errors
make: *** [node_modules] Error 2

Comment 6

[Julien Wajsberg [:julienw]]

Thanks Alexandre !

The issue is that "tar" does not use UNC paths to extract the archive.

I don't think this is easy to workaround this. It may be easier to either:

* wait for a version of npm that flatten the dependencies
* don't use gaia-node-modules on Windows and fallback to npm install.
* fallback to npm install anyway in case using gaia-node-modules is failing ?

What do you think kevin?

Comment 7

[Kevin Grandon :kgrandon]

I am ok falling back to npm install if we want. I'd like to redirect the needinfo to James/Aus first though. I believe the FxOS automation team had some plan to finally kill gaia-node-modules and make things nice. I'm not sure of what kind of timeline though, so if we want a quick fix, falling back to npm install works for me.

Comment 8

[Aus Lacroix [:aus]]

Quick fix would be to fall back to 'npm install' on Windows as kgrandon mentioned.

Longer term (starting with Linux on Taskcluster) is to pre-built a node module cache tarball that is platform specific and should extract properly. This tarball will be indexed on taskcluster (like Mulet builds currently) and will be retrievable as part of 'make node_modules'.

Comment 9

[Julien Wajsberg [:julienw]]

aus, if you look closely to the comments here, you'll see one of the issue is that _tar_ itself does not work properly on Windows with too long paths. You'll need to find a solution to that.
(NI just so that you notice this)

Comment 10

[Aus Lacroix [:aus]]

Julien, I think we'll be able to dodge that issue as we'll be building the tar.gz file on a windows machine itself and it will be the same package you get if you were to do 'npm install' from 'gaia/' on a windows box today.

Comment 11

[Julien Wajsberg [:julienw]]

If we can do that, this resolves all problems :)

Do you want to keep this bug for working on this or should we close WONTFIX ?

Comment 12

[Aus Lacroix [:aus]]

Meh, we can mark as WONTFIX. We'll get it for free when we start using taskcluster-npm-cache node module as part of the build instead of just using it on taskcluster.