1033472 - Gaia linter, gaia-build tests aren't using NODE_MODULES_GIT_URL and so connect to Github ("make: *** [node_modules] Error 2")

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::Build, defect)

Description

[Ed Morley [:emorley]]

There was an infra blip earlier today, which caused:
https://tbpl.mozilla.org/php/getParsedLog.php?id=42947409&tree=Mozilla-Inbound
{
08:45:58     INFO -  Running gjslint...
08:47:06     INFO -  541 files checked, no errors found.
08:47:07     INFO -  Note: gjslint only checked the files that are xfailed for jshint.
08:47:07     INFO -  wget -c https://github.com/mozilla-b2g/gaia-node-modules/tarball/7b1dc9d9345fd3c3d1a97ee0146bf416d3aa97c7 &&\
08:47:07     INFO -  	mv 7b1dc9d9345fd3c3d1a97ee0146bf416d3aa97c7 "modules.tar"
08:47:07     INFO -  --2014-07-02 08:47:07--  https://github.com/mozilla-b2g/gaia-node-modules/tarball/7b1dc9d9345fd3c3d1a97ee0146bf416d3aa97c7
08:47:07     INFO -  Resolving github.com (github.com)... 192.30.252.131
08:47:07     INFO -  Connecting to github.com (github.com)|192.30.252.131|:443... connected.
08:47:07     INFO -  HTTP request sent, awaiting response... 302 Found
08:47:07     INFO -  Location: https://codeload.github.com/mozilla-b2g/gaia-node-modules/legacy.tar.gz/7b1dc9d9345fd3c3d1a97ee0146bf416d3aa97c7 [following]
08:47:07     INFO -  --2014-07-02 08:47:07--  https://codeload.github.com/mozilla-b2g/gaia-node-modules/legacy.tar.gz/7b1dc9d9345fd3c3d1a97ee0146bf416d3aa97c7
08:47:07     INFO -  Resolving codeload.github.com (codeload.github.com)... 192.30.252.144
08:47:08     INFO -  Connecting to codeload.github.com (codeload.github.com)|192.30.252.144|:443... connected.
08:47:08     INFO -  HTTP request sent, awaiting response... 502 Bad Gateway
08:47:08     INFO -  2014-07-02 08:47:08 ERROR 502: Bad Gateway.
08:47:08     INFO -  make[1]: [modules.tar] Error 8 (ignored)
08:47:08     INFO -  tar --wildcards --strip-components 1 -x -m -f modules.tar "mozilla-b2g-gaia-node-modules-*/node_modules"
08:47:08     INFO -  tar: modules.tar: Cannot open: No such file or directory
08:47:08     INFO -  tar: Error is not recoverable: exiting now
08:47:08     INFO - TEST-UNEXPECTED-FAIL | make lint | make[1]: *** [node_modules] Error 2
08:47:08     INFO -  make[1]: *** [node_modules] Error 2
08:47:08     INFO -  make[1]: Target `hint' not remade because of errors.
08:47:08     INFO -  make[1]: Leaving directory `/builds/slave/test/gaia'
08:47:08     INFO - TEST-UNEXPECTED-FAIL | make lint | make: *** [lint] Error 2
}

We should not be using github.com as part of our automation runs - instead the repo should be mirrored to git.m.o and used from there.

This means that the Linter tests (and anything else that uses these node modules) do not comply with:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.29_Must_avoid_patterns_known_to_cause_non_deterministic_failures

It looks like bug 977366 has regressed (or don't work for the gaia linter tests at least).

An owner for this is needed soon, or the relevant jobs will sadly need to be hidden from the default view.

Comment 1

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Ed Morley,

we can assign git.m.o by NODE_MODULES_GIT_URL to clone from git.m.o and as my experiment, it works well:

> NODE_MODULES_GIT_URL=https://git.mozilla.org/b2g/gaia-node-modules.git make node_modules && make lint

I also see gaia_linter.py[1] in mozharness but looks we have used NODE_MODULES_GIT_URL for cloning node_modules, not sure where the problem is.

[1] http://hg.mozilla.org/build/mozharness/file/141595cb922b/scripts/gaia_linter.py#l121

Comment 2

[Julien Wajsberg [:julienw]]

Yep, just always set "NODE_MODULES_GIT_URL" for all run, and you should be fine.

Comment 3

[Ed Morley [:emorley]]

Thank you for that - updating summary.

This has also occurred during the "gaia build" job, eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=43041210&tree=B2g-Inbound

Please could one of you drive this? I'd like to avoid hiding these two suites if possible.

Comment 4

[Julien Wajsberg [:julienw]]

Hey Jonathan, I think it's something that you or someone from your team can do here?

Comment 5

[Jonathan Griffin (:jgriffin)]

Attachment: Pass NODE_MODULES_GIT_URL to gaia build and linter tests, r=:julienw

This resolves the problem locally.

Comment 6

[Julien Wajsberg [:julienw]]

Jonathan, why can't we use something like [1] (with the right env variables obviously) for all Gaia jobs ?

[1] http://hg.mozilla.org/build/mozharness/file/e76fdaa99480/configs/b2g/releng-try.py#l33

Comment 7

[Jonathan Griffin (:jgriffin)]

(In reply to Julien Wajsberg [:julienw] from comment #6)
> Jonathan, why can't we use something like [1] (with the right env variables
> obviously) for all Gaia jobs ?
> 
> [1]
> http://hg.mozilla.org/build/mozharness/file/e76fdaa99480/configs/b2g/releng-
> try.py#l33

We can't execute gaia's 'make' with env variables, because of a weird bug in make/xpcshell, see bug 1028816.

We could, however, use a similar mechanism to globally apply the variables to any command-line invoked by the gaia-based scripts.

Comment 8

[Julien Wajsberg [:julienw]]

(In reply to Jonathan Griffin (:jgriffin) from comment #7)
> (In reply to Julien Wajsberg [:julienw] from comment #6)
> > Jonathan, why can't we use something like [1] (with the right env variables
> > obviously) for all Gaia jobs ?
> > 
> > [1]
> > http://hg.mozilla.org/build/mozharness/file/e76fdaa99480/configs/b2g/releng-
> > try.py#l33
> 
> We can't execute gaia's 'make' with env variables, because of a weird bug in
> make/xpcshell, see bug 1028816.

Oh, funny...
Is there a bug somewhere to at least fix the underlying bug? Because this is painful :)


> We could, however, use a similar mechanism to globally apply the variables
> to any command-line invoked by the gaia-based scripts.

Well, my main concern was that we might download node_modules anytime now, so it would be better to set this variable globally. And maintenance-wise, it's easier if it's automatically applied (for example, for new jobs too) rather than defined in the job script itself.

An alternative is to call "make_node_modules" (which uses this variable) in all scripts before doing the real call.

What do you think?

Comment 9

[Jonathan Griffin (:jgriffin)]

(In reply to Julien Wajsberg [:julienw] from comment #8)
> (In reply to Jonathan Griffin (:jgriffin) from comment #7)
> > (In reply to Julien Wajsberg [:julienw] from comment #6)
> > > Jonathan, why can't we use something like [1] (with the right env variables
> > > obviously) for all Gaia jobs ?
> > > 
> > > [1]
> > > http://hg.mozilla.org/build/mozharness/file/e76fdaa99480/configs/b2g/releng-
> > > try.py#l33
> > 
> > We can't execute gaia's 'make' with env variables, because of a weird bug in
> > make/xpcshell, see bug 1028816.
> 
> Oh, funny...
> Is there a bug somewhere to at least fix the underlying bug? Because this is
> painful :)
> 

There isn't; I don't think anyone understands exactly where the bug is.

> 
> > We could, however, use a similar mechanism to globally apply the variables
> > to any command-line invoked by the gaia-based scripts.
> 
> Well, my main concern was that we might download node_modules anytime now,
> so it would be better to set this variable globally. And maintenance-wise,
> it's easier if it's automatically applied (for example, for new jobs too)
> rather than defined in the job script itself.
> 
> An alternative is to call "make_node_modules" (which uses this variable) in
> all scripts before doing the real call.
> 
> What do you think?

We already do that as of this patch, but that isn't sufficient....even when doing that before 'make lint', you still need to specify NODE_MODULES_GIT_URL during 'make lint' in order to avoid a github download.

I also wonder if we will ever need to modify the env variables we apply to commands per-branch?  If so, we should place a config file in gaia and read that in the mozharness script.  If not, I can make a more generic solution.

Comment 10

[Julien Wajsberg [:julienw]]

(In reply to Jonathan Griffin (:jgriffin) from comment #9)
> (In reply to Julien Wajsberg [:julienw] from comment #8)
> > (In reply to Jonathan Griffin (:jgriffin) from comment #7)
> > > (In reply to Julien Wajsberg [:julienw] from comment #6)
> > > > Jonathan, why can't we use something like [1] (with the right env variables
> > > > obviously) for all Gaia jobs ?
> > > > 
> > > > [1]
> > > > http://hg.mozilla.org/build/mozharness/file/e76fdaa99480/configs/b2g/releng-
> > > > try.py#l33
> > > 
> > > We can't execute gaia's 'make' with env variables, because of a weird bug in
> > > make/xpcshell, see bug 1028816.
> > 
> > Oh, funny...
> > Is there a bug somewhere to at least fix the underlying bug? Because this is
> > painful :)
> > 
> 
> There isn't; I don't think anyone understands exactly where the bug is.

The first step is to file it; could you do it (because you understand already better than me what the problem is) ?

> 
> > 
> > > We could, however, use a similar mechanism to globally apply the variables
> > > to any command-line invoked by the gaia-based scripts.
> > 
> > Well, my main concern was that we might download node_modules anytime now,
> > so it would be better to set this variable globally. And maintenance-wise,
> > it's easier if it's automatically applied (for example, for new jobs too)
> > rather than defined in the job script itself.
> > 
> > An alternative is to call "make_node_modules" (which uses this variable) in
> > all scripts before doing the real call.
> > 
> > What do you think?
> 
> We already do that as of this patch, but that isn't sufficient....even when
> doing that before 'make lint', you still need to specify
> NODE_MODULES_GIT_URL during 'make lint' in order to avoid a github download.

Ah right, because NODE_MODULES_SRC changes when NODE_MODULES_GIT_URL is set. Not a very good design from our side here :(

I have an idea to resolve this. Let me try and maybe you won't have to change anything in harness then.

> 
> I also wonder if we will ever need to modify the env variables we apply to
> commands per-branch?  If so, we should place a config file in gaia and read
> that in the mozharness script.  If not, I can make a more generic solution.

I don't really know, but a config file inside gaia sounds good (reminds me of the Travis config files ;) ).

Comment 11

[Julien Wajsberg [:julienw]]

Attachment: github PR to change how node_modules is generated

Hey Yuren, 

What do you think of these changes?

Hey Jonathan, do you have a way to test this on your side?

Thanks !

Comment 12

[Jonathan Griffin (:jgriffin)]

> The first step is to file it; could you do it (because you understand already better than me what the 
> problem is) ?

Looking at this, I think bug 1028816 is a good description of the problem.  That was marked fixed with a workaround, but feel free to re-open it if you want to investigate the fundamental problem.

> Hey Jonathan, do you have a way to test this on your side?

It's possible, but time-consuming.  If it works locally, I'd be tempted to just land it and see if it works.

Comment 13

[Yuren [:yurenju] (AFK ~ Oct. 12) (volunteer mode)]

Comment on attachment 8453157 [details] [review]
github PR to change how node_modules is generated

great, thanks!

and please use $(MAKE) instead of make :)

Comment 14

[Julien Wajsberg [:julienw]]

Thanks !
Fixed the nits, rebased and pushed; waiting for Travis and Gaia Try.

Comment 15

[Julien Wajsberg [:julienw]]

Comment on attachment 8451945 [details] [diff] [review]
Pass NODE_MODULES_GIT_URL to gaia build and linter tests, r=:julienw

Review of attachment 8451945 [details] [diff] [review]:
-----------------------------------------------------------------

Removing the review flag since we'll move to the other solution.

Comment 16

[Julien Wajsberg [:julienw]]

everything looks fine besides intermittents

master: 10f141af83815940d1bd2b00a05c4f2e0965a0e3

Comment 17

[Julien Wajsberg [:julienw]]

Ed, do you think we should land this on the branches too? (at least v2.0) ?

Comment 18

[Ryan VanderMeulen [:RyanVM]]

Absolutely.

Comment 19

[Julien Wajsberg [:julienw]]

Attachment: v2.0 github PR

Comment 20

[Julien Wajsberg [:julienw]]

Attachment: v1.4 github PR

Contains also the patch for bug 994622

Comment 21

[Julien Wajsberg [:julienw]]

a=tests
v1.4: c78d1ab13586da4db62690fab915cf8256d09c35
v2.0: e1aa5fcc3e1ee2dde9dd935efb1d660c09cf64ff

Comment 22

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=gaia&job_id=704608
repository: gaia
start_time: 2015-03-26T19:17:36
who: aus[at]mozilla[dot]com
machine: unknown
revision: 4177c233a11fd5d82679cfee7355eea64910579d

2015-03-27 02:19:13 ERROR 502: Bad Gateway.
make: *** [node_modules] Error 2