1087831 - Gaia node modules timing out when fetching from git mirror

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Kevin Grandon :kgrandon]

We've been often seeing gaia-node-modules timeout when fetching from the git mirror recently.

17:48:36     INFO - Automation Error: mozprocess timed out after 300 seconds running ['make', 'node_modules', 'NODE_MODULES_GIT_URL=https://git.mozilla.org/b2g/gaia-node-modules.git']
17:48:36    ERROR - timed out after 300 seconds of no output
17:48:36    ERROR - Return code: -9

See: https://tbpl.mozilla.org/php/getParsedLog.php?id=50835014&tree=B2g-Inbound

Comment 1

[Kevin Grandon :kgrandon]

Jonathan - anything else we can do here, or any ideas why it's timing out? Can we try fetching from github in the event of a timeout perhaps?

Comment 2

[Jonathan Griffin (:jgriffin)]

(In reply to Kevin Grandon :kgrandon from comment #1)
> Jonathan - anything else we can do here, or any ideas why it's timing out?
> Can we try fetching from github in the event of a timeout perhaps?

This is a curious problem.  We already have logic to retry this operation in case it times out, but the retry is failing.  You can reproduce this problem manually by running |make node_modules NODE_MODULES_GIT_URL=https://git.mozilla.org/b2g/gaia-node-modules.git|, hitting CTRL+C after 30s or so, then re-running that command.

In the log you link, the output of the retry is:

17:49:36     INFO - Calling ['make', 'node_modules', 'NODE_MODULES_GIT_URL=https://git.mozilla.org/b2g/gaia-node-modules.git'] with output_timeout 300
17:49:36     INFO -  # Running make without using a dependency ensures that we can run
17:49:36     INFO -  # "make node_modules" with a custom NODE_MODULES_GIT_URL variable, and then
17:49:36     INFO -  # run another target without specifying the variable
17:49:36     INFO -  make git-gaia-node-modules
17:49:36     INFO -  make[1]: Entering directory `/builds/slave/test/gaia'
17:49:36     INFO -  make[1]: `git-gaia-node-modules' is up to date.
17:49:36     INFO -  make[1]: Leaving directory `/builds/slave/test/gaia'
17:49:36     INFO -  rm -fr node_modules
17:49:36     INFO -  cp -R git-gaia-node-modules/node_modules node_modules
17:49:36     INFO -  cp: cannot stat `git-gaia-node-modules/node_modules': No such file or directory

We could potentially address this either in mozharness or in the gaia makefile; it's probably a bug in the gaia makefile so may make most sense to handle there.  If you want to work around this in mozharness instead, needinfo me again.