928487 - [marionette-js-runner] Update and rerun npm-mirror

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task)

Description

[Gareth Aye [:gaye] (back from PTO)]

1. Get on the http://npm-mirror.pub.build.mozilla.org machine.
2. Nuke all of the existing packages.
3. `git clone [e-mail]:mozilla-b2g/gaia.git`
4. `npm install npm-mirror@0.0.1-alpha-2`
5. `cd npm-mirror`
6. `DEBUG=* bin/npm-mirror \
        --master http://registry.npmjs.org \
        --manifest /path/to/gaia/package.json \
        --hostname http://npm-mirror.pub.build.mozilla.org \
        --root /path/to/wherever/the/webserver/should/live`
7. `cd /path/to/wherever/the/webserver/should/live`
8. Make sure the webserver is up.
9. `curl http://npm-mirror.pub.build.mozilla.org/marionette-js-runner` should send back some useful JSON.

Comment 1

[James Lal [:lightsofapollo]]

Could we get an ETA of when the deploy will happen?

Comment 2

[Jason Crowe [:jd]]

This was assigned to my queue but I am unable to do it as I do not know enough about the service. I pinged dustin on IRC but he was not around. I will need to be read in on the ins and outs of the service and get clarification from him prior to accomplishing this task. Otherwise dustin may be able to do this now and we can discuss the how later.

Comment 3

[Amy Rich [:arr] [:arich]]

dustin will be back from pto tomorrow.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

:jd, the docs are here (as expected):

https://mana.mozilla.org/wiki/display/websites/npm-mirror.pub.build.mozilla.org

I admit Gareth's instructions are confusing, since you shouldn't actually follow them.  A better bug would have been to just ask for the desired result :)

In particular, from the docs:

"The tool that does the mirroring (https://github.com/mozilla-b2g/npm-mirror) is cloned at /data/releng/src/npm-mirror/npm-mirror, and can be updated there with 'git pull' if necessary." (that's the "update" part)

and

"There is a mirror script at /data/releng/src/npm-mirror/mirror.sh which invokes the npm-mirror tool with the proper arguments.  It needs a package.json file as input, and extracts the dependencies from that file for download to the mirror.
curl -o /tmp/package.json https://raw.github.com/mozilla-b2g/gaia/master/package.json
./mirror.sh /tmp/package.json" (that's the "rerun" part)

Reassigning since the intent here is that this not be a service I maintain.  I'll be happy to update any docs that are found lacking.

Comment 5

[Jason Crowe [:jd]]

okay, i did see those docs, I guess my question was more along the lines of the instructions in the request. It does seem strange to me that I would ignore the entirety of the bug request, especially when the requester is asking me to "nuke" the entire project/packages/nfs mount (this part is unclear to me).

So just to be clear, I am to ignore every instruction from the requester and simply do a git pull plus the one curl and script on the admin server? Nothing more? No push to web servers no nuking anything etc...???

Apparently (as expected) means I am just supposed to know all of this from the docs? Do you recommend in the future I ignore all requests in the bugs and blindly follow your docs? It will be good to know the policy you have set up here.

Thanks

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Yeah, the bug is definitely confusing - not your fault.  As for nuking, I read that as nuking the existing npm-mirror package.  Gareth, do you want us to delete all of the existing mirrored packages?

The package data itself is on an NFS share, so no pushes.  The mirror script runs on the admin host on demand, and writes to the NFS share.

That said, trying to re-run the mirror script failed:

[[e-mail]3 dmitchell]# cd /data/releng/src/npm-mirror/npm-mirror/
[[e-mail]3 npm-mirror]# git pull
remote: Counting objects: 176, done.
remote: Compressing objects: 100% (78/78), done.
remote: Total 158 (delta 82), reused 137 (delta 65)
Receiving objects: 100% (158/158), 25.32 KiB, done.
Resolving deltas: 100% (82/82), completed with 11 local objects.
From https://github.com/mozilla-b2g/npm-mirror
   c5d7ff8..bd454b9  master     -> origin/master
Updating c5d7ff8..bd454b9
Fast-forward
 .jshintignore                 |   2 +
 .jshintrc                     |  16 ++++++-
 .travis.yml                   |   1 +
 CONTRIBUTING.md               |  16 +++++++
 Makefile                      |   2 +-
 README.md                     |  43 +++++++++++++----
 bin/npm-mirror                |  79 ++++++++++++++++---------------
 lib/download.js               | 101 ++++++++++++++++++++++++++++++++++++++++
 lib/maybemkdir.js             |  19 ++++++++
 lib/package.js                | 164 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/syncmanager.js            | 512 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------------------------------------------------------------
 lib/urlcheck.js               |  32 +++++++++++++
 package.json                  |   5 +-
 playground/example.sh         |   6 +++
 test/helper.js                |   7 +--
 test/integration/sync_test.js | 151 +++++++++++++++++++++++++++++++++++------------------------
 test/mock/fs.js               |   4 ++
 test/unit/maybemkdir_test.js  |  44 ++++++++++++++++++
 test/unit/package_test.js     |  66 ++++++++++++++++++++++++++
 test/unit/syncmanager_test.js | 281 +++-----------------------------------------------------------------------------------------------------------
 test/unit/urlcheck_test.js    |  30 ++++++++++++
 21 files changed, 950 insertions(+), 631 deletions(-)
 create mode 100644 .jshintignore
 create mode 100644 CONTRIBUTING.md
 create mode 100644 lib/download.js
 create mode 100644 lib/maybemkdir.js
 create mode 100644 lib/package.js
 create mode 100644 lib/urlcheck.js
 create mode 100755 playground/example.sh
 create mode 100644 test/unit/maybemkdir_test.js
 create mode 100644 test/unit/package_test.js
 create mode 100644 test/unit/urlcheck_test.js
[[e-mail]3 npm-mirror]# cd ..
[[e-mail]3 npm-mirror]# curl -o /tmp/package.json https://raw.github.com/mozilla-b2g/gaia/master/package.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
104   729  104   729    0     0   3671      0 --:--:-- --:--:-- --:--:--  5879
[[e-mail]3 npm-mirror]# ./mirror.sh /tmp/package.json

module.js:340
    throw err;
          ^
Error: Cannot find module 'graceful-fs'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/data/releng/src/npm-mirror/npm-mirror/lib/download.js:2:10)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

That was using a 'git pull' in the npm-mirror repo.  So it seems I'll need to change those instructions.  However, using npm install doesn't work either:

[[e-mail]3 npm-mirror]# npm install npm-mirror@0.0.1-alpha-2
npm http GET https://registry.npmjs.org/npm-mirror/0.0.1-alpha-2
npm http 404 https://registry.npmjs.org/npm-mirror/0.0.1-alpha-2
npm ERR! Error: version not found: 0.0.1-alpha-2 : npm-mirror/0.0.1-alpha-2
npm ERR!     at RegClient.<anonymous> (/usr/lib/node_modules/npm-registry-client/lib/request.js:272:14)
npm ERR!     at Request.self.callback (/usr/lib/node_modules/request/index.js:148:22)
npm ERR!     at Request.EventEmitter.emit (events.js:98:17)
npm ERR!     at Request.<anonymous> (/usr/lib/node_modules/request/index.js:891:14)
npm ERR!     at Request.EventEmitter.emit (events.js:117:20)
npm ERR!     at IncomingMessage.<anonymous> (/usr/lib/node_modules/request/index.js:842:12)
npm ERR!     at IncomingMessage.EventEmitter.emit (events.js:117:20)
npm ERR!     at _stream_readable.js:920:16
npm ERR!     at process._tickCallback (node.js:415:13)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <[e-mail]>

npm ERR! System Linux 2.6.32-279.2.1.el6.x86_64
npm ERR! command "node" "/usr/bin/npm" "install" "npm-mirror@0.0.1-alpha-2"
npm ERR! cwd /data/releng/src/npm-mirror
npm ERR! node -v v0.10.18
npm ERR! npm -v 1.3.6
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /data/releng/src/npm-mirror/npm-debug.log
npm ERR! not ok code 0

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

And of course, once I post that, I see the typo: 0.0.1-alpha-2 should be 1.0.0-alpha-2.  I also see that you changed all of the command-line options (but you were consistent and changed *all* of them)!

I used the 'npm install' option, and re-ran mirror.sh with its new contents:

#! /bin/bash

# note that this is only run on the admin host, not the webheads

manifest="${1}"
if [ -z "${manifest}" ]; then
    echo "USAGE: ./update manifest-file"
    echo "The manifest file is a package.json for the package to mirror, e.g., curl'd from"
    echo "https://raw.github.com/mozilla-b2g/marionette-js-runner/master/package.json"
    exit 1
fi

SITE_DIR=`cd $(dirname $0); pwd`
cd $SITE_DIR

DEBUG=* ./node_modules/npm-mirror/bin/npm-mirror \
    --manifest "${manifest}" \
    --hostname http://npm-mirror.pub.build.mozilla.org \
    --master http://registry.npmjs.org \
    --root /mnt/netapp/relengweb/npm-mirror

and things seem to have succeeded.  Gareth, if you still need me to nuke and re-mirror, let me know.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I rm'd the dir on the NFS mount and re-ran mirror.sh, per :gaye's request in IRC.