Closed Bug 912973 Opened 11 years ago Closed 11 years ago

Create a resilient NPM package repository

Categories

(Infrastructure & Operations :: IT-Managed Tools, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dustin, Assigned: gaye)

References

Details

Per bug 900663, we need to start supplying a local repository of Node modules.  William's found a way to do this with little more than Apache and a directory, so we need to set things up to host that now.

This would look like:
  http://npm-mirror.pvt.build.mozilla.org
  http://npm-mirror.pub.build.mozilla.org
both hosting the same content from a NFS mount (a subdir of releng_web), generated from William's script.  Uploads and updates will involve running that script, which mirrors a single package from registry.npmjs.org.

There's a little bit of Apache configuration to do to ensure that index.json is served for paths ending in /, but that's simple.

I don't think there's any need to host "private" packages that would only be available on the internal mirror.

Initially, traffic would be low - nothing is providing this service right now - but might grow as the build system starts hitting this service.  I'll ask about that in bug 900663.

So, Jake: any other questions or concerns before I start hacking on this?
Assignee: relops → dustin
Component: RelOps → WebOps: IT-Managed Tools
QA Contact: arich → nmaul
Deployment notes (for docs, later):

The vhosts in comment 0 point directly to /mnt/netapp/relengweb/npm-mirror.  There's no puppet required for the webheads, aside from that Apache config.

The script is on the admin host, and set up as follows:
 - python-virtualenv installed by hand
 - git clone https://github.com/wlach/mirror-npm.git /data/releng/src/npm-mirror/script
 - virtualenv /data/releng/src/npm-mirror/script
 - /data/releng/src/npm-mirror/script/bin/pip install mozhttpd httplib2

and then

# script/bin/python script/mirror-npm.py http://registry.npmjs.org /mnt/netapp/relengweb/npm-mirror/ http://npm-mirror.pvt.build.mozilla.org marionette-js-runner

A few changes would be nice, so this can be a but more usable and upgrade-able:

 - clone to a repo in the mozilla github org

 - add a setup.py -- enough that we can 'pip install' with git+git://github.com/mozilla/npm-mirror.git#egg=npm-mirror and it will pull in the necessary dependencies

 - Include a console_scripts entry point so that scripts/bin/npm-mirror can be executed directly

 - Is it possible to use relative URLs in the index.jsons?  As it is, and as you'll see at the site now, the pub vhost contains URLs with pvt in them.  If need be, we can just host everything on the pub vhost, and have releng automation talk to that, but that has some minor disadvantages in terms of reliability of the releng automation.

 - Holy cow, this mirrors a lot.  Is it really necessary to get *every* version of every package?  Just the marionette-js-runner mirror above is 200M.  This is OK if it's required - a few MB here and there is fine - but it would be great if it could be avoided.

You can see the results -- although they point to the internal mirror -- at http://npm-mirror.pub.build.mozilla.org/marionette-js-runner/
Flags: needinfo?(wlachance)
(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> A few changes would be nice, so this can be a but more usable and
> upgrade-able:
> 
>  - clone to a repo in the mozilla github org
> 
>  - add a setup.py -- enough that we can 'pip install' with
> git+git://github.com/mozilla/npm-mirror.git#egg=npm-mirror and it will pull
> in the necessary dependencies
> 
>  - Include a console_scripts entry point so that scripts/bin/npm-mirror can
> be executed directly

As discussed, I'm going to rewrite this tool in nodejs as I suspect that this is of use outside of Mozilla. Filed bug 916971 for that.
 
>  - Is it possible to use relative URLs in the index.jsons?  As it is, and as
> you'll see at the site now, the pub vhost contains URLs with pvt in them. 
> If need be, we can just host everything on the pub vhost, and have releng
> automation talk to that, but that has some minor disadvantages in terms of
> reliability of the releng automation.

As discussed on irc:

<dustin> the issue of absolute URLs is probably the most important to figure out soon
<wlach> dustin: could you not run the script twice, one for public and one for private? or even just modify the index.json's with a sed script
<dustin> wlach: yes, but then we have two copies of all those tarballs
<wlach> dustin: true. I have a feeling npm requires absolute urls
<dustin> I suppose we could have DirectoryIndex index-pub.json / index-pvt.json
<wlach> that would work too
<dustin> yeah, that makes sense

>  - Holy cow, this mirrors a lot.  Is it really necessary to get *every*
> version of every package?  Just the marionette-js-runner mirror above is
> 200M.  This is OK if it's required - a few MB here and there is fine - but
> it would be great if it could be avoided.

The current implementation is a bit naive. I think we want to get all the versions of the packages we maintain, but we can probably get by with only getting the required versions of the dependencies. I can add this as a feature request. :)
Flags: needinfo?(wlachance)
So on the multiple-URLs issue, the choice we've selected is to not implmement a npm-mirror.pvt.b.m.o origin and just host everything on pub.
The new node script is set up as follows:

cd /data/releng/src/npm-mirror
git clone https://github.com/mozilla-b2g/npm-mirror
cd npm-mirror
npm install

cd ..
curl -o /tmp/package.json https://raw.github.com/mozilla-b2g/marionette-js-runner/master/package.json
./mirror.sh /tmp/package.json

where mirror.sh runs npm-mirror with the appropriate options.  I'll add the necessary websites docs and whatnot.
https://mana.mozilla.org/wiki/display/websites/npm-mirror.pub.build.mozilla.org
https://wiki.mozilla.org/ReleaseEngineering/How_To/Mirror_NPM_Packages

So this is deployed AFAIC.  It's not currently mirroring devDependencies, which is a problem, but Gareth's filing a separate bug for that.
Assignee: dustin → gaye
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.