Closed
Bug 912973
Opened 11 years ago
Closed 11 years ago
Create a resilient NPM package repository
Categories
(Infrastructure & Operations :: IT-Managed Tools, task)
Infrastructure & Operations
IT-Managed Tools
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: dustin, Assigned: gaye)
References
Details
Per bug 900663, we need to start supplying a local repository of Node modules. William's found a way to do this with little more than Apache and a directory, so we need to set things up to host that now. This would look like: http://npm-mirror.pvt.build.mozilla.org http://npm-mirror.pub.build.mozilla.org both hosting the same content from a NFS mount (a subdir of releng_web), generated from William's script. Uploads and updates will involve running that script, which mirrors a single package from registry.npmjs.org. There's a little bit of Apache configuration to do to ensure that index.json is served for paths ending in /, but that's simple. I don't think there's any need to host "private" packages that would only be available on the internal mirror. Initially, traffic would be low - nothing is providing this service right now - but might grow as the build system starts hitting this service. I'll ask about that in bug 900663. So, Jake: any other questions or concerns before I start hacking on this?
Reporter | ||
Updated•11 years ago
|
Assignee: relops → dustin
Component: RelOps → WebOps: IT-Managed Tools
QA Contact: arich → nmaul
Reporter | ||
Comment 1•11 years ago
|
||
Deployment notes (for docs, later): The vhosts in comment 0 point directly to /mnt/netapp/relengweb/npm-mirror. There's no puppet required for the webheads, aside from that Apache config. The script is on the admin host, and set up as follows: - python-virtualenv installed by hand - git clone https://github.com/wlach/mirror-npm.git /data/releng/src/npm-mirror/script - virtualenv /data/releng/src/npm-mirror/script - /data/releng/src/npm-mirror/script/bin/pip install mozhttpd httplib2 and then # script/bin/python script/mirror-npm.py http://registry.npmjs.org /mnt/netapp/relengweb/npm-mirror/ http://npm-mirror.pvt.build.mozilla.org marionette-js-runner A few changes would be nice, so this can be a but more usable and upgrade-able: - clone to a repo in the mozilla github org - add a setup.py -- enough that we can 'pip install' with git+git://github.com/mozilla/npm-mirror.git#egg=npm-mirror and it will pull in the necessary dependencies - Include a console_scripts entry point so that scripts/bin/npm-mirror can be executed directly - Is it possible to use relative URLs in the index.jsons? As it is, and as you'll see at the site now, the pub vhost contains URLs with pvt in them. If need be, we can just host everything on the pub vhost, and have releng automation talk to that, but that has some minor disadvantages in terms of reliability of the releng automation. - Holy cow, this mirrors a lot. Is it really necessary to get *every* version of every package? Just the marionette-js-runner mirror above is 200M. This is OK if it's required - a few MB here and there is fine - but it would be great if it could be avoided. You can see the results -- although they point to the internal mirror -- at http://npm-mirror.pub.build.mozilla.org/marionette-js-runner/
Flags: needinfo?(wlachance)
Comment 2•11 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] from comment #1) > A few changes would be nice, so this can be a but more usable and > upgrade-able: > > - clone to a repo in the mozilla github org > > - add a setup.py -- enough that we can 'pip install' with > git+git://github.com/mozilla/npm-mirror.git#egg=npm-mirror and it will pull > in the necessary dependencies > > - Include a console_scripts entry point so that scripts/bin/npm-mirror can > be executed directly As discussed, I'm going to rewrite this tool in nodejs as I suspect that this is of use outside of Mozilla. Filed bug 916971 for that. > - Is it possible to use relative URLs in the index.jsons? As it is, and as > you'll see at the site now, the pub vhost contains URLs with pvt in them. > If need be, we can just host everything on the pub vhost, and have releng > automation talk to that, but that has some minor disadvantages in terms of > reliability of the releng automation. As discussed on irc: <dustin> the issue of absolute URLs is probably the most important to figure out soon <wlach> dustin: could you not run the script twice, one for public and one for private? or even just modify the index.json's with a sed script <dustin> wlach: yes, but then we have two copies of all those tarballs <wlach> dustin: true. I have a feeling npm requires absolute urls <dustin> I suppose we could have DirectoryIndex index-pub.json / index-pvt.json <wlach> that would work too <dustin> yeah, that makes sense > - Holy cow, this mirrors a lot. Is it really necessary to get *every* > version of every package? Just the marionette-js-runner mirror above is > 200M. This is OK if it's required - a few MB here and there is fine - but > it would be great if it could be avoided. The current implementation is a bit naive. I think we want to get all the versions of the packages we maintain, but we can probably get by with only getting the required versions of the dependencies. I can add this as a feature request. :)
Flags: needinfo?(wlachance)
Reporter | ||
Comment 3•11 years ago
|
||
So on the multiple-URLs issue, the choice we've selected is to not implmement a npm-mirror.pvt.b.m.o origin and just host everything on pub.
Reporter | ||
Comment 4•11 years ago
|
||
The new node script is set up as follows: cd /data/releng/src/npm-mirror git clone https://github.com/mozilla-b2g/npm-mirror cd npm-mirror npm install cd .. curl -o /tmp/package.json https://raw.github.com/mozilla-b2g/marionette-js-runner/master/package.json ./mirror.sh /tmp/package.json where mirror.sh runs npm-mirror with the appropriate options. I'll add the necessary websites docs and whatnot.
Reporter | ||
Comment 5•11 years ago
|
||
https://mana.mozilla.org/wiki/display/websites/npm-mirror.pub.build.mozilla.org https://wiki.mozilla.org/ReleaseEngineering/How_To/Mirror_NPM_Packages So this is deployed AFAIC. It's not currently mirroring devDependencies, which is a problem, but Gareth's filing a separate bug for that.
Assignee: dustin → gaye
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•