952566 - [npm-mirror] npm-mirror should support dependencies expressed as git urls

Status: RESOLVED INCOMPLETE

(Testing Graveyard :: JSMarionette, defect)

Description

[Gareth Aye [:gaye] (back from PTO)]

Npm nicely allows you to express your dependencies as either versions (which can be found in the registry) or git urls. Right now npm-mirror which pulls all of the packages we need into the build intranet can't sync packages expressed as git urls like the ones in https://github.com/krampstudio/grunt-jsdoc-plugin/blob/master/package.json.

Comment 1

[[e-mail]]

@gaye, can you point me where is the npm-mirror source used for TBPL?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

https://github.com/mozilla-b2g/npm-mirror

Comment 3

[[e-mail]]

thanks!

Comment 4

[[e-mail]]

Attachment: pull request redirect to github

handle git+https:// edge case

Comment 5

[[e-mail]]

Comment on attachment 8355412 [details] [review]
pull request redirect to github

The patch does not fix the download part

Comment 6

[[e-mail]]

after checking code for vain,
I start thinking maybe we could use some outside solution for npm local cache,

ex: https://github.com/mixu/npm_lazy

Comment 7

[Gareth Aye [:gaye] (back from PTO)]

Hi Fred -

In addition to the patch you proposed we also need to add code to clone a dependency in case of a git url here https://github.com/mozilla-b2g/npm-mirror/blob/master/lib/syncmanager.js#L124. We can probably just shell out to git. Thoughts?

Comment 8

[[e-mail]]

After above trial, the git+https:// prefix may not consist with real git or https link (followed by #version tag)

So I think shell out to git still cause some problems for edge cases.


The great thing is I've contact with the plugin author and he provided a new version to not use git url. So the bug 950080 is no longer required.


@greg, for module separation experiment, do you spot the need for external git url dependency in pacakage.json?

Comment 9

[[e-mail]]

@greg, for module separation experiment, do you spot the need for external git url dependency in pacakage.json?

Comment 10

[Greg Weng [:snowmantw][:gweng][:λ]]

I don't think git URL is a edge case. Of course if we don't have much resources to implement such feature we can temporarily disable it. But a NPM "mirror" without such abilitiy in our case can't guarantee we will not encounter this glitch, especially when a indirectly dependent package may someday add one line in its package.json. Thus we may have a npm-mirror server in some potential unstable situation.

And in our experiment, we may (in near future) use NPM to manage Gaia apps' dependencies, especially when app need to depend on some other apps with specific version or commit. So this feature may be necessary (but our experiment still an experiment, you know).

Comment 11

[James Lal [:lightsofapollo]]

FWIW- The best way to deal with the case where you need something that has not been published to the registry is by including it in the gaia repo and moving it to node modules (or even putting all of node_modules in gaia)

Comment 12

[Greg Weng [:snowmantw][:gweng][:λ]]

In this way our Gaia repo would contain more and more code that are not so directly coupling with the Gaia itself, and this is what we want to prevent. We even want to do the opposite thing that except those are really core part of Gaia, libraries which are already and can be isolated and reusable, should be packed as individual repos and publish on the internet, just like other open source libraries.

And what I worry about is not our code, but the 3rd-party modules that may depends on Git URL in the future. If we leave this protocol un-implemented, we must understand that someday our NPM server may broken because of this case. If this risk is in our plan and the problem is tolerable, then there is no problem to implement no Git protocol.

Comment 13

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Fred Lin [:gasolin] from comment #6)
> after checking code for vain,
> I start thinking maybe we could use some outside solution for npm local
> cache,
> 
> ex: https://github.com/mixu/npm_lazy

I think npm-mirror is equivalent to an npm cache, but npm-mirror currently does not support git urls. I think this would be fairly trivial to implement, but if you would rather work with releng and ateam to migrate from npm-mirror onto npm_lazy, I wouldn't protest. We need to make sure, however, that the cache fetches packages eagerly (we cannot wait until test time to sync packages). We also need to make sure that the cache is resilient to npmjs downtimes.

Once we've satisfied these constraints, I think it would be great to use a non-mozilla solution.

Comment 14

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Greg Weng [:snowmantw][:gweng][:λ] from comment #10)
> I don't think git URL is a edge case. Of course if we don't have much
> resources to implement such feature we can temporarily disable it. But a NPM
> "mirror" without such abilitiy in our case can't guarantee we will not
> encounter this glitch, especially when a indirectly dependent package may
> someday add one line in its package.json. Thus we may have a npm-mirror
> server in some potential unstable situation.

We *can* ensure that we won't encounter this glitch by using npm shrinkwrap and not depending on anything that depends on a git resource. Another solution is to check in node_modules like lightsofapollo suggested.

> And in our experiment, we may (in near future) use NPM to manage Gaia apps'
> dependencies, especially when app need to depend on some other apps with
> specific version or commit. So this feature may be necessary (but our
> experiment still an experiment, you know).

I think this is trivial to implement in npm-mirror, but it's not my highest priority right now.

Comment 15

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Greg Weng [:snowmantw][:gweng][:λ] from comment #12)
> In this way our Gaia repo would contain more and more code that are not so
> directly coupling with the Gaia itself, and this is what we want to prevent.

I think what we want to prevent is the situation (like you described in your slides) where whenever you make a change to one app, the tests for every app have to run which slows us down. I don't think repo size is anyone's main concern.

> We even want to do the opposite thing that except those are really core part
> of Gaia, libraries which are already and can be isolated and reusable,
> should be packed as individual repos and publish on the internet, just like
> other open source libraries.

Yes, but this is unrelated I think.

> And what I worry about is not our code, but the 3rd-party modules that may
> depends on Git URL in the future. If we leave this protocol un-implemented,
> we must understand that someday our NPM server may broken because of this
> case. If this risk is in our plan and the problem is tolerable, then there
> is no problem to implement no Git protocol.

Not if we check in node_modules or use npm shrinkwrap and make sure when we're upgrading dependencies that we aren't depending on any libs which don't live in npmjs.

Comment 16

[[e-mail]]

I think it may not solve problem via neither git mirror nor lazy cache. 

It's not worth to handle git url redirect if we can help plugin author do the right setup to solve the problem, which also benefit the world outside of mozilla.

Comment 17

[Firefox Bug Husbandry Bot]

Bulk closed as part of https://bugzilla.mozilla.org/show_bug.cgi?id=1422750