1453496 - Mozprofile 0.29 can't properly install addons in nightly

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Thom Chiovoloni [:tcsc] (ex-moco)]

My understanding of this is somewhat limited, but it seems like bug 1448221 (specifically the last patch) introduced a change to both Firefox and mozprofile, without which users of the latest version of mozprofile can't successfully install addons in nightly. They end up in <profilepath>/extensions/staged, and that's the end of it.

Sadly, the version of mozprofile wasn't updated at the same time. I'm not sure if updating the version is enough to do this, or if a publish/deployment step is required.

This broke TPS (a sync test suite that runs separately from normal test automation because of reasons), but there's no real fix at the moment because of it, beyond manually copying the mozprofile contents into my venv, which does work (there might be a better solution than doing that, but I'm not a python guy so I don't know of it).

Comment 1

[Andrew Swan [:aswan]]

I didn't quite follow comment 0, is mozprofile published to npm or some other external location?

Comment 2

[Thom Chiovoloni [:tcsc] (ex-moco)]

Yes, it's here: https://pypi.python.org/pypi/mozprofile.

It's possible there's a way to specify the version info in https://searchfox.org/mozilla-central/source/testing/tps/setup.py#17 to make it use the local version of mozprofile, but I don't know how to do that.

(Ignore the fact that the versions in that link are out of date; after I update them locally to the most recent version of everything, it doesn't work because 0.29 is still out of date, as mentioned, doesn't have the recent changes).

Comment 3

[Andrew Halberstadt [:ahal]]

Thanks for bringing this up, I can help with the version bump. Note that I also landed some backwards incompatible changes to the Addon installer, so we'll want to do a major version bump here (I'd like us to start transitioning to at least a semblance of SemVer).

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1453496 - [mozprofile] Bump version to 1.0.0 for Addon installation changes,

This is mainly to pick up bug 1448221 since the version of mozprofile on pypi can't
install addons with nightly anymore (due to the profile/extensions/staged directory
not being supported).

But since that change I've also landed several backwards incompatible API changes to
how addons are installed.

Bumping to 1.0.0 because I'd like us to start (attempting) to follow SemVer:
https://semver.org/

Review commit: https://reviewboard.mozilla.org/r/235890/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/235890/

Comment 5

[Andrew Halberstadt [:ahal]]

(In reply to Thom Chiovoloni [:tcsc] from comment #2)
> It's possible there's a way to specify the version info in
> https://searchfox.org/mozilla-central/source/testing/tps/setup.py#17 to make
> it use the local version of mozprofile, but I don't know how to do that.

Is tps used outside of mozilla-central/taskcluster? If so then yes, ideally you would use the versions that live in-tree. If this is the case, they should automatically just be available when running with mach, and should automatically be bootstrapped when running in CI (no need to depend on them in a requirements file). I can help you out with this.

But if you want to be able to `pip install tps` outside of mozilla-central and have mozbase get downloaded along with that, then yes, you'll need to use the pypi versions.

Either way, we need to get this uploaded to pypi, as I'm sure other external consumers will start noticing this problem soon too.

Comment 6

[Andrew Halberstadt [:ahal]]

s/If so/If not

Comment 7

[Thom Chiovoloni [:tcsc] (ex-moco)]

(In reply to Andrew Halberstadt [:ahal] from comment #5)
> (In reply to Thom Chiovoloni [:tcsc] from comment #2)
> > It's possible there's a way to specify the version info in
> > https://searchfox.org/mozilla-central/source/testing/tps/setup.py#17 to make
> > it use the local version of mozprofile, but I don't know how to do that.
> 
> Is tps used outside of mozilla-central/taskcluster? ...

It's never run inside taskcluster, and it's only sometimes run inside M-C (only when run locally).

For background, TPS is an end-to-end sync test, and runs (by default) against our production or staging FxA and Sync servers -- it's a test of the client code, but the idea is to avoid mocking quite as many things as we do in the normal tests. Since it does network requests, my understanding is that it's a bad fit for taskcluster (and running a local FxA + Sync server dev environment isn't trivial).

Regardless, when run in automation, it runs on a Jenkins box, and AFAIK only a subset of M-C is available, which is pulled out of a zipfile from M-C. AFAICT expanding that subset isn't hard, it would just require unzipping more things here [0]. (CCing rpapa on this bug since he's the one that manages TPS in automation)

When run locally it typically has all of M-C available.

> ... If not then yes, ideally
> you would use the versions that live in-tree. If this is the case, they
> should automatically just be available when running with mach, and should
> automatically be bootstrapped when running in CI (no need to depend on them
> in a requirements file). I can help you out with this.

Sadly, TPS isn't even run through mach. I suspect it would be a lot easier to run locally if it were though (it has it's own venv which is a hassle), although it would still require some setup even then, so it's never been obviously worth doing.

> But if you want to be able to `pip install tps` outside of mozilla-central
> and have mozbase get downloaded along with that, then yes, you'll need to
> use the pypi versions.

I don't think anybody cares about `pip install tps` working. TPS tends to get set up using [1], and is very awkward to use since it uses it's own venv. At some point I'd like to improve this, but don't really want to break the way it's setup in automation, and as mentioned, it would still need some setup.

[0]: https://github.com/mozilla-services/sync-tps-setup/blob/master/Dockerfile#L16-L31
[1]: https://searchfox.org/mozilla-central/source/testing/tps/create_venv.py

Comment 8

[Geoff Brown [:gbrown]]

Comment on attachment 8967216 [details]
Bug 1453496 - [mozprofile] Bump version to 1.0.0 for Addon installation changes,

https://reviewboard.mozilla.org/r/235890/#review241674

SemVer seems like a good thing.

On the other hand, I note that most (maybe all) other mozbase components use an x.y version rather than x.y.z. There is a little voice in my head asking "Is there a reason for that? Is something going to be upset by a different format?" I'm just saying...

Comment 9

[Andrew Halberstadt [:ahal]]

(In reply to Thom Chiovoloni [:tcsc] from comment #7)

That's fine. Sounds like you should keep using the versions that live on pypi in that case. If you could get the in-tree version of mozbase included as part of your mozilla-central subset, it might help you avoid issues like this in the future, but it might not be worth the effort.

> I don't think anybody cares about `pip install tps` working. TPS tends to
> get set up using [1], and is very awkward to use since it uses it's own
> venv. At some point I'd like to improve this, but don't really want to break
> the way it's setup in automation, and as mentioned, it would still need some
> setup.

I assume you run `python setup.py install` instead of pip then, that's basically the same thing when it comes to installing dependencies.


(In reply to Geoff Brown [:gbrown] from comment #8)
> SemVer seems like a good thing.
> 
> On the other hand, I note that most (maybe all) other mozbase components use
> an x.y version rather than x.y.z. There is a little voice in my head asking
> "Is there a reason for that? Is something going to be upset by a different
> format?" I'm just saying...

Good point. I'm *almost* sure that this will be fine. As long as the overall version is larger than the previous one, it shouldn't matter how many dots there are. For example the specifier "mozprofile<1" will work whether the version is 1, 1.0 or 1.0.0.0.0.

I do agree it's weird that different mozbase packages have a different amount of dots. But I'm going to start adding the 3rd version number every time I bump a mozbase package going forward, so hopefully it'll end up being consistent eventually.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7c2c4073bac6
[mozprofile] Bump version to 1.0.0 for Addon installation changes, r=gbrown

Comment 11

[Andrew Halberstadt [:ahal]]

I'll wait for this to merge before releasing to pypi, just on the off chance it gets backed out.

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/7c2c4073bac6

Comment 13

[Andrew Halberstadt [:ahal]]

Submitted to pypi:
https://pypi.org/project/mozprofile/