1280233 - [tracking] Allow TestPilot to be a WebExtension

Status: RESOLVED FIXED

(WebExtensions :: Compatibility, defect, P5)

Description

[Andy McKay]

This is a tracker to allow TestPilot to be a WebExtension. As I understand it there are two main functions of the TestPilot add-on: install, update and uninstall addons and send telemetry to Mozilla.

Comment 1

[Andy McKay]

After chatting to rhelmer in London, I'm not even sure this makes sense any more. If a test pilot add-on can be installed from the website really simply using mozAddonManager, then the value of add-on is reduced. All that remains is sending Telemetry to the servers for each experiment, and that can be accomplished by adding it as a library.

I'm tempted to close this in favour of the new website powers test pilot is going to gain.

I'd still like to send telemetry to a Mozilla server though.

Comment 2

[Les Orchard [:lorchard]]

FWIW, there are / will be other things the Test Pilot add-on does / will do. For example:

* Rating & survey requests at random intervals

* Notifications of new available experiments

* Telemetry pings with all the environment / client data that goes into those pings - which I don't think WebExtensions can gather themselves.

* In-client management of enabled experiments - kind of redundant with the add-on manager, but we kind of want to keep Test Pilot experiments conceptually separate from add-ons (even if they are literally add-ons at present)

That said, most of the above is already feasible in a WebExtension, I think. It's mainly the AddonManager powers that we need an old-school add-on for.

Comment 3

[Wil Clouser [:clouserw]]

+cc Georg and Rob.  This bug came up in a meeting about webextensions using telemetry last week and they had some concerns.  :)

Comment 4

[Georg Fritzsche [:gfritzsche]]

I have concerns about generally allowing WebExtensions to submit Telemetry.
Generally, we'd want Telemetry submissions to be limited to a controlled set of developers.

Do we win anything from moving test-pilot over to WebExtensions and trying to solve this concern?

Comment 5

[Andy McKay]

(In reply to Georg Fritzsche [:gfritzsche] from comment #4)
> Generally, we'd want Telemetry submissions to be limited to a controlled set
> of developers.

Please see bug 1280235.

Comment 6

[Andy McKay]

I actually think that Test Pilot should become an Experiment:

http://webextensions-experiments.readthedocs.io/

Comment 7

[Wil Clouser [:clouserw]]

This is likely going to become a 2017 goal for the Test Pilot team.

Comment 8

[Wil Clouser [:clouserw]]

(In reply to Andy McKay [:andym] from comment #6)
> I actually think that Test Pilot should become an Experiment:
> 
> http://webextensions-experiments.readthedocs.io/

Can you expand on what that would mean?  From those docs it sounds like WebExtension Experiments are for prototyping new APIs before they land in Firefox.  Aside from a couple of APIs around installing add-ons, Test Pilot doesn't do much in that arena.  Additionally, Web Extension Experiments only work in Nightly and Dev Edition.  Or am I misunderstanding?

Comment 9

[Chuck Harmston [:chuck]]

Clarifying what andym said in comment 6:

He thinks Test Pilot should be a Web Extension Experiment that adds the APIs for experiments. There are a  host of implications that we'll need to consider:

- Experiments are strictly coupled to Test Pilot. It can't run without it, hard stop.
- It strictly couples experiment versions with Test Pilot versions.
- It strictly couples experiment releases with Test Pilot releases.
- It probably removes the ability to graduate retired experiments to AMO. It either ships in Firefox or goes away.
- Experiment development will be much more complicated, since experiment authors' work will have to happen across multiple repositories that are also tightly coupled (i.e. pull requests will have to happen in both Test Pilot and experiment repositories.
- I'm not sure how CI will work.

Comment 10

[Andy McKay]

Just to muddy the waters, Kris just suggested another approach.

- The test pilot add-on becomes a WebExtension
- We make each test pilot experiment a bundled WebExtension Experiment (containing the API) with an Embedded WebExtension (containing the actual add-on) inside it

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Embedded_WebExtensions

That should alleviate a couple of Chucks points.

Either this solution or comment #6 would, as comment #8 points out require a few changes in add-ons manager to allow this to happen. For example: allowing a certain set of limited WebExtensions Experiments from the Test Pilot team to be allowed to run on release.

Comment 11

[Andy McKay]

(In reply to Chuck Harmston [:chuck] from comment #9)
> - It probably removes the ability to graduate retired experiments to AMO. It
> either ships in Firefox or goes away.

I don't think that's a bad thing. If a Test Pilot Experiment introduces a new API (and I doubt all experiments will do that) and we find the API it creates is useful, it would be great to graduate it up into Firefox. If we find that the experiment isn't useful, we could look at the API and evaluate it. Perhaps the API needs changing or is useful enough to be merged, or maybe it isn't.

One struggle we have is that any API we implement in Firefox we need to maintain and support, so we are trying to be cautious about what APIs we do accept. The ability for Test Pilot to test out not only features, but APIs is pretty neat.

Comment 12

[John Gruen [:jgruen]]

Just going to go through Chuck's list with some annotations

(In reply to Chuck Harmston [:chuck] from comment #9)
> Clarifying what andym said in comment 6:
> 
> He thinks Test Pilot should be a Web Extension Experiment that adds the APIs
> for experiments. There are a  host of implications that we'll need to
> consider:
> 
> - Experiments are strictly coupled to Test Pilot. It can't run without it,
> hard stop.

This seems reasonable-ish for production, but the required coupling seems like it would be pretty burdensome for development of both Test Pilot and add-ons.

> - It strictly couples experiment versions with Test Pilot versions.

Yeah, that might actually be a positive since it will prevent users from getting into weird states per our last launch. We'll have to rigidly enforce updates which we can probably do mostly silently, or at least charmingly. For the product we're building this might be a net win. 

> - It strictly couples experiment releases with Test Pilot releases.

I could see this being a QA headache though. It could also lead to weird cherry picking if there were some immediate need to push a fix to an experiment. Howeer, in practical terms I'm not sure this would affect much given the release cadence of experiments right now. We're not pushing changes to all of them all the time.

> - It probably removes the ability to graduate retired experiments to AMO. It
> either ships in Firefox or goes away.

This one is is only true should the core functionality of the experiment depend on custom APIs, correct?

I'm a little torn on this one. There's a case to be made that experiments could inform the roadmap of the WebExtension API, in the same way we're informing the Firefox Roadmap. Experiments don't just plop into FF, so maybe (if they're using custom APIs) they don't get to go straight to AMO either. I'd argue however, that we would need to work out some kind of special casing for APIs vetted for several months in Test Pilot, seems like we could trade the testing in Test Pilot for reduced time spent riding trains.


> - Experiment development will be much more complicated, since experiment
> authors' work will have to happen across multiple repositories that are also
> tightly coupled (i.e. pull requests will have to happen in both Test Pilot
> and experiment repositories.

Yeah, this seems like the largest issue right now. I don't want to yoke our team or experiment devs (or me) with a ton of extra coordination work. I think there's probably some not-so-painful mitigations here, but right now this looks like a sticking point. 

> - I'm not sure how CI will work.

Ditto this.

Comment 13

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

I'd like to try to take a step back and reframe the discussion.

The goal of Test Pilot is to prototype user-facing Firefox features. This means that Test Pilot experiments need unlimited power to modify Firefox.
  For example, Tab Center rearranges the Firefox UI by transforming the XUL DOM.
  To take another example, Min Vid is using js-ctypes to call native window manager APIs, because the window needs to do things platform doesn't yet support.

One goal of WebExtensions, on the other hand, is to provide a safer and more performant Firefox experience, by _reducing_ the power addons have to change Firefox.

Since these goals are opposed and aimed at different audiences, I'm not sure I see the value in trying to turn Test Pilot or Test Pilot experiments into WebExtensions.

The goal of WebExtension Experiments seems to be trying out APIs that could be made fully public later. I don't see overlap between the APIs needed by Test Pilot and those appropriate to expose to all WebExtensions--again, because the goals are dramatically different.

Now, it looks like _Embedded_ WebExtensions provide the same access to Firefox internals as the chrome authority in SDK addons, so we could use those. Even so, I'm not sure I see a good reason to rewrite the existing Test Pilot addon.


A couple of followup questions:

1. What would we hope to gain from moving the Test Pilot addon (and experiments) to WebExtensions? I'm sure this has been discussed, but it hasn't been written down in this bug.

2. What's the long-term plan for supporting Embedded WebExtensions? If they will eventually be fully deprecated, should we look at using system addons for Test Pilot experiments?

Comment 14

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Hi Andy - the ni? missed you, because you were tagged on a previous comment. But I'd like to get your feedback on #c13 :-)

Comment 15

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

BMO fail. sorry for the bug spam :-P

Comment 16

[Wil Clouser [:clouserw]]

Thanks for all the feedback in this bug.

Andy and I have been talking about this for a few weeks and we've got a plan that addresses all the concerns (we think!):
    
1) Test Pilot will continue to pursue being a Web Extension (using this bug as a tracker)
2) We acknowledge that Test Pilot uses some esoteric APIs and features that should be exposed to everyone, but will not be high priority for the imminent Great Web Extension Migration.  Test Pilot will continue as a restartless add-on for as long as it takes to finish out those APIs and then will expect to become a Web Extension.
3) Test Pilot Experiments are where we need to move quickly and sometimes work outside the confines of Web Extensions.  To allow this Test Pilot Experiments will continue to be either Web Extensions or restartless add-ons for the foreseeable future, with no Web Extension-only requirement.

To accomplish the above, we will:
    
A) [sec team] Create a new signing key
B) [andym] Add the new signing key to Firefox (uplift if reasonable).  Add-ons signed with the new key will be allowed to be non Web Extensions.
C) [jason?] Build an API around the signing key
D) [clouserw] Test Pilot uses new signing API

Potential Concerns

1) Add-ons signed with the new key won't work in old versions of Firefox.  The sooner we land the new key, the less of a problem this will be (plus any uplifting we can do).  Test Pilot targets the current version and previous version currently.

Feedback & Comments?

ni?jason because he got thrown under the bus in C up there ^.

Comment 17

[Jason Thomas [:jason]]

(In reply to Wil Clouser [:clouserw] from comment #16)
> A) [sec team] Create a new signing key

Does this mean we want to create a new Root CA for Test Pilot related add-ons or create unique signing certificate similar to what we do for Hotfix signing? 
Today we differentiate between different add-ons (system vs add-ons signed via AMO) using the OU field in the signing certificate. Can we do that instead? 

Will these add-ons be signed outside of AMO?

I've cc'd :ulfr here to comment since he might have a opinion on how we proceed with this.

[1] https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/internal/XPIProvider.jsm#1759

Comment 18

[Wil Clouser [:clouserw]]

Makes no difference from the Test Pilot POV.  ni? andy & julien for opinions.

Comment 19

[Julien Vehent]

> Does this mean we want to create a new Root CA for Test Pilot related add-ons

Hell No!

> Today we differentiate between different add-ons (system vs add-ons signed via AMO) using the OU field in the signing certificate. Can we do that instead?

That uses a different intermediate CA, right? If so, I'm leaning towards that. Creating new intermediates is relatively easy and solves this particular problem without much code changes. It's still going to take some work to setup though, so not free.

> Will these add-ons be signed outside of AMO?

And if so, how? We don't have a good way to handle out-of-amo addons signing today. If testpilot needs an internal-only process, we can probably integrate that to trunion or autograph, but if we need a new public API, that might become complex.

Comment 20

[Jason Thomas [:jason]]

Wil, Julien, and I discussed this today and we will need to do the following:

* Ops + Sec will create a new intermediate certificate based of the current Add-ons Root (offline) certificate. This intermediate will be used to generate signing certificates for Test Pilot Web Extensions.

* Sec will implement a signing API within the Autograph service configured with this new Intermediate Certificate. This service will only be accessibly internally.

* We will need to figure out how to differentiate the Intermediate Certificate and possibly generated signing certificate within Firefox for Test Pilot Web Extensions. Wil will find out who can help with this.

Comment 21

[Robert Helmer [:rhelmer]]

aswan and I were chatting yesterday - is it necessary to make a new key for this? He was trying to convince me that it was not :)

Currently system addons expect the OU to be "Mozilla Components". We can tell what sort of add-on this is (system add-on, GMP, test pilot, normal add-on, etc) based on the update location.

Do you think it's valuable to also require a unique key? One thing that comes to mind is that MITMing an update service would allow serving a test pilot add-on as a system add-on, which would then be hidden from the UI and not easy to remove by the user. I might be missing something though, what do you think?

Comment 22

[Julien Vehent]

> Do you think it's valuable to also require a unique key?

Following up on this: it was decided to use a new OU for privileged web extensions, and thus not create a new key.

Comment 23

[Dave Justice]

Webextension version of the testpilot addon was merged in https://github.com/mozilla/testpilot/pull/3456

Next Test Pilot release is scheduled for 4/12 https://calendar.google.com/event?action=TEMPLATE&tmeid=czc3MGowc2Fzamk4Z3ViMGpocnZzNTR1OTRfMjAxODA0MTJUMjMwMDAwWiBtb3ppbGxhLmNvbV82bjJoa3BiZ2hyYWVvb3Rxa2JqZGUybzNyNEBn&tmsrc=mozilla.com_6n2hkpbghraeootqkbjde2o3r4%40group.calendar.google.com&scp=ALL

Comment 24

[Andrew Swan [:aswan]]

Closing per comment 23

Comment 25

[ExE Boss]

The Testpilot version that is downloadable from is https://testpilot.firefox.com/files/@testpilot-addon/updates.json still at 2.0.8-dev-259fe19, which is the bootstrapped extension, whereas the WebExtension is version 3.0.0, which hasn’t been released on the Test Pilot website yet due to issues with the extension release script.

Comment 26

[Andrew Swan [:aswan]]

The title of this bug is "allow" testpilot to be a webextension, which is done.