578522 - add-ons should be tested in a new browser instance, like CFX does

Status: RESOLVED WONTFIX

(addons.mozilla.org Graveyard :: Add-on Builder, defect, P1)

Description

[Dietrich Ayala (:dietrich)]

while add-ons should have the option to be tested in the browser instance that Flightdeck is running in, it should not be the default.

a new empty browser instance should be started, and the add-on code run there.

Comment 1

[Myk Melez [:myk] [@mykmelez]]

I'm not so sure, since the current model has several valuable benefits.

First, it's the simplest way to test an add-on, since it doesn't require the builder to start a new instance of Firefox, which would take longer and require the developer to switch between windows to start or stop testing the add-on.

Second, it enables developers to test their add-ons with their existing bookmarks, cookies, and other browser data, which makes testing easier in the case of add-ons that access that information (f.e. a Twitter add-on that relies on the user to be signed into Twitter, or a bookmark add-on that displays bookmarks in a different way).

And third, it causes developers to test that their add-on works when installed in the middle of a browser session, which we want to encourage add-ons to support because it means users don't have to restart their browser to use them, a key benefit of building add-ons using Add-on Builder.

(This latter benefit is the reason Atul started working on the |cfx develop| SDK feature, which among other things allows you to test add-ons in an existing browser session.)

I can certainly imagine that there are cases for which it is useful to allow developers to test add-ons in separate sessions with their own profiles, as does the SDK.  But for the casual developers we're targeting with the Add-on Builder, testing in the existing browser session seems like the better option more of the time, and I would be inclined to reserve this feature for the SDK, at least in the short term.

Nevertheless, I'm open to hearing more about why it would be useful to do this.  Perhaps there are considerations about which I'm unaware?

Comment 2

[Dietrich Ayala (:dietrich)]

I filed this because many addons that people were discussing in #jetpack alter the browser content such that testing them could cause dataloss. One guy had an add-on that started out by removing all open tabs except the active one.

I've hit this problem where I wanted to test my add-on, but was worried about the repercussions to my active session, so switched to a new profile on a different instance of the browser.

I think that at the very least there should be a big warning to the user - either on or near the test button, or a warning the first time, that the user can choose to dismiss forever.

I'd prefer to be able have the seamless "clean room" environment that the SDK provides as an option.

Comment 3

[Piotr Zalewa [:zalun]]

Although I think having advanced options will be a plus in the future, implementing this one might be difficult. Is it possible to run firefox within a empty profile from a plugin? 
And to be honest I think sandboxing is good - hitting Test might crash FF and if code was not saved (which should be almost always the case) user will loose the code.

Comment 4

[Dietrich Ayala (:dietrich)]

(In reply to comment #3)
> Although I think having advanced options will be a plus in the future,
> implementing this one might be difficult. Is it possible to run firefox within
> a empty profile from a plugin? 

You'd probably have to make a shell command, opening the current application binary w/ the right cmd line arguments to make this happen. I wrote an add-on that had an embedded xulrunner app, and did this to launch the app from within Firefox.

Comment 5

[Daniel Buchner [:dbuc]]

In this case I thing the right move is to present the user with info describing the test flow and let them make the decision to use a separate profile when developing with FlightDeck or not. 

Developing a "clean room" solution is far more difficult a task, and encouraging developers to code in a different profile nearly identical results. (arguably better in many ways)

Let's do the info and advise step in the very next release, then see if we can augment that with something more substantial as we move forward.

Comment 6

[Atul Varma [:atul]]

Developing a "clean room" solution is interesting, particularly if we're able to use the same debugging tools that we'll eventually be able to use for debugging remote processes (e.g. e10s, mobile). It's also fortunate that the immense work that's gone into making Firefox faster can also help reduce the code-test loop.

It's definitely something that we need to budget time for though, as we currently rely on the Python-based Mozrunner package to perform lots of seemingly-trivial but actually-nontrivial tasks like finding and killing off runaway Firefox processes.

I also think it's important to still let a user just use their current browser for development. This makes it really easy to just jump in and start playing around, without having to do any extra setup like manually modifying a bunch of otherwise-annoying preferences in a sandboxed profile (e.g. the "omg you're submitting unencrypted data over the internets!" dialog).

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Have there been any updates on this (or has it been fully abandoned) since it was last prioritized in April? Is this blocked on any of the e10s or Add-on add-on work?

I'm trying to clear out a bunch of bugs that haven't had any progress in a few months, and this is one in that list.

Comment 8

[Daniel Buchner [:dbuc]]

I believe we want this, but at this point it is far future. At first thought, it feels like something we'd tackle post-client-side add-on building integration.

Comment 9

[Wil Clouser [:clouserw]]

the add-on builder project has been retired.  More info at https://blog.mozilla.org/addons/2013/12/18/add-on-builder/