847566 - Developing in the browser is painful

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Kevin Grandon :kgrandon]

FirefoxOS development in the browser is painful for a number of reasons:

- System/Homescreen apps don't work
- Not enough shims in apps, and adding shims to apps slows down the production app.
- Missing APIs

I have create a plugin that can address these concerns, and this bug is to track the implementation of that plugin within gaia.

Comment 1

[Kevin Grandon :kgrandon]

Attachment: Github pull request pointer

Hi Vivien,

I wanted to avoid having a constantly changing ~300kb binary in the source code, so I believe a remote download of the plugin is our best option for now. I strongly believe that implementing this will make development much more efficient on the platform. Please let me know what you think. Thanks!

Comment 2

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Comment on attachment 720867 [details]
Github pull request pointer

Let's see if this efforts can be combined with FFOs simulator at some point.

Comment 3

[Alexandre Poirot [:ochameau]]

Kevin, Depending on external resources like this is a bit scary.
You should be able to land your addon source code directly into gaia, like httpd addon.
Unfortunately, you are using jetpack, and SDK addon traditionally requires a build step. So it would means requiring cfx as build dependency.
It doesn't sound reasonable either.
*But* the cool thing is that, starting with FF21, SDK modules are in firefox.
It basically means that it would be possible to write a bootstrap addon using SDK APIs. I can try to do that with your addon.

Otherwise, as vivien said, we can also try to use your effort in the simulator. Support for using firefox as FirefoxOS runtime is a long term goal for the simulator. It will take quite some time before we will be able to run the whole OS on top of Firefox runtime, but it could be worth integrating your addon in the simulator in order to have a decent "single app support" right now, in firefox. The benefit of this is that we can gain from existing firefox devtools by opening one given app in a new firefox tab. And if we integrate it in the simulator, your work will have a bigger visibility.

Comment 4

[Kevin Grandon :kgrandon]

It's not scary to rely on external development dependencies as we already do this with XULRunner and even other unit test resources.

I appreciate the work that you guys are doing with the simulator, and our end goals are the same. This plugin solves the long-term goals that you guys have toda. Once you have support for a native firefox environment, I'm more than happy to kill this plugin.

Comment 5

[Ben Francis [:benfrancis]]

(In reply to Alexandre Poirot (:ochameau) from comment #3)
> Support for using firefox as FirefoxOS runtime is a long term
> goal for the simulator.

What does this mean exactly?

The two big issues that prevent hacking on privileged apps inside Firefox are:
1) Privileged apps are not supported by Firefox (either hosted due to the security model not allowing it or packaged due to the lack of support for packaged apps).
2) Therefore developers can't use the Firefox developer tools.

Since apps in Firefox stopped launching in a tab in the browser UI and started launching in special runtime windows, I don't think you can use the developer tools at all for installed apps launched in the Firefox runtime.

One option is to add the ability to grant any privileged permissions to hosted apps during development so the user can just load the app as a hosted app by typing its URL. This has the huge benefit of being able to just refresh the page every time you make a change and would instantly give access to developer tools. We used to have this feature behind a pref, but it got removed due to security concerns.

Another option is to add support for privileged packaged apps to Firefox and make the developer tools work for the Firefox runtime windows. This would give access to the developer tools but mean that developers have to re-package (and re-install?) their app every time they make a change.

Another option is to go the route of developing inside a B2G Desktop/Simulator but make all the developer tools remoteable. This still requires repackaging & re-installing on every change but requires less changes to Firefox.

Comment 6

[Alexandre Poirot [:ochameau]]

(In reply to Ben Francis [:benfrancis] from comment #5)
> (In reply to Alexandre Poirot (:ochameau) from comment #3)
> > Support for using firefox as FirefoxOS runtime is a long term
> > goal for the simulator.
> 
> What does this mean exactly?

I had that one in mind:

> One option is to add the ability to grant any privileged permissions to
> hosted apps during development so the user can just load the app as a hosted
> app by typing its URL. This has the huge benefit of being able to just
> refresh the page every time you make a change and would instantly give
> access to developer tools. We used to have this feature behind a pref, but
> it got removed due to security concerns.

I think that's what Kevin is doing with this addon.
Currently the simulator only target device or b2g desktop.
It looks like we could easily add a firefox target by using this addon.
As the simulator is a jetpack addon, it will be very easy to use its code.

Comment 7

[Kevin Grandon :kgrandon]

I would love to see a path of convergence for the simulator and the runtime plugin - it makes total sense to focus our efforts on a single solution.

If we were to head down this path, I'd want to check with the simulator team and see what kind of timeframe we were looking at. At a minimum here's what I think needs to happen in the simulator: 

1 - Firefox becomes a valid target for the simulator.
2 - Ability to dynamically read out of a gaia directory on disk.

Comment 8

[Ben Francis [:benfrancis]]

(In reply to Kevin Grandon :kgrandon from comment #7)
> 1 - Firefox becomes a valid target for the simulator.

What does this mean? The simulator has its own gecko build so provides a user interface to install apps into that gecko runtime. What's the benefit of using the simulator UI to install an app into Firefox when Firefox already has UI to install apps? If Firefox can run privileged apps then surely its own install interface would work just as well?

> 2 - Ability to dynamically read out of a gaia directory on disk.

Over file://, app:// or http:// ? I think there are issues with running web apps over file://

Comment 9

[Kevin Grandon :kgrandon]

(In reply to Ben Francis [:benfrancis] from comment #8)
> 
> What does this mean? The simulator has its own gecko build so provides a
> user interface to install apps into that gecko runtime. What's the benefit
> of using the simulator UI to install an app into Firefox when Firefox
> already has UI to install apps? If Firefox can run privileged apps then
> surely its own install interface would work just as well?

I'm not too sure on the current app interface in Firefox. I'll try to take a look, and even if we have to do some XUL work in Firefox, that seems like the right way to go.


(In reply to Ben Francis [:benfrancis] from comment #8)
> 
> Over file://, app:// or http:// ? I think there are issues with running web
> apps over file://

Sorry, should've been more clear here. I just want to be able to save a file in the system or homescreen app, then click refresh to see my changes.

Comment 10

[Myk Melez [:myk] [@mykmelez]]

(In reply to Alexandre Poirot (:ochameau) from comment #3)
> You should be able to land your addon source code directly into gaia, like
> httpd addon.
> Unfortunately, you are using jetpack, and SDK addon traditionally requires a
> build step. So it would means requiring cfx as build dependency.
> It doesn't sound reasonable either.

This doesn't actually seem that unreasonable.  The Simulator does it by including the Add-on SDK repository as a submodule, and that works pretty well, although there are some tricky details to Git submodules.  For Gaia, it'd probably work better for the build scripts to download the Add-on SDK the same way they download the XULRunner SDK.


(In reply to Kevin Grandon :kgrandon from comment #4)
> It's not scary to rely on external development dependencies as we already do
> this with XULRunner and even other unit test resources.

Indeed, it seems reasonable enough.  It isn't perfect (B2G builds recently had trouble downloading external dependencies due to server availability issues), but it should generally work fine.  And the Simulator does it, too, as its build scripts download pre-built copies of B2G and ADB.


> I appreciate the work that you guys are doing with the simulator, and our
> end goals are the same. This plugin solves the long-term goals that you guys
> have toda.

That's somewhat true, as we too plan to reuse Firefox instead of shipping our own XULRunner.  But we intend to continue loading Gaia inside B2G instead of a regular Firefox browser window to provide a more complete simulation.


(In reply to Ben Francis [:benfrancis] from comment #5)
> (In reply to Alexandre Poirot (:ochameau) from comment #3)
> > Support for using firefox as FirefoxOS runtime is a long term
> > goal for the simulator.
> 
> What does this mean exactly?

It means running B2G/Gaia on Firefox's XULRunner instead of a separate copy of XULRunner that we ship with the Simulator.


> Since apps in Firefox stopped launching in a tab in the browser UI and
> started launching in special runtime windows, I don't think you can use the
> developer tools at all for installed apps launched in the Firefox runtime.

I suspect you actually could use them if you set the relevant preferences, although I haven't tested it.  If not, though, then it should be straightforward to make it possible, since the desktop runtime is a XUL application that includes (or can be made to include) the same Toolkit components for remote debugging as Firefox and other XUL apps.


> One option is to add the ability to grant any privileged permissions to
> hosted apps during development so the user can just load the app as a hosted
> app by typing its URL. This has the huge benefit of being able to just
> refresh the page every time you make a change and would instantly give
> access to developer tools. We used to have this feature behind a pref, but
> it got removed due to security concerns.

We already added this feature to the Simulator at your request last fall for a developer event you were running, but I never heard back from you after the event about how it worked for you.  Nevertheless, the feature remains in the Simulator, and I remain open to improving it to meet this use case better!


> Another option is to add support for privileged packaged apps to Firefox and
> make the developer tools work for the Firefox runtime windows.

bwalker (cc:ed) is working to prioritize privileged packaged apps on desktop.  His aim is to provide the feature to users, not developers, but you're absolutely right that it could benefit developers as well.


> This would
> give access to the developer tools but mean that developers have to
> re-package (and re-install?) their app every time they make a change.

Not necessarily.  Recently I've been working out a Simulator feature that will automatically repackage, reinstall, and restart a packaged app, i.e. the equivalent of Ctrl-R for packaged apps (and likely to be attached to the Ctrl-R shortcut within the Simulator process, so it is literally just as easy).  I could imagine a "developer mode" for the desktop runtime that provides the same functionality (in addition to enabling the developer tools by default).


> Another option is to go the route of developing inside a B2G
> Desktop/Simulator but make all the developer tools remoteable. This still
> requires repackaging & re-installing on every change but requires less
> changes to Firefox.

dcamp (cc:ed) says it's a top priority for his organization to make all existing devtools remoteable by the end of Q2 (and a hard requirement for all new devtools to be remoteable by default).  So this is going to happen soon regardless.


(In reply to Kevin Grandon :kgrandon from comment #7)
> I would love to see a path of convergence for the simulator and the runtime
> plugin - it makes total sense to focus our efforts on a single solution.

I'm not sure this is entirely true.  The Simulator targets third-party app developers, while the Firefox OS Runtime targets Gaia developers, and those audiences seem to have different needs and desires.  In particular, Gaia developers seem to prefer developing inside Firefox tabs, whereas third-party developers seem to prefer the more isolated simulation that the Simulator provides.

Those differences might be mere artifacts of immature tools and documentation, but they could also reflect real distinctions between the audiences.  Nevertheless, it's certainly worth digging deeper into them and investigating potential paths of convergence.  Product managers like those on bmaggs's team (cc:ed) may be able to help.


(In reply to Ben Francis [:benfrancis] from comment #8)
> (In reply to Kevin Grandon :kgrandon from comment #7)
> > 1 - Firefox becomes a valid target for the simulator.
> 
> What does this mean? The simulator has its own gecko build so provides a
> user interface to install apps into that gecko runtime. What's the benefit
> of using the simulator UI to install an app into Firefox when Firefox
> already has UI to install apps? If Firefox can run privileged apps then
> surely its own install interface would work just as well?

The Simulator's Dashboard tab was originally designed to be an interface for installing an app into the simulation.  But we've since overloaded it with an affordance for pushing an app to a physical device.  And we've discussed adding Firefox tabs as a third target.

So the benefit of using the Simulator's interface to install/run an app in Firefox is that it provides a single, memorable, developer-centric interface for accessing the various available targets.

(The UX challenge, however, is making it clear to a developer what the various options are and when to choose which one.  Without such guidance, more options will just impose a tyranny of choice.)

Comment 11

[Myk Melez [:myk] [@mykmelez]]

Comment on attachment 720867 [details]
Github pull request pointer

Regardless of what we eventually do with developer tools in general, and even though it adds an external dependency, it still seems reasonable to add this to the Gaia build scripts as a non-default option.

Comment 12

[Vivien Nicolas (:vingtetun) (:21) - (NOT reading bugmails, needinfo? please)]

Attachment: Patch

I repack a little bit your code. Great work, thanks Kevin!

Comment 13

[Kevin Grandon :kgrandon]

Attachment: Github pull request pointer

Comment 14

[Kevin Grandon :kgrandon]

Comment on attachment 735307 [details]
Github pull request pointer

First PR has landed in master: https://github.com/mozilla-b2g/gaia/commit/68efb2e61ef183d4b4a2b05421c7562bd67de8a6

Comment 15

[Ben Francis [:benfrancis]]

This is awesome!

Do we need a separate bug for "Developing third party privileged apps is painful"?

Comment 16

[Kevin Grandon :kgrandon]

Landed a small fix here: https://github.com/mozilla-b2g/gaia/pull/9078

closing this out for now. Thanks all!

Comment 17

[Kevin Grandon :kgrandon]

(In reply to Ben Francis [:benfrancis] from comment #15)
> This is awesome!
> 
> Do we need a separate bug for "Developing third party privileged apps is
> painful"?

Thanks for the suggestion. I have filed Bug 860219 for this, please chime in with your suggestions!

Comment 18

[Kevin Grandon :kgrandon]

Attachment: Github pull request pointer 2

Comment 19

[Kevin Grandon :kgrandon]

Attachment: Github pull request pointer 3

Received a review from James on this one.

Comment 20

[Kevin Grandon :kgrandon]

Part 2 landed: https://github.com/mozilla-b2g/gaia/commit/cc6f54a9a85ce08c202cfc1dbd6bc58c9ed79907

Part 3 landed: https://github.com/mozilla-b2g/gaia/commit/f6587418cc5fd8dfe885539ae1a798473890c7f5

Comment 21

[Jan Jongboom [:janjongboom] (inactive)]

Attachment: mozSetMessageHandler shim

Comment 22

[Kevin Grandon :kgrandon]

Attachment: Initial dev panel implementation

Comment 23

[Alexandre Poirot [:ochameau]]

Comment on attachment 736213 [details]
Initial dev panel implementation

Looks good and... open doors for awesometies!

Comment 24

[Kevin Grandon :kgrandon]

Comment on attachment 736213 [details]
Initial dev panel implementation

This has landed in master: https://github.com/mozilla-b2g/gaia/commit/bc68664d822582e85b1ef285db5004d6055b977e

Comment 25

[Alexandre Poirot [:ochameau]]

Attachment: Pull request 9115

Here is a followup to PR 9113, to avoid using __exposedProps__ (that is going to be deprecated) and ensure that we always start firefox with the system app and the devtool panel being opened.

Comment 26

[Ben Francis [:benfrancis]]

Hmm, this is now starting to overlap with the Firefox OS Simulator quite a lot.

It's very cool, but this bug has already been marked resolved and I would argue this new "FFOS Control" UI is outside of the scope of the original bug report.

It's one thing to get Gaia apps running in Firefox to provide access to developer tools, it's another to start simulating hardware buttons of a device.

I'd be careful about going too much further with this without some kind of roadmap for where these two tools are heading.

Comment 27

[Kevin Grandon :kgrandon]

@Ben - you're totally right here. This was more of a proof of concept to see what it would be like, and potentially drive a roadmap forward. I'm going to land this last patch, then we will stop using this bug to track further changes. Thanks for your feedback!

Comment 28

[Kevin Grandon :kgrandon]

Last patch has landed here: https://github.com/mozilla-b2g/gaia/commit/170565b7789c2d5fa4a7b8b2fac43702b35bd2af. We'll use new bugs if we decide on any new improvements for this.