Open Bug 1244163 Opened 4 years ago Updated 2 years ago

DevTools inaccessible to XUL apps using "firefox -app"

Categories

(DevTools :: General, defect, P4)

defect

Tracking

(firefox47 affected)

Tracking Status
firefox47 --- affected

People

(Reporter: jryans, Unassigned)

References

(Blocks 1 open bug)

Details

In bug 912121 and bug 1203159, we moved DevTools files in several ways.

By the end of bug 1203159, all DevTools files (client and server) end up inside the "dist/browser" directory, meaning they are specific to the browser app that Firefox runs.

The reason that both client and server ended up together is we wanted to have a single directory structure to map "resource://devtools" to.

However, this presents (at least) 2 problems for XUL apps using "firefox -app" to run something other than the browser:

1. You can no longer start a DevTools server from your app, since the files are hidden inside browser
2. The add-on manager fails because it tries to Cu.isModuleLoaded a DevTools file it can't access
:glandium, we spent a while talking about non-Firefox apps.  Can you see any way for them to access the server files (#1)?  Currently it seems like they would need a custom build / would need to extract them from the Firefox omni.ja.

For #2, we can work around this with a try block.
Flags: needinfo?(mh+mozilla)
The only way would be to move the files to the top-level non-browser omni.ja. I thought that's what we ended up doing...
Flags: needinfo?(mh+mozilla)
Before Firefox 44 (which means before bug 912121 and bug 1203159), the state was:

* /browser/devtools files are packaged into the browser omni.ja
* /toolkit/devtools files are packaged into the top-level non-browser omni.ja
* Non-Firefox XUL apps (using -app) have a top-level omni.ja with only DevTools server files
* Non-Firefox XUL apps (using -app) are able to start the DevTools server, etc.
* comm-central apps are able to start the DevTools server, etc.

After bug 912121 which includes the major file migration to /devtools, the state became:

* /devtools/client files are packaged into the browser omni.ja
* /devtools/(server|shared) files into the top-level non-browser omni.ja
* Non-Firefox XUL apps (using -app) have a top-level omni.ja with only DevTools server files
* Non-Firefox XUL apps (using -app) are able to start the DevTools server, etc.
* comm-central apps are able to start the DevTools server, etc.

So, after bug 912121, we have preserved both slim file size in the top omni.ja and the ability for non-Firefox XUL apps to start the server.

After bug 1203159 (and follow up bug 1217687) which desired to expose DevTools files under resource://devtools, the state became:

* For desktop Firefox, /devtools/* is packaged into the browser omni.ja
  * MOZ_DEVTOOLS set to "all", so /devtools/client is included
* For other m-c apps, /devtools/* is packaged into the top-level omni.ja
  * MOZ_DEVTOOLS set to "server" (the default), so /devtools/client is excluded
* Non-Firefox XUL apps (using -app) have a top-level omni.ja with **NO** DevTools files
* Non-Firefox XUL apps (using -app) **CANNOT** start the DevTools server, etc.
* comm-central apps are able to start the DevTools server, etc.
  * Still works for c-c because they use the full build process with MOZ_DEVTOOLS=server

So, we lost DevTools support for external XUL apps after bug 1203159.

I do not see any easy fix here, unfortunately...  One path forward could be for a DevTools server add-on to be created that these apps can install as needed.  Since this only impacts external XUL apps, I would imagine such work would be low priority.

For the specific problem in the add-on manager (#2 in comment 0), I have filed bug 1245251.
I managed to extract files/dirs in firefox/browser/omni.ja which are required to enable "firefox -app" debugging:


chrome/
  devtools/
    content/
    modules/
        devtools/
          acorn/
          server/
          shared/
  en-US/locale/en-US/devtools/ # or other locales

components/
  interfaces.xpt


Note that interfaces.xpt is also required, or Ci.nsIJSInspector is not available to "firefox -app", and causes a hard-to-debug error when trying to connect to the debugger server.

I zipped them (except interfaces.xpt) as `devtools.zip`, and put it into my app dir, and add such lines to the chrome.manifest:


locale devtools en-US jar:devtools.zip!/chrome/devtools/en-US/locale/en-US/devtools/client/
locale devtools-shared en-US jar:devtools.zip!/chrome/en-US/locale/en-US/devtools/shared/
content devtools jar:devtools.zip!/chrome/devtools/content/
resource devtools jar:devtools.zip!/chrome/devtools/modules/devtools/
interfaces interfaces.xpt


Now may app can be remotely debugged again.

However, may app will crash if I use the JS debugger for tens seconds (45-47 are affected). Still trying to figure out why.

Can you split those devtools files into a single .ja file in firefox? Then at least we (external XUL apps authors) can ship it with our apps.
Duplicate of this bug: 1257477
(In reply to Duan Yao from comment #4)
> However, may app will crash if I use the JS debugger for tens seconds (45-47
> are affected). Still trying to figure out why.

This sounds like bug 1225160 which is being investigated.

> Can you split those devtools files into a single .ja file in firefox? Then
> at least we (external XUL apps authors) can ship it with our apps.

I am not sure we have a good way to express what's needed here with the current build system.  But I agree something like this would simplify things for debugging external XUL apps.
(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #6)
> I am not sure we have a good way to express what's needed here with the
> current build system.

It's not only a build system problem. Firefox itself doesn't support doing that. But what you're describing, essentially, is an addon...
So the server files are not included? If it is just about starting the server, check out https://addons.mozilla.org/en-US/firefox/addon/remote-developer-tools-server/ (I hope it still works)
(In reply to Philipp Kewisch [:Fallen] from comment #8)
> So the server files are not included? If it is just about starting the
> server, check out
> https://addons.mozilla.org/en-US/firefox/addon/remote-developer-tools-server/
> (I hope it still works)

Can you verify that this works in firefox 45.0.1? so firefox -app
(In reply to Philipp Kewisch [:Fallen] from comment #8)
> So the server files are not included? If it is just about starting the
> server, check out
> https://addons.mozilla.org/en-US/firefox/addon/remote-developer-tools-server/
> (I hope it still works)
This addon(released in 2014) doesn't include server codes of devtools, just call them.
In recent versions of mozilla-central, there is an install.rdf in /devtools.  It's meant to be used as a way to contribute to DevTools in Firefox without having to build[1].  It's possible this could be used to install DevTools into an XUL app, though I don't have an external XUL app on hand to test this with.

I would imagine if you are able to load the DevTools files via this add-on, you'll still need something to start the server, so that could be the add-on mentioned in comment 8 or a different approach.  It would also be possible to integrate the DevTools UI directly into the XUL app, though that's likely more work.

(In reply to Philipp Kewisch [:Fallen] from comment #8)
> So the server files are not included? If it is just about starting the
> server, check out
> https://addons.mozilla.org/en-US/firefox/addon/remote-developer-tools-server/
> (I hope it still works)

Currently no DevTools files are accessible to external XUL apps, so the server doesn't exist.

[1]: https://developer.mozilla.org/en-US/docs/Tools/Contributing/Contribute_on_nightly#Load_Nightly%27s_devtools_from_the_source_code
A few thoughts, now that I've discovered this bug:

(1) How hard would it be, technically, to move manifest chrome/devtools.manifest from browser/chrome.manifest to toolkit/chrome.manifest, along with the necessary infrastructure in packaging to make that happen?

(2) How big would the perf hits (file size, startup time) be from doing so?  (I'm wondering about impact on the general population, which may not need devtools.)

(3) Regarding comment 4 from Duan Yao, how was the devtools part of omni.ja extracted?  As I understand it, strictly speaking omni.ja is no longer a proper ZIP file.

(4) I've had success loading browser/chrome.manifest in a XULRunner app when Firefox is built with jarred chrome, not omnijar chrome.  How hard would it be to convert a stock Firefox from omnijar to jarred chrome on-the-fly via a Python script, perhaps?

(5) Would it be feasible to add a new build target that packages up the devtools only for external use in a devtools folder, and requiring the developer to add "manifest devtools/manifest" to their chrome.manifest?  If so, this could simply be another build artifact published to http://archive.mozilla.org/pub/firefox/releases/45.3.0esr/ or http://archive.mozilla.org/pub/firefox/releases/49.0b1/, as examples.

I'm willing to do the necessary work, especially if we can get it in for the next ESR.  I'm just throwing some ideas out to see what people think.
Why not make devtools a system addon?
(In reply to Mike Hommey [:glandium] from comment #14)
> Why not make devtools a system addon?

If that's an acceptable solution, sure, I'd be happy to do build system legwork for that.  It's been a long while since I crafted an addon, though.  I need some references on what makes an addon a system addon.  Also, there's still the question of AMO support - moving the addon from build to hosting.

In particular, I'm used to working with install.rdf and chrome.manifest, and it sounds like those are considered obsolete, if this page is any indication:
https://developer.mozilla.org/en-US/Add-ons

Documentation on system addons is a little lacking:
http://gecko.readthedocs.io/en/latest/toolkit/mozapps/extensions/addon-manager/SystemAddons.html

I would love it if someone could cite an example in a mozilla-* repo of a system addon being built right now, as an example to study.  Even test code would be helpful.
Flags: needinfo?(jryans)
(In reply to Alex Vincent [:WeirdAl] from comment #15)
> (In reply to Mike Hommey [:glandium] from comment #14)
> > Why not make devtools a system addon?
> 
> If that's an acceptable solution, sure, I'd be happy to do build system
> legwork for that.  It's been a long while since I crafted an addon, though. 
> I need some references on what makes an addon a system addon.  Also, there's
> still the question of AMO support - moving the addon from build to hosting.


There isn't any difference on the add-on side between a system and normal add-on - the distinction is made on the AddonManager side. Technically this is based on the install location, which is "browser/features" directory for built-in, and profile "features" directory for updates.

If an add-on appears in one of these locations, then it is hidden from the UI, and can't be user disabled. Normal or temporary add-ons with the same ID will override these, so it's possible to allow users to install over system add-ons (this is also useful for dev purposes).

System add-ons are generally built and shipped along with Firefox, and not hosted on AMO. We can ship updates to them via the Firefox update mechanism (AUS aka Balrog) but this is not strictly necessary, it can ship along with Firefox just as any other code does.


> In particular, I'm used to working with install.rdf and chrome.manifest, and
> it sounds like those are considered obsolete, if this page is any indication:
> https://developer.mozilla.org/en-US/Add-ons


install.rdf still works, but restartless add-ons should be used instead - bootstrap.js instead of chrome.manifest - examples at the DXR link below.


> Documentation on system addons is a little lacking:
> http://gecko.readthedocs.io/en/latest/toolkit/mozapps/extensions/addon-
> manager/SystemAddons.html
> 
> I would love it if someone could cite an example in a mozilla-* repo of a
> system addon being built right now, as an example to study.  Even test code
> would be helpful.


Everything in https://dxr.mozilla.org/mozilla-central/source/browser/extensions except for pdfjs are system add-ons that are built and shipped with Firefox (pdf.js is an add-on upstream, but it just gets rolled into the omni.ja by the build system)
(In reply to Alex Vincent [:WeirdAl] from comment #13)
> A few thoughts, now that I've discovered this bug:
> 
> (1) How hard would it be, technically, to move manifest
> chrome/devtools.manifest from browser/chrome.manifest to
> toolkit/chrome.manifest, along with the necessary infrastructure in
> packaging to make that happen?

For the devtools team, it is a useful property to be able to access all of DevTools (client and server) through resource://devtools.  This means all the files must be packaged together.  If we packaged it as part of toolkit instead of browser, it would cause the toolkit omni.ja to include the DevTools client and server, and the client is basically wasted space for an external XUL app.

So, doing this would get the server working again, but at the cost of a extra disk space in the toolkit omni.ja for unnecessary files.  In the past, :glandium has said this would be a bad choice to make for external XUL apps, but I have no opinion about what external XUL apps would prefer myself.

The build system changes to do this aren't too hard.  There may be various client modules that are meant for desktop Firefox that would need to be disabled from starting in other apps.  In-tree apps like Fennec shouldn't be affected, since we can use build flags to strip out the client from their build.

> (2) How big would the perf hits (file size, startup time) be from doing so? 
> (I'm wondering about impact on the general population, which may not need
> devtools.)

I don't think there is impact on Firefox.  External XUL apps might be impacted by DevTools client modules running by default that don't want or need.

> (3) Regarding comment 4 from Duan Yao, how was the devtools part of omni.ja
> extracted?  As I understand it, strictly speaking omni.ja is no longer a
> proper ZIP file.

I am pretty sure most zip utilities will operate on them...?  I've manually packaged them occasionally in the past, but it's been a while.

> (4) I've had success loading browser/chrome.manifest in a XULRunner app when
> Firefox is built with jarred chrome, not omnijar chrome.  How hard would it
> be to convert a stock Firefox from omnijar to jarred chrome on-the-fly via a
> Python script, perhaps?

I am not sure about the details of this route, perhaps build system peers may know more.

> (5) Would it be feasible to add a new build target that packages up the
> devtools only for external use in a devtools folder, and requiring the
> developer to add "manifest devtools/manifest" to their chrome.manifest?  If
> so, this could simply be another build artifact published to
> http://archive.mozilla.org/pub/firefox/releases/45.3.0esr/ or
> http://archive.mozilla.org/pub/firefox/releases/49.0b1/, as examples.

This sounds okay to me, but I don't know much about the details of archiving additional build outputs.
(In reply to Mike Hommey [:glandium] from comment #14)
> Why not make devtools a system addon?

So far I have avoided system add-ons because of the high complexity around updates.  However, :rhelmer has told me on IRC that some of that will be reduced soon (bug 1273709) so that built-in system add-ons continue running even if they are not present in the Balrog update reply.

DevTools has also discussed internally about future plans to use system add-ons on a panel by panel basis for tools that are Gecko specific.  (We are currently in the process of converting the DevTools client to remove Gecko specific and run more like a web site, but we'll likely only convert some portions.)

I'll discuss this ideally with the rest of DevTools team in the next few days.
Flags: needinfo?(jryans)
It's been two weeks, a third of a release cycle.  FF52 central cycle is a couple weeks away, meaning we have about eight weeks left to get a solution ready for the next ESR...
Flags: needinfo?(jryans)
Sorry for the delay.  I discussed this with the team a second time yesterday.

If someone wants to contribute work to package the DevTools server as a system add-on, we are open to accepting that, as long as the implementation doesn't restrict our future plans for the DevTools client.  We should ensure with :rhelmer that we won't need to mess with Balrog update data if we're only shipping the system add-on in Firefox (for now).  We'd like to scope this to just the server components if possible because we're working on moving the client in other directions (to become more like a regular website).  This could imply some complexity, since there are shared modules used by both the client and server at the moment.  

The /devtools directory already contains some packaging to use DevTools as an add-on, but that's for the whole of DevTools.  It was meant for making it easier to contribute DevTools changes[1].  Having said that, would it be possible for external XUL apps to load this existing install.rdf from a Gecko checkout as a way to load the DevTools server when debugging is wanted?

An alternate path forward relates to something :glandium brought up recently[2] on dev-platform: Firefox as a product could decide to stop separating browser and toolkit files.  IIUC, a side effect of that change would mean all the DevTools files are back in the same omni.ja again, so this problem would disappear for external XUL apps.

[1]: https://developer.mozilla.org/en-US/docs/Tools/Contributing/Contribute_on_nightly#Load_Nightly%27s_devtools_from_the_source_code
[2]: https://groups.google.com/d/msg/mozilla.dev.platform/qB-9gJl9Jgs/fVuq5nBwBwAJ
Flags: needinfo?(jryans)
It sounds like we want to do two things:
(1) Invoke the build on the devtools subdirectory only, with CONFIG['MOZ_DEVTOOLS'] = "server"
(2) Update install.rdf, which (strangely) has a hardcoded version number.

Unfortunately, I do not recognize the build system Mozilla uses anymore.  What I had hoped would be obvious is therefore not.  toolkit/moz.build is what decides to take us into the devtools, it seems.

Another possibility is a non-Firefox, non-B2G build might package up the parts we need.  browser/confvars.sh specified MOZ_DEVTOOLS = "all".

/devtools/chrome.manifest is pretty obvious about what it takes to build resource://devtools/ though.  I wonder, now that I see it, if we're really overthinking this and should just zip up devtools/server, devtools/shared, a special chrome.manifest and a special install.rdf.
Blocks: slimerjs
Just Cconfirming that by using Duan Yao's instructions in comment 4 as a reference we have successfully got devtools server running with our xul app using firefox -app

These are the steps we performed with firefox 51 developers edition 

* cd /opt/firefox_dev_51
* mkdir /tmp/omni
* cp browser/omni.ja /tmp/omni/
* cd /tmp/omni/
* unzip omni.ja 
* cp -r chrome/devtools /tmp/myapp/chrome/
* cp components/interfaces.xpt /tmp/myapp/components/
* cp -r chrome/en-US/locale/en-US/devtools /tmp/myapp/chrome/locale/
* Update myapp chrome.manifest and add these lines
<pre>
locale devtools en-US chrome/locale/devtools/client/
locale devtools-shared en-US chrome/locale/devtools/shared/
content devtools chrome/devtools/content/
resource devtools chrome/devtools/modules/devtools/
interfaces components/interfaces.xpt
</pre>

Then in our code that starts the debug server we have...
<code>
debugger: function (portOrPath) {
	try {
		const { devtools } = Cu.import("resource://devtools/shared/Loader.jsm", {});
		const { DebuggerServer } = devtools.require("devtools/server/main");

		if (!DebuggerServer.initialized) {
			DebuggerServer.init();
			DebuggerServer.allowChromeProcess = true;
			DebuggerServer.addBrowserActors("myapp");
		}

		let listener = DebuggerServer.createListener();
		listener.portOrPath = portOrPath;
		listener.open();

		dump("Debugger server started on " + portOrPath + "\n");
	} catch (e) {
		console.log(e);
		Cu.reportError(e);
	}
}
</code>
Based on FF50 going to release yesterday, it's pretty clear I missed FF52's departure to the Aurora track.  Damn university class scheduling.
The devtools addon has been removed from mailnews.  I did manage to get it working at some technical cost in my company's codebase yesterday... please let me know here if anyone is still interested.
At my company we would love to see the client and server moved to the toolkit omni.ja. Running the client directly within the app would be a lot nicer than remotely connecting from another firefox process. It would also enable the use of the web console which replaces the old error console that's been removed some time ago.

Alex: I'm a little unclear of what you got working, if you integrated the client somehow, I would be interested.
I got Firefox's DevTools working in qbrt via some hackery:

https://github.com/mozilla/qbrt/blob/dce317093dd73d579cab1df65bf42fa4593de364/bin/install-runtime.js#L201-L226
https://github.com/mozilla/qbrt/blob/1f9075e3350bdb0b873a660b10a672e3ff6c6298/devtools.manifest

In the long run, however, I suspect the solution here will be to integrate the DevTools extension being developed in bug 1356226.
Product: Firefox → DevTools
You need to log in before you can comment on or make changes to this bug.