[Meta] Support "1 UA multi-screen" on FxOS

RESOLVED WONTFIX

Status

Firefox OS
General
RESOLVED WONTFIX
3 years ago
2 months ago

People

(Reporter: shelly, Unassigned)

Tracking

(Depends on: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(URL)

User Story

Our mission is to support "1 UA" multi-screen rendering on FxOS, which allows a FxOS device connect and render contents to any non-FxOS display, or a device that doesn't have Firefox browser.

Here are some other key interests:

(very important so say it again)
- This is a "1UA multi-screen" support on FxOS, which says a FxOS device can connect to any external display that doesn't run Gecko, nor have the Firefox browser, but still display contents sent from that FxOS device.

- In our case (1UA, multi-screen), contents on remote screen are rendered by the primary device, but in the resolution of that external display.

- Two connection protocols are implemented here, the HDMI cord with a DisplayPort adaptor(wired) and Wifi Display(wireless).

-This is not mirroring, but treat the external screen as a remote displayer, displaying contents sent from the primary device.

To achieve our goal, the first fundamental change, is to allow the sprout of multiple top-level windows in FxOS. Please see the attachment "Structure of multi-screen on b2g for a clearer view.

Attachments

(3 attachments, 15 obsolete attachments)

178 bytes, text/html
Details
1.42 MB, application/pdf
Details
1.29 MB, application/pdf
Details
(Reporter)

Description

3 years ago
= Goal =

With a FxOS mobile device, users are able to project/launch an app or a tab of web content to another display device that does not have gecko running on it (e.g., monitor, tv screen...). The connection between a FxOS device to this non-FxOS display device could via a HDMI cord or other protocols.

= Current =

FxOS creates one xul window when booting up system, and this is the one and only "toplevel" window in FxOS.

= Plans and direction =

Kernel is able to provide a secondary frame buffer, for example, the default frame buffer is used for the built in display screen with type "fbmem0", and "fbmem1" refers to a virtual frame buffer(can be used for HDMI output).

- Restrict one toplevel window per "frame buffer type"
- Able to create another toplevel window via windowWatcher.openWindow(...)(?)
  (or something else)
- Target window is no longer the first element of sTopWindow
http://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsWindow.cpp#194
- A new mechanism to find the focused window when SetFocus() is called 
http://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsWindow.cpp#390
- Able to dequeue a specific type of frame buffer from GonkDisplay() per request
http://dxr.mozilla.org/mozilla-central/source/widget/gonk/nsWindow.cpp#530


The whole plan is still pretty vague, hoping I'm going to the right direction, if not, please feel free to give us any advise!
(Reporter)

Comment 1

3 years ago
Hi Michael, I've brought up this idea to roc at Portland, and he suggested me consulting you about the window creation and management for gonk platform. It would be great if we can have your feedback on this topic, thank you very much!
Flags: needinfo?(mwu)
This will almost certainly require developer documentation of some nature. Any new APIs implemented to support this functionality would need to be covered, but also existing materials would need to be updated to note compatibility of certain existing Window features.
Keywords: dev-doc-needed

Comment 3

3 years ago
Some information sent by email.
Flags: needinfo?(mwu)
(Reporter)

Updated

3 years ago
Summary: Support multi-window on FxOS → Support multi-screen on FxOS
(Reporter)

Comment 4

3 years ago
Created attachment 8561912 [details]
[Demo1] A FxOS device and a WifiDisplay receiver

We have prepared a FxOS device with WifiDisplay support, and a receiver device with other OS (Android, in this case), or none (for the case of TV).
(Reporter)

Comment 5

3 years ago
Created attachment 8561914 [details]
[Demo2] Connect with WifiDisplay protocol

Connect these two devices with WifiDisplay.
(Reporter)

Comment 6

3 years ago
Created attachment 8561921 [details]
[Demo3] Launch 'system-remote' when external screen is ready

When the connection of WifiDisplay is completed and the external screen is ready to be connected, launch 'system-remote' in the external display. In this example, 'system-remote' is an empty html document with blue background. Please reference later attachment for more details about 'system-remote'.
(Reporter)

Comment 7

3 years ago
Created attachment 8561927 [details]
[Demo4] Launch Clock app in the external display

Launch Clock app into the external display. Please note that this is not a mirror of primary display, we still have controls over the FxOS device.
(Reporter)

Comment 8

3 years ago
Created attachment 8561929 [details]
[Demo5] Launch a web site in the external display

Same as the forth demo, this example shows that it can also open a web page in the external display as well.
(Reporter)

Comment 9

3 years ago
Created attachment 8561933 [details]
Structure of multi-screen on b2g

A very basic structure of how b2g supports multi-screen. In this diagram, we only cover how system-remote receives an app URL, and how this app is launched in external display. How to 'control' the app launched in external display is still TBD, and which window gets the actual focus is still TBD too (most likely the remote window won't get the focus).
(Reporter)

Comment 10

3 years ago
Created attachment 8561943 [details]
Action Items

Action items of supporting multi-screen on b2g.
(Reporter)

Updated

3 years ago
Attachment #8561943 - Attachment mime type: text/plain → text/html

Comment 11

3 years ago
Do you have patches or a branch where we can see the changes?
(Reporter)

Comment 12

3 years ago
(In reply to Michael Wu [:mwu] from comment #11)
> Do you have patches or a branch where we can see the changes?

Sure, but they are pretty hacky, let me put them together.
(Reporter)

Comment 13

3 years ago
Created attachment 8562737 [details] [diff] [review]
[WIP] Add 'shell-remote' and some hacks in shell.js
(Reporter)

Comment 14

3 years ago
Created attachment 8562738 [details] [diff] [review]
[WIP] Impl of WifiDisplay
(Reporter)

Comment 15

3 years ago
Created attachment 8562739 [details] [diff] [review]
[WIP] Changes in nsWindow and other related modules
(Reporter)

Comment 16

3 years ago
Still putting together the change set of gfx.
(Reporter)

Updated

3 years ago
Attachment #8562739 - Attachment description: [WIP] Changeset of nsWindow and other related modules → [WIP] Changes in nsWindow and other related modules
(Reporter)

Comment 17

3 years ago
Created attachment 8563186 [details] [diff] [review]
[WIP] Changes in gfx
(Reporter)

Updated

3 years ago
Depends on: 1138258
(Reporter)

Updated

3 years ago
Depends on: 1138269
(Reporter)

Updated

3 years ago
Depends on: 1138271
(Reporter)

Updated

3 years ago
Depends on: 1138349
(Reporter)

Updated

3 years ago
No longer depends on: 1138349
Assignee: nobody → kechen
Assignee: kechen → administration

Updated

3 years ago
Blocks: 1146810
(Reporter)

Updated

3 years ago
Depends on: 1138287
(Reporter)

Updated

3 years ago
Depends on: 1147298
(Reporter)

Updated

3 years ago
No longer depends on: 1138258
(Reporter)

Updated

3 years ago
Summary: Support multi-screen on FxOS → [Meta] Support multi-screen on FxOS
(Reporter)

Updated

3 years ago
Depends on: 1147301
(Reporter)

Updated

3 years ago
Depends on: 1147303
Those demo pictures look fantastic!
Great job guys!!
(Reporter)

Updated

3 years ago
No longer depends on: 1147301
(Reporter)

Updated

3 years ago
No longer depends on: 1147298
(Reporter)

Updated

3 years ago
No longer depends on: 1147303
(Reporter)

Comment 19

3 years ago
Created attachment 8587256 [details]
[slide] Multi-screen on FxOS
Attachment #8561912 - Attachment is obsolete: true
Attachment #8561914 - Attachment is obsolete: true
Attachment #8561921 - Attachment is obsolete: true
Attachment #8561927 - Attachment is obsolete: true
Attachment #8561929 - Attachment is obsolete: true
Attachment #8561943 - Attachment is obsolete: true
Attachment #8562737 - Attachment is obsolete: true
Attachment #8562738 - Attachment is obsolete: true
Attachment #8562739 - Attachment is obsolete: true
Attachment #8563186 - Attachment is obsolete: true
(Reporter)

Comment 20

3 years ago
Created attachment 8587266 [details]
The demo video

Please see!
(Reporter)

Updated

3 years ago
Attachment #8587266 - Attachment mime type: text/plain → text/html

Updated

3 years ago
Depends on: 1150503

Updated

3 years ago
Depends on: 1144103

Updated

3 years ago
Depends on: 1152135

Updated

3 years ago
Depends on: 1151936

Updated

3 years ago
Blocks: 1138811

Updated

3 years ago
No longer blocks: 1138811

Comment 21

3 years ago
Hi Sotaro,

I tried to figure out why does this feature depends on [1] & [2].
For [1], I even think that it should be "this feature blocks [1]".
For [2], I don't see any relationship between this feature and screen recording (related to primary screen only). 

According to this feature is led by Shelly, I discussed with her but she doesn't know the reason as well. Could you share your thought so we can understand what is the dependency?

[1] Bug 1150503 - [Meta] Video/Camera stream output to to the connected external display
[2] Bug 1144103 - Support screen recording
Flags: needinfo?(sotaro.ikeda.g)
(In reply to Marco Chen [:mchen] from comment #21)
> Hi Sotaro,
> 
> I tried to figure out why does this feature depends on [1] & [2].
> For [1], I even think that it should be "this feature blocks [1]".
> For [2], I don't see any relationship between this feature and screen
> recording (related to primary screen only). 
> 
> According to this feature is led by Shelly, I discussed with her but she
> doesn't know the reason as well. Could you share your thought so we can
> understand what is the dependency?
> 
> [1] Bug 1150503 - [Meta] Video/Camera stream output to to the connected
> external display

Isn't it one use case of multi screen? Android already support it. And actually Wifi-Display's one of the major use case.

> [2] Bug 1144103 - Support screen recording

I just add it because it actually uses multi screen capability. 2nd screen renders same thing as primary screen and it is finally rendered to mp4 file.
Flags: needinfo?(sotaro.ikeda.g)
mchen, is there still unclear things?
Flags: needinfo?(mchen)
(In reply to Sotaro Ikeda [:sotaro] from comment #22)
> (In reply to Marco Chen [:mchen] from comment #21)
> > Hi Sotaro,
> > 
> > I tried to figure out why does this feature depends on [1] & [2].
> > For [1], I even think that it should be "this feature blocks [1]".
> > For [2], I don't see any relationship between this feature and screen
> > recording (related to primary screen only). 
> > 
> > According to this feature is led by Shelly, I discussed with her but she
> > doesn't know the reason as well. Could you share your thought so we can
> > understand what is the dependency?
> > 
> > [1] Bug 1150503 - [Meta] Video/Camera stream output to to the connected
> > external display
> 
> Isn't it one use case of multi screen? Android already support it. And
> actually Wifi-Display's one of the major use case.
> 
> > [2] Bug 1144103 - Support screen recording
> 
> I just add it because it actually uses multi screen capability. 2nd screen
> renders same thing as primary screen and it is finally rendered to mp4 file.

About the above, I just want to have a almost same capability to android's Presentation class also on Firefox OS.
  http://developer.android.com/reference/android/app/Presentation.html

I do not understand well about presentation API, therefore I might misunderstand something.

Comment 25

3 years ago
Hi Sotaro,

Thanks for your prompt reply and see my comment as below:

>> For [1], I even think that it should be "this feature blocks [1]".
> Isn't it one use case of multi screen? Android already support it. And actually Wifi-Display's one of the major use case.

Yes, you are right. I agree it is one use case of multi-screen. But as my last comment, it should be "this feature BLOCKS use case [1] (user story)" not depends on [1]. how do you think?

>> [2] Bug 1144103 - Support screen recording
> I just add it because it actually uses multi screen capability. 2nd screen renders same thing as 
> primary screen and it is finally rendered to mp4 file.

Got your point.
I would said "see also" might be a proper one for your case because
  "depends" means that this meta bug can not be closed without [2] landed but I think it is not the case. 
That said I don't think "screen recording" will block this feature. So I suggest to remove this dependency.
Flags: needinfo?(mchen) → needinfo?(sotaro.ikeda.g)

Comment 26

3 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #24)
> > 
> > > [2] Bug 1144103 - Support screen recording
> > 
> > I just add it because it actually uses multi screen capability. 2nd screen
> > renders same thing as primary screen and it is finally rendered to mp4 file.
> 
> About the above, I just want to have a almost same capability to android's
> Presentation class also on Firefox OS.
>   http://developer.android.com/reference/android/app/Presentation.html
> 
> I do not understand well about presentation API, therefore I might
> misunderstand something.

It is cool that you have another idea for second screen capability but I suggest you can split it from "screen recording" bug. I think Screen Recording can't let me link to send screen capability.

Comment 27

3 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #24)
> I do not understand well about presentation API, therefore I might
> misunderstand something.

Shelly is preparing the material so please stay tuned. Thanks.
(In reply to Marco Chen [:mchen] from comment #25)
> Hi Sotaro,
> 
> Thanks for your prompt reply and see my comment as below:
> 
> >> For [1], I even think that it should be "this feature blocks [1]".
> > Isn't it one use case of multi screen? Android already support it. And actually Wifi-Display's one of the major use case.

Sorry, I could not understand  'For [1], I even think that it should be "this feature blocks [1]" ' can you explain by different words?
Flags: needinfo?(sotaro.ikeda.g)
(In reply to Marco Chen [:mchen] from comment #26)
> (In reply to Sotaro Ikeda [:sotaro] from comment #24)
> > > 
> > > > [2] Bug 1144103 - Support screen recording
> > > 
> > > I just add it because it actually uses multi screen capability. 2nd screen
> > > renders same thing as primary screen and it is finally rendered to mp4 file.
> > 
> > About the above, I just want to have a almost same capability to android's
> > Presentation class also on Firefox OS.
> >   http://developer.android.com/reference/android/app/Presentation.html
> > 
> > I do not understand well about presentation API, therefore I might
> > misunderstand something.
> 
> It is cool that you have another idea for second screen capability but I
> suggest you can split it from "screen recording" bug. I think Screen
> Recording can't let me link to send screen capability.

I do not related the above to "screen recording" bug. I just put "screen recording" bug in this bug, because it uses multiscreen capability. How to use this meta bug seems different from I thought. I do not care about to remove "screen recording" bug from this bug.

Updated

3 years ago
No longer depends on: 1144103
From my point of view "screen recording" uses minimum multi screen capability. Therefore its use case was easiest first step toward multi screen support.
(In reply to Sotaro Ikeda [:sotaro] from comment #28)
> (In reply to Marco Chen [:mchen] from comment #25)
> > Hi Sotaro,
> > 
> > Thanks for your prompt reply and see my comment as below:
> > 
> > >> For [1], I even think that it should be "this feature blocks [1]".
> > > Isn't it one use case of multi screen? Android already support it. And actually Wifi-Display's one of the major use case.
> 
> Sorry, I could not understand  'For [1], I even think that it should be
> "this feature blocks [1]" ' can you explain by different words?

mchen, can you explain differently? Your sentence was too abstract for me.
Flags: needinfo?(mchen)
Okey, I seems to understand what you saying. I still think bug 1150503 is very important for multiple display support. But I do not care whether it blocks this bug.
Flags: needinfo?(mchen)

Comment 33

3 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #28)
> > >> For [1], I even think that it should be "this feature blocks [1]".
> > > Isn't it one use case of multi screen? Android already support it. And actually Wifi-Display's one of the major use case.
> 
> Sorry, I could not understand  'For [1], I even think that it should be
> "this feature blocks [1]" ' can you explain by different words?

hm....

Sorry for my poor description. 
I think "An user story should depends on a feature to achieve" so "this feature blocks this user story". 
  Now the multi-screen is a feature and [1] is a user story. So "multi-screen blocks [1]" not depends on [1]. (refer to bug 1142883 - dual display is on the Blocks list not on the depends on list) 

I agree [1] is a very important use case so please allow me to set [1] in the blocks list from depends on list first.

And your cool idea in the screen recording bug is very valuable, please wait for a sync between your idea & us because we are already align with :mwu for our proposal.

[1] Bug 1150503 - [Meta] Video/Camera stream output to to the connected external display
Blocks: 1150503
No longer depends on: 1150503
Thanks for explanation, it becomes clear!
(Reporter)

Updated

3 years ago
User Story: (updated)
Summary: [Meta] Support multi-screen on FxOS → [Meta] Support "1 UA multi-screen" on FxOS
Created attachment 8590917 [details]
nsWindow-GonkDisplay diagram
For now we have a couple of similar features being developed such as this one (multi-screen), 
Wifi Display (mirror mode only) and screen recording (Bug 1144103). After reviewing and 
rethinking all these works, I believe it's the good time to figure out how to put them together. 

Before we go further, please allow me to elaborate what we have done for 
project multi-screen. The user story is what you can see in the demo video. 
(http://goo.gl/gjYlV4). Following is the summary of the story we present in the video:

1) The phone can handle multiple remote displays at a time. Due to the hardware limitation, 
   the phone can only connect to one HDMI display. We use “Miracast” protocol to let the 
   phone be able to have “multiple remote displays”.

2) We can open an app on arbitrary display.

3) We can open a web page on arbitrary display.

4) Deliver API for app developers to open a page on arbitrary display. The way we make the 
   video and picture be playing remotely is to use the API we deliver. (window.open)

5) The content which displays remotely should be rendered in the resolution of the 
   resident display.

In order to carry out the story, we choose “multiple top level windows” approach. 
In the current B2G architecture, only one top level window would be opened 
(in nsDefaultCLH.js) to contain all the contents. Since gecko already supports 
multiple top level windows (and the subsequent multiple layer trees and compositors) 
and to have a the simplicity design, we decide to create a top level window for each 
display and associate its display context with the window. Block diagram in the
attachment 8590917 [details] reveals the idea. (very rough but representative enough)

As you can see, each top level window (nsWindow in the diagram) will be associated 
with one display context owned by GonkDisplay. Different type of display context has 
different consumer. For example, primary and HDMI display context would be consumed by HWC; 
mediaserver would be consuming Wifi Display context while running via Miracast.

The “multiple-top-level-window” design has been implemented and proved to perform 
very well. We also have spoken to Michael Wu regarding this approach. The rest of work 
to get this fundamental infrastructure of “multiscreen” landed is to get rid of 
all the bad smell code and unnecessary dependency between components. 

Everything beyond nsWindow-nsGonkDisplay-HWC is not in the diagram. If you are interested in it, 
we are making a wiki page for the entire story. Will keep you posted.

However, our current design of multi-screen breaks the “mirror mode” function when using Miracast. 
I found that you are doing a great job in Bug 1144103 for “screen recording” and it inspired me 
how to fulfil the user story we defined while not screwing "mirror mode" up. The only concern is 
there GonkDisplay is some duplicate enhancement to deal with multiple render targets / display context 
(Our enhancement of GonkDisplay also owns similar “DisplayDevice” structure). 
It’s not a big deal and can be definitely integrated to each other.

To sum up, we will do our best to land Bug 1138287 first. To address the duplicate GonkDisplay 
enhancement issue, since we are still picking up the screen recording patch, would you mind 
letting us know your plan on landing the screen recording patches? In the mean time, 
could you please review our GonkDisplay changes based on the multiple-top-level-window 
design in mind to facilitate the integration?

Thanks!

Updated

3 years ago
Flags: needinfo?(sotaro.ikeda.g)
(In reply to Henry Chang [:henry] from comment #36)

> 
> 4) Deliver API for app developers to open a page on arbitrary display. The
> way we make the 
>    video and picture be playing remotely is to use the API we deliver.
> (window.open)

Thanks for the summary. Is there a bug or spec how to use "window.open" in this use case?

The reason why I quickly implement screen recording(Bug 1144103) is that Bug 1138287 looks totally different from what I expected. In the bug, I asked the architectural overview, but seem not provided yet.

I am interested in only "local rendering result to external display" use case withing multi-display use case. It requests architectural change. If we want to evaluate if Bug 1138287 is correct from the architectural change point of view. We need a some level of architectural blueprint. I personally want to seek more concrete diagram than attachment 8590917 [details]. attachment 8590917 [details] is great for test implementation. But before implementing for the product, we might want more concrete one.
Flags: needinfo?(sotaro.ikeda.g) → needinfo?(hchang)
Created attachment 8591668 [details]
multi display's architecutal overview diagram

This is my current multi-display's architectural overview. There are a lot of problems needs to address. From this point of view, I prefer incremental implementation approach. Each one cycle fix one relatively small problem.
(In reply to Sotaro Ikeda [:sotaro] from comment #38)
> This is my current multi-display's architectural overview. There are a lot
> of problems needs to address. From this point of view, I prefer incremental
> implementation approach. Each one cycle fix one relatively small problem.

As the incremental approach, it might be easier to extend the used like the following.
[1] screen recording
      + It require minimum architectural change.
[2] output same content of primary display to HDMI
      + It is default behavior on android device.
      + It does not require new document of external display.
        We could limit the change to widget and gfx.
[3] system app could output different contents to 2nd display.
     User could change the output between just mirror and different content.
[4] any apps could output different content to 2nd display if they want.
(In reply to Sotaro Ikeda [:sotaro] from comment #37)
> 
> I am interested in only "local rendering result to external display" use
> case withing multi-display use case.

For this use case, I imaged android capabilities like the followings. For Firefox OS to be competitive as a platform, similar capabilities seems necessary.

  http://developer.android.com/reference/android/app/Presentation.html
  http://developer.android.com/reference/android/media/MediaRouter.html
  http://developer.android.com/reference/android/hardware/display/DisplayManager.html
  http://developer.android.com/reference/android/view/Display.html


About android Presentation class, I created the following.
https://github.com/sotaroikeda/android-diagrams/raw/master/graphics/Presentation_5.1.pdf

Updated

3 years ago
Depends on: 1152370
(In reply to Sotaro Ikeda [:sotaro] from comment #39)
> The following diagrams might be related to attachment 8591668 [details]
> 
> *nsWindow
> https://github.com/sotaroikeda/firefox-diagrams/blob/master/widget/
> widget_nsWindow_FirefoxOS_2_2.pdf
> *window.open()
> https://github.com/sotaroikeda/firefox-diagrams/blob/master/dom/
> dom_window_open_FirefoxOS_2_2.pdf
> *VsyncSource
> https://github.com/sotaroikeda/firefox-diagrams/blob/master/gfx/
> gfx_VsyncSource_FirefoxOS_2_2.pdf
> *ScreenProxy
> https://github.com/sotaroikeda/firefox-diagrams/raw/master/widget/
> widget_ScreenProxy_FirefoxOS_2_2.pdf
> *screen
> https://github.com/sotaroikeda/firefox-diagrams/blob/master/dom/
> dom_screen_FirefoxOS_1_01.pdf
> 
> Another diagrams exits in the following.
>    https://github.com/sotaroikeda/firefox-diagrams/wiki/Firefox-Diagrams

Nice diagrams!

Btw, may I know what tool did you use to make these diagrams? Thanks!
Flags: needinfo?(hchang)
(In reply to Henry Chang [:henry] from comment #42)
> Nice diagrams!
+1 Awesome diagrams, which are really helpful! Great job!

> Btw, may I know what tool did you use to make these diagrams? Thanks!
The extension vsdx is new file format of Microsoft Visio.
I think Sotaro made those diagrams using Visio 2013.
(Reporter)

Comment 44

3 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #37)
> (In reply to Henry Chang [:henry] from comment #36)
> 
> > 
> > 4) Deliver API for app developers to open a page on arbitrary display. The
> > way we make the 
> >    video and picture be playing remotely is to use the API we deliver.
> > (window.open)
> 
> Thanks for the summary. Is there a bug or spec how to use "window.open" in
> this use case?
If you are curious about how gaia is hacked for multi-screen, please see:
https://github.com/shellylin/gaia/commits/multiscreen

Although we are hacking window.open, but after discussing with Alive and Luke about Gaia's implementation, we are not going to address the change to window.open at this phase because it involves API design. Some Gaia's folks like the idea of using window.open but some don't.

As for how system app manages multiple screens, it involves design from UX as well. Luke will takeover bug 1142391 and he will keep us updated.

> 
> The reason why I quickly implement screen recording(Bug 1144103) is that Bug
> 1138287 looks totally different from what I expected. In the bug, I asked
> the architectural overview, but seem not provided yet.
> 
> I am interested in only "local rendering result to external display" use
> case withing multi-display use case. It requests architectural change. If we
> want to evaluate if Bug 1138287 is correct from the architectural change
> point of view. We need a some level of architectural blueprint. I personally
> want to seek more concrete diagram than attachment 8590917 [details].
> attachment 8590917 [details] is great for test implementation. But before
> implementing for the product, we might want more concrete one.

The wiki is only half way done but please feel free to take a look
https://wiki.mozilla.org/Firefox_OS/1UA_Multi-Screen

I'm sorry but I don't get your question here...what kinds of use case would you like to know, the user story? Or if multi-screen capable of the use case of recording/mirror?
And what kinds of information you like to know more from attachment 8590917 [details]?
(Reporter)

Comment 45

3 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #40)
> (In reply to Sotaro Ikeda [:sotaro] from comment #38)
> > This is my current multi-display's architectural overview. There are a lot
> > of problems needs to address. From this point of view, I prefer incremental
> > implementation approach. Each one cycle fix one relatively small problem.
> 
> As the incremental approach, it might be easier to extend the used like the
> following.
> [1] screen recording
>       + It require minimum architectural change.
> [2] output same content of primary display to HDMI
>       + It is default behavior on android device.
>       + It does not require new document of external display.
>         We could limit the change to widget and gfx.
Before we got to multi-screen, we have experiment on mirroring this already, please see my repo here:
https://github.com/shellylin/gecko-dev/commits/multiscreen-mirror-hdmi
They are pretty nasty though...

> [3] system app could output different contents to 2nd display.
>      User could change the output between just mirror and different content.
> [4] any apps could output different content to 2nd display if they want.

Sounds like a nice approach! but I think they are pretty independent and can be implemented parallel. Our goal is to enable the support of opening multiple top-level windows, which is a fundamental feature of gecko browser. Sure we need to think of a way for users to choose which "connection type" is for this remote display(mirror or extended), but it also requires the UX team to involve.
(In reply to Henry Chang [:henry] from comment #42)
> 
> Nice diagrams!
> 
> Btw, may I know what tool did you use to make these diagrams? Thanks!

I use "Visio Professional". I do not know a free tool that has similar usability.
(In reply to Shelly Lin [:shelly] from comment #45)
> 
> Sounds like a nice approach! but I think they are pretty independent and can
> be implemented parallel. Our goal is to enable the support of opening
> multiple top-level windows, which is a fundamental feature of gecko browser.
> Sure we need to think of a way for users to choose which "connection type"
> is for this remote display(mirror or extended), but it also requires the UX
> team to involve.

Yes, [3][4] have a lot of things that could be done independently from [1][2]. I am most concerned about WebAPI part. It could take longer time.

And we need to have concrete UX design to make clear the user experience. As you mentioned, UX team help seems to help it.
(In reply to Ethan Tseng [:ethan] from comment #43)
> I think Sotaro made those diagrams using Visio 2013.

Yes, I used Visio 2013 professional.
Hi Sotaro san,

Let me combine all the replies in this comment.

Q: Is there a bug or spec how to use "window.open" in this use case?
A: Not yet. Since the actual user story is not decided. However, we have a gaia repo for the development.  ( https://github.com/shellylin/gaia/tree/multiscreen ). Briefly speaking, the system app would take all the controls of “window.open”. When system app receives ‘mozbrowserwindowopen’ event, the system app will notify the “remote system app”, which is running in the remote top level window, to load URI in the remote top level window.

Q: The reason why I quickly implement screen recording(Bug 1144103) is that Bug 1138287 looks totally different from what I expected. In the bug, I asked the architectural overview, but seem not provided yet.
A: Sorry about that. attachment 8590917 [details] is the core idea of the implementation. I think the reason the patch in Bug 1138287 looks totally different from what you expected is our core idea is for “multiple top level window”. Multiple top level windows have multiple compositors. In the screen recording case, there is still only one top level window and one compositor. The common part among our intentions is GonkDisplay would be enhanced to contain multiple display context. The screen recording use case is like one compostor maps multiple DisplaySurfaces (one to many) and our use case is like one compositor maps one DisplaySurface (one to one). 

Q: From this point of view, I prefer incremental implementation approach. Each one cycle fix one relatively small problem.
A: I think screen recording and multiscreen is orthogonal to some degree. The fundamental common work is to make GonkDisplay contain multiple DisplaySurface. Based on this change, we can implement screen recording and multiscreen at the same time. Do you agree my argument?

We are reviewing your multiscreen diagram and trying to address the questions. Thanks!
(In reply to Shelly Lin [:shelly] from comment #44)
> I'm sorry but I don't get your question here...what kinds of use case would
> you like to know, the user story? Or if multi-screen capable of the use case
> of recording/mirror?
> And what kinds of information you like to know more from attachment 8590917 [details]
> [details]?

When this bug was created, the scope of this bug is not clear for me. It seems not clear that user stories in Comment 0 is enough to be competitive to android about multi-screen support. For example, bug 1150503 seemed lacking. But from Bug 1138287 Comment 11, it seems not cared.

I feel that only attachment 8590917 [details] is too abstract as a overview architecture. From it we could not know that an application could do bug 1150503 in that architecture. And we could not know which kinds of problems exist in current gecko. Then I feel that diagrams like attachment 8591668 [details] seems also necessary. Then I created one.
(In reply to Henry Chang [:henry] from comment #49)
> Q: From this point of view, I prefer incremental implementation approach.
> Each one cycle fix one relatively small problem.
> A: I think screen recording and multiscreen is orthogonal to some degree.
> The fundamental common work is to make GonkDisplay contain multiple
> DisplaySurface. Based on this change, we can implement screen recording and
> multiscreen at the same time. Do you agree my argument?

screen recording and multiscreen have a lot of things we could do independently. But from gfx point of view, gfx related code needs incremental change towards multi-screen support. It depends on the related components. gfx and widget have a tight relationship. therefore I commented to bug 1138287.
(In reply to Shelly Lin [:shelly] from comment #44)
> The wiki is only half way done but please feel free to take a look
> https://wiki.mozilla.org/Firefox_OS/1UA_Multi-Screen
> 

Thanks for creating the wiki!
(In reply to Sotaro Ikeda [:sotaro] from comment #51)
> (In reply to Henry Chang [:henry] from comment #49)
> > Q: From this point of view, I prefer incremental implementation approach.
> > Each one cycle fix one relatively small problem.
> > A: I think screen recording and multiscreen is orthogonal to some degree.
> > The fundamental common work is to make GonkDisplay contain multiple
> > DisplaySurface. Based on this change, we can implement screen recording and
> > multiscreen at the same time. Do you agree my argument?
> 
> screen recording and multiscreen have a lot of things we could do
> independently. But from gfx point of view, gfx related code needs
> incremental change towards multi-screen support. It depends on the related
> components. gfx and widget have a tight relationship. therefore I commented
> to bug 1138287.

Hi Sotaro san,

(I will simply use "multiscreen" to refer to "showing different content on different device"
in the following.)

For multiscreen case, only few gfx code changes need to be made because all the gfx 
code has considered multiple top level windows and multiple compositors case. 
That's the main reason we choose the approach. The minor gfx changes includes 
(from patch in Bug 1138287)

1) In GLContextProviderEGL.cpp, when to create GLContext with "offscreen == true"
2) When using HwcComposer, what display is needed to draw (PRIMARY_DISPAY or EXTERNAL_DISPLAY)

(Do you regard it as an architectural change?)

For nsWindow part, we will add supported flags to nsIWindowWatcher.openWindow to
deliver "mDisplayType" to nsWindow. Every original nsWindow operation associated
with GonkDisplay could use "mDisplayType" to obtain corresponding display context
from GonkDisplay.

For "screen recording", it requires more gfx changes like the need of two-pass rendering,
which is not natively considered.

---------------------------------------------------------------------------------

attachment 8591668 [details] is really really good! The following are my answers to your questions:

Q: Can't we open a top document of non-primary display?
A: We could but what's the downside of opening a window rather than a document?

Q: How do we specify display id as argument?
A: TBD. In the current implementation, we only use a special flag to tell
   system app "this URL is going to show on the remote display" and system app
   would prompt user to choose a display to show. It's actually not the realm of gecko.

Q: How can gaia side get all displays info?
A: We will propose something like DisplayManager

Q: Can't we need android's MediaRouter kind of capabilities?
A: I think we need it eventually! However, since the user story is not finalised,
   the current user story is too simple to need it.

Thanks!
(In reply to Henry Chang [:henry] from comment #53)
> 
> For multiscreen case, only few gfx code changes need to be made because all
> the gfx 
> code has considered multiple top level windows and multiple compositors
> case. 
> That's the main reason we choose the approach. The minor gfx changes
> includes 
> (from patch in Bug 1138287)
> 

The above is not correct. For a test implementation it might be OK. But gfx code needs some big modification if we want to implement correctly. We reuse android's low level implementations. It is same to hardware composer. On android, all compositions of display are done in sync to primary display's vsync. On b2g, we also have to do same thing. On current gecko, vsync is handled independently by CompositorParent. And there is no way to compose all CompositorParents in one pass.

SurfaceFlinger::onVSyncReceived()
  http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp#784
SurfaceFlinger::setUpHWComposer()
  http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp#1016
HWComposer::prepare()
  http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp#784
HWComposer::commit()
  http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/surfaceflinger/DisplayHardware/HWComposer.cpp#739
(In reply to Henry Chang [:henry] from comment #53)
> 
> For nsWindow part, we will add supported flags to
> nsIWindowWatcher.openWindow to
> deliver "mDisplayType" to nsWindow. Every original nsWindow operation
> associated
> with GonkDisplay could use "mDisplayType" to obtain corresponding display
> context
> from GonkDisplay.
> 

I am not fun of using DisplayType here. DisplayId might be better. On gonk, HWC handle display is limited to at most 3 even on latest HWC. But actual number of display should not be limited to it. Android's SurfaceFlinger does not have such limiation. At first, the SurfaceFlinger tried to allocate HwcDisplayId and if it failed, the created display is composed only by using OpenGL.


http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/surfaceflinger/SurfaceFlinger.cpp#1358
(In reply to Henry Chang [:henry] from comment #53)
> For "screen recording", it requires more gfx changes like the need of
> two-pass rendering,
> which is not natively considered.

From my point of view, two pass rendering is mandatory for "HDMI display output" use case. On android device, if we just connected the android device to HDMI display, normally primary display's content is also shown HDMI connected display by default.

HDMI display connection's use case, if there is no application that take the HDMI display explicitly, it should show primary display's content, I think.
(In reply to Henry Chang [:henry] from comment #53)
> 
> Q: Can't we open a top document of non-primary display?
> A: We could but what's the downside of opening a window rather than a
> document?

The questions were just for myself. It expect to use window.open() as a function. If there is a way to open a document without using window.open(), it should be OK.
(In reply to Sotaro Ikeda [:sotaro] from comment #56)
> 
> From my point of view, two pass rendering is mandatory for "HDMI display
> output" use case. On android device, if we just connected the android device
> to HDMI display, normally primary display's content is also shown HDMI
> connected display by default.

It means that primary display's content is shown to the HDMI display by default.
(In reply to Sotaro Ikeda [:sotaro] from comment #55)
> 
> I am not fun of using DisplayType here. DisplayId might be better. On gonk,
> HWC handle display is limited to at most 3 even on latest HWC. But actual
> number of display should not be limited to it. Android's SurfaceFlinger does
> not have such limiation. At first, the SurfaceFlinger tried to allocate
> HwcDisplayId and if it failed, the created display is composed only by using
> OpenGL.
> 

By the way, on android, display id is allocated in DisplayManagerService.addLogicalDisplayLocked().

http://androidxref.com/5.1.0_r1/xref/frameworks/base/services/core/java/com/android/server/display/DisplayManagerService.java#734
(In reply to Sotaro Ikeda [:sotaro] from comment #56)
> 
> HDMI display connection's use case, if there is no application that take the
> HDMI display explicitly, it should show primary display's content, I think.

It mean there should be a mechanism to switch between "mirror mode" and "display document composition mode" safely. My image to it is similar to SurfaceControl.setDisplayLayerStack().

http://androidxref.com/5.1.0_r1/xref/frameworks/base/core/java/android/view/SurfaceControl.java#571
(In reply to Sotaro Ikeda [:sotaro] from comment #54)
> (In reply to Henry Chang [:henry] from comment #53)
> > 
> > For multiscreen case, only few gfx code changes need to be made because all
> > the gfx 
> > code has considered multiple top level windows and multiple compositors
> > case. 
> > That's the main reason we choose the approach. The minor gfx changes
> > includes 
> > (from patch in Bug 1138287)
> > 
> 
> The above is not correct. For a test implementation it might be OK. But gfx
> code needs some big modification if we want to implement correctly. We reuse
> android's low level implementations. It is same to hardware composer. On
> android, all compositions of display are done in sync to primary display's
> vsync. On b2g, we also have to do same thing. On current gecko, vsync is
> handled independently by CompositorParent. And there is no way to compose
> all CompositorParents in one pass.
> 
> SurfaceFlinger::onVSyncReceived()
>  
> http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/
> surfaceflinger/SurfaceFlinger.cpp#784
> SurfaceFlinger::setUpHWComposer()
>  
> http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/
> surfaceflinger/SurfaceFlinger.cpp#1016
> HWComposer::prepare()
>  
> http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/
> surfaceflinger/SurfaceFlinger.cpp#784
> HWComposer::commit()
>  
> http://androidxref.com/5.1.0_r1/xref/frameworks/native/services/
> surfaceflinger/DisplayHardware/HWComposer.cpp#739

Hi Sotaro san,

1) We had discussed with Jerry earlier on and he said he didn't know
   if HWC is able to hook each vsync for each display but his gecko
   implementation has the flexibility to provide vsync for different display.
   It might require him to answer this question.

2) Regarding "there is no way to compose all CompositorParents in one pass.",
   I wonder if there is any reason that we need to compose all CompositorParents
   in one pass? We are not the expert but I would imagine different display
   would have different refresh rate and the subsequent vsync. Having compose
   all CompositorParents in one pass might not be a MUST? Sorry for my guess
   and please correct me if I am wrong.   

3) I buy your mDisplayId idea and we do need such a mapping from this logical
   display id (defined on our own) to physical display id (defined by HWC)
   in this case.

4) For "showing same content on HDMI" case, what if we reuse the fbHandle
   grabbed from FrameBufferSurface? We tried something like that (ignore the fence though)
   and it worked with some tearing. Do you think the attempt to reuse the fbHandle is on
   the right track?

We really appreciate all your comments! Thanks!
(In reply to Henry Chang [:henry] from comment #61)
> 
> 2) Regarding "there is no way to compose all CompositorParents in one pass.",
>    I wonder if there is any reason that we need to compose all
> CompositorParents
>    in one pass? We are not the expert but I would imagine different display
>    would have different refresh rate and the subsequent vsync. Having compose
>    all CompositorParents in one pass might not be a MUST? Sorry for my guess
>    and please correct me if I am wrong.

About hwc display, we need to do it, because current hwc hal api is implemented to do it and SurfaceFlinger works such way.
 
> 4) For "showing same content on HDMI" case, what if we reuse the fbHandle
>    grabbed from FrameBufferSurface? We tried something like that (ignore the
> fence though)
>    and it worked with some tearing. Do you think the attempt to reuse the
> fbHandle is on
>    the right track?

It could be one option for the optimization in future. It has one drawback. If ImageLayer(like video) is composed by OpenGL and primary display and HDMI display's resolution is different, composition result is not optimal. Because ImageLayer's image scaling is done in final composition.

And to prevent hidden problems, at first, I prefer same way to SurfaceFlinger. That is the most low risk implementation.
(In reply to Sotaro Ikeda [:sotaro] from comment #62)
> > 4) For "showing same content on HDMI" case, what if we reuse the fbHandle
> >    grabbed from FrameBufferSurface? We tried something like that (ignore the
> > fence though)
> >    and it worked with some tearing. Do you think the attempt to reuse the
> > fbHandle is on
> >    the right track?
> 
> It could be one option for the optimization in future. It has one drawback.
> If ImageLayer(like video) is composed by OpenGL and primary display and HDMI
> display's resolution is different, composition result is not optimal.
> Because ImageLayer's image scaling is done in final composition.
> 
> And to prevent hidden problems, at first, I prefer same way to
> SurfaceFlinger. That is the most low risk implementation.

If there is "GlobalCompositor?/CompositorManager?" kind of class exist, the switching between Mirror and original content and hwc display's sync composition could be managed safely, I think.
Hi, there,

I'd like to update the current progress on gaia.

Since we are in the proof-of-concept stage, after discussing with Shelly and Alive, we cut down some user stories in order to merge them back to master first. To avoid altering the API, we agreed not to implement "window.open(..)" hack at this time and will keep finding the proper way.

Followings are our plan:

1. Add a preference in settings for enabling multiscreen support and turn it off by default. (Followings happen only when the preference is on and an external display is connected.)
2. Add options to BrowserContextMenu for launching the current web page to the connected external display.
3. Launch every app to the connected external display when clicking icons on homescreen.
4. There will always be one app at a time in the remote desktop so the previous app will be terminated when a new app is launched.
5. No interaction involved in the remote desktop so users can't control the remote app.

I'll break down these tasks into bugs and update the wiki soon.

Any suggestions are welcome. Thanks.
Depends on: 1156647
No longer depends on: 1156647

Updated

3 years ago
Depends on: 1156981

Updated

3 years ago
Depends on: 1157441

Updated

3 years ago
Depends on: 1157874

Updated

3 years ago
Depends on: 1161874
(Reporter)

Comment 65

3 years ago
Created attachment 8601949 [details]
[slides] Multi-screen on FxOS

Some minor fixes.
Attachment #8587256 - Attachment is obsolete: true
(Reporter)

Updated

3 years ago
Depends on: 1169176
(Reporter)

Updated

3 years ago
Depends on: 1173258
(Reporter)

Comment 66

3 years ago
Created attachment 8621429 [details]
[slides] Multi-screen on FxOS (overview, involve no tech)

Fix typo.
Attachment #8561933 - Attachment is obsolete: true
Attachment #8590917 - Attachment is obsolete: true
Attachment #8591668 - Attachment is obsolete: true
Attachment #8601949 - Attachment is obsolete: true
(Reporter)

Comment 67

3 years ago
Created attachment 8621431 [details]
[slides] Multi-screen on FxOS (focus on technical discussion, implementation design)
feature-b2g: --- → 2.2r+
feature-b2g: 2.2r+ → 2.2r?

Updated

3 years ago
Depends on: 1186000
Depends on: 1186800
No longer a feature-b2g:2.2r since we can't take the efforts to backport all necessary changes to 2.2r.
feature-b2g: 2.2r? → ---
No longer blocks: 1142883

Comment 69

3 months ago
Firefox OS is not being worked on
Status: NEW → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → WONTFIX
Keywords: dev-doc-needed
You need to log in before you can comment on or make changes to this bug.