1182132 - Eliminate user action requirement when requestFullscreen({vrDisplay: vrHmd})

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, enhancement)

Description

[Diego Marcos [:dmarcos]]

VR content should be displayed within a headset without requiring any user action. When I visit a website that has a VR experience it should be displayed on page load if a an HMD is available. The headset will be used in extended mode and the main display will render the 2D / non VR content as it would do without any HMD present. There won't be window resizing or resolution changes to match the dimensions of the HMD display. This way the developer tools will remain functional and content development workflow will keep the update code, refresh page, check changes cycle that characterizes the Web. VR content will be also able to link other VR content without forcing the user to take the headset off and click on a VR mode button. Some sort of VR trusted UI will be necessary to introduce a loading bar and possible error messages ("No VR content available for site XXX") when transitioning between sites. Users will consume 2D content on the computer main display and VR within the headset. This will allow any website to easily include VR experiences without impacting the current content. It opens the opportunity to use VR as a complementary medium. Imagine Wikipedia augmenting content with VR pieces to illustrate articles. For instance, the dinosaurs entry could offer a 3D side by side view of different species that will strongly convey their sense of scale. In the future more and more pieces of the traditional Web will transition to the VR medium: We will watch video and read the news within the headset. The technology is still not mature enough to support usage for extended periods of time and many interaction techniques like text input or reading or 2d page navigation have to be refined or totally reinvented. In some cases it's still unclear if VR will offer any advantage over the traditional mouse and keyboard. At this stage is more sensible to keep VR and traditional web media separated and converge them over time as technology improves.

Comment 1

[Josh Carpenter [:jcarpenter]]

Need to discuss this further before anything is implemented. 

* Is this a UX that we want?
* Where does this sit in the priority stack?
* What is degree of engineering work?
* What is performance impact of having two instances of the same site open at same time?

Etc. 

We can also break this bug into smaller pieces. 

1) The flow for entering VR mode (this bug suggests it's automatic, always running in the background)
2) The flow for moving between links when in VR mode

The later we want regardless of what the flow for the former is.

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to Diego Marcos [:dmarcos] from comment #0)
> VR content should be displayed within a headset without requiring any user
> action.

No, I don't think we should do this.

Accessing the HMD isn't much different from accessing any other hardware.  We should ask the user, ideally via a doorhanger, when a site wants to access the HMD.  The user can then choose to Allow & Remember, or just Allow once.  If they Allow & Remember, then future visits will work like you describe, with no user action.  But the first time they need to be aware of what is going on.

From a developer's/content's perspective, it would look like this:

1) code does e.mozRequestFullScreen({ vrDisplay: hmd });
2-a) if the user has allowed & remembered in the past, display a doorhanger notification that says "The site at foo.com is displaying content on your VR headset.", that fades out after a few seconds like the other similar notices do (e.g. pointer lock)
2-b) if the user has not remembered a pref, display a doorhanger saying something like "The site at foo.com would like to access your VR headset.  Continue?"
3) code will receive a fullscreen transition event if it successfully entered fullscreen

Comment 3

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Also, supporting simultaneous VR and non-VR usage should be a separate bug, but we're already 99% there already -- likely all that's needed is something like an extension to window.open() to say "open on VR HMD".  When a page is opened that way, the current page and the target page can communicate via postMessage, which then provides a way to build the shared VR+non-VR experience.  I'm not sure I buy that simultaneous usage being all that interesting though, but we should talk about that in a separate bug.

Comment 4

[Diego Marcos [:dmarcos]]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #2)
> (In reply to Diego Marcos [:dmarcos] from comment #0)
> > VR content should be displayed within a headset without requiring any user
> > action.
> 
> No, I don't think we should do this.
> 
> Accessing the HMD isn't much different from accessing any other hardware. 
> We should ask the user, ideally via a doorhanger, when a site wants to
> access the HMD.  The user can then choose to Allow & Remember, or just Allow
> once.  If they Allow & Remember, then future visits will work like you
> describe, with no user action.  But the first time they need to be aware of
> what is going on.
> 
> From a developer's/content's perspective, it would look like this:
> 
> 1) code does e.mozRequestFullScreen({ vrDisplay: hmd });
> 2-a) if the user has allowed & remembered in the past, display a doorhanger
> notification that says "The site at foo.com is displaying content on your VR
> headset.", that fades out after a few seconds like the other similar notices
> do (e.g. pointer lock)
> 2-b) if the user has not remembered a pref, display a doorhanger saying
> something like "The site at foo.com would like to access your VR headset. 
> Continue?"
> 3) code will receive a fullscreen transition event if it successfully
> entered fullscreen

Getting rid of the user action requirement to display content is important to let VR sites to link other VR sites without forcing the user to approve for any new domain she visits. How can this be achieved without getting of the user action requirement?

Comment 5

[Diego Marcos [:dmarcos]]

Relying on fullscreen API to push VR content into a headset might be giving us unnecessary headaches. I understand VR as another type of media that should play seamlessly if available. The same way sound on a website will play on your headphones without requiring any user action or extra permissions.

Comment 6

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to Diego Marcos [:dmarcos] from comment #4)
> Getting rid of the user action requirement to display content is important
> to let VR sites to link other VR sites without forcing the user to approve
> for any new domain she visits. How can this be achieved without getting of
> the user action requirement?

It can't, in Firefox.  That was never a goal for Firefox itself -- to do that kind of seamless linking requires a VR-aware browser, i.e. Horizon.

Comment 7

[Diego Marcos [:dmarcos]]

We can deliver cross VR site navigation at pretty low cost by relaxing the user action requirement. If the security concerns are insurmountable we could use a CSS attribute to indicate that a specific DOM element contains VR content and that it will be pushed to a headset if there's one available. 

VR is going to be a tethered + desktop experience by the time it hits the consumer market in 2016. Traditional websites consumption should remain in the main display and VR should happen within the headset. As the current state of technology VR doesn't yield a better experience for navigating traditional sites and reading text than normal mouse and keyboard. I'm in favor to use each environment (VR vs 2D) when appropriate instead of shoehorning all the Web into a headset from day one.

Comment 8

[Xidorn Quan [:xidorn] UTC+10]

This should also be marked as fixed because of bug 1160017, no?