1002350 - [Stingray] An algorithm for finding the nearest element on 2D plane

Status: RESOLVED INVALID

(Firefox OS Graveyard :: Gaia::Homescreen, defect)

Description

[Luke Chang [:lchang] (inactive)]

There may be no touch events on Stingray so we may support alternate methods like spatial navigation and selection border to let users can interact with a hardware keyboard (or a remote controller).

Comment 1

[John Hu [:johnhu][:johu][:醬糊小弟]]

As we discussion offline, we move selection border as another bug and keep this bug to trace spatial navigation.

Comment 2

[Luke Chang [:lchang] (inactive)]

Attachment: Pull Request 19012

Let's add spatial navigation for stingray!

Comment 3

[Luke Chang [:lchang] (inactive)]

Hi John, I've addressed the opinions. Please help to review it again. Thanks a lot.

Comment 4

[John Hu [:johnhu][:johu][:醬糊小弟]]

Comment on attachment 8418507 [details] [review]
Pull Request 19012

Cool. I don't have no more questions.

Comment 5

[John Hu [:johnhu][:johu][:醬糊小弟]]

Don't forget to rename the test folder as "_test" to disable the unit tests.

Comment 6

[[:fabrice] Fabrice Desré]

It's sad to do that in gaia. We have (or had) support on the gecko side. Brad, didn't you work on that for fennec? Do we still have some platform support?

Comment 7

[John Hu [:johnhu][:johu][:醬糊小弟]]

We know there is an implementation in gecko. But it only supports input element to be focused[1]. In our case, we may want to navigate some "DIV" or "SPAN" or other non-input element.

BTW, the documentation of spatial navigation is too hard to find. We don't know how to enable it in nightly.


[1] http://dxr.mozilla.org/mozilla-central/source/toolkit/modules/SpatialNavigation.jsm#347-362

Comment 8

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to John Hu [:johnhu][:johu][:醬糊小弟] from comment #7)
> We know there is an implementation in gecko. But it only supports input
> element to be focused[1]. In our case, we may want to navigate some "DIV" or
> "SPAN" or other non-input element.
Then improve the gecko implementation to suit your needs.


> BTW, the documentation of spatial navigation is too hard to find. We don't
> know how to enable it in nightly.
Desktop doesn't support spatial nav anymore. For Fennec, you can see from the code that you set the "snav.enabled" pref to enable it and various other "snav.*" prefs to configure it.

https://mxr.mozilla.org/mozilla-central/source/toolkit/modules/SpatialNavigation.jsm#411

Comment 9

[John Hu [:johnhu][:johu][:醬糊小弟]]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #8)
> Desktop doesn't support spatial nav anymore. For Fennec, you can see from
> the code that you set the "snav.enabled" pref to enable it and various other
> "snav.*" prefs to configure it.
> 
> https://mxr.mozilla.org/mozilla-central/source/toolkit/modules/
> SpatialNavigation.jsm#411

Ok, got it. That's why I cannot have it in Desktop browser no matter how many snav.* I put.

How do we enable it in Firefox OS? And may we only enable it at a specific iframe?

Comment 10

[John Hu [:johnhu][:johu][:醬糊小弟]]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #8)
> Then improve the gecko implementation to suit your needs.

Ok! We should do that. But I would like to land this to gaia for short-term workaround and file another bug to improve the gecko implementation.

Comment 11

[GaryChen [:GaryChen][:PYChen][:gchen][:陳柏宇]]

Comment on attachment 8418507 [details] [review]
Pull Request 19012

r=me, good job!!

Comment 12

[[:fabrice] Fabrice Desré]

Please don't land a gaia workaround. We all know they tend to live for way too long.

Comment 13

[GaryChen [:GaryChen][:PYChen][:gchen][:陳柏宇]]

(In reply to Fabrice Desré [:fabrice] from comment #12)
> Please don't land a gaia workaround. We all know they tend to live for way
> too long.

Hi John and Fabrice,
   As I know this is not a workaround, this is a UX improvement for "homescreen-stingray".
   Seems 'Spatial Navigation' make people confusing, maybe we need to rename this feature.
  
Hi Marco,
   We should consider how to extend 'Spatial Navigation' in gecko to suit our requirements. (refer to comment 7 and comment 8)
   Can we put 'Spatial Navigation' improvement in S3's action item?

Comment 14

[Luke Chang [:lchang] (inactive)]

Sorry to make everybody confused. The initial idea when I filed this bug is to support a behavior that users can interact with "homescren-stingray" app by arrow keys not only for focusing some elements but also changing some states.

In fact, we don't really want to "focus" some elements in this case. For example, there are some <input> elements on the screen and we may want to change the style of one of them (e.g. add yellow borders) in order to indicate where user's "pointer" is when user press any arrow keys. However, the <input> element with yellow borders is not really focused (from gecko's perspective) at that moment so we don't want to see a text cursor on it and software keyboard shouldn't appear either until users press "enter".

Actually, what I want to do in this bug is to implement a shortcut to let user can *use* our homescreen app just like everybody can use keyboard only to interact with Zimbra system but doesn't need any gecko support.

The "spatial navigation" may be a necessary feature of stingray in many use cases (e.g. browsing a page in the browser app) if they don't provide a mouse-like device. It, indeed, should be supported by gecko but this scope is beyond this bug.

For this reason, I guess change title to "A helper to let user interact with homescreen-stingray by arrow keys" would be more precise. Any thought?

Comment 15

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to John Hu [:johnhu][:johu][:醬糊小弟] from comment #9)
> (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #8)
> > Desktop doesn't support spatial nav anymore. For Fennec, you can see from
> > the code that you set the "snav.enabled" pref to enable it and various other
> > "snav.*" prefs to configure it.
> > 
> > https://mxr.mozilla.org/mozilla-central/source/toolkit/modules/
> > SpatialNavigation.jsm#411
> 
> Ok, got it. That's why I cannot have it in Desktop browser no matter how
> many snav.* I put.
> 
> How do we enable it in Firefox OS? And may we only enable it at a specific
> iframe?

You'll need to add a line like this [1] in your chrome code, passing in the element (the iframe you mention) that you want to have spatial navigation for.

[1] http://mxr.mozilla.org/mozilla-central/source/mobile/android/chrome/content/browser.js#4607

Comment 16

[Marco Chen [:mchen]]

(In reply to GaryChen [:GaryChen][:PYChen][:陳柏宇] from comment #13)
> Hi Marco,
>    We should consider how to extend 'Spatial Navigation' in gecko to suit
> our requirements. (refer to comment 7 and comment 8)
>    Can we put 'Spatial Navigation' improvement in S3's action item?

Hi Gary,

Yes, we can evaluate it since we don't want each apps to implement their own spatial navigation.
Could you help to collect the requirements or use cases then we can decide how much Gecko should be improved for support them? 

ex: 1. add yellow borders to show focus.
    2. move focus between iframes.
    3. move focus between input elements.
    4. others.

By the way, this can be updated to new bug. Thanks.

Comment 17

[Evelyn Hung]

(In reply to Luke Chang [:lchang] from comment #14)
> For this reason, I guess change title to "A helper to let user interact with
> homescreen-stingray by arrow keys" would be more precise. Any thought?

Yes, please rename this bug as something like "a indicator of highlighted app controlled by arrow keys" to reflect it's more UI presentation stuff.

Enabling spatial navigation in Gecko makes things much easier here but there are still some UI tuning needed. So is this bug blocking our rest work? If no, I tend to leave this WIP patch here before we figure out the effort of enabling spatial navigation in Gecko.

Do we have a bug for enabling spatial navigation in Gecko? If no, please create one. Thanks!

Comment 18

[Luke Chang [:lchang] (inactive)]

This module is more likely to be a helper / algorithm just for finding the nearest element on 2D plane.

I'll file another bug for enabling spatial navigation in Gecko. However, to support the same function of this WIP module in Gecko instead of Gaia, all we need is an API like "dom.findNearestElement('right', domElementsForTraversal)". Is it right?

Comment 19

[Luke Chang [:lchang] (inactive)]

Bug 449165 - expose "nearest element" api

Comment 20

[Luke Chang [:lchang] (inactive)]

I've filed Bug 1008862 for enabling spatial navigation in B2G. Let's move relevant discussion to there.

Comment 21

[Luke Chang [:lchang] (inactive)]

Hi Marco and Evelyn,

We're now clear that all we need in "homescreen-stingray" is just an algorithm for finding the nearest element no matter it's implemented in Gaia or Gecko. It's different from supporting full spatial navigating function so I've separated the issues into two bugs, this one (bug 449165) and bug 1008862.

I'd like to know if it's possible to implement an API about exposing "the nearest elements in the specified direction" at this moment.

However, I propose landing this patch first because it blocks some of the other modules in "homescreen-stingray". It doesn't affect any other apps in Gaia and won't take much time for us to switch to Gecko version once it's ready.

Comment 22

[Marco Chen [:mchen]]

Hi Luke,

Some questions here could you help to answer? Thanks.

Q1: What kind of elements you considered to be in the scope of "finding nearest elements"?
    ex: focusable elements or others.

Q2: If you mean focusable elements only then gecko can move the focus with out gaia work since gecko can find nearest element in one direction already. 
  Q2-a Then why do you need this kind of Web API exposed to gaia?
  Q2-b Is the spatial navigation from fennec not enough for this case?

Q3: If not only focusable elements, maybe Gaia app should take care by it self. (ex: move between different embeded i-frames). Could you talk to S.C. first? 

Thanks.

Comment 23

[Luke Chang [:lchang] (inactive)]

(In reply to Marco Chen [:mchen] from comment #22)
> Q1: What kind of elements you considered to be in the scope of "finding
> nearest elements"?
>     ex: focusable elements or others.

Any kind of elements such as DIV, SPAN or even just a set of coordinates (e.g. {left: 10, top: 20, width: 100, height: 200}).
 
> Q2: If you mean focusable elements only then gecko can move the focus with
> out gaia work since gecko can find nearest element in one direction already. 
>   Q2-a Then why do you need this kind of Web API exposed to gaia?
>   Q2-b Is the spatial navigation from fennec not enough for this case?

Even though it's focusable elements only, we may still need this API because:

Sometimes we need to know which element is the next element in advance so that we're able to prevent focusing it in certain cases (without putting some "hack" in "focus"/"keydown" event).

For instance, if there is a custom model dialog overlaid other elements, we don't expect that elements which are not contained in that dialog should be traversed by arrow keys. 

> Q3: If not only focusable elements, maybe Gaia app should take care by it
> self. (ex: move between different embeded i-frames).

Yes, S.C. and I both think it's more reasonable to implement it in Gaia.

Comment 24

[Luke Chang [:lchang] (inactive)]

(In reply to Luke Chang [:lchang] from comment #23)
> Yes, S.C. and I both think it's more reasonable to implement it in Gaia.

Also, implementing it in Gaia may be easier to add some animation for focus-switching.

Comment 25

[Luke Chang [:lchang] (inactive)]

Hi Fabrice and Brad,

Thank you for raising the issue to us.

As the discussion above, we're now clear that what we need in homescreen app is just an algorithm for finding the next element to highlight. This algorithm may need to be easily customized based on the UX design. Indeed, to enable spatial navigation on Stingray is an important task on the gecko side. For example, the pages in the browser app may need to support navigating by the remote controller because of accessibility. However, it's a different issue and seems that we can't just utilize or improve it to meet our requirements in homescreen because:

- Every apps (even every UI modules in the same app) should have ability for specifying their own traversing order based on their own purpose. For instance, when we highlight an element on the right most of an UI container (e.g. AppList), the next element to be highlighted after pressing "right" key may be another UI container located below it no matter there's an element exactly on the right of it or not.

- Not every elements on the screen can be highlighted. Only elements which can interact with remote controllers at the moment need to be highlighted. It doesn't depend on accessibility, the original purpose of spatial navigation in gecko.

- We use this algorithm to "highlight" elements instead of "focusing" them. That means we don't expect to see the software keyboard or a blinking text cursor when we highlight an input element. The "highlight" is just a visual look on the front-end side.

For this reason, I propose addressing this bug on the gaia side. What do you think?

Comment 26

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Luke Chang [:lchang] from comment #25)
> Hi Fabrice and Brad,
> 
> Thank you for raising the issue to us.
> 
> As the discussion above, we're now clear that what we need in homescreen app
> is just an algorithm for finding the next element to highlight. This
> algorithm may need to be easily customized based on the UX design. Indeed,
> to enable spatial navigation on Stingray is an important task on the gecko
> side. For example, the pages in the browser app may need to support
> navigating by the remote controller because of accessibility. However, it's
> a different issue and seems that we can't just utilize or improve it to meet
> our requirements in homescreen because:
> 
> - Every apps (even every UI modules in the same app) should have ability for
> specifying their own traversing order based on their own purpose. For
> instance, when we highlight an element on the right most of an UI container
> (e.g. AppList), the next element to be highlighted after pressing "right"
> key may be another UI container located below it no matter there's an
> element exactly on the right of it or not.
this is a situation that is common to all spatial navigation
> 
> - Not every elements on the screen can be highlighted. Only elements which
> can interact with remote controllers at the moment need to be highlighted.
these are focasable elements, which is what spatial nav is designed for
> It doesn't depend on accessibility, the original purpose of spatial
> navigation in gecko.
The original purpose of spatial nav was navigating a cable box UI with a remote control, not accessibility.
> 
> - We use this algorithm to "highlight" elements instead of "focusing" them.
> That means we don't expect to see the software keyboard or a blinking text
> cursor when we highlight an input element. The "highlight" is just a visual
> look on the front-end side.
This is already supported. You need to click on an input box to bring up the keyboard.
> 
> For this reason, I propose addressing this bug on the gaia side. What do you
> think?

I think what you describe is spatial navigation and you should use and/or improve the existing mechanism in gecko.

Comment 27

[Luke Chang [:lchang] (inactive)]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #26)
> (In reply to Luke Chang [:lchang] from comment #25)
> > - Every apps (even every UI modules in the same app) should have ability for
> > specifying their own traversing order based on their own purpose.
> this is a situation that is common to all spatial navigation

Do you mean that we may create / extend the API to let users can replace the algorithm at runtime in gecko?

> > - Not every elements on the screen can be highlighted. Only elements which
> > can interact with remote controllers at the moment need to be highlighted.
> these are focasable elements, which is what spatial nav is designed for

I guess you meant the "tabindex" attribute. 

The use case is: There are many focusable elements and 2 non-focusable boxes on the screen. Some elements are in the first box visually (not structurally) and the others are in the second box. The only way to switch "highlight" state between 2 boxes is pressing "STAR" key on the remote controller (not by arrow keys). Only the elements in the highlighted box can be traversed. In other words, we need to change the "tabindex" attribute of every focusable elements each time the highlight state of boxes is changed. Also, we still need an algorithm to determine which box each element is visually located to so that we can change the "tabindex" attribute of relevant elements.

Another use case is: Some focusable elements may be just regions painted in canvas. I'm not sure how to make it focusable.

Thanks.

Comment 28

[John Hu [:johnhu][:johu][:醬糊小弟]]

Hi Brad,

I am trying to enable the Spatial Navigation in shell.html/shell.js of B2G. The spatial navigation module only lives in chrome window.

May we enable it in nested content window within another content window?

This is because the System app of gaia is in content window which is created by shell.html which is chrome window. Currently, it is only enabled at System app's window. But other apps which lives in the content windows(iframe) of System app's window.

Comment 29

[Brad Lassey [:blassey] (use needinfo?)]

I don't know how spatial navigation works under e10s

Comment 30

[Marco Chen [:mchen]]

Hi Brad,

It is great and thanks for your consultant here.

And could you help to clarify the questions from Luke on comment 27?

I think the key point is that Gecko can implement one algorithm for one kind of spatial navigation.
But the UX design didn't always follow this algorithm in Gecko. In that case the gaia can make it's own custimized algorithm. (just like second question from comment 27)

Thanks.

Comment 31

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Luke Chang [:lchang] from comment #27)
> (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #26)
> > (In reply to Luke Chang [:lchang] from comment #25)
> > > - Every apps (even every UI modules in the same app) should have ability for
> > > specifying their own traversing order based on their own purpose.
> > this is a situation that is common to all spatial navigation
> 
> Do you mean that we may create / extend the API to let users can replace the
> algorithm at runtime in gecko?
My recommendation would be to add tags to elements such as <button nextElementRight="id-of-next-element"/> and if spatial navigation is navigating from an element with such a tag, follow that hint. 
> 
> > > - Not every elements on the screen can be highlighted. Only elements which
> > > can interact with remote controllers at the moment need to be highlighted.
> > these are focasable elements, which is what spatial nav is designed for
> 
> I guess you meant the "tabindex" attribute. 
> 
> The use case is: There are many focusable elements and 2 non-focusable boxes
> on the screen. Some elements are in the first box visually (not
> structurally) and the others are in the second box. The only way to switch
> "highlight" state between 2 boxes is pressing "STAR" key on the remote
> controller (not by arrow keys). Only the elements in the highlighted box can
> be traversed. In other words, we need to change the "tabindex" attribute of
> every focusable elements each time the highlight state of boxes is changed.
> Also, we still need an algorithm to determine which box each element is
> visually located to so that we can change the "tabindex" attribute of
> relevant elements.
> 
> Another use case is: Some focusable elements may be just regions painted in
> canvas. I'm not sure how to make it focusable.
I think you'd want to do something like an image map. It looks like this has been discussed at the w3c and WHATWG. No idea what our implementation status is, but it says Mozilla backed the proposal in 2010 http://www.w3.org/html/wg/wiki/ChangeProposals/addimagemaptocanvas.

dbaron, do you know what the status of useMap in canvas is?


> 
> Thanks.

Comment 32

[Antonio Gomes (tonikitoo)]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #31)
> (In reply to Luke Chang [:lchang] from comment #27)
> > (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #26)
> > > (In reply to Luke Chang [:lchang] from comment #25)
> > > > - Every apps (even every UI modules in the same app) should have ability for
> > > > specifying their own traversing order based on their own purpose.
> > > this is a situation that is common to all spatial navigation
> > 
> > Do you mean that we may create / extend the API to let users can replace the
> > algorithm at runtime in gecko?
> My recommendation would be to add tags to elements such as <button
> nextElementRight="id-of-next-element"/> and if spatial navigation is
> navigating from an element with such a tag, follow that hint. 

You are probably talking about this http://dev.w3.org/csswg/css-ui/#nav-dir ?

Comment 33

[Luke Chang [:lchang] (inactive)]

(In reply to Antonio Gomes (tonikitoo) from comment #32)
> (In reply to Brad Lassey [:blassey] (use needinfo?) from comment #31)
> > My recommendation would be to add tags to elements such as <button
> > nextElementRight="id-of-next-element"/> and if spatial navigation is
> > navigating from an element with such a tag, follow that hint. 
> 
> You are probably talking about this http://dev.w3.org/csswg/css-ui/#nav-dir ?

Yes, we've surveyed this draft. However, each time we change the DOM structure, we still need an algorithm implemented in Gaia and based on UX design to assign this CSS properties accordingly. Doesn't it?

Comment 34

[Luke Chang [:lchang] (inactive)]

Let me summarize a little bit:

All we need in this bug is a simple algorithm to find the next element for UI behavior. If we prefer to implement it in gecko, we need:

1. enable spatial navigation in B2G
2. enable spatial navigation in Firefox Nightly
3. implement useMap in canvas (http://www.w3.org/html/wg/wiki/ChangeProposals/addimagemaptocanvas)
4. implement "Keyboard control" CSS properties (http://dev.w3.org/csswg/css-ui/#keyboard)
5. a customizable algorithm to let Gaia set "useMap" and "nav-dir" accordingly (bug 449165)

Hi Marco, is it possible to put these in S3's backlog? Thanks.

Comment 35

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #31)
> (In reply to Luke Chang [:lchang] from comment #27)
> > Another use case is: Some focusable elements may be just regions painted in
> > canvas. I'm not sure how to make it focusable.
> I think you'd want to do something like an image map. It looks like this has
> been discussed at the w3c and WHATWG. No idea what our implementation status
> is, but it says Mozilla backed the proposal in 2010
> http://www.w3.org/html/wg/wiki/ChangeProposals/addimagemaptocanvas.
> 
> dbaron, do you know what the status of useMap in canvas is?

I think the most recent canvas accessibility work was done in bug 966591.  I think the people in that bug probably know the status better if you need more information.

Comment 36

[Luke Chang [:lchang] (inactive)]

Clear NI since this bug is unnecessary for now.

Comment 37

[Luke Chang [:lchang] (inactive)]

Close this bug since we don't need it anymore.