Simulate a scrolling gesture if the user hit the utility tray

NEW
Assigned to

Status

Firefox OS
Gaia::System
5 years ago
a year ago

People

(Reporter: vingtetun, Assigned: vingtetun)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Created attachment 752724 [details] [diff] [review]
Patch

This is a small simple fix that forward a panning gesture to apps when the user click on the utility tray. This works for all embedded apps I tried and as well as the browser apps and some apps from the marketplace.

One can object that it is not perfect - which is true. But this is a simple and cheap solution that allow to scroll app content without any knowledge of the content itself.

Another possible solution would have been to send a mousemove event at position 0,0 but it would have force the app to implement a custom code path for supporting this and it looks like a hack to me. The proposed solution is more like a shortcut.

Let's ping UX to see what they think about that too. Josh I'm asking about the feature not the code for you :)
Attachment #752724 - Flags: review?(timdream)
Attachment #752724 - Flags: feedback?(jcarpenter)
Comment on attachment 752724 [details] [diff] [review]
Patch

Review of attachment 752724 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think this will work for all apps.

Essentially, you are sending a "swipe" to the center of the screen, but the active scrollable area might not even at the center.

iOS implement the API in a way that it would allow the developer to specify which scrollable area to receive the |scrollToTop| event. 
http://stackoverflow.com/questions/15589292/scroll-to-top-when-tapping-the-status-bar

I think we should follow the same, at least kick off the discussion on dev-webapi, before landing this hacky solution. (even if this eventually leads us to |-moz-overflow-to-top:|)
Attachment #752724 - Flags: review?(timdream) → review-
(In reply to Tim Guan-tin Chien [:timdream] (MoCo-TPE) from comment #1)
> Comment on attachment 752724 [details] [diff] [review]
> Patch
> 
> Review of attachment 752724 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I don't think this will work for all apps.
> 
> Essentially, you are sending a "swipe" to the center of the screen, but the
> active scrollable area might not even at the center.

The code send a sequence of events that are interpreted as a swipe by BrowserElementPanning.js. This is fine if the active scrollable area is not at the center. In practice it seems like most of mobile apps where you want such behavior have this kind of layout. 

At least it works for most (if not all) of the apps I tried where I want such behavior 

So as I said above this solution is a kind of shortcut that seems to work on 99% percent of the case (numbers out of the hat!).

> 
> iOS implement the API in a way that it would allow the developer to specify
> which scrollable area to receive the |scrollToTop| event. 
> http://stackoverflow.com/questions/15589292/scroll-to-top-when-tapping-the-
> status-bar
>
> I think we should follow the same, at least kick off the discussion on
> dev-webapi, before landing this hacky solution. (even if this eventually
> leads us to |-moz-overflow-to-top:|)


My intention is far from creating a new standard element in this case since I feel like this depends mostly on the UI/layout of the apps. Currently apps are organized in such a way that this gesture is useful.

Also the iOS solution (which is really different in terms of impl as you said) force a scrolling and the developer have to take care of the visibility of their lists. This is for free here (again for most cases).

The sequence of events could also be handle differently by an application if they want to (translate as something else).

I'm usually the one fighting for landing stuff in the platform but I don't think it fits in this case but using the status bar as a button that offers a shortcut (which can be different based on embedder) sounds useful to me.


But I can be completely wrong here! (that won't be the first time :))
So let's needinfo'ed some of the guys that may have an opinion here. Jonas and Mounir for the API question and Justin for the mozbrowser relation.
Flags: needinfo?(mounir)
Flags: needinfo?(justin.lebar+bug)
Flags: needinfo?(jonas)
I don't understand what the purpose of this bug is.

When the user clicks on the status bar (the thing at the top of the screen with the clock, signal strength, etc, right?), we'll send a series of mouse events to the main app frame which correspond to an upward scrolling gesture?

Why do we want to do that?
Flags: needinfo?(justin.lebar+bug)
(In reply to Justin Lebar [:jlebar] from comment #3)
> I don't understand what the purpose of this bug is.
> 
> When the user clicks on the status bar (the thing at the top of the screen
> with the clock, signal strength, etc, right?), we'll send a series of mouse
> events to the main app frame which correspond to an upward scrolling gesture?
> 
> Why do we want to do that?

It is useful for some apps that put 'content' first. The bigger consumer being the browser app.
Why is tapping the status bar preferable to actually scrolling up?

I guess this would make more sense to me if tapping the status bar scrolled all the way to the top, or something.  As is, this seems a bit strange...
On long pages it's definitely nice to have the ability to scroll to the top quickly, without having to repeatedly swipe upwards.

However simply sending a bunch of mouse move events and hope that that hits a scrollable area doesn't sound like a safe solution to me. We should look inside of the app and try to find scrollable region and then scroll that.

Olli: What would be the appropriate set of events to fire here? Does Pointer Events prescribe something here at all?

Another tricky issue here is that it'd be very nice to not have to hit the main event loop of the app before starting to scroll. Pointer Events are generally designed to avoid that I think, so maybe they provide a solution here?

What does Safari do on iOS?
Flags: needinfo?(jonas) → needinfo?(bugs)
If we just want to scroll to the very top, certainly we can do that in the platform  (scrollTo the main window to y=0).  We could even add a browser-api method for that.

This code was trying to scroll up "some", which I still don't quite get.
(In reply to Justin Lebar [:jlebar] from comment #7)
> If we just want to scroll to the very top, certainly we can do that in the
> platform  (scrollTo the main window to y=0).  We could even add a
> browser-api method for that.
> 
> This code was trying to scroll up "some", which I still don't quite get.

I agree with Justin. It seems that adding a method to the browser API that does scrolling would be better.
Flags: needinfo?(mounir)
(In reply to Justin Lebar [:jlebar] from comment #5)
> Why is tapping the status bar preferable to actually scrolling up?
> 
> I guess this would make more sense to me if tapping the status bar scrolled
> all the way to the top, or something.  As is, this seems a bit strange...

The generated scrolling gesture is big enough to scroll to top most of the encountered content (so it does what you say in practice - it scrolled y*y where y = height / 2). 


(In reply to Justin Lebar [:jlebar] from comment #7)
> If we just want to scroll to the very top, certainly we can do that in the
> platform  (scrollTo the main window to y=0).  We could even add a
> browser-api method for that.
>

heh! I would have implement it this way if it was that simple :) (or I could have missed something obvious!)
 
> This code was trying to scroll up "some", which I still don't quite get.

In many cases what should be scrolled up is not the main application window but some content inside it. 
For the case of the contacts app for example you want to scroll the inner scrollable area that contains the list of contacts. For the case of the browser app you want to scroll the remote view first and then scroll the parent.

So scrolling the main window won't fit here.

The reasons why I was in favor of this cheap solution versus adding a new API are:
 - the use case seems valid only for mobile content where scrolling to the top is painful and you don't have many shortcuts to do it. (like the VK_HOME key on desktop, or a usable scrollbar).
 - the use case seems not really useful for embedders (like other browsers apps) except the system app.
 - I have no idea if there any patents on this but providing a shortcut that does something the same way it is done by user is probably not patented.

On a personal note I learned with Fennec that trying to do something simple theoretically with random web content (like a form assistant) is a good way to find a good number of use cases where the solution does not do what you expect. When the shortcut solution do a scroll to top gesture on what the user see. Nothing more. 



(In reply to Jonas Sicking (:sicking) from comment #6)
> However simply sending a bunch of mouse move events and hope that that hits
> a scrollable area doesn't sound like a safe solution to me. We should look
> inside of the app and try to find scrollable region and then scroll that.

I generally agree. But looking at mobile app it seems to me that it is a bit like the way we save the entered name/password in Firefox. It is weak but it works for most of the case because everybody layout those the same way.

This is just my feeling. I won't push for a weak solution if I didn't felt it is probably simpler and better than other stuffs.

> What does Safari do on iOS?

I don't know about the internal but the link given in #c1 explained that they scrolled the first scrollable area they found below the statusbar. For most case it is for free. But in some cases the developer have to explicitly set an attribute on the scrollable area if there are multiple of them and some are hidden/visible.


Again I'm not pretending that the shortcut solution is perfect (as I said on #c0) and if the current use case makes it a good reason to add a new API which will solve the same things without requiring developer to change anything in their app that's fine by me. 
It could also be a simple solution that I have missed. Otherwise I wonder how expensive it will be to impl such a feature where you have to walk the dom to find the first scrollable item (which can be not what the user expect to scroll as well).


I hate to be in the position of the guy that spent time defending the 'do not fix the platform' solution or to waste your time if I'm blind here :(
> In many cases what should be scrolled up is not the main application window but some 
> content inside it. 

So do you just want to scroll the main frame and all nested frames to y=0?  I think we can do that...
(In reply to Jonas Sicking (:sicking) from comment #6)
 
> Olli: What would be the appropriate set of events to fire here? Does Pointer
> Events prescribe something here at all?
No sure I understand the question. Which events? Just scroll to top as jlebar says and perhaps fire scroll
event after that.
Flags: needinfo?(bugs)
> So do you just want to scroll the main frame and all nested frames to y=0?
> I think we can do that...

Yes, the behavior should be to scroll to top of content, and not just a fixed distance. The later need is already well covered by swiping up / down.

That said I am throwing this out to the broader UX team to discuss before giving thumbs up on this feature. It's very handy for long web pages, lists, etc. But I have a few concerns that the team needs to discuss first. We'll follow up. Adding myself for needsinfo as a reminder. 

Vivien, one question: with this patch enabled, do you see any impact on the speed and reliability of opening the top drawer? I'm concerned that this may slow the time to recognize the user's down-ward swipe. We always want that top drawer to be as responsive as possible, as a rule.
Flags: needinfo?(jcarpenter)
> > Olli: What would be the appropriate set of events to fire here? Does Pointer
> > Events prescribe something here at all?
> No sure I understand the question. Which events? Just scroll to top as
> jlebar says and perhaps fire scroll event after that.

Why "perhaps"?

Anyhow, the bigger question is, how do we allow the page to cancel this action? I.e. pages should be able to prevent the scrolling in some way, no?
Flags: needinfo?(bugs)
Do we need to allow page to cancel the action? Well, I guess it is possible even when
one scrolls using scrollbar (desktop builds), if the mouse events are prevented.
Though, for example on osx we support gesture events which aren't exposed to web, and it
is possible to scroll pages using those.

Anyhow, should we simulate wheel events here? Web page can prevent those ones, and if so, the
page is not scrolled.
Flags: needinfo?(bugs)
I like this idea, spent time on it during Madrid work week as well. The approach that I took there was to emit an event to the active application and then that was responsible for handling that event itself. Two downsides: a) to get an event to active app from system I needed a patch in Gecko (but I still think there should be a way around that, but soit). b) the browser API doesn't have a `scrollTo` function, so had to add that as well. Therefore I abandoned it.

This however sounds like a pretty nice workaround.
Yes, I think applications need to be in control over this so that they can implement their own scrolling.

However simply firing an event before we scroll wouldn't let us do off-main-thread scrolling, right? Since we'd have to do a roundtrip to the main thread before we could start scrolling.

This is why I was curious how Pointer Events handle this since I believe they support off-main-thread scrolling without having to roundtrip to the main thread first.
Flags: needinfo?(bugs)
pointer events help here, since there is the CSS property which can be effectively transferred to
off-main-thread.
Flags: needinfo?(bugs)
Olli, you are providing very incomplete answers. If you'd rather not be involved here and would like us to simply make something up, please feel free to say so. I just thought it would be great to get your help with ensuring that what we come up with here fits the designs of existing and upcoming UI events.

The question is this:

How do we implement a feature which lets the user press the top of the screen which lets the user "scroll visible content up". I.e. as long as the user presses some part of the screen that is outside of the actual content window, we want to scroll the content up. Once the user stops pressing the scrolling should stop.

We have the following requirements:
* Content should be able to prevent the scrolling. I.e. just like content today is
  able to prevent scrolling when the user touches the screen and drags the finger,
  or when the user uses a scroll wheel on a mouse, the page can override the
  request to scroll. We want that to be possible here too.
  How does pointer events handle this scenario?
* Content should be able to designate an area that is to be scrolled when the user
  performs this scrolling gesture. Most likely we will have heuristics which finds
  the most appropriate scrollable region and scrolls that. But it would be good if
  content can explicitly control which region will be scrolled when this UI gesture
  is used.
  I expect this to be an API that is specific for this type of gesture, and thus
  for now FirefoxOS specific. But it's good if it is more generic of course. And
  we should definitely reach out to other mobile web browser vendors to see if
  they are interested in implementing the same feature and thus the same API.
* We have to be able to start scrolling without hitting the main thread of the
  content. I.e. we don't want to fire an event and check if it got cancelled or
  otherwise modified.


It would be great if you could help design this feature. What I'm looking for here is a description of what events to fire when, and what JS-APIs and/or CSS properties to add.

Again, if you'd rather not be involved, we can definitely come up with something. But any help with pointers to existing or upcoming features to align with would be most appreciated.
Flags: needinfo?(bugs)
(In reply to Jonas Sicking (:sicking) from comment #18)
> Olli, you are providing very incomplete answers. If you'd rather not be
> involved here and would like us to simply make something up, please feel
> free to say so. I just thought it would be great to get your help with
> ensuring that what we come up with here fits the designs of existing and
> upcoming UI events.
> 
> The question is this:
> 
> How do we implement a feature which lets the user press the top of the
> screen which lets the user "scroll visible content up". I.e. as long as the
> user presses some part of the screen that is outside of the actual content
> window, we want to scroll the content up. Once the user stops pressing the
> scrolling should stop.

Better then actually ask the question rather than randomly set needinfo flag ;)

You question was
"However simply firing an event before we scroll wouldn't let us do off-main-thread scrolling, right? "
and my answer covered that.


I'll come back to this bug hopefully within next few days when I'm reviewing the patches which implement
pointer events.
My initial question in comment 6, as well as followup questions in later comments, asked much more than that. Anyway, looking forward to your answers when you are done with pointer events.
(In reply to Jonas Sicking (:sicking) from comment #18)

> We have the following requirements:
> * Content should be able to prevent the scrolling. I.e. just like content
> today is
>   able to prevent scrolling when the user touches the screen and drags the
> finger,
>   or when the user uses a scroll wheel on a mouse, the page can override the
>   request to scroll. We want that to be possible here too.
>   How does pointer events handle this scenario?
CSS property touch-action
https://dvcs.w3.org/hg/pointerevents/raw-file/tip/pointerEvents.html#the-touch-action-css-property
touch-action: none;



> * Content should be able to designate an area that is to be scrolled when
> the user
>   performs this scrolling gesture. Most likely we will have heuristics which
> finds
>   the most appropriate scrollable region and scrolls that. But it would be
> good if
>   content can explicitly control which region will be scrolled when this UI
> gesture
>   is used.
>   I expect this to be an API that is specific for this type of gesture, and
> thus
>   for now FirefoxOS specific. But it's good if it is more generic of course.
> And
>   we should definitely reach out to other mobile web browser vendors to see
> if
>   they are interested in implementing the same feature and thus the same API.
touch-action: auto; or touch-action: pan-x/y



> * We have to be able to start scrolling without hitting the main thread of
> the
>   content. I.e. we don't want to fire an event and check if it got cancelled
> or
>   otherwise modified.
The state of the CSS property can be pushed to off-main-thread and hit-testing done there.
Based on roc we do already some hit-testing off-main-thread.
Flags: needinfo?(bugs)
> > * Content should be able to designate an area that is to be scrolled when
> > the user
> >   performs this scrolling gesture. Most likely we will have heuristics which
> > finds
> >   the most appropriate scrollable region and scrolls that. But it would be
> > good if
> >   content can explicitly control which region will be scrolled when this UI
> > gesture
> >   is used.
> >   I expect this to be an API that is specific for this type of gesture, and
> > thus
> >   for now FirefoxOS specific. But it's good if it is more generic of course.
> > And
> >   we should definitely reach out to other mobile web browser vendors to see
> > if
> >   they are interested in implementing the same feature and thus the same API.
> touch-action: auto; or touch-action: pan-x/y

Olli: I don't understand the answer here. Can you give an example on how a developer would designate the area to be scrolled when the user touches the top of the screen?
Flags: needinfo?(bugs)
(In reply to Olli Pettay [:smaug] from comment #21)
> (In reply to Jonas Sicking (:sicking) from comment #18)
> 
> > We have the following requirements:
> > * Content should be able to prevent the scrolling. I.e. just like content
> > today is
> >   able to prevent scrolling when the user touches the screen and drags the
> > finger,
> >   or when the user uses a scroll wheel on a mouse, the page can override the
> >   request to scroll. We want that to be possible here too.
> >   How does pointer events handle this scenario?
> CSS property touch-action
> https://dvcs.w3.org/hg/pointerevents/raw-file/tip/pointerEvents.html#the-
> touch-action-css-property
> touch-action: none;

Actually, I don't understand the answer here either. Which element would the author set "touch-action: none" on in order to prevent any and all scrolling when the user presses the top of the screen?
Perhaps I don't understand your question.
I thought you mean with "designate an area" something like <div style="overflow: scroll">, which
browser should of course handle automatically.
By using touch-action, the page gets even more control.
Flags: needinfo?(bugs)
(In reply to Jonas Sicking (:sicking) from comment #23)
> Actually, I don't understand the answer here either. Which element would the
> author set "touch-action: none" on in order to prevent any and all scrolling
> when the user presses the top of the screen?
I don't know where "presses the top of the screen" came to the question, but 
the question was about "Content should be able to prevent the scrolling".
And that can be done by setting touch-action: none to the areas which shouldn't be scrolled.

I think I'm missing what information your looking after.
Olli: The feature we are trying to build works something like this:

1. The user navigates to a website, for example an webmail website.
2. This website contains a header with some buttons and other UI, as well as a scrollable area which displays email contents.
3. The user scrolls down to read further down in the email.
4. Once the user is done reading, the user wants to scroll back up to the top of the email to see the email details again (i.e. who the email was sent to and from).
5. This "scroll back up" action is a very common action on small screen devices such as mobile phones. Hence we want to create a feature in FirefoxOS which makes it easier for the user to "scroll back up". This bug is about implementing a "scroll back up" feature. I.e. it is about implementing the *whole* such feature.

There are several possible UIs we can imagine building to enable such a "scroll back up" feature. I.e. we could have a button in the browser app which the user can press. We could say that a quick press at the "status bar" (i.e. that contains notification icons and the clock) should trigger the feature. We could in the future add a voice command to trigger the feature.

What exact UI to use is one question here.

The other question, to you, is what web platform features do we need to expose to webpages as part of building this feature. I.e. I'm not looking for an answer of the form "maybe we can use CSS feature X", but rather "here is a comprehensive solution".

Comment 18 contains some of the requirements that we have on the feature. I.e. when the user triggers the "scroll back up" action (whatever it is), we want to by default do something sensible. But we also want to enable the web page to override that sensible solution as to better control what is happening.

For example in the web based email reader described above, it would be sad if nothing scrolled because the outermost <div> isn't scrollable. And it would be sad if the UI piece scrolled up 2 pixels because the developer accidentally made the button 2 pixels higher than the UI container. I.e. I would like the developer to be able to say that the email area is the one that should be scrolled up.

Your answers in comment 21 seems like it could help towards building this feature. But is is not enough to build the feature we are attempting to build here.
Flags: needinfo?(bugs)
Ok, I start to understand. Nothing to do with touch events or pointer events or touch-action.
More about somehow checking whether the page wants to default-handle certain user/UA action, or should the 
UA do its default, and if so, using which target element.

thinking...
The simplest approach, which wouldn't require any changes to web apis would just send [home] key events
to the web page. Key events go to the focused element.
Sending the event could be done async. If preventDefault() isn't called, the child process can
tell which scrollable area to scroll and inform parent process/layer system about that, asynchronously.

The normal "must be done off-main-thread" doesn't quite apply here, since there isn't hit-testing
involved, nor requirement to get the results from the content process for some decision making in
parent process.
Flags: needinfo?(bugs)
The "must be done off-main-thread" requirement has nothing to do with hit testing. The reason for "must be done off-main-thread" is that we want to be able to start updating the UI even if the page that the user is looking at is perfoming an expensive operation in JS.

So "must be done off-main-thread" definitely applies here, which is why I listed it as the 3rd requirement in comment 18.

Additionally, simply sending a "home" button event wouldn't fulfill the "page can designate area to scroll" requirement. I.e. the second requirement in comment 18.

That requirement still needs to be fulfilled even if the page contains focusable areas outside of the scrollable region.

Also, simply sending a home button wouldn't work if the user currently has an editable area focused since that would just move the cursor rather scroll to top.


Finally, a requirement that I forgot to mention in comment 18 is that I think it'd be good to support only scrolling while the user is activating the "scroll to top" button. I.e. it would be nice to be able to build a UX where as long as the user presses the top of the screen we rapidly scroll upwards. But if the user sees something that he is interested in, he should be able to lift his finger and the scrolling should stop where we are.

Updated

4 years ago
Duplicate of this bug: 922891
Flags: needinfo?(jcarpenter)
Attachment #752724 - Flags: feedback?(jcarpenter)
You need to log in before you can comment on or make changes to this bug.