Make <video> and <audio> controls keyboard navigable

NEW
Unassigned

Status

()

Toolkit
Video/Audio Controls
8 years ago
8 months ago

People

(Reporter: davidb, Unassigned)

Tracking

(Blocks: 1 bug, {access, ux-control, ux-discovery})

unspecified
access, ux-control, ux-discovery
Points:
---
Dependency tree / graph
Bug Flags:
wanted1.9.2 +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [parity Chrome, IE, Opera])

Attachments

(1 attachment, 1 obsolete attachment)

21.00 KB, patch
MarcoZ
: review+
Dolske
: review-
Details | Diff | Splinter Review
(Reporter)

Description

8 years ago
Follow up for bug 486899.

That bug adds support for tabbing to media elements and supporting direct keyboard control of playback.

This bug is about making the media controls (which appear on mouse over) keyboard accessible. This will require defining a keystroke to invoke the controls HUD, and making sure that the control elements have tabindex=0 and are keyboard operable; and that they expose accessibility information.
(Reporter)

Comment 1

8 years ago
Support for media keys (found on some keyboards), should be added on this bug, or filed separately. (See bug 486899 comment #59)
(Reporter)

Updated

8 years ago
Blocks: 492516
(Reporter)

Comment 2

8 years ago
Things to test before we consider this fixed/landable:

1. Mouse controls still work.
2. Keyboard control works (without screen reader).
3. There is visual feedback during keyboard only use
4. Keyboard control works with screen reader (sr).
5. Accessible role, state, and action exposure works, virtual and focus-interaction sr modes change as expected.
6. Accessible name changes work for sr.
7. In general the sr feedback for using the controls is decent.
8. Mochitests for name changes, focus, and action names, disabled on bug 486899 now pass.
9. Assuming we want shortcuts *and* keyboard-able controls, we need to make sure we don't break shortcuts. This includes trying the shortcut keys while controls are focussed, to check for potential conflicts/double-ups.

The a11y team should definitely be tapped for at least 4-8, and probably 1-9.

Comment 3

8 years ago
Created attachment 379346 [details] [diff] [review]
patch

1) Make controls focusable
2) Draw dotted border around focused controls
3) Draw dotted border around focused video 
4) Show controls always if focus is in or inside of video (even if mouse pointer is outside of video)
5) Hide controls if focus goes away
6) Show volume slider if focus goes on mute button
7) Hide volume slider if focus goes outside of mute button or volume slider
Assignee: nobody → surkov.alexander
Status: NEW → ASSIGNED
Attachment #379346 - Flags: review?(mconnor)
Attachment #379346 - Flags: review?(marco.zehe)
Attachment #379346 - Flags: review?(david.bolter)

Comment 4

8 years ago
8) Make video's keyboard shortcuts work if video is focused only.
(Reporter)

Comment 5

8 years ago
Comment on attachment 379346 [details] [diff] [review]
patch

>                 keyHandler : function(event) {
>-                    // Ignore keys when content might be providing its own.
>-                    if (!this.video.hasAttribute("controls"))
>+                    if (event.originalTarget != this.video)
>                         return;

I assume the original intention of checking !this.video.hasAttribute("controls") was to see if the content developer had specified their own controls?  Is that not working?

I understand the your check for event.originalTarget, but we might want to || with the controls check?
I think we need to take a step back and figure out how things *should* work before rushing into implementations...

Comment 7

8 years ago
In principle, this works quite well already. One thing I noticed wat that the left and right arrow keys don't do anything when focused on the scrubber. This is regardless of whether the video is playing or paused: The left and right arrows simply do not change the current position. When focused on the video element grouping itself, left and right do change the position as expected.

Other than this, focusing the individual controls works very well. NVDA even copes reasonably well with the showing and hiding of the volume slider.

Comment 8

8 years ago
(In reply to comment #5)

> I assume the original intention of checking
> !this.video.hasAttribute("controls") was to see if the content developer had
> specified their own controls?  Is that not working?
> 
> I understand the your check for event.originalTarget, but we might want to ||
> with the controls check?

Controls attribute means nothing because if I get right then it specifies if controls are visible initially. So moveout from video and controls will be hidden. Bt what was your point to keep check for controls attribute?

Comment 9

8 years ago
(In reply to comment #6)
> I think we need to take a step back and figure out how things *should* work
> before rushing into implementations...

We discussed a lot in 486899 and we got an agreement controls should be focusable. So I decided to play with video starting from this point and tired to get result which seems logically complete. So I created prototype what you can try it and say what is good and what is not. Theoretical discussions are good but sometimes you need to touch it. So at least I needed to see how it is going to work before I can say how it should be.

Comment 10

8 years ago
(In reply to comment #7)
> In principle, this works quite well already. One thing I noticed wat that the
> left and right arrow keys don't do anything when focused on the scrubber. This
> is regardless of whether the video is playing or paused: The left and right
> arrows simply do not change the current position. 

They works and change it :) by miliseconds I think, so you should press 1000 times to move to one second. I'll try to change this.

> Other than this, focusing the individual controls works very well. NVDA even
> copes reasonably well with the showing and hiding of the volume slider.

That's really great! Because that is the thing I wasn't completely ensured.

Comment 11

8 years ago
Created attachment 379385 [details] [diff] [review]
patch2

progress slider moves by seconds
Attachment #379346 - Attachment is obsolete: true
Attachment #379385 - Flags: review?(mconnor)
Attachment #379385 - Flags: review?(marco.zehe)
Attachment #379385 - Flags: review?(david.bolter)
Attachment #379346 - Flags: review?(mconnor)
Attachment #379346 - Flags: review?(marco.zehe)
Attachment #379346 - Flags: review?(david.bolter)

Comment 12

8 years ago
So if the video has no controls attribute then the video will no longer be focusable and no keyboard shortcuts will work, right? I assume that if there is no controls attribute then the page author has decided to take responsibility for this.

Comment 13

8 years ago
(In reply to comment #12)
> So if the video has no controls attribute then the video will no longer be
> focusable and no keyboard shortcuts will work, right? I assume that if there is
> no controls attribute then the page author has decided to take responsibility
> for this.

video is focusable always, it doesn't depend on controls attribute. I didn't change this. If focus on video then its keyboard shortcuts works, if focus is somewhere on controls then they don't.
(Reporter)

Comment 14

8 years ago
(In reply to comment #8)
> (In reply to comment #5)
> 
> > I assume the original intention of checking
> > !this.video.hasAttribute("controls") was to see if the content developer had
> > specified their own controls?  Is that not working?
> > 
> > I understand the your check for event.originalTarget, but we might want to ||
> > with the controls check?
> 
> Controls attribute means nothing because if I get right then it specifies if
> controls are visible initially. So moveout from video and controls will be
> hidden. Bt what was your point to keep check for controls attribute?

The controls attribute:

"If this attribute is present, Firefox will offer controls to allow the user to control video playback, including volume, seeking, and pause/resume playback."
from: https://developer.mozilla.org/en/html/element/video)

"The controls attribute is a boolean attribute. If present, it indicates that the author has not provided a scripted controller and would like the user agent to provide its own set of controls."
from: http://www.whatwg.org/specs/web-apps/current-work/multipage/video.html#attr-media-controls
Comment on attachment 379385 [details] [diff] [review]
patch2

r=me for the pure functional part. I think having the arrows skip by second rather than millisecond on the scrubber itself makes much more sense. We're not in a video editing environment here, and even there, millisecond browsing should be an option rather than the default I think. :)
Attachment #379385 - Flags: review?(marco.zehe) → review+
Comment on attachment 379385 [details] [diff] [review]
patch2

(In reply to comment #9)
> > I think we need to take a step back and figure out how things *should* work
> > before rushing into implementations...
> 
> We discussed a lot in 486899 and we got an agreement controls should be
> focusable.

Discussed, yes, but I don't see any agreement other than what needed to be done to limit scope for 1.9.1 RC1.

In particular, as I said on IRC, I think this is the wrong approach because it makes keyboard commands act bizarrely. EG, depending on where you click (or tab to) sometimes space will toggle pause, sometimes it will toggle mute, sometimes it does nothing.

A proper design also needs to sort out the correct behavior of tabindex when the controls attribute isn't present (or is added/removed), and in general how A11Y should deal with media elements with page-provided controls. I'll note that such custom controls can (and will!) make all the mistakes our XUL controls have made, so I wonder if we're even solving these problems at the right level.

I have some more thoughts to write up, but wanted to make sure we're not charging down the wrong path only to change direction again.

(In reply to comment #15)
> I think having the arrows skip by second
> rather than millisecond on the scrubber itself makes much more sense. We're not
> in a video editing environment here, and even there, millisecond browsing
> should be an option rather than the default I think. :)

This breaks using the scrubber with a mouse. See bug 473103. That bug needs to be fixed before we can set the scale's increment to a sensible value. Millisecond movements are not at all desired, but is a limitation due to the other bug.
Attachment #379385 - Flags: review-

Comment 17

8 years ago
(In reply to comment #16)
> (From update of attachment 379385 [details] [diff] [review])

> > We discussed a lot in 486899 and we got an agreement controls should be
> > focusable.
> 
> Discussed, yes, but I don't see any agreement other than what needed to be done
> to limit scope for 1.9.1 RC1.

Ok, probably an agreement between a11y team members. Let's get it between us.

> In particular, as I said on IRC, I think this is the wrong approach because it
> makes keyboard commands act bizarrely. EG, depending on where you click (or tab
> to) sometimes space will toggle pause, sometimes it will toggle mute, sometimes
> it does nothing.

Sorry but really don't understand your point. Do you care about that space key clicks a button depending on which button is focused?

> A proper design also needs to sort out the correct behavior of tabindex when
> the controls attribute isn't present (or is added/removed),

I'm not sure we should define tablindex behaviour. I think it should correspond to order of controls.

> and in general how
> A11Y should deal with media elements with page-provided controls. I'll note
> that such custom controls can (and will!) make all the mistakes our XUL
> controls have made, so I wonder if we're even solving these problems at the
> right level.

Please clarify what right level means. I have no idea how we can affect on accessibility of custom controls.

> I have some more thoughts to write up, but wanted to make sure we're not
> charging down the wrong path only to change direction again.

Nice. That is a reason to r-?

> (In reply to comment #15)
> > I think having the arrows skip by second
> > rather than millisecond on the scrubber itself makes much more sense. We're not
> > in a video editing environment here, and even there, millisecond browsing
> > should be an option rather than the default I think. :)
> 
> This breaks using the scrubber with a mouse. See bug 473103. That bug needs to
> be fixed before we can set the scale's increment to a sensible value.
> Millisecond movements are not at all desired, but is a limitation due to the
> other bug.

I would say it's not a bug it's a feature. I don' believe someone can operate with slider by milliseconds with a mouse.
(Reporter)

Comment 18

8 years ago
(In reply to comment #16)
> I have some more thoughts to write up, but wanted to make sure we're not
> charging down the wrong path only to change direction again.

Looking forward to these (though I know this is busy-busy-time).
As seen in bug 495162 we should also take care of the following issue:

Further why do we hide the controls when having set the focus on the video tag
but moving the mouse out of the control? It's one level higher and similar to
this bug report.
(Reporter)

Comment 20

8 years ago
Yeah if <video> or FF supplied video controls have focus, I think the mouse out shouldn't hide the controls. Dolske?
(Reporter)

Comment 21

8 years ago
The corollary to this is, since mouse only users would want the controls to hide on mouse out, we would need to make sure mouse usage of the video controls does not give them focus. (Hmm)
(Reporter)

Comment 22

8 years ago
Alex, looks like we need more testing, specifically:
Update test_elm_media.html which is currently broken and disabled.
Add <video> tests to test_elm_media.html, currently tests only <audio>.
Work with Justin to make sure we have tests for keyboard + mouse interactions.
Generally expand automated tests to make landing pleasant.
So, first off, I think at some point (now? later?) there needs to be some kind of UX/A11Y meetup so we can get a better understanding of actual A11Y requirements and use cases. Talking with other UX folks, it seems clear we really want to get a better handle on this stuff (beyond <video>) because we're running into it more often, and it really needs to be considered holistically with the rest of the product UX. I'm not sure what the right forum is for this -- a conference call feels like too little, but given our geographic diversity a face-to-face could be difficult to arrange. Thoughts?


That said, here's an outline of the direction I think we should be taking. (Now would be a good time to go grab a Snickers Bar):

1. Fix <video> tabindex.

Currently the video controls enable tabindex on the media element, and it's always set even if the page provides its own controls (and we might be racy if the page sets a different tab order). Instead, media elements should enable a tabindex=0 on themselves when the "controls" attribute is set, and clear it when the attribute is not set. Of course, if the page explicitly sets a tabindex that should always be preferred and not cleared.


2. The individual elements of the browser-provided video controls should not be focusable.

For the reasons described in comment 16. If they're focusable, keyboard UI becomes bizarre depending on where you happened to click (or tab to). It's also really sucky to use, see bug 486899 comment 30. [Side issue: this avoids the focus rings appearing on the controls when you click on them, bug 492415, though there are other workarounds. The focus ring makes the controls look really bad. They makes sense when tabbing, but until bug 418521 is fixed it's all-or-nothing.]

The HUD concept (focus video, press something, get access to individual control elements) doesn't seem like an effective solution to me. Discoverability is still an issue (the "press something"), and it feels like having to support 2 different methods of interaction with the same controls.

A point I raised in the email thread a couple of weeks ago ("Video/Audio a11y and l10n", which kicked off the realization there were A11Y issues with video), was that it might make sense for A11Y stuff to be aware of and interact more directly with the media element, and less with the XUL controls (which, in turn, are just interacting with the media element). In other words, treat <video> as a unique type of UI widget, like as a <button>, <select>, or <tab>, but with more complex interactions. This is seeming more and more like a better approach, and I think it merits further investigation (but I don't understand the A11Y side of things all that well).


3. Invoking a keyboard command should probably result in some visual indication.

Broader issue from bug 495162. We should ideally show something when keyboard commands perform an action. Say, an icon overlay along the lines of what OS X does when adjusting the system volume with the keyboard. I'm not completely sold on this -- the biggest problem is that this creates a visually noisy/busy UI if you're trying to quickly control the media with keyboard shortcuts. But it does seem like we should do *something* for volume, since it has the least obvious feedback that the keypress actually did something. This issue needs the most new-UI thinking.

What we should *not* do, though, is activate the existing controls upon keyboard presses. Too noisy and distracting. One of the design constraints with in-content controls is that the controls must overlay the video, and as a result we have to be careful about them becoming annoying when they're in the way (because they're obscuring the thing you're trying to watch). The current mouseover/mouseout fading minimizes this pain; having the full controls jump into view when using the keyboard to adjust volume or skip around is not a good experience. There's also the question of when to hide the controls after a keyboard interaction -- 1 second? 5 seconds? Never? What happens when you interact with both the keyboard and mouse? Best to be able to avoid this in the first place.


4. Add support for media-enabled keyboards, and controlling a page's media element(s)

EG, after load page that has a media element, pressing the keyboard's play/mute
should do something without having to focus/click it. (And if there are multiple media elements... hmmm.) "Mute this tab" would also be supremely awesome.
(Reporter)

Comment 24

8 years ago
100% agreed that Ux and A11y should jam! A meetup is ideal, but why don't we start with a conference call?

I'll comment further when I've found a snickers bar.

Comment 25

8 years ago
Ok. It seems we found stumbling block. It's a bug 418521. They don't want to
draw focus ring when user click by mouse, therefore Justin's point about
keyboard bizarre makes sense. It's impossible to see what widget is focused and
therefore key pressing will confuse (for example, space on play button and
space on mute button). So if I get right this is a main point why Justin
doesn't want to have controls focusable as I proposed in the patch. Justin,
please correct me if I'm wrong.
The focus rings are ugly, but that's basically a separate problem from the "bizarre" keyboard UI... Having keys do different things depending on where you click/tab on the video is the primary issue. That a single simple solution (nonfocusable controls) resolves both problems is just a bonus.

Comment 27

8 years ago
(In reply to comment #26)
> The focus rings are ugly, but that's basically a separate problem from the
> "bizarre" keyboard UI... Having keys do different things depending on where you
> click/tab on the video is the primary issue. That a single simple solution
> (nonfocusable controls) resolves both problems is just a bonus.

I think this is because you tend to consider video control as indivisible control. I tend to consider it as element consisted from several elements. And from this point of view it's definitely good that keys make different things depending what is currently focused. I think video control is too complex and it's not correct to think of it as unique thing. So that for example I believe non focusable slider doesn't make sense because it behaves like an image showing what keys user presses rather than real control element. I think we become broken on keyboard compatibility level.

Comment 28

8 years ago
(In reply to comment #1)
> Support for media keys (found on some keyboards), should be added on this bug,
> or filed separately. (See bug 486899 comment #59)
Already filed bug 448910

Updated

8 years ago
Blocks: 470629
(Reporter)

Updated

8 years ago
Attachment #379385 - Flags: review?(bolterbugz)
(Reporter)

Comment 29

8 years ago
Comment on attachment 379385 [details] [diff] [review]
patch2

Cancelling my review until we get dolske on board :)

Updated

8 years ago
Attachment #379385 - Flags: review?(mconnor)
(Reporter)

Comment 30

8 years ago
Community feedback:
http://mindforks.blogspot.com/2009/09/keyboard-control-of-html5-video.html
(Reporter)

Comment 31

8 years ago
Marking 1.9.2? because we need to decide if we will go with this interaction for improved accessibility (see comment #30) and if so, we should probably get it into 3.6.
Flags: blocking1.9.2?
(Reporter)

Comment 32

8 years ago
Alexander, you are currently assigned this bug but I know you are busy. Is the latest patch up to date?

Boriss, do you think comment #30 is the way to go?
I don't think anything has really changed since my last comment 23. Blog comments can be a useful data point, but they shouldn't drive UI design.
(Reporter)

Comment 34

8 years ago
(In reply to comment #33)
> I don't think anything has really changed since my last comment 23. Blog
> comments can be a useful data point, but they shouldn't drive UI design.

Yep I agree blog comments should not drive UI design but the blog itself was something Limi seemed to agree with when I explained it verbally at the face to face. It would be good to hear a consensus from the Ux team on this bug. (I personally don't have a strong opinion what is the best approach).

If we can get comment 23 part 3 done, that would be great. One thing I would request is adding some visual affordance for when the user tabs to the media element that says "yeah you are on this thing now and you can use the keyboard to do stuff". For accessibility with AT (e.g. screen readers) we can provide our own similar hinting through API. I think we can also listen to media events to provide feedback as well.
Justin, do you really want to block on this? I'm thinking no.
(Reporter)

Comment 36

8 years ago
Demoting to 'wanted' 1.9.2.
Flags: blocking1.9.2? → wanted1.9.2?
Right, I don't think this blocks now.
Flags: wanted1.9.2? → wanted1.9.2+
Component: Video/Audio → Video/Audio Controls
Product: Core → Toolkit
QA Contact: video.audio → video.audio

Comment 38

7 years ago
Dose this bug cover Media Keys ?

I can't control video/audio using Media Keys (Play, Pause, Forward, Backward, ..etc.)

Comment 39

7 years ago
(In reply to comment #38)
> Dose this bug cover Media Keys ?
> 
> I can't control video/audio using Media Keys (Play, Pause, Forward, Backward,
> ..etc.)

Sorry, I didn't notice Comment#1 !!!

Comment 40

4 years ago
ARIA media player design pattern - Keyboard Control Proposal:
http://lists.w3.org/Archives/Public/wai-xtech/2013Feb/0018.html
(Reporter)

Updated

3 years ago
Keywords: ux-control, ux-discovery
Whiteboard: [parity Chrome]

Comment 41

3 years ago
Anybody knows how Chrome solved this issue?
(Reporter)

Updated

2 years ago
Whiteboard: [parity Chrome] → [parity Chrome, IE, Opera]

Comment 42

a year ago
Any update on this? chrome/IE/Edge all include the individual controls in the focus order.
Flags: needinfo?(surkov.alexander)
(Reporter)

Comment 43

a year ago
This needs a new owner.
Assignee: surkov.alexander → nobody
Status: ASSIGNED → NEW
Flags: needinfo?(surkov.alexander)

Updated

8 months ago
Duplicate of this bug: 1311888

Comment 45

8 months ago
How can we move this 7 year old bug forward? 

Other major browsers have embraced accessibility as a priority and added keyboard navigability along with an obvious visual focus state to the <video><audio> elements and their controls. (IE/Edge/Opera/Chrome).

Is keyboard navigability a minimum requirement for the FF team when working with native elements?

If an author uses a native <video><audio> element without the aid of 3rd party JavaScript plugins they end up with elements on the page that fail basic WCAG 2.0 guidelines. e.g. 2.1.1 Keyboard Level A

Only sighted users with a mouse and fine motor skills can enjoy the content in these native tags when using Firefox. It is well proven many users with disabilities use Firefox as their primary web browser.

Let me know if I can help in any way in terms of testing / user testing and such. ;)
(In reply to Joe Watkins from comment #45)
> How can we move this 7 year old bug forward? 

This bug needs someone who can help write the code for it. See comment #23 and comment #40 for details, as well as we should try to meet compat with other browsers where possible.
You need to log in before you can comment on or make changes to this bug.