audio "autoplay after page interaction" ignores keyboard events

VERIFIED FIXED in Firefox 66

Status

()

defect
P2
normal
VERIFIED FIXED
2 months ago
2 months ago

People

(Reporter: pomax, Assigned: alwu)

Tracking

(Blocks 1 bug, {dev-doc-needed})

64 Branch
mozilla67
Points:
---
Bug Flags:
qe-verify +

Firefox Tracking Flags

(firefox65 wontfix, firefox66 verified, firefox67 verified)

Details

Attachments

(4 attachments, 1 obsolete attachment)

(Reporter)

Description

2 months ago

Audio elements are not allowed to play on a page that has not been interacted with by the user, and that's fine, but apparently "keyboard interaction" does not count as interaction for this purpose and that's not okay. People who can't use mice should still be able to play web games with sound even though they're navigating purely with a keyboard.

STR:

<html>
<audio id="clip" src="some-random-file.mp3"></audio>
<button id="test">play sound</button>
<script>
let clip = document.getElementById('clip');
let btn = document.getElementById('test');
btn.addEventListener('click', evt => {
clip.play();
});
btn.focus();
</script>
</html>

load page (which will make sure all the user has to do is press enter), and press enter.

Expected:

The clip plays because the user has interacted with the page.

Actual:

Console error "Autoplay is only allowed when approved by the user, the site is activated by the user, or media is muted."

Additional:

This works 100% fine if the user clicks the button using the mouse, so something's forgetting to record the fact that keyboard interaction is just as valid as mouse interaction.

What do other browsers do:

Edge: plays audio on "enter" from keyboard
Chrome: plays audio on "enter" from keyboard

Updated

2 months ago
Priority: -- → P2
(Assignee)

Updated

2 months ago
Assignee: nobody → alwu
(Assignee)

Comment 1

2 months ago

carriage return and space are common keys which user might use to start media, so we should take account them as vaild user gesture inputs.

tab is also a common interactive key which user will use, and Chrome treats it as a valid input as well.

As their pseudo char code are zero, we have to check their key code in order to distinguish them from other controls keys such as shift, alt...

(Assignee)

Comment 2

2 months ago

Except printable keys, we would treat carriage return, space and tab as valid user gesture inputs. Other keys won't activate the document.

(Assignee)

Comment 3

2 months ago

We can listen for the block event to know whether AudioContext hasn't started yet which can speed up the test.

(Assignee)

Comment 4

2 months ago

Testing web audio with GUM is not related with the original purpose of this test. In order to reduce the complexity of this test, separate this part as another new test.

(Reporter)

Comment 5

2 months ago

If I'm reading #2 correctly, that would rule out "space" and "tab", which should definitely count as the user having activated the document. (tab for obvious reasons, space because it acts as the same signal as carriage return for buttons etc)

(Assignee)

Comment 6

2 months ago

(In reply to Pomax [:pomax] from comment #5)

If I'm reading #2 correctly, that would rule out "space" and "tab", which should definitely count as the user having activated the document. (tab for obvious reasons, space because it acts as the same signal as carriage return for buttons etc)

What #2 are you referring to? Did you mean the patch1?

It won't rule out the space, it will still treat space as a valid user gesture input. However, before applying my patches, we haven't supported tab.

I tested tab on Chrome, they treat it as a valid input. So maybe we should also allow it in order to reduce the compat issue.

(Reporter)

Comment 7

2 months ago

Ah sorry, no: I meant comment #2.

Attachment #9046807 - Attachment description: Bug 1530220 - part1 : allow 'carriage return' as a vaild user gesture input. → Bug 1530220 - part1 : allow some non-printalble keys as vaild user gesture inputs.
(Assignee)

Comment 8

2 months ago

(In reply to Pomax [:pomax] from comment #7)

sorry, no: comment #2.

Has added tab as a valid input, thank you for reminding.

(Assignee)

Comment 9

2 months ago

In order to reduce web compat issue, we should also allow arrow keys and the tab key as valid user gesture input, which Chrome has allowed to active the document.

Attachment #9046807 - Attachment description: Bug 1530220 - part1 : allow some non-printalble keys as vaild user gesture inputs. → Bug 1530220 - part1 : allow some non-printalble keys as supported user gesture inputs to activate document.

First off, our intent with gesture activation is to unblock media autoplay when the user has demonstrated intent to autoplay, rather than just when they've interacted with a page.

We don't consider interaction with the browser as demonstration of intent to play media. So pressing keys which cause the browser to scroll the viewport, for example, should not gesture activate the page.

I think space is already gesture activating.

Try this:

  1. Open https://pearce.org.nz/video/autoplay.html in Firefox with autoplay blocked
  2. Press space.
  3. Wait... after a few seconds, a timeout (set on page load) runs and script will call play().
  4. Observe that video starts to play.

So space is unblocking autoplay today in Firefox. While pressing space does cause the browser to scroll its viewport, it also is commonly used as a keyboard shortcut to play/pause media in desktop media player software. Firefox's built in video controls also use space to play/pause media. So I'm OK with it gesture activating.

If you try the above procedure but press Tab or Enter instead of Space, autoplay does not start. So Tab and Enter don't unblock autoplay, as pointed out in comment 0 above.

I don't think it makes sense to gesture activate on Tab press. Tab is used to shift focus, which doesn't demonstrate intent to autoplay media.

An Enter keypress outside of an editable element isn't interaction with the browser, so I'd be OK with Enter on non-editable controls causing gesture activation.

Attachment #9046882 - Attachment is obsolete: true

Comment 11

2 months ago
Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7c1b053e161b
part1 : allow some non-printalble keys as supported user gesture inputs to activate document. r=masayuki,cpearce
https://hg.mozilla.org/integration/autoland/rev/f706441a86b5
part2 : modify test in order to test different kinds of keycode. r=cpearce
https://hg.mozilla.org/integration/autoland/rev/9ce43f1e7bc0
part3 : speed up test. r=padenot
https://hg.mozilla.org/integration/autoland/rev/07c49703b70c
part4 : separate tesing web audio with gum to another test. r=cpearce
(Assignee)

Comment 13

2 months ago

Comment on attachment 9046807 [details]
Bug 1530220 - part1 : allow some non-printalble keys as supported user gesture inputs to activate document.

Beta/Release Uplift Approval Request

  • Feature/Bug causing the regression: Bug 1452536
  • User impact if declined: User won't be able to activate docuement to allow autoplay by pressing enter key.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce:
  1. goto https://alastor0325.github.io/htmltests/autoplay_tests/autoplay_test2.html
  2. press enter key
  3. video should start playing
  • List of other uplifts needed: all patches in this bug
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): These patches doesn't introduce any other behavioural changes, we just extend our support key list in order to let enter key activate the document. In addition, we also have an automation test for this issue.
  • String changes made/needed: none
Attachment #9046807 - Attachment description: Bug 1530220 - part1 : allow some non-printalble keys as supported user gesture inputs to activate document. → Bug 1530220 - part1 : allow 'carriage return' as a vaild user gesture input.
Attachment #9046807 - Flags: approval-mozilla-beta?
Flags: qe-verify+

Observed the issue with 66.0b13.
Fix verified with 67.0a1 (2019-03-05) on Windows 10, macOS 10.13.6, Ubuntu 18.04.

Status: RESOLVED → VERIFIED

Comment on attachment 9046807 [details]
Bug 1530220 - part1 : allow some non-printalble keys as supported user gesture inputs to activate document.

Adds keyboard control for block autoplay, verified in nightly.
OK for uplift to beta 14.

Attachment #9046807 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9046807 - Attachment description: Bug 1530220 - part1 : allow 'carriage return' as a vaild user gesture input. → Bug 1530220 - part1 : allow some non-printalble keys as supported user gesture inputs to activate document.
QA Whiteboard: [qa-triaged]
(Reporter)

Comment 17

2 months ago

@cpearce it feels like your argument for why space is supported, but tab isn't, undermines itself: if space is supported because it's commonly used in desktop media players, then there should be no objection to supporting tab as well, as that's commonly used to activate the document in desktop browsers:

  1. put https://pearce.org.nz/video/autoplay.html in chrome's URL bar and hit enter,
  2. wait a few seconds,
  3. tab into the document,
  4. the video starts playing.

Given that Chrome effectively defines what is "common" these days (as much as I wish that weren't true), that feels reason enough to add support for tab as well, as that's what people will be used to by now.

(I'd mention what Edge does, but given that that's about to become Chrome, its behaviour is going to be "whatever Chrome already does", so that just adds to the argument that tab is commonly used to activate the document).

Verified with 66.0b14 as well.

You need to log in before you can comment on or make changes to this bug.