1509933 - Use transient user activation as the default way to detect if the action is an user-input for `click-to-play`

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Alastor Wu [:alwu]]

Use `media.autoplay.blocking_policy` to decide the current autoplay policy.
Use `dom.user_activation.transient.timeout` to adjust the timeout value for the policy `1` (transient activation)

Comment 2

[Chris Mills [:cmills]]

Probably needs detailing somewhere

Comment 3

[Edgar Chen [:edgar]]

Alastor, we had implemented an API in Document to track user gesture. It is kinda global state of the page indicating whether a page is considered as having user gesture within a recent time frame (5 sec). It also works in async callback or promise resolution as long as the callbacks happen within the time frame that user interacts with the page.

Basically, replacing UserActivation::IsHandlingUserInput() calls in https://searchfox.org/mozilla-central/rev/cce8b90aece0f42e5025e45282de16066eeaa662/dom/media/AutoplayPolicy.cpp#170 with the new API should just work.

Comment 4

[Alastor Wu [:alwu]]

Attachment: Bug 1509933 - part1 : add new pref 'media.autoplay.blocking_policy'.

Comment 5

[Alastor Wu [:alwu]]

Attachment: Bug 1509933 - part2 : migrate the old pref to the new pref.

Comment 6

[Alastor Wu [:alwu]]

Attachment: Bug 1509933 - part3 : use the new pref in tests.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3abb24d6dfc9
part1 : add new pref 'media.autoplay.blocking_policy'. r=geckoview-reviewers,snorp,padenot
https://hg.mozilla.org/integration/autoland/rev/951ea1d0f42a
part2 : migrate the old pref to the new pref. r=mconley
https://hg.mozilla.org/integration/autoland/rev/c15ecdcd5d13
part3 : use the new pref in tests. r=padenot

Comment 8

[Dorel Luca [:dluca]]

Backed out 3 changesets (bug 1509933) for Linting failure in gecko/mobile/android/app/mobile.js. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=302376999&repo=autoland&lineNumber=303

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=c15ecdcd5d135ff216901ae76857467ac972a570&selectedTaskRun=Ja9O-KkDTz6XAYyvpd37IA-0

Backout:
https://hg.mozilla.org/integration/autoland/rev/4d3907096e4995eb5c9c344404463b977d2d8979

Comment 9

[Alastor Wu [:alwu]]

Any idea why this lint failure happens? I can't understand this lint error because they're just simple preference declaration.

Comment 10

[Dorel Luca [:dluca]]

Sorry, no clue on that one,

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5b34a70e2378
part1 : add new pref 'media.autoplay.blocking_policy'. r=geckoview-reviewers,snorp,padenot
https://hg.mozilla.org/integration/autoland/rev/59985cb69387
part2 : migrate the old pref to the new pref. r=mconley
https://hg.mozilla.org/integration/autoland/rev/95805db3a142
part3 : use the new pref in tests. r=padenot

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/5b34a70e2378
https://hg.mozilla.org/mozilla-central/rev/59985cb69387
https://hg.mozilla.org/mozilla-central/rev/95805db3a142

Comment 13

[uBlock-user]

Seems commits in comment 12 broke click-to-play for good. media.autoplay.blocking_policy is set to 1 in Nightly and autoplay is getting bypassed on YT, bringing back https://bugzilla.mozilla.org/show_bug.cgi?id=1588138 issue like before.

Comment 14

[uBlock-user]

Setting dom.user_activation.transient.timeout to 100 works successfully, but setting this pref to 1 blocks the autoplay on YT and the video won't play, on the other hand setting the value to 1000 and the autoplay would occur.

Summary : media.autoplay.blocking_policy is already set to 1 before changing the aformentioned pref and autoplay doesn't get blocked without changing that pref.

Comment 15

[Alastor Wu [:alwu]]

So when setting media.autoplay.blocking_policy=1 would use a transitent user gesture for the autoplay policy. That is the action of requesting autoplay would be treated as an user's intention if the action happens within a certain period of time after user interacts with the page.

The dom.user_activation.transient.timeout controls how long that time is. So if you set it to 100 (ms), which means only the play invocation that happens within 100ms after you interacts with the page would be allowed. And from your cases, the timing of Youtube calling play() is longer than 100ms and shorter than 1000ms. So probably we should use something smaller than 1000ms but larger than 100ms as our threshold.

The reason we wanted to introduce this policy is because the previous policy didn't work on certain situations and that is actually a spec issue and it probably won't be fixed. Currently, the new spec for the user input is [1], which addresses the user input by another different concept. In addition, a lots of users complain that the autoplay happens unexpectedly after they click something unrelated with the media, so I think introducing an expire concept for autoplay might be a good thing and we wanted to see how well users like or dislike this new policy.

Finally, setting media.autoplay.blocking_policy=2 can fallback to the old policy.

[1] https://html.spec.whatwg.org/multipage/interaction.html#user-activation-data-model

Comment 16

[uBlock-user]

So probably we should use something smaller than 1000ms but larger than 100ms as our threshold.

I tested few more values for dom.user_activation.transient.timeout and it works with 200/350/500/750 ms, not tested beyond 750 as 1000 fails which I mentioned before. I suggest that value should be changed from 5000 and set to 500 which I feel is adequate. WDYT ?

Comment 17

[Alastor Wu [:alwu]]

According to the spec, The transient activation duration is expected be at most a few seconds [1], but it apperantly doesn't fit for the autoplay.

[1] https://html.spec.whatwg.org/multipage/interaction.html#user-activation-data-model

---

Hi, Edgar,

How do you think if we create a special timeout threshold just for the autoplay which would be much shorter than the value of dom.user_activation.transient.timeout? Then, maybe have a new function like HasValidTransientUserGestureActivationForAutoplay() to allow us to use a shorter time for autoplay check, so that we can achieve an ideal result for autoplay and won't affect other APIs which are using transitent activation at the same time.

Thank you.

Comment 18

[Edgar Chen [:edgar]]

(In reply to Alastor Wu [:alwu] from comment #17)

How do you think if we create a special timeout threshold just for the autoplay which would be much shorter than the value of dom.user_activation.transient.timeout?

I don't like having multiple thresholds, but have no other better idea than this.
So Google Chrome still uses the sticky flag for autoplay, does my understanding correct?

Comment 19

[Alastor Wu [:alwu]]

(In reply to Edgar Chen [:edgar] from comment #18)

I don't like having multiple thresholds, but have no other better idea than thctanding correct?

Yes, and that is one of our autoplay policies [1] when media.autoplay.blocking_policy=0, but that policy doesn't fit all users' requirement because not all user-activations are related with media directly, such as GDPR Cookie Law.

So that is why I wanted to provide another option ( media.autoplay.blocking_policy=1), which is to use transient activation for autoplay. We don't have an explicit spec to define if blocking autoplay should use sticky activation or transitent activation, so I would like to explore it as both ways to allow users to have more choices.

Or do you think who should be involved in this discussion as well if I want to introduce a specific threshold for blocking autoplay?

[1] https://docs.google.com/document/d/1p81bYntlLMTxXoANnQXS9KH0TZ4y8aKKeH9x3GVS3eY/edit#heading=h.k3q4co83ntet

Comment 20

[Edgar Chen [:edgar]]

(In reply to Alastor Wu [:alwu] from comment #19)

We don't have an explicit spec to define if blocking autoplay should use sticky activation or transient activation

The major concern is compatibility, that is why we use the same threshold as Chrome.
If the specific threshold is used for blocking autoplay only, I think it should be fine (Hope that it is the only case that we need to introduce a different threshold).
But if the spec has defined autoplay to use transient activation explicitly, I think we should use the same threshold as others then.

Comment 21

[Alastor Wu [:alwu]]

Ok, thank you for your suggestion, then I will add a new threshold for the autoplay only. The spec didn't restrict what way should be used for determining the autoplay [1], the user agent can have its own implementation.

[1] https://html.spec.whatwg.org/multipage/media.html#allowed-to-play

Comment 22

[Alastor Wu [:alwu]]

Hmm, I tested some websites to see if the transient activation can work well on them, but I found that the result is not consistent because different website has varied time to trigger media.

For example, use 750ms can work well for Twich, Vimeo, Youtube and Soundcloud, but it would break the normal playback for BBC because after pressing play button, it needs more time to prepare its playback and BBC didn't handle the situation of the play promise being rejected after user presses the play button. So the video would be stuck, which is like loading forever.

And the Netflix is also affected when we use 750ms, when you press play in the homepage, the video won't start playing at the beginning. But luckily, they've addressed the play rejected case, so they would provide anohter play button to allow user to start playing video again.

That seems indicating that the transient activation is not robost enough to replace the old click-to-play (which is media.autoplay.blocking_policy=2), but we are not able to fix the old problem (user-input flag can't be propagated into an async event handler), so it seems that the policy of transient activation can only be used as an enhencement for the existing policy which is using sticky activation :(

Comment 23

[uBlock-user]

After using transient activation for three days, it seems I was mistaken about the last two values I suggested, there's a race condition involved sometimes and those two don't work, so anything beyond 350 ms is running on race condition. Speaking of media.autoplay.blocking_policy fallback mode, is this temporary/will get removed soon ?

Comment 24

[Chris Mills [:cmills]]

Hrm, looking at this again, it doesn't look like this is a developer-facing feature that needs documenting on MDN. This is more of an internals thing?

Removing dev-doc-needed keyword as a result. Let me know if you do want this mentioned on MDN anywhere, along with some suggested wording. Thanks!

Comment 25

[Alastor Wu [:alwu]]

Sorry for my late reply. The fallback mode (media.autoplay.blocking_policy=2) would still be kept existing for a long while until we really figure out a good solution for blocking autoplay. The ideal situation is to allow that mode work for all situations, but that issue seems not possible to be solved because of the spec issue :(

Comment 26

[bughit]

This migration broke your recommendation in Bug 1427966 to use media.autoplay.enabled.user-gestures-needed=false to block autoplay on youtube in-place navs. Why wasn't this migrated to media.autoplay.blocking_policy=2, which seems to work?

Comment 27

[nabiel.omair]

(In reply to Edgar Chen [:edgar] (away ~ 02/25) from comment #3)

Alastor, we had implemented an API in Document to track user gesture. It is kinda global state of the page indicating whether a page is considered as having user gesture within a recent time frame (5 sec). It also works in async callback or promise resolution as long as the callbacks happen within the time frame that user interacts with the page.

Basically, replacing UserActivation::IsHandlingUserInput() calls in https://searchfox.org/mozilla-central/rev/cce8b90aece0f42e5025e45282de16066eeaa662/dom/media/AutoplayPolicy.cpp#170 with the new API should just work.

What about sending a message, within a user clicked function, through browser.runtime.sendMessage from extension's Popup to background.js script ?
In background script this throws
DOMException: getDisplayMedia requires transient activation from a user gesture