1483703 - Don't block the autoplay media element without audio track

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P2)

Description

[Alastor Wu [:alwu]]

Because the media element without audio track is non-audible, we should allow it to autoplay like muted media.

Comment 1

[Alastor Wu [:alwu]]

Hi, Chris, 

I think we should block play() if the media element hasn’t loaded metadata yet.

Although you think it’s racy (from bug1464922 comment0), but I think it’s not, because of following reasons. 

From the threads perspective, changeReadyState() and Play() are both running in main thread, so it won’t race.

From the depending relationship perspective, we have already made the result of play promise depending on the value of different attributes, such as mute, volume and user activation flag. Therefore, it seems reasonable to me to make play() adding one more dependency, ready state.

—

My idea is that, 

(1) if media element have not loaded metadata when it requests play(), start to load the resource of media element, and reject the play promise.

(2) if media element which had loaded metadata and no audio track requests play, reject play promise.

How do you think?

Thank you!

Comment 2

[Chris Pearce [:cpearce (Not reading bugmail)]]

The cases I'm worried about:

{
    // case 1
    let v = document.createElement("video");
    v.src = "http://pearce.org.nz/video/320x240.ogg";
    v.play();
    document.body.appendChild(v);
}
{
    // case 2
    let v = document.createElement("video");
    v.src = "http://pearce.org.nz/video/320x240.ogg";
    v.addEventListener("loadedmetadata", () => v.play(), { once: true});
    document.body.appendChild(v);            
}

A lot of sites do case 1. By the time the site calls play(), we won't have reached loadedmetadata yet, and so the play would be blocked because we don't know that the video doesn't have an audio track yet. But in case 2, we'll have reached loadedmetadata by the time we call play(), and so we'll know that 320x240.ogg doesn't have an audio track, and so the play will not be blocked.

Basically, I think we should try to make both these cases work.

Comment 3

[Chris Pearce [:cpearce (Not reading bugmail)]]

Alwu and I discussed this, and decided that if we've reached loadedmetadata at the time that play() is called we'll allow autoplay if the media doesn't have an audio track, but we won't inject a "wait until we've reached loadedmetadata" step.

So potentially we'll get different results if play() is called before loadedmetadata than if it's called after, but at least we'll fix the apple.com case.

Comment 5

[Alastor Wu [:alwu]]

Attachment: Bug 1483703 - part1 : allow media without audio track to autoplay.

We would allow media without audio track to autoplay after it had loaded the metadata.

If media hasn't loaded metadata yet, we would treat it as audible media and then block it.

Comment 6

[Alastor Wu [:alwu]]

Attachment: Bug 1483703 - part2 : add telemetry for the media which was blocked before loading metadata and ended up being without audio track.

Add two telemetry scarlar,

"MEDIA_BLOCKED_NO_METADATA" records how many media which was blocked because it hadn't loaded metadata yet.
"MEDIA_BLOCKED_NO_METADATA_ENDUP_NO_AUDIO_TRACK" records how many media which was blocked because it hadn't loaded metadata and ended up for being no audio track.

By collecting those data, we can know the proportion of media which should be autoplay but was blocked because of lacking metadata.

Comment 7

[Alastor Wu [:alwu]]

Attachment: Bug 1483703 - part3 : modify test.

Comment 8

[Alastor Wu [:alwu]]

Attachment: Bug 1483703 - part4 : modify current telemtry scalar because we won't block media without audio track anymore.

Since we don't block media without audio track anymore, the original telemetry scalar becomes useless.

We need to change its meaning in order to know the number of allowed autoplay without audio track.

Comment 9

[Alastor Wu [:alwu]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=413969cbfbcd32034b4547e845f2837b4765740f&selectedJob=194610913

Comment 10

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 9002132 [details]
Bug 1483703 - part1 : allow media without audio track to autoplay.

Chris Pearce (:cpearce) has approved the revision.

Comment 11

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 9002134 [details]
Bug 1483703 - part2 : add telemetry for the media which was blocked before loading metadata and ended up being without audio track.

Chris Pearce (:cpearce) has approved the revision.

Comment 12

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 9002135 [details]
Bug 1483703 - part3 : modify test.

Chris Pearce (:cpearce) has approved the revision.

Comment 13

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 9002138 [details]
Bug 1483703 - part4 : modify current telemtry scalar because we won't block media without audio track anymore.

Chris Pearce (:cpearce) has approved the revision.

Comment 14

[Alastor Wu [:alwu]]

Attachment: Data review request

Comment 15

[François Marier [:francois]]

Comment on attachment 9002593 [details]
Data review request

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in Scalars.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Not permanent.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1.

5) Is the data collection request for default-on or default-off?

Default ON, only in pre-release channels.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No, telemetry alerts are enough.

Comment 16

[François Marier [:francois]]

Comment on attachment 9002138 [details]
Bug 1483703 - part4 : modify current telemtry scalar because we won't block media without audio track anymore.

François Marier [:francois] has approved the revision.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd6a42517bb9
part1 : allow media without audio track to autoplay. r=cpearce

Comment 18

[Alastor Wu [:alwu]]

hmm, need to wait the patch2 getting r+, and then I could push other patches.

Comment 19

[François Marier [:francois]]

Comment on attachment 9002134 [details]
Bug 1483703 - part2 : add telemetry for the media which was blocked before loading metadata and ended up being without audio track.

François Marier [:francois] has approved the revision.

Comment 20

[Alastor Wu [:alwu]]

The comment17 I pushed is the wrong version which became closed and I can't update it again. 
still trying to find the way to reopen and update it...

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/71ad474cdeeb
part1 : allow media without audio track to autoplay. r=cpearce

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3326c0e920d4
part2 : add telemetry for the media which was blocked before loading metadata and ended up being without audio track. r=cpearce,francois

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d915d805e964
part3 : modify test. r=cpearce

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bb9759881cd7
part4 : modify current telemtry scalar because we won't block media without audio track anymore. r=cpearce,francois

Comment 25

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/cd6a42517bb9
https://hg.mozilla.org/mozilla-central/rev/71ad474cdeeb
https://hg.mozilla.org/mozilla-central/rev/3326c0e920d4
https://hg.mozilla.org/mozilla-central/rev/d915d805e964
https://hg.mozilla.org/mozilla-central/rev/bb9759881cd7