1182329 - WebRTC streams should be able to be automatically played even with media.autoplay.enabled=false

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P3)

Description

[Gerhard Großmann]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0
Build ID: 20150511103818

Steps to reproduce:

To prevent mp3 files from playing automatically I opened about:config and set media.autoplay.enabled to false.


Actual results:

Not just HTML audio and video files didn’t start automatically but also the webcam didn’t work any more within WebRTC applications like https://appear.in/ or Firefox Hello.


Expected results:

Audio and video files have a play button where I’m able to start the media which isn’t autoplayed. With WebRTC there’s no possibility to use video anymore.

Is it intented that Firefox Hello stops sending a webcam image when autoplay is deactivated? I don’t think so. WebRTC video shouldn’t be affected by the autoplay setting.

Comment 1

[Mark Banner (:standard8)]

From what I can tell this is a problem in the sdk. It might be improved in the latest nightly/dev edition builds where the Hello code add some play calls, but I suspect remote audio is still broken.

I can't see any .play() calls in the sdk except for when an element is moved in the DOM tree. Hence neexinfo to Tokbox to take a look.

Comment 2

[msander]

Adam, could you or someone on your team take a look at this?

(In reply to Mark Banner (:standard8) (afk until 20th July) from comment #1)
> From what I can tell this is a problem in the sdk. It might be improved in
> the latest nightly/dev edition builds where the Hello code add some play
> calls, but I suspect remote audio is still broken.
> 
> I can't see any .play() calls in the sdk except for when an element is moved
> in the DOM tree. Hence neexinfo to Tokbox to take a look.

Comment 3

[Adam Ullman]

@msander Yeah sure. I have filed an issue in our bug database and we'll take a look at it.

Comment 4

[u515900]

I'm working on this here at TokBox and according to this comment: https://bugzilla.mozilla.org/show_bug.cgi?id=659285#c113 confirmed by my tests calling "play()" doesn't solve the problem since the call would need to be dispatched from a task initiated by a user interaction which we can't ensure.

Do you have more info here, or any work around suggestion?


(In reply to Mark Banner (:standard8) (away until 24 Aug) from comment #1)
> From what I can tell this is a problem in the sdk. It might be improved in
> the latest nightly/dev edition builds where the Hello code add some play
> calls, but I suspect remote audio is still broken.
> 
> I can't see any .play() calls in the sdk except for when an element is moved
> in the DOM tree. Hence neexinfo to Tokbox to take a look.

Comment 5

[Gerhard Großmann]

Sorry, Jonathan, I have no idea about SDKs or play() calls. I just wondered why my WebRTC video didn’t work anymore and identified the media.autoplay.enabled setting by deleting and restoring different settings until only one was left.

As said, not only Hello is affected but also https://appear.in/ , http://webr.tc , http://talky.io or http://webrtc.github.io/samples/src/content/devices/input-output/ . Strangely the service https://vline.com/ does work despite autoplay is disabled.

Is it possible to force displaying a play button when autoplay is disabled? So the WebRTC video could be started by the user.

Comment 6

[Gerhard Großmann]

Maybe Maire Reavy [:mreavy] misunderstood what this bug’s about and therefore changed my bug title from “WebRTC video doesn’t work if media.autoplay.enabled = false” to the summary with only Hello and play(). I still think the bug is more general.

Comment 7

[Mike de Boer [:mikedeboer]]

Perhaps, let's ask her! :)

Comment 8

[Maire Reavy [:mreavy]]

It's very common for sites to use autoplay since turning off autoplay is not standard and not user exposed.  IIUC the reporter turned off autoplay because he didn't want it for video playback.  (I agree ads that automagically play can be annoying.) 

However, "autoplay" is the same one used for all video (WebRTC as well as MP4/WebM/etc).  So if a user turns off autoplay in the browser, it is disabled for video elements fed by WebRTC.  This has nothing to do with WebRTC Platform.  

It so happens that many of the popular sites/services (e.g. webr.tc, talky) use autoplay though probably not all;  that is likely why vline works (it probably doesn't use autoplay -- I'd have to confirm to be certain).  

Hello (as a service) can choose not use autoplay -- or force it to be used even if the pref is off.  To do so, I believe that would require a change to the SDK.  I defer to TokBox and folks like Standard8 on those details.  But this is not a WebRTC Platform issue -- and Mozilla can't control how other sites chose to handle autoplay.  We can only control Hello, which is why I re-titled this to be a Hello-specific bug/request.

Comment 9

[Adam Ullman]

Maire, our issue is that we're not sure how to "force it to be used even if the pref is off". Calling the play() method doesn't work unless it is triggered by user interaction as per Jonathan's comment above.

Good point though about VLine. I'm not sure how they get it to work.

Comment 10

[u515900]

Actually I can't get Vline to work either when I set the autoplay disabled flag to true. This is not surprising since they don't do anything special. Gerhard can you confirm that it works for you? If yes, could you specify which version of FF? FYI, I'm testing in FF DE (42.0a2).

Adam, I think what Maire was saying is that Hello (as a "trusted" application embedded in FF) can probably override the user preference to fix the problem in its context. We, as OpenTok library provider, can't do much beside what Gerhard is saying, which is essentially calling ".play()" on a user action. I don't really know how to offer a nice UX with that though.

FYI what Chrome is doing in its mobile version is that it prevents autoplay on "regular" video but overrides it when it is used in WebRTC context so that it always works.

Comment 11

[Maire Reavy [:mreavy]]

I still believe this is an issue we want to solve for Hello, not in general.  IMO the best person to find a solution to this is Standard8 (Mark Banner), who should now be back from PTO.  

Mark -- If you have questions about what to do or how to do it (or if you think there's something I missed in my analysis), please feel free to reach out to me or Randell with questions/suggestions.

Comment 12

[Gerhard Großmann]

Yes, I can confirm that the service vline.com works for me, even with autoplay.enabled set to false (this setting seems to be the other way round in FF42). I use Firefox 40.0 Mozilla Firefox for Ubuntu, canonical - 1.0.

The Chrome solution Jonathan described makes a lot of sense to me: Blocking only regular video and always allow WebRTC – latter has to ask for accessing the cam/mic anyway.

Comment 13

[u515900]

Gerhard, thanks for the details. It makes sense since the behaviour changed in FF 41 (see https://bugzilla.mozilla.org/show_bug.cgi?id=659285#c113), so Vline who's calling play programmatically, instead of relying on "autostart" attribute, is stil working in FF 40- but stops working in FF 41+.

Comment 15

[Mark Banner (:standard8)]

I've taken a look. Here's what I think are the facts:

- When "media.autoplay.enabled" is false, using the autoplay attribute doesn't work.
-- Using a different route of calling videoElement.play() will work, up until Firefox 41 (as per bug 659285 comment 113).

- Post Firefox 41, there's nothing a web page using WebRTC can do if autoplay is disabled, unless they somehow detect that and present something to the user to click on.

- Per comment 10, Google Chrome mobile has a way to deny autoplay except for WebRTC contexts.

Maybe we can find a way to make this work for WebRTC contexts in Firefox?

In any case, this is obviously a core bug in the first instance, so I'm re-summarising, moving and marking it as a regression from bug 659285.

Comment 16

[Chris Peterson [:cpeterson]]

This is a WebRTC bug.

Comment 17

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Chris Peterson [:cpeterson] from comment #16)
> This is a WebRTC bug.

Chris - we (from webrtc) can't override the operation of the media element pref that disables autoplay...  The pref disables autoplay, which websites using realtime media really rely on (though they could use play(), but apparently this pref also blocks that by default as well). Unlike streaming media, realtime media can't "wait" for the user to click on it. 

At minimum, we need to have some way (read by the media element we attach it to) to mark a (PeerConnection-sourced) MediaStream as autoplayable - with that, we can set it.  We can't just make autoplay work for mediastreams in general, or the pref is trivial to bypass.

Chrome's architecture is different, and some of their shortcuts we don't take with audio/video playback.  They may tag it with an autoplay-is-ok boolean, of course.

So we can mark our streams -- but we need a mark to use; some sort of API surface to allow autoplay to work.  And while not very complex, this is also non-trivial to add to media elements.

Comment 18

[Chris Peterson [:cpeterson]]

OK. I'll move this bug back to the Audio/Video:Playback component because it requires support from the media element.

This bug is won't be a high priority, however, because the STR require a non-default pref value (media.autoplay.enabled=false) and few users will run into the problem.

Comment 19

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Chris Peterson [:cpeterson] from comment #18)
> OK. I'll move this bug back to the Audio/Video:Playback component because it
> requires support from the media element.
> 
> This bug is won't be a high priority, however, because the STR require a
> non-default pref value (media.autoplay.enabled=false) and few users will run
> into the problem.

Yup, I agree, though it's not an uncommon about:config tweak (due to people hating autoplaying videos.... which Chrome has disabled by default nowadays I think).  Also some extensions flip it (FlashBlock did IIRC)

Comment 20

[mirh]

(In reply to Mark Banner (:standard8) from comment #15)
> - Post Firefox 41, there's nothing a web page using WebRTC can do if
> autoplay is disabled, unless they somehow detect that and present something
> to the user to click on.

Well, it seems this somehow manages. https://www.visagetechnologies.com/HTML5/beta/Samples/FaceTracker/sampleTracker.html

Aside of that, anyhow even if it was impossible, can't you just make the "Share" camera selection button send the play() signal?

Comment 21

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Mass change P2 -> P3

Comment 22

[Alastor Wu [:alwu]]

(In reply to Randell Jesup [:jesup] from comment #17)
> At minimum, we need to have some way (read by the media element we attach it
> to) to mark a (PeerConnection-sourced) MediaStream as autoplayable - with
> that, we can set it.  We can't just make autoplay work for mediastreams in
> general, or the pref is trivial to bypass.

Hi, Randell,

Could you help us mark MediaStream as autoplayable and then we can let it pass the check in media element?

This bug is important now, because we would plan to turn on this pref by default in the future (maybe 59 or later) and we need to make sure it won't break any webRTC websites.

Thanks!

Comment 23

[Randell Jesup [:jesup] (needinfo me)]

passing this on to other people in media...  If this pref is turned on, it will break many webrtc sites I believe.  We could check the major ones, but I'm sure tons of demos would break (not to mention our own tests/demos/fiddles).

Comment 24

[Andreas Pehrson [:pehrsons]]

Can we simply make the pref only apply when HTMLMediaElement has an mDecoder member, at [1]?

For audio I can see workarounds with web audio, that would apply to MediaStreams as well. And why not send the videos from a server through WebRTC... Hmm. Someone will have to make a call on how open this should be.


[1] http://searchfox.org/mozilla-central/source/dom/html/HTMLMediaElement.cpp#6919-6921

Comment 25

[Stephan Sokolow]

Also, implementation of any such exception should keep Bug 1405827 (Add an API to allow fine-grained control over media autoplay) in mind so that "scoped autoplay permissions" extensions can give users the power to customize their WebRTC autoplay policy.

(eg. Suppose they frequent sites which use WebTorrent-based video players)

Comment 26

[Alastor Wu [:alwu]]

(In reply to Andreas Pehrson [:pehrsons] from comment #24)
> Can we simply make the pref only apply when HTMLMediaElement has an mDecoder
> member, at [1]?

What I observed is, those webRTC sites uses autoplay keyword, so the actual place we block them is here [1].

It sounds not a good idea, autoplay element with mSrcStream might still cause user problem. The better way would still be to know which stream is used to webRTC and then don't block them. 

[1] http://searchfox.org/mozilla-central/rev/1ebd2eff44617df3b82eea7d2f3ca1b60cc591a0/dom/html/HTMLMediaElement.cpp#6116

Comment 27

[Andreas Pehrson [:pehrsons]]

(In reply to Alastor Wu [:alwu][please needinfo me][GMT+8] from comment #26)
> (In reply to Andreas Pehrson [:pehrsons] from comment #24)
> > Can we simply make the pref only apply when HTMLMediaElement has an mDecoder
> > member, at [1]?
> 
> What I observed is, those webRTC sites uses autoplay keyword, so the actual
> place we block them is here [1].

That's fine, my comment still applies from what I can see.


> It sounds not a good idea, autoplay element with mSrcStream might still
> cause user problem. The better way would still be to know which stream is
> used to webRTC and then don't block them. 

But how do we decide which streams are ok to autoplay and which are not?


> [1]
> http://searchfox.org/mozilla-central/rev/
> 1ebd2eff44617df3b82eea7d2f3ca1b60cc591a0/dom/html/HTMLMediaElement.cpp#6116

Comment 28

[Alastor Wu [:alwu]]

(In reply to Andreas Pehrson [:pehrsons] from comment #27)
> > It sounds not a good idea, autoplay element with mSrcStream might still
> > cause user problem. The better way would still be to know which stream is
> > used to webRTC and then don't block them. 
> 
> But how do we decide which streams are ok to autoplay and which are not?

That is also what I want to know ;)

I ask this question because of comment 17, so I'm wondering whether you have any good way to do it.
> At minimum, we need to have some way (read by the media element we attach it to) to mark a 
> (PeerConnection-sourced) MediaStream as autoplayable - with that, we can set it.

Thanks!

Comment 29

[Andreas Pehrson [:pehrsons]]

I can invent a way to tell which type of source a *track* comes from. But someone else (product manager?) will have to make the call on which tracks we want to poke holes for. Any poked holes mean abusers get through as well.

Comment 30

[Alastor Wu [:alwu]]

(In reply to Andreas Pehrson [:pehrsons] from comment #29)
> I can invent a way to tell which type of source a *track* comes from. But
> someone else (product manager?) will have to make the call on which tracks
> we want to poke holes for. Any poked holes mean abusers get through as well.

Thank you very much!

---

Hi, Blake,
Could you provide your opinion according to the comment29?
Thanks!

Comment 31

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Alastor Wu [:alwu][please needinfo me][GMT+8] from comment #22)
> This bug is important now, because we would plan to turn on this pref by
> default in the future (maybe 59 or later) and we need to make sure it won't
> break any webRTC websites.

Alastor, is there a bug tracking this? Can we link it here? Would love more information about the overall plan and intent. Is it matching Chrome?

The problem with poking holes is nothing prevents advertisers from using peer connection. 

Are we going to track an ad playing into element.captureStream going into a PeerConnection and out again from another PeerConnection on the same machine in a local loop, which is then being played in a local element? That's going to be hard.

Conversely, are we going to track a real incoming webrtc call whose audio on the receiving end goes through web audio for additional processing before being played?

This is not how we want to boost our peer connection usage numbers. ;-P

Comment 32

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to mirh from comment #20)
> can't you just make the "Share" camera selection button send the play() signal?

That would certainly count as a "user interaction". We could even make skipping the prompt due to "Remember this decision" count as a user interaction. Except these permission prompts are for the *outgoing* audio and video, not the incoming, so it's hardly perfect.

But such an approach might still beat trying to poke holes for some types of tracks, and trying to track sources through the mediastream graph.

Comment 33

[Stephan Sokolow]

Another option would be a modified form of how geolocation is handled, with...

1. An address bar icon that ensures there's always a way to trigger playback for elements where `autoplay` was blocked.
2. A way to specify "show the icon, but start with the popup collapsed away" as the default behaviour so it doesn't become an annoyance once users have learned where to look for it.
3. A new entry under View Page Info > Permissions so that it's possible to have the browser remember the user's choice in a coarse fashion.

Comment 34

[Randell Jesup [:jesup] (needinfo me)]

Note that Chrome has implemented a block on autoplay, which IIRC does not block WebRTC-sourced or getUserMedia-sourced streams.  So far as I know ad sites (and news/etc sites that like to autoplay video *with audio*, like CNN) have not switched to Webrtc, which would be more complex and resource-intensive for them to deal with than streaming video.  

I don't know if Chrome blocks videoelement.captureStream()-sourced streams (ad site could bypass by video1.src = http:...; stream = video1.captureStream(...); video2.srcObject = stream;)  --- I don't know if anyone has done this or if chrome blocks it (easy to test), but I'd be ok with blocking that from autoplay.  Of course, if video1.src is blocked from autoplay, then this trick gets blocked, so it may not be an issue.

Comment 35

[Andreas Pehrson [:pehrsons]]

Chrome seems to also block some stuff in iframes on android; compare https://jsfiddle.net/0z7o1vpw/ to https://fiddle.jshell.net/0z7o1vpw/show. The former doesn't even show that it has an audio track (the volume/mute button) while the latter does.

Comment 36

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

So we don't waste time reinventing things in this bug, who owns the autoplay design, and is there a doc?

Have we talked with other vendors about this, and how did that go? An email thread "Converging around an autoplay behaviour?" from around September appeared to discuss this topic, with links to what Chrome is doing on, like:

  Unified autoplay policy - https://blog.chromium.org/2017/09/unified-autoplay.html
  ...more on autoplay     - https://sites.google.com/a/chromium.org/dev/audio-video/autoplay

  gesture delegation API  - https://github.com/WICG/gesture-delegation/blob/master/explainer.md

There seems a bit of a disconnect between that and what's being proposed here, so I was curious how those discussions went. Would unification here be good for the web?

Comment 37

[Alastor Wu [:alwu]]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #31)
> Alastor, is there a bug tracking this? Can we link it here? Would love more
> information about the overall plan and intent. Is it matching Chrome?

Bug 1376321, here are some related doc [1]

[1]
https://docs.google.com/document/d/1xhXcnIKCUdEzYWHAXPaYkgbyBDvRcKRrLDybqNovwgA/edit#heading=h.ykaoqltr9eni
https://docs.google.com/document/d/1VvJNMasDQu05ayAlkUrW6mkJT4f-JOd2iB75aFRBG0M/edit

Comment 38

[Blake Wu [:bwu][:blakewu]]

Jib, 
We are discussing with product manager and UX for block autoplay story on Firefox. There are many details not locked down yet, like should we do something like Chrome or Apple or both. But two things for sure are a)we need to block autoplay for some conditions and b)we don't want to break WebRTC either. :-) No matter which design could be, the media.autoplay.enabled=false is a fundamental function we need to make it work. 
I am cc-ing product manager, Cindy, to keep her in loop.

Comment 39

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Maire and I just had our 1-1, and we ask that autoplay be left alone in 59.

We have a ton of WebRTC patches landing in 59, and we don't have cycles to deal with this atm. 60 maybe.

Comment 40

[Paul Adenot (:padenot)]

clearing NI.

Comment 41

[Alastor Wu [:alwu]]

We've decided to change the way blocking autoplay, we would align our policy to Chrome's (not all of them), the website would be allowed to autoplay after it has been activated by specific user gestures. It would be implemented in bug1382574.

Therefore, most WebRTC sites should be allowed to autoplay since websites often ask for camera permission after user clicked the button.

Keep this bug open to track whether there are still any broken WebRTC websites after fixing bug1382574. (If needed, we would consider to give the autoplay permission for website once it got the camera or microphone permission)

Comment 42

[Paul Adenot (:padenot)]

(In reply to Alastor Wu [:alwu][please needinfo me][GMT+8] from comment #41) 
> Therefore, most WebRTC sites should be allowed to autoplay since websites
> often ask for camera permission after user clicked the button.

This wouldn't work for unidirectional WebRTC applications, where you're only receiving video and audio, but not sending anything.

Comment 43

[Alastor Wu [:alwu]]

(In reply to Paul Adenot (:padenot) from comment #42)
> This wouldn't work for unidirectional WebRTC applications, where you're only
> receiving video and audio, but not sending anything.

If so, that kinds of websites are equal to normal media playback, it's reasonable that they can only be allowed to playback after getting user interaction.

Comment 44

[Randell Jesup [:jesup] (needinfo me)]

If it matters: Some apps negotiate a one-way reception stream, then later in response to input ("I want to talk") or simply immediately ask for cam/mic and renegotiate, or just continue to use two one-way conduits.  It's conceivable that a site might start with send-only, then later make it two-way or add a receive-only stream.

Also, actual video being received may occur well after the negotiation.

None of this may affect the decisions here; I just wanted to point out some of the less-common usecases here.  An advantage to :alwu's design would be avoiding using webrtc as a way to bypass the restriction in ads.

Comment 45

[feedback]

I just ran into this issue with my web app that does not send or receive any data and only does local video and audio processing https://www.seeingwithsound.com/webvoice.htm With Firefox 58 for Android, live video does not start no matter what buttons I press (so despite user interaction!) if autoplay is disabled in Settings. With autoplay enabled it works fine. This is totally counter-intuitive and it took me a long time to notice this link with the autoplay setting. Thanks.

Comment 46

[Chris Pearce [:cpearce (Not reading bugmail)]]

I am thinking that as well as copying Chrome's user-gesture policy, we should also treat sites which have had camera/microphone permissions granted in a getUserMedia() call as if they also have autoplay permission. This would mean if you open a link to a WebRTC call in a new tab, it should be able to play once cam/mic permission is granted.

The code that checks whether permission has been granted can be found around here:
https://searchfox.org/mozilla-central/rev/a539f8f7fe288d0b4d8d6a32510be20f52bb7a0f/dom/media/MediaManager.cpp#2402

As pointed out above, that won't solve the unidirectional use case, but that's functionally equivalent to the regular playback case.

Comment 47

[Chris Pearce [:cpearce (Not reading bugmail)]]

Calling MediaManager::IsActivelyCapturingOrHasAPermission() tells us whether camera/microphone permission has been granted, so we can use that to achieve comment #46 above. Jib also suggested looking up whether we've stored a deviceId during a navigator. mediaDevices.enumerateDevices() call as an indicator that a successful GUM call has been made in the past. That looks like it's async, so it's easier to just use IsActivelyCapturingOrHasAPermission(). Using IsActivelyCapturingOrHasAPermission() also fixes the problem when loading an appear.in call directly doesn't play any audio.

Comment 48

[Chris Pearce [:cpearce (Not reading bugmail)]]

Attachment: Bug 1182329 - Allow pages which have camera/microphone permission to autoplay audible media.

It seems reasonable to assume that when a page has been granted permission
to capture camera/microphone, the user intends it to play audible media.

Review commit: https://reviewboard.mozilla.org/r/218628/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/218628/

Comment 49

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8949263 [details]
Bug 1182329 - Allow pages which have camera/microphone permission to autoplay audible media.

https://reviewboard.mozilla.org/r/218628/#review224964

::: dom/media/AutoplayPolicy.cpp:33
(Diff revision 1)
> +  MediaManager* manager = MediaManager::Get();
> +  if (manager) {

MediaManager is lazy-loaded, so this would launch MediaManager and its media thread unnecessarily on a lot of browsers where it isn't loaded today.

We should probably use GetIfExists() instead to avoid that (this is the one that can return nullptr, which we're already checking for here).

This leaves an edge-case where a site may have been granted persistent gum permissions, but hasn't accessed navigator.mediaDevices yet. In theory I think it's possible to read these permissions without loading MediaManager. In practice it may not matter, in case we want to increment and file a follow-up bug on this limitation.

Comment 50

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8949263 [details]
Bug 1182329 - Allow pages which have camera/microphone permission to autoplay audible media.

https://reviewboard.mozilla.org/r/218628/#review224972

I meant to r+ in case we're happy with s/Get/GetIfExists/ (for now).

Comment 51

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 8949263 [details]
Bug 1182329 - Allow pages which have camera/microphone permission to autoplay audible media.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/218628/diff/1-2/

Comment 52

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/80c32bd74840
Allow pages which have camera/microphone permission to autoplay audible media. r=jib

Comment 53

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/80c32bd74840