Closed Bug 1577596 Opened 1 year ago Closed 2 months ago

Fenix needs autoplay content permissions to allow granular allow/block of audio or video

Categories

(GeckoView :: General, enhancement, P1)

Unspecified
Android
enhancement

Tracking

(firefox69 wontfix, firefox70 wontfix, firefox71 wontfix, firefox72 wontfix, firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox69 --- wontfix
firefox70 --- wontfix
firefox71 --- wontfix
firefox72 --- wontfix
firefox73 --- fixed

People

(Reporter: colee, Assigned: snorp)

References

Details

(Whiteboard: [geckoview:m1909] [geckoview:m1910][geckoview:m1912])

Attachments

(1 file)

Similar to Firefox desktop, Fenix would like to offer a granular site permission for allowing or blocking audio or video independently on a per-site basis. Our product management marked the autoplay controls as a Must for the quarter. We may be able to implement small parts of it. That ticket is here:

https://github.com/mozilla-mobile/fenix/issues/255

It appears GV today only offers a global runtime flag setAutoplayDefault to either enable or disable autoplay of all media. It is not possible to implement blocking or allowing autoplay with per-site exceptions, blocking only audio autoplay, or blocking only on wifi with per-site exceptions.

When the autoplay stuff was being introduced in Gecko, it used to be something we could enable it per-origin. Then something changed and it didn't look like we could do that anymore. Andreas, Alastor, you two seem to be the most involved -- is there a way for us to set autoplay permission on a per-origin basis?

Flags: needinfo?(apehrson)
Flags: needinfo?(alwu)

I think GV can implement a new API to use nsIPermissionManager to modify sites' autoplay-media permission [1], like what we do on desktop. Then AutoplayPolicy would check document's principle [2] to decide if we allow autoplay or not.

[1] https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/browser/modules/SitePermissions.jsm#695
[2] https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/dom/media/AutoplayPolicy.cpp#69-82

Flags: needinfo?(apehrson)
Flags: needinfo?(alwu)

Adding [geckoview:fenix:m8] whiteboard tag for consideration in GV's September sprint because the Fenix issue is in their Q3 backlog.

https://github.com/mozilla-mobile/fenix/issues/255

Priority: -- → P2
Whiteboard: [geckoview:fenix:m8]

We need to add a new API constant for the autoplay site permission. James recommends we also remove the global autoplay permission from GV.

Adding this bug to GV's September sprint.

Priority: P2 → P1
Whiteboard: [geckoview:fenix:m8] → [geckoview:m1909]

James recommends we also remove the global autoplay permission from GV.

+1 for that. AC has its own mechanism to handle "default rules" for site permissions. Having a global switch makes it just more complicated. :)

Assignee: nobody → rbarker

(In reply to Alastor Wu [:alwu] from comment #2)

I think GV can implement a new API to use nsIPermissionManager to modify sites' autoplay-media permission [1], like what we do on desktop. Then AutoplayPolicy would check document's principle [2] to decide if we allow autoplay or not.

[1] https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/browser/modules/SitePermissions.jsm#695
[2] https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/dom/media/AutoplayPolicy.cpp#69-82

From what I can tell, this will not work with GeckoView. How was this expected to work in GeckoView?

This is how permissions are handled in GeckoView: https://searchfox.org/mozilla-central/source/mobile/android/components/geckoview/GeckoViewPermission.js#266

From what I can tell there is no way to know that the permission is needed so there is no way to prompt for it.

Flags: needinfo?(alwu)

There is nothing to do with promt, you just have to set autoplay-media permission with site's url and correspond state, then Gecko Autoplay policy would check nsIPermissionManager to see if the website user is browsing has autoplay-media permission or not, to decide whether we have to block or allow this site to autoplay. Maybe you can take bug1543812 as a reference?

Flags: needinfo?(alwu)

With other permissions (like for audio/video, geolocation, etc), GeckoView does not store the permission settings in Gecko. Instead, we delegate this to the app. This has worked out well for things that use nsIContentPermissionPrompt, but I guess autoplay doesn't do that. Could we add that? I think it would just prompt once when autoplay media is encountered on a page. Desktop could just ignore that if they don't need it.

Note that I don't believe we would ever show an actual UI prompt. This would just be a mechanism to allow Fenix to provide the permission value on-demand.

Flags: needinfo?(alwu)

I don't know if prompting to user if a good idea, because in our previous shield-study result, it shows that user really don't like to have any prompt. Should we need to discuss with UX about what the best way is to add those sites to white/black list on mobile?

Flags: needinfo?(alwu)

(In reply to Alastor Wu [:alwu] from comment #9)

I don't know if prompting to user if a good idea, because in our previous shield-study result, it shows that user really don't like to have any prompt. Should we need to discuss with UX about what the best way is to add those sites to white/black list on mobile?

In GeckoView, the prompt might not be a pop-up but just something displayed in the URL bar to indicate something was blocked. Right now we have no way to know if something was blocked by auto play or even a way for the user to white list sites in gecko view. The prompt is how we handle interactions like that. The UI policy is up to the embedding app.

(In reply to Randall Barker [:rbarker] from comment #10)

In GeckoView, the prompt might not be a pop-up but just something displayed in the URL bar to indicate something was blocked. Right now we have no way to know if something was blocked by auto play or even a way for the user to white list sites in gecko view. The prompt is how we handle interactions like that. The UI policy is up to the embedding app.

It sounds similar with what we currently have on desktop? A little icon displayed just near the url bar. Because I don't understand what the plan is here, are you going to have something calling nsIContentPermissionPrompt in the chrome JS on Fenix? or using other way?

(In reply to Alastor Wu [:alwu] from comment #11)

(In reply to Randall Barker [:rbarker] from comment #10)

In GeckoView, the prompt might not be a pop-up but just something displayed in the URL bar to indicate something was blocked. Right now we have no way to know if something was blocked by auto play or even a way for the user to white list sites in gecko view. The prompt is how we handle interactions like that. The UI policy is up to the embedding app.

It sounds similar with what we currently have on desktop? A little icon displayed just near the url bar. Because I don't understand what the plan is here, are you going to have something calling nsIContentPermissionPrompt in the chrome JS on Fenix? or using other way?

Fenix would have no knowledge of the underlying implementation. Fenix and other embedding applications such as Firefox Reality interface only with the GeckoView API. So we need to add an API that gives embedding applications flexibility. How we have done that for other permissions such as pop-up, camera, microphone, location, etc is through gecko providing a promise that GeckoView can then give to the embedder to either allow or deny. How that decision is made is up to the embedding application, it could be a pop up or it could be some other process the embedder uses such as a white list.

OK, so I guess you can probably add an new GeckoView API and call nsIPermissionManager API [1] to add url with permission autoplay-media, and then AutoplayPolicy can check if the media, which requests autoplay, is in that list or not.

[1] https://searchfox.org/mozilla-central/rev/7531325c8660cfa61bf71725f83501028178cbb9/netwerk/base/nsIPermissionManager.idl#79-107

Tracking for GV's October sprint

Whiteboard: [geckoview:m1909] → [geckoview:m1910]

Let me conclude what I have understood so far, please correct me if I am wrong.


GV needs two different kinds of blocking autoplay related APIs.

The first one is to notify the embedding application to know if the page is being blocked, in order to allow the embedding application to show an icon in the URL bar.

This is easy to be implemented, because everytime when autoplay media gets blocked, we would send an event [1] to notify Chrome JS in order to show the blocked icon on the URL bars. So GV can listen to this event and then tells the embedding application that blocking autoplay has occurred, either telling them via callback function, or by a specific Java event. (or any other way you like)

[1] https://searchfox.org/mozilla-central/rev/35cc00a25c4471993fdaa5761952bd3afd4f1731/dom/html/HTMLMediaElement.cpp#3865


The second one is to allow the embedding application adding or removing specific URL into the white list or the black list, where autoplay is enabled or disabled by default. Normally, when a site starts requesting a permission for autoplay, the AutoplayPolicy would check nsIPermissionManager to know if the site has this permission or not [2]. However, as GV doesn't store any permission inside, so all permissions should be stored in the side of the embedding application, which means that we are not able to query any permission from nsIPermissionManager. In this situation, we should instead ask the embedding application if the site requesting autoplay has permission or not, which is stored in the embedding application side.

So I suppose this new API would look like other permission-like methods in [3], which would be called whenever Gecko wants to know if this site is in the embedding application's white list or black list. That means, every time those functions being called [4], we would have to communicate between Gecko (C++) and the embedding application (Java).

Now, there are some problems. First, these autoplay APIs are designed as synchronous, we don't want them to be asynchronous. There is also an API document.autoplayPolicy, which would synchronously return the result to tell if the document is allowed to autoplay or not. We have some discussions about whether it should be sync or asyn in [5], and we and Apple believe that it should be a sync API.

Therefore, we would have to synchronously ask the embedding app for the result, which seems not pratical. Because we can't garantee the embedding app would return the result to us immediately, and the cost of communication cross Gecko and the embedding app is not very cheap and I believe that we don't want to have synchronous communication between Gecko and the embedding app if possible.

If what I mentioned above is correct, I would prefer to take autoplay permission as a special case, to store its permission inside GV, instead of storing in the embedding app.

[2] https://searchfox.org/mozilla-central/rev/35cc00a25c4471993fdaa5761952bd3afd4f1731/dom/media/AutoplayPolicy.cpp#69
[3] https://mozilla.github.io/geckoview/consumer/docs/permissions
[4] https://searchfox.org/mozilla-central/rev/35cc00a25c4471993fdaa5761952bd3afd4f1731/dom/media/AutoplayPolicy.h#33-56
[5] https://github.com/w3c/autoplay/issues/6

Flags: needinfo?(snorp)
Flags: needinfo?(rbarker)
Whiteboard: [geckoview:m1910] → [geckoview:m1909] [geckoview:m1910]

The conversation here[1] is interesting, but this comment[2] from cpearce is not accurate when considering GeckoView, which is disappointing. We may have to do disk access, cross processes, prompt, etc. Also, I don't really get why this is a document property instead of something similar to Notification.requestPermission(). This is fundamentally a permission request, is it not? I would like us to change our position on this API if possible.

Right now I don't think permanently storing the permission in Gecko is an option. It breaks encapsulation in a way that we should not tolerate if at all avoidable.

[1] https://github.com/w3c/autoplay/issues/6
[2] https://github.com/w3c/autoplay/issues/6#issuecomment-447174685

Flags: needinfo?(snorp)
Flags: needinfo?(rbarker)

(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #16)

The conversation here[1] is interesting, but this comment[2] from cpearce is not accurate when considering GeckoView, which is disappointing. We may have to do disk access, cross processes, prompt, etc. Also, I don't really get why this is a document property instead of something similar to Notification.requestPermission(). This is fundamentally a permission request, is it not? I would like us to change our position on this API if possible.

I won't say that autoplay is a permission request, because the final decision is still made by user agent, not by user. For in the user-gesture-activation policy, we would check if the document has been activated by user gesture, it's a key rule to decide if we allow autoplay or not. That is why this property has to been in the document.

In addition, its behavior is also very different from other permissions. For example, even if user puts the website on the black list (disable autoplay), the website is still allowed to autoplay after it got activated by user gesture.

One important argument is that, this API represents current status of allowing autoplay, so it should be as direct as it could be. That says if we have a promise, and it would be resolved later. But at the moment the promise is being resolved, the autoplay status might be changed again, so what the user gets is the previous information, not the current one.


Let me explain how media autoplay policy works.

First, we have two different autoplay policy, user-gesture-activation (this is our default policy) and click-to-play (the one we used before, but it still can be used by changing the pref).

The user-gesture-activation policy is to block any autoplay media until the document gets activated by user gestures, such as keyboard press, mouse click or finger touch. That means, it doesn't block media all the time, it's more like to block the first couples of play invocations, before user interacts with the site.

Therefore, if we block autoplay by default, whenever a media is preparing to start, we would first check if the site is in the white list. If so, then we would allow it to start. If it's not in the white list, or it's in the black list, then we would check if the document it belongs to has been activated by user gesture or not. If the document hasn't been activated, then we would block autoplay, otherwise, we would allow it to start.

In these procedure, the most important check of allowing or blocking autoplay is to know if the document gets activated or not. So in this model, it's really not matching the permission model we have for GeckoView. Note that, even we use the same model as other permission, and the embedding app denies the autoplay request from Gecko, the autoplay can still be allowed to start if the document has been activated by user gestures. That means, you spent some time to ask if this site is stored in list, which the embedding app maintains, but the answer from the embedding app would not be the final result of allowing or denying autoplay, because there are still other conditions which we have to consider.

Based on this reason, I think it's not a good API design and would confuse developers who are using GV to handle autoplay.

I still think a possible solution would be having APIs like AllowSiteToAutoplayByDefault(URL) or DisableSiteToAutoplayByDefault(URL) to allow embedding app to add or remove a permission, which stored in Gecko.


However, as I said in comment15, it's easy to notify the embedding app that blocking autoplay has occurred, comparing with adding a finer granular autoplay control for each site.

We can listen this event [1] and then wrap it as a certain API to notify the embedding app to do something, such as showing blocked icon on the URL bar.

[1] https://searchfox.org/mozilla-central/rev/35cc00a25c4471993fdaa5761952bd3afd4f1731/dom/html/HTMLMediaElement.cpp#3865

We talked about this today and decided that it should be ok to add a "prompt" (via nsContentPermissionPrompt or similar) for the first autoplay attempt per origin per session. GV will respond to this and the media will be played if allowed. GV will also insert an EXPIRE_SESSION perm into the permission manager as we do today for other permissions, so future autoplay attempts will use that value.

The only way to allows GV to add or remove a permission, which would expire when the session ends, to Gecko's permission manager is following the GV's permission model, which is to ask embedding app for an approval when the requesting behaviors happen (ex. geo, camera).

Per yesterday discussion, we would like to add a prompt, which won't really promt user, it's just a mechanism to ask embedding app for a permission, as a trigger point in order to make GV add a permission property to Gecko's permission manager, so that we can directly require permission manager to know if the site has permission or not.


However, there are still some problems if we only promt at the first attempt per origin.

(1) The result of autoplay permission are not just allow or deny

From [1], we can see that GV would only add value either allow or deny to permission manager, but when asking embedding app if the site is in the blacklist or whitelist, except it's in one of those lists, the answer might also be not in any list at all. If the site is not in any list, then we should not insert value to permission manager. (keep its value unknown)

That requires a change of the PermissionDelegate.Callback interface, because the interface only has grant() and reject(), and they are not enough to tell us if the site is not allowed to autoplay (in the black list) or the site is not in any list at all.

In addition, autoplay permission also has another special value BLOCKED_ALL [2]. For above issues, it seems that we should extend the original callback interface or have a special interface for autoplay, in order to handle "in the black list v.s. not in any list" and a special value "BLOCKED_ALL"?

[1] https://searchfox.org/mozilla-central/rev/55aa17110091deef24b913d033ccaf58f9c6d337/mobile/android/components/geckoview/GeckoViewPermission.js#266-271

[2] https://searchfox.org/mozilla-central/rev/8a63fc190b39ed6951abb4aef4a56487a43962bc/browser/modules/SitePermissions.jsm#265


(2) How to synchronize the user-custom permission list from Fenix?

If we only prompt at the first attemp, and the permission, which GV inserts to permission manager, is session-based. That means if user modify the black/white list later, we couldn't know Fenix has changed the list in this session, unless user restarts Fenix.

For example, Fenix adds Youtube into its black list, so when we first prompt to Fenix, GV would add Youtube to Gecko's black list, and then we would check that value directly, instead of prompting again. If user changes their mind, removing Youtube from black list, and add it to the white list, Gecko is not able to know that, which results in using wrong permission result in the remaining session.

As we don't have any API directly update permission from GV, so even if Fenix update its list, Gecko doesn't know the permission for some sites has been changed, unless we would like to introduce new API to notify Gecko that we should try to prompt to GV again, in order to get the latest infomation.

I think if we would really want to have a list for managing specifc permission, we should have another method to directly control permission in Gecko, not relying the present mechanism all the time. Because having a list means that, we have already decide how to handle these sites for certain permission, so we could update all of them to Gecko whenever we starts GV. It's like an initialization and those permissions could still be expired when the session ends.


(3) How to notify embedding app that blocking autoplay has occurred?

As I said before, the final blocking decision is made in Gecko. Even if the embedding app denies to start autoplay (maybe the site is in the black list), the site could still be allowed to autoplay if the site has been activated by user gesture.

That means we couldn't use current mechanism to know if the site is finally blocked or not, we have to have new API to notify embedding app that blocking occurs, like what we do for desktop, dispatching an event [3].

[3] https://searchfox.org/mozilla-central/rev/35cc00a25c4471993fdaa5761952bd3afd4f1731/dom/html/HTMLMediaElement.cpp#3865

Flags: needinfo?(snorp)
Flags: needinfo?(rbarker)

(In reply to Alastor Wu [:alwu] from comment #19)

The only way to allows GV to add or remove a permission, which would expire when the session ends, to Gecko's permission manager is following the GV's permission model, which is to ask embedding app for an approval when the requesting behaviors happen (ex. geo, camera).

Per yesterday discussion, we would like to add a prompt, which won't really promt user, it's just a mechanism to ask embedding app for a permission, as a trigger point in order to make GV add a permission property to Gecko's permission manager, so that we can directly require permission manager to know if the site has permission or not.


However, there are still some problems if we only promt at the first attempt per origin.

(1) The result of autoplay permission are not just allow or deny

From [1], we can see that GV would only add value either allow or deny to permission manager, but when asking embedding app if the site is in the blacklist or whitelist, except it's in one of those lists, the answer might also be not in any list at all. If the site is not in any list, then we should not insert value to permission manager. (keep its value unknown)

AFAICT, the third option of 'unknown' results in some default policy being applied. This can be decided by the app itself, so it will never need to reply with this, only 'allow' or 'deny'.

(2) How to synchronize the user-custom permission list from Fenix?

If we only prompt at the first attemp, and the permission, which GV inserts to permission manager, is session-based. That means if user modify the black/white list later, we couldn't know Fenix has changed the list in this session, unless user restarts Fenix.

We can just prompt on every autoplay attempt, right?

(3) How to notify embedding app that blocking autoplay has occurred?

As I said before, the final blocking decision is made in Gecko. Even if the embedding app denies to start autoplay (maybe the site is in the black list), the site could still be allowed to autoplay if the site has been activated by user gesture.

If we're going to disregard the result of the prompt then we shouldn't do it at all, as in the case of a user gesture.

That means we couldn't use current mechanism to know if the site is finally blocked or not, we have to have new API to notify embedding app that blocking occurs, like what we do for desktop, dispatching an event [3].

[3] https://searchfox.org/mozilla-central/rev/35cc00a25c4471993fdaa5761952bd3afd4f1731/dom/html/HTMLMediaElement.cpp#3865

We could just listen to that event and propagate it to the app if we think it's necessary, but I think it'll be good to avoid it altogether. If we only block when the app respond with block, then we don't need this at all.

Flags: needinfo?(snorp)

(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #20)

AFAICT, the third option of 'unknown' results in some default policy being applied. This can be decided by the app itself, so it will never need to reply with this, only 'allow' or 'deny'.

I still don’t really understand how GV insert unknown to permission manager, because we only have allow() and deny()two options which the embedding app can do.

We can just prompt on every autoplay attempt, right?

Prompting to GV everytime when gecko calling IsAllowedToPlay() is what we would like to avoid, because it would definitely slow down the process. In addition, IsAllowedToPlay() is not only called when media starts playing, it’s a method which can be called anywhere and anytime, we also use it to check or assert some certain conditions. In those situations, we don't want to get the result asynchronously.

If we're going to disregard the result of the prompt then we shouldn't do it at all, as in the case of a user gesture.

Yes, that’s the key! The result of the prompt CAN NOT present the final decision of autoplay, because prompting to embedding app is just asking if the site in the black/white list or not, it’s just one factor of the decision process of blocking autoplay. We have to consider user activation as well.

We could just listen to that event and propagate it to the app if we think it's necessary, but I think it'll be good to avoid it altogether. If we only block when the app respond with block, then we don't need this at all.

As I mentioned above, the app is not able to know the final blocking result even if the app deny the prompt. So we have to add another API to notify the app, which is not elegant but seems unfortunately necessary.

Don't roll over to GV's November sprint, but keep as a high priority P2.

Priority: P1 → P2
Rank: 1

After discussed with Nils today, now we have another appoarch which could fit in GV's mechanism a little bit more and solve some problems I mentioned in comment19.

What GV team would like to do is (1) allow the embedding app to decide the final autoplay decision (2) show the blocked icon when the embedding app denies the request.

What Media team would like to do is "to avoid unneccesary asynchronous communication betweens process", which would slow down the play invocation and affect our sync API.

Therefore, the new approach (still like a workaround) is to "when we create media element and we are 100% sure it can't be allowed to autoplay by checking its property (ex. is element audible) and document activation first, then we notify GV and make the embedding app to decide the final decision".

So it could decouple the request process from the real timing when media element starts to play, we ask GV beforehead in order to get the reply way earlier than the time media prepares to start.


Then, let's see the problems I mentioned in comment19.

(1) The result of autoplay permission are not just allow or deny

From the snorp's comment in comment20, I assume that if the embedding app doesn't reply to the request, then the GV won't do anything, so the permission status for that site would be unknown? That would be equal to "the site is not in any list". Please help me clarify this point because I still don't really understand that.

In addition, as we start asking GV way earlier before the real play occurs, so it should have enough time to get the decision from the embedding app.

(2) How to synchronize the user-custom permission list from Fenix?

This issue is still unsolved by this method, but I think we can solve it from another direction, which is to request user restarts the Fenix once they changes the white/blacklist.

(3) How to notify embedding app that blocking autoplay has occurred?

As we could change our internal logic, only sending request to the embedding app when we 100% sure the media can't be play after checking the media's property and document's user activation, so that the embedding app can decide the final decision, which make the app can show the blocked icon when it denies the request.


Please feel free to give me any suggestion or ask me question if I was not clear at all.
Thank you.

Flags: needinfo?(snorp)

(In reply to Alastor Wu [:alwu] from comment #23)

After discussed with Nils today, now we have another appoarch which could fit in GV's mechanism a little bit more and solve some problems I mentioned in comment19.

What GV team would like to do is (1) allow the embedding app to decide the final autoplay decision (2) show the blocked icon when the embedding app denies the request.

What Media team would like to do is "to avoid unneccesary asynchronous communication betweens process", which would slow down the play invocation and affect our sync API.

Therefore, the new approach (still like a workaround) is to "when we create media element and we are 100% sure it can't be allowed to autoplay by checking its property (ex. is element audible) and document activation first, then we notify GV and make the embedding app to decide the final decision".

So it could decouple the request process from the real timing when media element starts to play, we ask GV beforehead in order to get the reply way earlier than the time media prepares to start.

All of that sounds fine to me.

Then, let's see the problems I mentioned in comment19.

(1) The result of autoplay permission are not just allow or deny

From the snorp's comment in comment20, I assume that if the embedding app doesn't reply to the request, then the GV won't do anything, so the permission status for that site would be unknown? That would be equal to "the site is not in any list". Please help me clarify this point because I still don't really understand that.

Yes, we would just decide on some default behavior (probably block?) if the app isn't handling this "prompt".

In addition, as we start asking GV way earlier before the real play occurs, so it should have enough time to get the decision from the embedding app.

(2) How to synchronize the user-custom permission list from Fenix?

This issue is still unsolved by this method, but I think we can solve it from another direction, which is to request user restarts the Fenix once they changes the white/blacklist.

Now I'm confused again. Can't we prompt on each page load if there are autoplay elements? If the user changes their whitelist settings, a simple reload would get the new changes.

(3) How to notify embedding app that blocking autoplay has occurred?

As we could change our internal logic, only sending request to the embedding app when we 100% sure the media can't be play after checking the media's property and document's user activation, so that the embedding app can decide the final decision, which make the app can show the blocked icon when it denies the request.

Very good.

Flags: needinfo?(snorp) → needinfo?(alwu)

(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #24)

(2) How to synchronize the user-custom permission list from Fenix?

This issue is still unsolved by this method, but I think we can solve it from another direction, which is to request user restarts the Fenix once they changes the white/blacklist.

Now I'm confused again. Can't we prompt on each page load if there are autoplay elements? If the user changes their whitelist settings, a simple reload would get the new changes.

Yes, but we would still get a cases where users change their setting during one page load.

Ex.

  1. add site A to blacklist
  2. open site A
  3. site A reqeust play, and it would be rejected by the embedding app, and then we cache this result
  4. remove site A from blakclist, and add it to the whitelist
  5. site A reqeust play again, as we are still using cached value, so site A fails to start (but it should be allowed)

(3) How to notify embedding app that blocking autoplay has occurred?

As we could change our internal logic, only sending request to the embedding app when we 100% sure the media can't be play after checking the media's property and document's user activation, so that the embedding app can decide the final decision, which make the app can show the blocked icon when it denies the request.

Very good.

But notice that, if we do so, the timing showing the blocked icon would be different from the real timing website requests play. However, from the user perspective, it should probably be fine, because "having a media element but doesn't request play" is basically equal to "blocking a media elementw with play requst", because both of them are still being paused. So I think user maybe won't feel too weird for showing a blocked icon too early?

Flags: needinfo?(alwu)
Flags: needinfo?(rbarker)

Note that the Fenix team wants to block both muted and unmuted videos. They want to prevent autoplay of those annoying floating videos on news websites (because they are annoying and waste bandwidth and battery on mobile):

https://github.com/mozilla-mobile/fenix/issues/5636

Randall says Gecko's media.autoplay.default pref level 5 (nsIAutoplay::BLOCKED_ALL) might do what we need here, but Gecko would need to provide a proper API for GV to expose to Fenix.

The level is a pref value so some one would need to set the pref to 5 to test it. We already are setting the pref in GeckoRuntimeSettings, we just don't have a flag for level 5.

Depends on: 1593843
See Also: → 1592004

Because of bug 1592004, now I'm not quite sure we would still want to follow the way we mentioned in comment23, because it might work for blocking autoplay case, but it doesn't work for audio focus case.

I'll discuss with Nils again to see if what is the way we can both accept. Now I'm thinking about maybe we should just ask GV for a approval every time and see how it's going to be. Maybe user won't feel any difference about the delaying caused by IPC round trip...?

Now my plan is to implement a similar plan proposed by GV team before, which is to send a permission request to the app at the first time when (1) calling media.play() and (2) media with attribute autoplay has enough data and ready to start playing. But if the document has been activated, we would simply allow all autoplay, so we won't send any request.

The reason is because if we use the way in comment23, then we can't know if the media in the page is audible or not, which is also a factor of deciding blocking autoplay result, because we can decide to block audible media only or to block both audible and inaudible media. Because of that, it seems that we have to send the audible state to the embedding app as well?

So I would like to know " How would Fenix handle the permission request from autoplay? "

If we have to send an additional parameter to indicate if media is audible or not, which seems not happening on current mechanism, because we usually send only permission name to the app. Therefore, either it requires a change of API, or we have to hack on the name of permission in order to allow it reflect the state (ex. autoplay-audible, autoplay-inaudible). If we don't do so, then when the request happen, Fenix is not able to know if they would like to show the blocked icon or not.

Ex. the site A has permission block (block audible media only), and then a muted media starts, so we send a request. How does Fenix decide if they should show the blocked icon in this situation?


In addition, we can ignore the problem in bug 1592004 for now, because I think it can use another way to solve, we don't have to mix them together.

Flags: needinfo?(snorp)
Flags: needinfo?(rbarker)

You can probably just add an isAudible or similar to nsIContentPermissionRequest[1], and then we can pass that along in GV. As you said, this would allow Fenix to make the decision to allow inaudible content or to block all autoplay media. How Fenix decides that is not super important, IMO. I think you need to consider that a response to the permission request may take an arbitrary amount of time, though. Fenix may choose to put a very subtle UI up (like a badge) which will allow the user to accept the autoplay request. This would resolve the GeckoResult and in turn respond to the permission request. HTMLMediaElement would begin playback at that point.

[1] https://searchfox.org/mozilla-central/source/dom/interfaces/base/nsIContentPermissionPrompt.idl#65

Flags: needinfo?(snorp)
Flags: needinfo?(rbarker)

Update my progress here, I'm working on bug1593843, which would dispatch a permission request to GV. It's almost ready for reviewing, but still needs some testing. Once it finishes, GV team can start implementing the handling logic in GV and Fennix.

Update progress again, now I have started a review process for bug1593843.

James, how would an app provide settings like useTrackingProtection or userAgentMode if these were per-site settings? These settings need to be known at document load time.

I assume GeckoSessionSettings are not per-site?

Providing the autoplay settings also on document load would be more suitable here, because the current status of specifications is that a blocked request to play is reported synchronously, and there is no attempt by the web engine to play the same media when the setting changes. (Imagine attempts to play several sounds being blocked and then all sounds being played at once on user activation or whatever.) With an async permission request, at least the first attempt to play is likely to fail regardless of the response from the app.

Flags: needinfo?(snorp)

(In reply to Karl Tomlinson (:karlt) from comment #33)

James, how would an app provide settings like useTrackingProtection or userAgentMode if these were per-site settings? These settings need to be known at document load time.

We don't currently have any mechanism for per-site settings.

I assume GeckoSessionSettings are not per-site?

Correct, they are essentially per-tab settings.

Providing the autoplay settings also on document load would be more suitable here, because the current status of specifications is that a blocked request to play is reported synchronously, and there is no attempt by the web engine to play the same media when the setting changes. (Imagine attempts to play several sounds being blocked and then all sounds being played at once on user activation or whatever.)

Yeah, it may be worthwhile to think about what per-origin settings for some things would look like. It's still not going to be synchronous, though, so I am not sure it will solve the problems you're describing.

With an async permission request, at least the first attempt to play is likely to fail regardless of the response from the app.

Not fail, but perhaps delayed by 100ms or so? If we ask Fenix and it says "yup go ahead" without prompting the user, I don't think that should fail.

Flags: needinfo?(snorp)

First version document for the usage in GeckoView

After landing bug 1593843, now GeckoView can receive the permission request from Gecko. First, remember to turn on the pref media.geckoview.autoplay.request.

Then when a page first creates media element, we would dispatch two request with different permission names, autoplay-media-audible and autoplay-media-inaudible, to ask for the permission for both types. One page would only has one audible request and one inaudible request.

The reason we decide to dispatch the request earlier, not dispatch the request when media start, is that
(1) If we dispatch the request when media.play() is being called, then return a pending play promise to the page
=> it violates the spec requirement, see part 1 in [1]

(2) if we dispatch the request when media.play() is being called, then return a rejected play promise because we haven't got the response from Fenix
=> if the page only calls play() once, then even if we get the allowed response, the page is still not going to start.
Although this issue might also happen on current mechanism when the page immediately call media.play() after creating it. But we assume most site would do some processing before calling play(), which might take some time and give us a chance to receive the response from Fenix.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1593843#c18


As the implementation of Gecko side is separate from the implementation of GeckoView and Fenix, so I think we can start with this implementation, and if you guys think current implementation can't fit your requirement, then we can discuss how to modify it.

Thank you.

See Also: → 1587522
See Also: 1587522

Hey snorp, it looks like the Gecko pieces are done, but is there still a dependency on bug 1587522? Or would Fenix just waiting on GV to expose this media API?

Flags: needinfo?(snorp)
Whiteboard: [geckoview:m1909] [geckoview:m1910] → [geckoview:m1909] [geckoview:m1910][geckoview:m1912]
Flags: needinfo?(snorp)
Priority: P2 → P1
Assignee: rbarker → snorp
Pushed by jwillcox@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1a9be6d049a7
Expose autoplay permission requests in GeckoView r=geckoview-reviewers,rbarker,agi
Pushed by jwillcox@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/73bdaad89840
Expose autoplay permission requests in GeckoView r=geckoview-reviewers,rbarker,agi
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
You need to log in before you can comment on or make changes to this bug.