1623715 - Support platform level MediaControl and MediaSession API implementation

Status: RESOLVED FIXED

(GeckoView :: Media, task, P1)

Description

[Alastor Wu [:alwu]]

Currently, Fenix is using their own media component [1] to implement the media control, but that component can't suppport MediaSession API [2].

We have already implemented the platform-level media control feature on MacOS, Windows and Linux that is also supporting MediaSession API. Using that API means that websites can customize the metadata that would be shown on the virtual control interface and implement the calllback functions when users press media control keys, such as play/pause, next track and previous track.

We need Fenix to define some API to allow us set the metadata on the virtual interface and handle the task of receving/notifying media control keys, then we can implement something like [3] to enable our platform-level media control on Fenix.

[1] https://github.com/mozilla-mobile/android-components/tree/1795ba83459b6ff2f4cd82a547c5cad5bb98ba1e/components/feature/media/src/main/java/mozilla/components/feature/media
[2] https://w3c.github.io/mediasession/
[3] Platform-specific implementation for media control
MacOS
https://searchfox.org/mozilla-central/source/widget/cocoa/MediaHardwareKeysEventSourceMacMediaCenter.h
Linux
https://searchfox.org/mozilla-central/source/widget/gtk/MPRISServiceHandler.h
Windows
https://searchfox.org/mozilla-central/source/widget/windows/WindowsSMTCProvider.h

Comment 1

[Alastor Wu [:alwu]]

Hi, Sebastian,
I wonder if GeckoView team has any interest or resource in this topic?
Thank you.

Comment 2

[Sebastian Kaspari (:sebastian; :pocmo)]

From AC side I'd definitely be interested in working on a new / better media API - there are multiple issues/bugs that we could address. Flagging emily and snorp to see what they think.

Comment 3

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

Do we want to move to move this bug and discussion to github.com/mozilla-mobile/android-components ?

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

Yeah, we could do something different here. I guess it depends on how bad we want the DOM API and how it helps the bugs Sebastian is referring to.

Comment 5

[Emily Kager [:ekager]]

Would like to discuss with product after getting some answers here because fixing Media issues is one of the top Fenix priorities.
Sebastian, could we list off some of the bugs we think this would help resolve to get an idea of how high priority this should be?
Snorp, do we have any idea of the GV effort that would be required here?

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

Agi has already started looking at this, but I'm not sure it will solve the problems you're referring to. This bug is about a new DOM API which allows Web content to expose an API allowing external control. We might be able to solve some existing problems with the MediaElement GeckoView API by emulating this new DOM API, but it's not clear to me yet.

Comment 7

[[ex-Mozilla] Agi Sferro | :agi]

It will solve some bugs since the code paths are different (e.g. the notification for soundcloud.com would work), I would like to avoid investing a lot in MediaElement so if any of the bugfixes require big changes I would rather implement the Media Session API and make sure the case works there. I guess I would need to see the list of bugs to know whether this can help or not.

Comment 8

[Sebastian Kaspari (:sebastian; :pocmo)]

Agi, Alastor: Does it make sense to have a very brief meeting this week to discuss the API and what issues we are facing in AC with the current approaches and whether we can create something that fits more the needs we have? The TL;DR is that we jump through a lot of hoops trying to figure out which tab is actively playing (or paused) and whether it is worth exposing controls for it, requesting media focus or showing a media notification. Some of those things desktop already does (better than us) and I would love to just re-use that if we can find a way to expose this nicely as a GeckoView API.

Comment 9

[[ex-Mozilla] Agi Sferro | :agi]

I think it makes sense to meet! I've been looking at simplifying that API to make your job simpler and fix some of the problems you mention, here's the draft doc: https://docs.google.com/document/d/1cv7dhLvf7HYr34qNP7SYBNCARBZ2HFbegRsK4PGZHZQ/edit if you can look at it and tell me what you think I'd really appreciate it!

Comment 10

[[ex-Mozilla] Agi Sferro | :agi]

alwu: could you or someone on your team look over the doc in Comment 9 too?

Comment 11

[Alastor Wu [:alwu]]

Sure, feel free to send a meeting invitation to me!
In addition, I've left some comment on the doc in the comment9. Thank for documenting that!

Comment 12

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [1.8] Add MediaSession API for (DOM) media session control delegation.

Comment 13

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [2.4] Add MediaSession API tests.

Depends on D84189

Comment 14

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [3.2] Update changelog.

Depends on D84190

Comment 15

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [4.1] Fix LinkedList::length() for a const context.

Depends on D84405

Comment 16

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [5.1] Add missing media control API for GV implementation.

Depends on D84543

Comment 17

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [6.0] Add prefs to control media controller deactivation.

Depends on D86043

Comment 18

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [7.2] Extract media utility functions.

Depends on D86043

Comment 19

[Eugen Sawin [:esawin]]

Attachment: Bug 1623715 - [8.2] Move media fullscreen event to JS and extend its metadata.

Depends on D86349

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd585490e79b
[1.7] Add MediaSession API for (DOM) media session control delegation. r=snorp,alwu,geckoview-reviewers,agi
https://hg.mozilla.org/integration/autoland/rev/ddff358f800e
[2.4] Add MediaSession API tests. r=geckoview-reviewers,droeh,snorp
https://hg.mozilla.org/integration/autoland/rev/6e7ce9e5f89b
[3.2] Update changelog. r=geckoview-reviewers,droeh
https://hg.mozilla.org/integration/autoland/rev/47f3eb481909
[4.1] Fix LinkedList::length() for a const context. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/4887dea37231
[5.1] Add missing media control API for GV implementation. r=alwu
https://hg.mozilla.org/integration/autoland/rev/1b7c3fb0da3b
[7.2] Extract media utility functions. r=geckoview-reviewers,owlish
https://hg.mozilla.org/integration/autoland/rev/c8ca1d1866e7
[8.2] Move media fullscreen event to JS and extend its metadata. r=geckoview-reviewers,snorp,alwu

Comment 21

[Dorel Luca [:dluca]]

Backed out 7 changesets (bug 1623715) for Linting failure and android build bustage. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313260118&repo=autoland&lineNumber=832

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313260118&repo=autoland&lineNumber=832

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=c8ca1d1866e7e3591d2df84c2a4f0204d43386ed

Backout:
https://hg.mozilla.org/integration/autoland/rev/ed9c0db48e518e144ea8134ee500cc0dcea62433

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4ff55894774d
[1.8] Add MediaSession API for (DOM) media session control delegation. r=snorp,alwu,geckoview-reviewers,agi
https://hg.mozilla.org/integration/autoland/rev/a118a4d72bef
[2.4] Add MediaSession API tests. r=geckoview-reviewers,droeh,snorp
https://hg.mozilla.org/integration/autoland/rev/898972ec680c
[3.2] Update changelog. r=geckoview-reviewers,droeh
https://hg.mozilla.org/integration/autoland/rev/4361ef84e780
[4.1] Fix LinkedList::length() for a const context. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/579df1a653ee
[5.1] Add missing media control API for GV implementation. r=alwu
https://hg.mozilla.org/integration/autoland/rev/2e7e298cfe27
[7.2] Extract media utility functions. r=geckoview-reviewers,owlish
https://hg.mozilla.org/integration/autoland/rev/9704afdeefb1
[8.2] Move media fullscreen event to JS and extend its metadata. r=geckoview-reviewers,snorp,alwu

Comment 23

[Bob Clary [:bc] (inactive)]

I just pulled and hit this build error from this bug I think.

6:42.98 In file included from Unified_cpp_widget_android1.cpp:65:
 6:42.98 /home/bclary/mozilla/builds/autoland/mozilla/widget/android/nsWindow.cpp:582:41: error: no member named 'GetState' in
 'mozilla::dom::MediaController'
 6:42.98       PlaybackChanged(mMediaController->GetState());
 6:42.98                       ~~~~~~~~~~~~~~~~  ^
 6:42.98 1 error generated.

Comment 24

[Razvan Maries]

Backed out for build bustages.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=VqROqQ-pQ7SI-FZs6AisXg.0&resultStatus=testfailed%2Cbusted%2Cexception&revision=9704afdeefb196923a8e54e1a4fb03392c12cc76

Log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313331452&repo=autoland&lineNumber=47145

Backout: https://hg.mozilla.org/integration/autoland/rev/5007c1c9c60c0b0d93ad11eedc89e629c4fab91c

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3b946d54d6db
[1.8] Add MediaSession API for (DOM) media session control delegation. r=snorp,alwu,geckoview-reviewers,agi
https://hg.mozilla.org/integration/autoland/rev/feecebd57035
[2.4] Add MediaSession API tests. r=geckoview-reviewers,droeh,snorp
https://hg.mozilla.org/integration/autoland/rev/8f3b66e90a27
[3.2] Update changelog. r=geckoview-reviewers,droeh
https://hg.mozilla.org/integration/autoland/rev/9fad7d603355
[4.1] Fix LinkedList::length() for a const context. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/36f1dd6d86c0
[5.1] Add missing media control API for GV implementation. r=alwu
https://hg.mozilla.org/integration/autoland/rev/971c63a075e1
[7.2] Extract media utility functions. r=geckoview-reviewers,owlish
https://hg.mozilla.org/integration/autoland/rev/e61166c9b808
[8.2] Move media fullscreen event to JS and extend its metadata. r=geckoview-reviewers,snorp,alwu

Comment 26

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/3b946d54d6db
https://hg.mozilla.org/mozilla-central/rev/feecebd57035
https://hg.mozilla.org/mozilla-central/rev/8f3b66e90a27
https://hg.mozilla.org/mozilla-central/rev/9fad7d603355
https://hg.mozilla.org/mozilla-central/rev/36f1dd6d86c0
https://hg.mozilla.org/mozilla-central/rev/971c63a075e1
https://hg.mozilla.org/mozilla-central/rev/e61166c9b808

Comment 27

[Chris Peterson [:cpeterson]]

Moving some media bugs to the new GeckoView::Media component.