1353652 - [MediaControl-Linux] Provide MPRIS API for music/video streaming web sites

Status: RESOLVED FIXED

(Core :: Widget: Gtk, enhancement, P3)

Description

[sedrubal]

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20170327113357




Expected results:

There should be an API for music and video streaming web sites like soundcloud/spotify/youtube/... to interact with the systems media player API like MPRIS.
So users are able to see the current playing track/video and control playback (pause, play, stop, forward, backward, ...) like they're used to do directly in their system.

Comment 1

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Do you mean hooking up https://developer.mozilla.org/en-US/docs/Web/API/Media_Session_API to MPRIS?

Comment 2

[sedrubal]

Yes. I didn't know that there is such an API but it looks  like it's the right place.

Comment 3

[sedrubal]

I found this addon for KDE Plasma called "Plasma Integration" (https://addons.mozilla.org/firefox/addon/plasma-integration/). I somehow detects the name of the media, firefox currently plays, and integrates it into the system.

Comment 4

[espectalll]

Should this be considered a duplicate or a dependency of #448910?

Comment 5

[sedrubal]

The other bug only covers media keys on keyboards. My intention was to integrate audio and video playback into the system in order that Firefox acts like a media player. Then other applications can for example pause playback when a phone call comes in or display the current track on SoundCloud or Spotify...

Comment 6

[espectalll]

You're right, I just thought they were directly related as MPRIS can provide multimedia keyboard support from the desktop environment

Comment 7

[Marc Streckfuß [:MeFisto94]]

I will try to work on implementing the native side of MPRIS so it can be combined with the MediaSession API when done.

Comment 8

[Alastor Wu [:alwu]]

FYI, as I've implemented a MediaController interface in bug1565689, we can use it to play/pause media in firefox if we would like to control media from the system level panel.

Comment 9

[Marc Streckfuß [:MeFisto94]]

Attachment: MPRISServiceHandler.h first draft

Comment 10

[Marc Streckfuß [:MeFisto94]]

Attachment: MPRISServiceHandler.cpp first draft

Comment 11

[Marc Streckfuß [:MeFisto94]]

So this is my first approach for a Skeleton Class to provide MPRIS Support.

Before going into the implementational details: I failed at extracting the class into it's own cpp file, for some reason other files break and namespaces get confused. So could someone maybe have a quick glance at what I did wrong, so I can compile and test everything again?

I added the file simply to the UNIFIED_SOURCES in alphabetical order:

    'mozcontainer.cpp',
    'MPRISServiceHandler.cpp',
    'NativeKeyBindings.cpp',

The file sometimes had the namespace ::mozilla::widget::mozila::widget amongst other things, while working fine as part of WindowServiceProvider. I am sorry for that inconvenience.

The following problems need consideration:

1. Multithreading. The Run() method will block the calling thread and all received events will be delivered from that Thread. This means that method calls/Events like Stop() and Seek() will come from the D-Bus Thread and changing Properties like the Metadata coming from a different thread has to be synchronized.
2. Note: The Parts guarded with defines are not yet implemented but are also optional
3. https://specifications.freedesktop.org/mpris-spec/2.2/Track_List_Interface.html and https://specifications.freedesktop.org/mpris-spec/2.2/Playlists_Interface.html haven't been implemented, they are optional as well and only rarely make sense for Firefox, especially given that the MediaSession API is (not yet) advanced enough.
4. The API is not generic enough in two ways:
   4.1 struct MPRISMetadata is difficult: I have to pass the Metadata as array of GVariant *, because the length of a track and the cd-number might well be an integer, where the artist certainly is a string. There are too much possible tags available, but:
   4.2 The whole API might need to be adjusted to what can be supported with the Media Session API and maybe what other platforms (Mac OS and Windows) offer. Ideally the OS-dependant Implementation can be switched out without the caller knowing. This would however mean we indeed need to translate metadata. If we wanted to support all media playing (like directly playing mp3 files) though, then we would limit ourselves too much when only allowing what the Media Session API specifies.
5. Binding of the Code to other parts: Obviously one wants to register Handlers for all the Events and Properties.
6. Feel free to try the code already. All that you need to do is get the class to compile again and invoke Initialize() and Run(), then you will see Mozilla Firefox Nightly appearing as Media Player in your system tray with some dummy values filled in and some printf() when you trigger events.

I think that is all for now, I'd love to hear some feedback

Comment 12

[Marc Streckfuß [:MeFisto94]]

Attachment: Bug 1353652 - Initial Draft of MPRIS API Provider (Media API on Linux)

Comment 14

[Alastor Wu [:alwu]]

Hi, Martin,

I heard that you might be a best person to ask for the question about gnome, could you give us some suggestion on [1]?
Thank you!

[1] https://phabricator.services.mozilla.com/D47999#1655569

Comment 15

[Martin Stránský [:stransky] (ni? me)]

I forwarded the question to Jean Felder. But it looks to me that pure MPRIS is a way to go, see this recent gnome-shell extension for instance:
https://github.com/JasonLG1979/gnome-shell-extension-mpris-indicator-button

and how it uses the MPRIS:
https://github.com/JasonLG1979/gnome-shell-extension-mpris-indicator-button/blob/master/mprisindicatorbutton%40JasonLG1979.github.io/dbus.js

Comment 16

[Marc Streckfuß [:MeFisto94]]

Attachment: Bug 1353652 - Fix unified builds

Comment 17

[Martin Stránský [:stransky] (ni? me)]

(In reply to Martin Stránský [:stransky] from comment #15)

I forwarded the question to Jean Felder. But it looks to me that pure MPRIS is a way to go, see this recent gnome-shell extension for instance:
https://github.com/JasonLG1979/gnome-shell-extension-mpris-indicator-button

Guys,

there's a reply from Jean Felder. From my understanding it's OK to use only MPRIS and
don't care about GNOME media keys.

---

Regarding GNOME Media keys and MPRIS:

GNOME settings daemon is able to detect the media key events and send this
information to the registered applications with its own api [1].
However, it also acts as an MPRIS controller. So, if an application has
MPRIS support it will use MPRIS instead of the media-keys API.
See the relevant part of the code [2] and the initial bug report [3].

To sum it up:

• (if) you don't have MPRIS support in you application and you want to handle
  the media keys: use the media-keys-API [1]
• (if) you have MPRIS support in your application: you don't need the media-keys
  API. The media keys events are handled by MPRIS (via the MediaPlayer.Player
  interface [4]).

[1]
https://gitlab.gnome.org/GNOME/gnome-settings-daemon/blob/master/plugins/media-keys/README.media-keys-API
[2]
https://gitlab.gnome.org/GNOME/gnome-settings-daemon/blob/master/plugins/media-keys/gsd-media-keys-manager.c#L1845
[3] https://bugzilla.gnome.org/show_bug.cgi?id=697810
[4]
https://specifications.freedesktop.org/mpris-spec/2.2/Player_Interface.html

---

Comment 18

[Karl Tomlinson (:karlt)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3d6293ddb5764064dd5fa71bd7c3ebd09901050d

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e167d7827094
Fix unified builds r=karlt

Comment 20

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/e167d7827094

Comment 21

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #24249 (as of Wed, 04 Dec 2019 11:10:27 GMT) ==

Improvements:

24% build times linux64 tsan taskcluster-m5.4xlarge tsan-fuzzing 1,475.89 -> 1,128.68

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24249

Comment 22

[peter]

I'm not familiar with Bugzilla or Phabricator but it looks like the suggested code set the mpris:artUrl to the icon of Firefox?

Should this be set to the artwork defined in the Media Session API? Also see the spec:
https://www.freedesktop.org/wiki/Specifications/mpris-spec/metadata/#mpris:arturl

This is what other clients on Linux like Spotify or Auryo do.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b4dc366c9161
Initial Draft of MPRIS API Provider (Media API on Linux) r=alwu

Comment 24

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/b4dc366c9161

Comment 25

[Cristina Coroiu [:ccoroiu]]

Backed out changeset b4dc366c9161 (bug 1353652) for build bustage on linux32 at build/src/widget/gtk/MPRISServiceHandler.cpp

Backout: https://hg.mozilla.org/mozilla-central/rev/f314754b7e11062f78e3f55e80e061ff9db7a326

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=b4dc366c916164498a62980e867c0cf5647e6d41

Merge push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=e192008ebc8080df7507b836c0ec8358cdf260f5

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=288489921&repo=mozilla-central&lineNumber=29794

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e7e12a4881b8
Initial Draft of MPRIS API Provider (Media API on Linux) r=alwu

Comment 27

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/e7e12a4881b8

Comment 29

[Ilya K]

This doesn't actually use the metadata from navigator.mediaSession, does it? Is this planned to be implemented?

Comment 30

[Marc Streckfuß [:MeFisto94]]

See the tracking bug here: https://bugzilla.mozilla.org/show_bug.cgi?id=1572869
First we will add the Favicon and Tab Title as Metadata and then we will have support for Media Session, but currently there is no Metadata at all.
We do it step by step and also want to support sites not using MediaSession API anyway :)

Comment 31

[Ilya K]

I see. Thanks a lot, going to follow the tracking bug. Also not sure if this should go here or there: the way Firefox currently interacts directly with the <audio>/<video> element seems to break some React websites for me that do their own state tracking.

Comment 32

[Marc Streckfuß [:MeFisto94]]

I guess the best would be to report this as a blocking bug to the media-control meta bug providing more detailed information (e.g. an example page and maybe even how the state can break).

This is specifically interesting because for the websites it shouldn't be different than a manual click, really, so it probably is some side-effect.

Comment 33

[Alastor Wu [:alwu]]

(In reply to Ilya K from comment #31)

I see. Thanks a lot, going to follow the tracking bug. Also not sure if this should go here or there: the way Firefox currently interacts directly with the <audio>/<video> element seems to break some React websites for me that do their own state tracking.

According to what we got on Fennec before (we have enabled media control on Android for a long while), if the web media player doesn't listen to play and pause event properly and only rely on touch event on their interface, then it's high chance to have a inconsistency on their interface when we control media element directrly by media control.

But free feel to open a new bug for that, and NI me on that bug.

Thank you.

Comment 34

[Ilya K]

Yep, that's basically the issue. I've checked on Fennec/Fenix, and it's the same thing there. I guess there's no reason to open a bug then.