Provide timeline information of media to Windows 10's SMTC
Categories
(Core :: Audio/Video: Playback, enhancement, P4)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox125 | --- | fixed |
People
(Reporter: Poopooracoocoo, Assigned: nerixdev)
References
(Blocks 1 open bug)
Details
Attachments
(5 files)
empty timeline info - modernflyouts and firefox.png
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0
Steps to reproduce:
- Played https://googlechrome.github.io/samples/media-session/video.html
- Pressed pause on my keyboard. Note that I am running Modern Flyouts on Windows 10.
Actual results:
No timeline information was displayed.
Expected results:
Timeline information should have been displayed.
|
||
Comment 1•4 years ago
|
||
Bugbug thinks this bug should belong to this component, but please revert this change in case of error.
|
Reporter | |
Comment 2•4 years ago
|
||
|
|
Reporter | |
Comment 3•4 years ago
|
||
Just as an FYI, Google said that they recently implemented this for macOS, and may do so for Windows.
|
|
Reporter | |
Comment 4•4 years ago
|
||
|
||
Updated•4 years ago
|
|
|
Reporter | |
Comment 5•3 years ago
|
||
has there been any progress on this bug btw?
I don't have much experience with C++, but I looked into this.
The main problem is that with the current setup, Firefox can't update the timeline.
As I explained in 1698283 (comment), Firefox (and on this note Chromium as well) uses the ISystemMediaTransportControls interface which can't update the timeline (as it's too old?).
To update the timeline, Firefox would need to get the SystemMediaTransportControls class from C++/WinRT. This could introduce breaking changes.
- Compatibility with older version of Windows. In the
WindowsSMTCProvider, there's a check for Windows 8.1 or later. The "new" SMTC would require at least Windows 10 (Build 10240, the first one). - From what I found, Firefox is not using
C++/WinRTso this would need to be added. - The "new" SMTC don't provide a way of getting the class/interface for a specific HWND, instead one just calls the static method. Currently, Firefox creates an invisible window to bind the SMTC to.
There also seems to exist ISystemMediaTransportControls2 (in Windows.Media.h) which I just found out about.
It's not documented anywhere but it's the missing interface. It doesn't need C++/WinRT, the ComPtr<ISystemMediaTransportControls> (in WindowsSMTCProvider) could be casted to a ComPtr<ISystemMediaTransportControls2> where the timeline could be updated.
I'm not too sure about compatibility with older Windows versions.
|
||
Comment 9•3 years ago
|
||
In order to do this, we might want to have a cross-platform mechanism to update the time to the platform specific MediaControlKeySource first. We have a method to set position if websites set the position state via MediaSession API, but I suppose that that API won't be called frequently or never be called. So we might need a way to calulate that by ourselve in order to update a rough position information and use above method to notify the key source to call platform API.
Also, if multiple media are playing in the same tab, we would also need to determine what media should be shown for the timeline update. (or don't show timeline in this situation)
In addition, as the default SMTC visual control interface on Windows 10 doesn't have timeline display, this bug is an enhancement for an external plugin (Modern Flyouts), so change its priority to P4.
|
Assignee | |
Comment 10•8 months ago
|
||
The PositionState is already saved as an optional value inside the MediaSession.
This makes the PositionState optional in all events emitted as it can be
explicitly reset by passing an empty object.
|
||
Updated•8 months ago
|
|
|
Assignee | |
Comment 11•8 months ago
|
||
When setting the position state for a media session, it might not be the
active. When activating a session it should be sent.
Depends on D198847
|
|
Assignee | |
Comment 12•8 months ago
|
||
Depends on D198848
|
|
Assignee | |
Comment 13•8 months ago
|
||
Depends on D198849
|
|
||
Comment 14•8 months ago
|
||
Sorry for my late response. I just came back from vacation. I just want to let you know that I will start reviewing your patches next week, thank you for your patience!
|
|
||
Updated•7 months ago
|
|
|
||
Comment 15•7 months ago
|
||
Thank you so much for your help on this issue. After getting r+ on your patches, you could consider to grant the try server permission and then pushing patches to the try server, or I can help you push them to the try server to see if they break any tests or not.
Let me know what path you would like to choose. Thank you!
|
|
Assignee | |
Comment 16•7 months ago
|
||
I updated the patches, and I'll try to get permission and push to the try server after getting r+.
For integration tests of SMTC, GSMTC could be used, but that might be flaky. I know that in some (unknown) cases, it doesn't capture running apps, but I don't know how rare that bug is. Windows itself uses the NowPlayingSessionManager, which is undocumented.
|
|
||
Comment 17•7 months ago
|
||
(In reply to Nerixyz from comment #16)
I updated the patches, and I'll try to get permission and push to the try server after getting r+.
I already r+ all you patches and the win-reviewers for D198850 is optional, so you're good to go now! I'm happy to vouch you for the level 1 access, you just need to follow this instruction and NI me on the bug.
For integration tests of SMTC, GSMTC could be used, but that might be flaky. I know that in some (unknown) cases, it doesn't capture running apps, but I don't know how rare that bug is. Windows itself uses the NowPlayingSessionManager, which is undocumented.
Yes unfortunately currently we're lacking of the platform level integration test for this feature, we mostly rely on the mediacontroller level integration tests. We could definitely do it better if there is a good and robust way to do so.
|
|
Assignee | |
Comment 18•7 months ago
|
||
I pushed the patches to try. The 974 failures don't seem to be related to the patches - at least they're not in the mediacontrol tests.
|
|
||
Comment 19•7 months ago
|
||
You can use ./mach try fuzzy --preset media-full to test our pre-defined media tests.
|
|
Assignee | |
Comment 20•7 months ago
|
||
Ah, thank you! This one got a few more failures, but the only one from mediacontrol was browser_audio_focus_management.js.
|
|
||
Comment 21•7 months ago
|
||
Hmm usually we shouldn't get that many errors, I would suggest that you can push two times, one with your patches and one without, to see the difference. Also, it would be good to push changes based on the latest mozilla-central to see if there is any conflict.
|
|
||
Comment 22•7 months ago
•
|
||
I pushed your changes to the try server, and the result looks good. I will land it later after the tasks are all completed and no broken tests on the CI.
|
|
Assignee | |
Comment 23•7 months ago
|
||
Thank you! Seems like we both pushed :)
|
||
Comment 24•7 months ago
|
||
Pushed by alwu@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/f94f409282ad Emit PositionState as an optional value. r=alwu https://hg.mozilla.org/integration/autoland/rev/3b2c031481cb Save PositionState for media sessions in MediaStatusManager. r=alwu https://hg.mozilla.org/integration/autoland/rev/0ab5cd414ce9 Implement current playback position algorithm for PositionState. r=alwu https://hg.mozilla.org/integration/autoland/rev/f81cf9bd7b9f Provide timeline information and handle SeekTo on Windows. r=alwu
|
||
Comment 25•7 months ago
|
||
Backed out for causing build bustage on TestPositionState.cpp
|
|
Assignee | |
Comment 26•7 months ago
|
||
Moved the test data out of the anonymous namespace (see D198849).
|
|
||
Comment 27•7 months ago
|
||
Pushed by alwu@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/72bc9a4787bc Emit PositionState as an optional value. r=alwu https://hg.mozilla.org/integration/autoland/rev/9e54467e852a Save PositionState for media sessions in MediaStatusManager. r=alwu https://hg.mozilla.org/integration/autoland/rev/41fd655c5c32 Implement current playback position algorithm for PositionState. r=alwu https://hg.mozilla.org/integration/autoland/rev/21d2a36f8f07 Provide timeline information and handle SeekTo on Windows. r=alwu
|
||
Comment 28•7 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/72bc9a4787bc
https://hg.mozilla.org/mozilla-central/rev/9e54467e852a
https://hg.mozilla.org/mozilla-central/rev/41fd655c5c32
https://hg.mozilla.org/mozilla-central/rev/21d2a36f8f07
Description
•