1707242 - Speaker icon on tab playing audio disappears at youtube.com

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Adrian Skar]

Attachment: speaker off.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0

Steps to reproduce:

1. Have a YouTube video as a background tab
2. Have hardwaremediakeys enabled
3. Pause and let the audio/speaker icon disappear on the tab
4. Change the mouse focus to a different window/app
5. Resume playback on YouTube using Play/Pause media keys

Actual results:

The speaker/audio icon disappears even though the audio is still playing.
The icon shows up again if the mouse cursor hovers over the playing tab.

Expected results:

The speaker/audio icon remains active as long as audio is playing (which is what happens on other sites like soundcloud.com).

Comment 1

[Adrian Skar]

Attachment: speaker on.png

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 3

[Alastor Wu [:alwu]]

In your step1 Have a YouTube video as a background tab, do you mean pin the tab and move to other foreground tabs? Because of Proton, the sound indicator seems only showing when the mouse is hovering on the tab. I can only see the sound indicator showing above the favicon when I pin a tab.

I couldn't reproduce this issue on Windows, MacOS and Linux, Would you mind to have a screen recording to let me see if I was doing anything wrong? Thank you.

Comment 4

[Adrian Skar]

(In reply to Alastor Wu [:alwu] from comment #3)

In your step1 Have a YouTube video as a background tab, do you mean pin the tab and move to other foreground tabs? Because of Proton, the sound indicator seems only showing when the mouse is hovering on the tab. I can only see the sound indicator showing above the favicon when I pin a tab.

I couldn't reproduce this issue on Windows, MacOS and Linux, Would you mind to have a screen recording to let me see if I was doing anything wrong? Thank you.

It happens both to the speaker icon when pinned and to the playing dialog when unpinned as long as the tab is in the background. I will attach the screen recording (the playing status disappears at 00:17 even though the sound/video is still playing). Thank you!

Comment 5

[Adrian Skar]

Attachment: screen recording.mkv

Comment 6

[Alastor Wu [:alwu]]

Thank you for the recording! Following is the timeline in your recording.

1. 9s : paused the video (the audio output became 0)
2. 12s : the playing hint disappeared on the tab
3. 15s : started playing again (the audio output started showing output level, abd playing also appeared again on the tab)
4. 18s : playing disappeared again <--- (this was the issue)
5. 27s : hovered mouse onto the tab, playing appeared again

---

I can sometime reproduce this issue (not 100%, not sure why) Once the tab goes foreground again (even if switching tabs by keyboard shotcut, not by mouse hovering) the playing would appear again.

Jared, I wonder if this is a regression bug by Proton. I did see Gecko notify audio-playback when resuming media via media control. Would you mind to take a look on this?

Thank you.

Comment 7

[Jared Wein [:jaws] (please needinfo? me)]

I'm able to reproduce it but only by following those timing steps very closely. I think it has something to do with the 3 second interval. That's the same 3 second interval that we use for the delayed removal of the audio indicator.

The audio indicator is getting removed by the presence of the "soundplaying-scheduledremoval" attribute, which is only added in the "DOMAudioPlaybackStopped" event handler. I added logging and when this bug happens we receive a "DOMAudioPlaybackStarted" event followed by a "DOMAudioPlaybackStopped" event. We shouldn't get a "Stopped" event right after the "Started" event if the media is still playing. Note that I used the media keys on my keyboard when testing this.

I looked through all of the front-end code and I can't see where we could be getting a second message or not cleaning up between these. Could this be a regression from https://bugzilla.mozilla.org/show_bug.cgi?id=1578615 ?

Comment 8

[Alastor Wu [:alwu]]

Ah, thank you for debugging that. If that is possibly related with the audio playback event, then I will debug it again.

Comment 9

[Alastor Wu [:alwu]]

Interesting, the root cause of this issue is actually from the pretty low level of the media stack.

tl;dr, this issue is related with suspending background video decoding.

---

So Gecko incorrectly dispatched another DOMAudioPlaybackStopped event right after DOMAudioPlaybackStarted event, because the media element explicitly stopped the audio channel agent due to this condition being false, which means the media element's ready state was lower than HAVE_CURRENT_DATA.

That was caused by here because we couldn't find the image frame from image container that changed the ready state to HAVE_METADATA again. (And the element would keep in this incorrectly state even if the element was playing!)

The call path of resetting the image in the image container was started from here in the SeekingState::SeekCompleted. Why did seeking happen? That was because the MDSM entered the DormatState first, which is used to release the decoder resource when the media is not playing. When leaving DormantState, we would seek to the previous play position.

When video sink reset the current image in Redraw(), then VideoFrameContainer would clear the current frame because it found that image was empty. Why image was empty? Because the background video decoding had been suspended, the video we decoded were all fake data.

---

So this issue was introduced long time ago (since bug1346120), and it causes the background media element always being in a wrong ready state (HAVE_METADATA), instead of HAVE_CURRENT_DATA or any other higher states.

Comment 10

[Alastor Wu [:alwu]]

Attachment: Bug 1707242 - part1 : use empty image in null video data.

Using a non-null image can prevent element from changing its ready state incorrectlt to HAVE_METADATA. See more detailed analysis in [1].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1707242#c9

Comment 11

[Alastor Wu [:alwu]]

Attachment: Bug 1707242 - part2 : add test.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1a5d3062bbe1
part1 : use empty image in null video data. r=bryce
https://hg.mozilla.org/integration/autoland/rev/9abb40a67736
part2 : add test. r=bryce

Comment 13

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/1a5d3062bbe1
https://hg.mozilla.org/mozilla-central/rev/9abb40a67736

Comment 15

[Jon Bauman [:jbauman:]]

Attachment: bug-1707242.png

Hey Alastor,

Bug 1744411 reported this same issue as not fixed. When I try to reproduce on macOS, I observe the "playing" text consistently on background tabs playing audio, but only get the speaker icon when hovering. Is this the expected behavior?

Comment 16

[Alastor Wu [:alwu]]

I believe that is a part of Proton design. Jared, could you help me confirm that? Because I think you know Proton design more than me. Thank you.

Comment 17

[Jared Wein [:jaws] (please needinfo? me)]

Yes. The speaker icon only appears when hovering, as the speaker doubles as a button to mute the tab. We wanted to show the favicon when not hovered because the favicon is used by many users to find a tab within the tabstrip.