Intermittent dom/media/test/test_streams_individual_pause.html | video1 video frame should not have updated since video1 paused - got "r0g0b0a0", expected "r0g255b0a255"

ASSIGNED
Assigned to

Status

()

defect
P3
normal
Rank:
25
ASSIGNED
3 years ago
6 days ago

People

(Reporter: intermittent-bug-filer, Assigned: pehrsons)

Tracking

({dev-doc-needed, intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

Component: Audio/Video → Audio/Video: MediaStreamGraph
Assignee

Comment 1

3 years ago
We pause a video element playing a MediaStream and it's visible image becomes black. Even with no alpha. It's as if the drawImage() call bailed and nothing was drawn to the canvas.
Rank: 35
Priority: -- → P3
Assignee

Comment 2

3 years ago
We actually have a screenshot here that shows both video elements displaying images at different times.

That definitely points to the drawImage() call.
Rank: 35
Component: Audio/Video: MediaStreamGraph → Canvas: 2D
Priority: P3 → --
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
https://wiki.mozilla.org/Bugmasters#Intermittent_Test_Failure_Cleanup
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Comment hidden (Intermittent Failures Robot)
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
Status: REOPENED → RESOLVED
Last Resolved: 2 years agoa year ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 19

7 months ago
I caught this in rr. The reference frame for v1 is captured when v1 is in HAVE_NOTHING, so clearly there's nothing to show.

The odd thing is that we get "playing" in HAVE_NOTHING, which is against the spec.

This is due to [1] where we don't check that we have a frame to display when checking whether we are eligible for autoplay.
And since this is for a MediaStream which has only live-data we won't have a frame to display until we are playing. We don't display the current frame as soon as it's been loaded, as we do for playback. Perhaps we should.

I don't think the spec is clear about how to handle this, but it will require some spec language archeology to figure out.


[1] https://searchfox.org/mozilla-central/rev/3d989d097fa35afe19e814c6d5bc2f2bf10867e1/dom/html/HTMLMediaElement.cpp#6330
Assignee

Updated

7 months ago
Rank: 25
Component: Canvas: 2D → WebRTC: Audio/Video
Priority: -- → P3
Assignee

Comment 20

7 months ago
I found [2]:
> When the video element is paused, the current playback position is the first frame of video, and the element's show poster flag is set
>     The video element represents its poster frame, if any, or else the first frame of the video.

This in itself sounds a bit contradictory. What if the show poster flag is not set? Then it do we do the "or else the first frame" or fall through to the next:


> The video element represents the last frame of the video to have been rendered.

There wouldn't be a "last frame to have been rendered" yet.


I suppose we'd go with the "or else the first frame of the video". This then becomes a question on whether MediaStreams have a "first frame of the video".


[2] https://html.spec.whatwg.org/multipage/media.html#the-video-element:dom-media-paused
Comment hidden (Intermittent Failures Robot)
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 23

6 months ago
Bug 1423241 has made this spike.
Assignee: nobody → apehrson
Blocks: 1423241
Status: REOPENED → ASSIGNED
Assignee

Comment 24

6 months ago
jya, do you have thoughts on how we should behave here?

In particular this bit:

(In reply to Andreas Pehrson [:pehrsons] from comment #20)
> I suppose we'd go with the "or else the first frame of the video". This then
> becomes a question on whether MediaStreams have a "first frame of the video".

Since MediaStreams are not buffered, a "first frame of the video" would be outdated immediately after it's been displayed.


How does this work for streaming through the playback stack? Such streams would be buffered, right?
Flags: needinfo?(jyavenard)
Assignee

Comment 26

6 months ago
This patch is a stop-gap fix for the intermittent. I plan to follow up with a proper fix after this lands, if we can reach come to a conclusion on how it should be.
Keywords: leave-open

Comment 27

6 months ago
Pushed by pehrsons@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/759ec08eebaa
Change from "playing" to "loadeddata" before assuming there's a video frame. r=jib
Comment hidden (Intermittent Failures Robot)
Flags: needinfo?(jyavenard)
Assignee

Comment 30

6 months ago
I'd still like your thoughts on comment 24 so we can proceed with a long-term solution to this.
Flags: needinfo?(jyavenard)
playing is fired when you call play(), irrespective of the readyState.

(In reply to Andreas Pehrson [:pehrsons] from comment #24)
> jya, do you have thoughts on how we should behave here?
> 
> In particular this bit:
> 
> (In reply to Andreas Pehrson [:pehrsons] from comment #20)
> > I suppose we'd go with the "or else the first frame of the video". This then
> > becomes a question on whether MediaStreams have a "first frame of the video".
> 
> Since MediaStreams are not buffered, a "first frame of the video" would be
> outdated immediately after it's been displayed.
> 
> 
> How does this work for streaming through the playback stack? Such streams
> would be buffered, right?

A frame is displayed until there's a new one ready to be displayed. Regardless of buffering.

I'm not sure I understand the problem at hand however.

"loadeddata" is fired when we have a frame, as such, if the element is playing you can't guarantee that what you see will be the first frame, it may be the next one and so forth.

loadeddata is *queued* when there's a first frame, playback is asynchronous so by the time loadeddata actually is fired it's likely it would no longer be the first frame.

With playback what you would do here is not call play at all. when loadeddata is fired you're sure to look at the first frame.

Waiting on playing was wrong regardless.
Flags: needinfo?(jyavenard)
Assignee

Comment 32

6 months ago
(In reply to Jean-Yves Avenard [:jya] from comment #31)
> playing is fired when you call play(), irrespective of the readyState.
This test doesn't play(). It's using autoplay. So "playing" should fire when we enter HAVE_ENOUGH_DATA.

> (In reply to Andreas Pehrson [:pehrsons] from comment #24)
> > jya, do you have thoughts on how we should behave here?
> > 
> > In particular this bit:
> > 
> > (In reply to Andreas Pehrson [:pehrsons] from comment #20)
> > > I suppose we'd go with the "or else the first frame of the video". This then
> > > becomes a question on whether MediaStreams have a "first frame of the video".
> > 
> > Since MediaStreams are not buffered, a "first frame of the video" would be
> > outdated immediately after it's been displayed.
> > 
> > 
> > How does this work for streaming through the playback stack? Such streams
> > would be buffered, right?
> 
> A frame is displayed until there's a new one ready to be displayed.
> Regardless of buffering.
> 
> I'm not sure I understand the problem at hand however.
> 
> "loadeddata" is fired when we have a frame, as such, if the element is
> playing you can't guarantee that what you see will be the first frame, it
> may be the next one and so forth.
> 
> loadeddata is *queued* when there's a first frame, playback is asynchronous
> so by the time loadeddata actually is fired it's likely it would no longer
> be the first frame.
> 
> With playback what you would do here is not call play at all. when
> loadeddata is fired you're sure to look at the first frame.
> 
> Waiting on playing was wrong regardless.

Well no, I interpret it as correct since we're relying on autoplay to fire "playing".

For autoplay, the first frame should be visible while paused, then we start playing automatically when entering HAVE_ENOUGH_DATA and video frames continue from the first frame.

But for a MediaStream we don't behave in this way. We don't check for a frame so we start playing in HAVE_NOTHING.

To me it then boils down to whether a MediaStream is considered having a first frame or not.

If it does, well that's weird since we don't buffer the frames and there'll be an observable jump once playing starts.
If it doesn't, well then I have followup questions on when playing should be fired.

I'm not sure which makes more sense and that's the dilemma here atm.
Flags: needinfo?(jyavenard)
Assignee

Comment 33

6 months ago
I filed https://github.com/whatwg/html/issues/4215 to try to progress this issue.
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 35

6 days ago

The spec issue seems clear -- we should display the first frame even when paused. This will lead to the jump once playing starts, but that's all right. We should be able to align the MediaStream handling in HTMLMediaElement more to that of playback with this change.

Adding dev-doc-needed because this will be a very observable change.

Flags: needinfo?(jyavenard)
Assignee

Updated

6 days ago
Duplicate of this bug: 1425643
You need to log in before you can comment on or make changes to this bug.