Extend metrics for dropped frames

RESOLVED FIXED in Firefox 68

Status

()

enhancement
P3
normal
RESOLVED FIXED
2 months ago
2 months ago

People

(Reporter: tarek, Assigned: tarek)

Tracking

57 Branch
mozilla68
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox68 fixed)

Details

Attachments

(1 attachment)

Assignee

Description

2 months ago

mozRequestDebugInfo gives us some information about the dropped frames in the decoder (1), but not in the sink (2) or the compositor (3).

the getVideoPlaybackQuality() API sums 1, 2 and 3

Let's add in our debug information the intermediate values (2 & 3) so we can see where the frames are dropped exactly.

The JSON content below is from our marionette-media test file and you can see a lot of dropped frames in getVideoPlaybackQuality() but the decoder is doing a good job.

	{
	"getVideoPlaybackQuality": {
		"corruptedVideoFrames": 0, 
		"creationTime": 32274, 
		"droppedVideoFrames": 198, 
		"totalVideoFrames": 205
	}, 
	"mozRequestDebugInfo": {
		"Audio Decoder(audio/opus, 2 channels @ 48.0kHz)": "opus audio decoder", 
		"Audio Frames Decoded": "860", 
		"Audio State": {
		"decoder": "0", 
		"demuxq": "0", 
		"demuxr": "0", 
		"ds": "0", 
		"eos": "0", 
		"in": "860", 
		"ni": "0", 
		"no": "0", 
		"out": "860", 
		"pending": "0", 
		"qs": "0", 
		"sid": "5", 
		"tt": "-1.0", 
		"tths": "-1", 
		"wfd": "0", 
		"wfk": "0", 
		"wp": "0"
		}, 
		"Audio Track Buffer Details": {
		"Evictable": "596944", 
		"NextGetSampleIndex": "860", 
		"NextInsertionIndex": "1500", 
		"NumSamples": "1500", 
		"Size": "1046366"
		}, 
		"Audio Track Buffered": "ranges=[(0.000000, 30.001000)]", 
		"AudioSinkWrapper": {
		"AudioEnded": "0", 
		"IsPlaying": "1", 
		"IsStarted": "1"
		}, 
		"Compositor dropped frame(including when element's invisible)": "0", 
		"Container Type": "MediaSource", 
		"Dumping Audio Track Buffer(audio/webm)": {
		"mLastAudioTime": "17.201000"
		}, 
		"Dumping Data for Demuxer": "7f6d1cedbc00", 
		"Dumping Video Track Buffer(video/webm)": {
		"mLastVideoTime": "3.433000"
		}, 
		"Hardware Video Decoding": "disabled", 
		"MDSM": {
		"GetClock": "15190500", 
		"GetMediaTime": "15188250", 
		"IsPlaying": "1", 
		"duration": "53701000", 
		"mAudioCompleted": "0", 
		"mAudioStatus": "idle", 
		"mDecodedAudioEndTime": "17194500", 
		"mDecodedVideoEndTime": "3417000", 
		"mIsPrerolling": "0", 
		"mMediaSink": "7f6d20093a10", 
		"mPlayState": "3", 
		"mSentFirstFrameLoadedEvent": "1", 
		"mVideoCompleted": "0", 
		"mVideoStatus": "pending", 
		"state": "DECODING"
		}, 
		"MediaDecoder=7f6d1fef0600": {
		"channels": "2", 
		"hasAudio": "1", 
		"hasVideo": "1", 
		"mPlayState": "PLAYING", 
		"rate": "48000"
		}, 
		"Video Decoder(video/vp9, 3840x2160 @ 60.05)": "ffvpx video decoder", 
		"Video Frames Decoded": "205 (skipped=0)", 
		"Video State": {
		"decoder": "1", 
		"demuxq": "0", 
		"demuxr": "0", 
		"ds": "0", 
		"eos": "0", 
		"in": "206", 
		"ni": "0", 
		"no": "1", 
		"out": "205", 
		"pending": "0", 
		"qs": "1", 
		"sid": "4", 
		"tt": "-1.0", 
		"tths": "-1", 
		"wfd": "0", 
		"wfk": "0", 
		"wp": "0"
		}, 
		"Video Track Buffer Details": {
		"Evictable": "0", 
		"NextGetSampleIndex": "206", 
		"NextInsertionIndex": "1581", 
		"NumSamples": "1581", 
		"Size": "81167006"
		}, 
		"Video Track Buffered": "ranges=[(0.000000, 26.350000)]", 
		"VideoSink": {
		"IsPlaying": "1", 
		"IsStarted": "1", 
		"VideoQueue": {
			"finished": "0", 
			"size": ""
		}, 
		"mEndPromiseHolder.IsEmpty()": "0", 
		"mHasVideo": "1", 
		"mVideoFrameEndTime": "3417000", 
		"mVideoSinkEndRequest.Exists()": "0"
		}
	}
	}
Assignee

Updated

2 months ago
Type: defect → enhancement
Assignee

Updated

2 months ago
Assignee: tarek → jyavenard
Assignee

Updated

2 months ago
Priority: -- → P3
Assignee

Comment 1

2 months ago

We can do Bug 1542674 once this one is done I guess

Blocks: 1542674
Assignee

Updated

2 months ago
Assignee: jyavenard → tarek
Assignee

Comment 2

2 months ago

My current understanding is that we can add those counters :

  • a total number of dropped frames for VideoSink
  • a total number of dropped frames for the video Compositor

And maybe:

  • a transient in/out/no counter for VideoSink, that will give us the frames that are in, out and get dropped at one point.

Pointers:

https://searchfox.org/mozilla-central/source/dom/media/mediasink/VideoSink.cpp#520
https://searchfox.org/mozilla-central/source/dom/media/mediasink/VideoSink.cpp#542
https://searchfox.org/mozilla-central/source/dom/html/HTMLMediaElement.cpp#1553
https://searchfox.org/mozilla-central/source/gfx/layers/ipc/ImageBridgeParent.cpp#198

Assignee

Comment 4

2 months ago

This patch adds the number of dropped frames for each step of the process
(read/sink/compositor) and gives us more insight about where frames are
dropped, as opposed to the getVideoPlaybackQuality() API which gives the grand
total.

Assignee

Comment 5

2 months ago

.

Comment 6

2 months ago
Pushed by tziade@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fbd1effe6fe4
Extend metrics for dropped frames - r=padenot

Comment 7

2 months ago
bugherder
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Depends on: 1548064
You need to log in before you can comment on or make changes to this bug.