1530996 - Store performance benchmark during decoding for use in MediaCapabilities

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, enhancement, P2)

Description

[Alex Chronopoulos [:achronop]]

We want a way to measure the ability of a system to decode AV1. We do something similar to VP9 in [1].

[1] https://searchfox.org/mozilla-central/source/dom/media/Benchmark.h#101

Comment 1

[Alex Chronopoulos [:achronop]]

Talking with jya was revealed that the way we are doing Benchmark is not ideal. Benchmark for VP9, the same stands for AV1, uses a static video buffer to measure performance. That practice increases the binary size. This is a problem especially in android thus right now we exclude benchmark check on android.

One solution mentioned is to download the video from somewhere. A second path to examine is to measure on the fly on the first video decode, like Chrome does. In every case we need to examine the options and do the correct design. This patch will implement the existing mechanism.

Comment 2

[Alex Chronopoulos [:achronop]]

Discussing this further with jya, he suggested to turn. Measuring quality capabilities on the fly, during decoding, is what we need rather than one more Benchmark that will increase the binary size. Then MediaCapabilities should query that information in order to make a decision.

The high level design on that would be to rely on a permanent storage system inside gecko. Then we could use the FrameStatisticsData object of HTMLMediaElement to report quality measures, like frames dropped, for a number of codec frame rates and amount of pixels.

The FrameStatisticsData is updated during decoding in VideoSink [1]. An example of using that data to create a percentage of frame dropped/all frames is from telemetry [2] inside HTMLMediaElemnt.

This idea depends on whether we can store data permanently in a efficient way. We need to find out what is the current status on that in order to see how we can proceed.

[1] https://searchfox.org/mozilla-central/rev/2a6f3dde00801374d3b2a704232de54a132af389/dom/media/mediasink/VideoSink.cpp#469
[2] https://searchfox.org/mozilla-central/rev/2a6f3dde00801374d3b2a704232de54a132af389/dom/html/HTMLMediaElement.cpp#4190

Comment 3

[Alex Chronopoulos [:achronop]]

Attachment: av1-benchmark-class.patch

Comment 4

[Alex Chronopoulos [:achronop]]

Attachment: av1-sample-as-include-file.patch

These are the patches of the initial idea. They were tested and ready for review but we are moving away from this approach now.

Comment 5

[Alex Chronopoulos [:achronop]]

Attachment: Bug 1530996 - Enhance MediaCapabilities to read from database.

I am dumping a big patch with everything I have so far. This is the initial idea. It has been tested locally and it's green on try. After a discussion in Whistler, we found out that some extra bits need to be added on the top of that. I am pushing what I have so far in case anyone wants to provide feedback.

The implementation so far includes.

• A wrapper around kvstrore module.
• An IPDL protocol that uses the wrapper in order to access the database from the parent process and avoid sandbox.
• A DecoderBenchmark class that reads the playback statistics, calculates a "score" and save it permanently in the database.
• Use of that functionality in HTMLMediaElement and MediaCapabilities to store and get the score, respectively.

The extra work that needs to be done is:

• Change the stored value to a moving average of 20 values, instead of the sole result of a random playback
• On Shutdown or EndOfStream step of MediaFormatReader, collect all the MediaInfo information and signal it to HTMLMediaElement to do the calculations and to trigger the storing.
• When a playback parameter changes on the fly, playback statistics are not reset so we need a mechanism to maintain some form of internal statistics per resolution or framerate and use that for the calculations.

Comment 6

[Alex Chronopoulos [:achronop]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3488e4724cce8f0a89e2219ee4865b12f7997f8c

Comment 7

[Alex Chronopoulos [:achronop]]

Attachment: Bug 1530996 - Create a wrapper around kvstore module. r?jya

Comment 8

[Alex Chronopoulos [:achronop]]

Attachment: Bug 1530996 - Create the IPC protocol to access the database from the content process. r?jya

The database is accessible from the parent process due to to the sandbox thus it is required an IPDL protocol that will transfer the queries and the results

Comment 9

[Alex Chronopoulos [:achronop]]

Attachment: Bug 1530996 - Store the benchmark calculated from the frame statistics. r?jya

Create a class that gets the video properties and the frame statistics, calculates the score in percentage for that video playback, creates a key string according to video properties and trigger the storage of the score.

The video properties used are the resolution, the frame rate and the bitdepth.

For the key, a range of levels has been created for each property and the video is categorised on the closest levels. The key consists of the various levels like: "ResolutionLevel5-FrameRateLevel1-8bit".

Finaly, it uses the IPDL protocol to store the pair of the score and the key.

Comment 10

[Alex Chronopoulos [:achronop]]

Attachment: Bug 1530996 - Use the benchmark class from MediaCapabilities. r?jya

Make use of the new DecoderBenchmark class in MediaCapabilities instead of the old Benchmark mechanism.

Comment 11

[Alex Chronopoulos [:achronop]]

Attachment: Bug 1530996 - Use the new storage mechanism to save the current playback benchmark. r?jya

Create an event in MediaFormatReader the will signal to the HTMLMediaElement to initiate a new storing according to the latest VideoInfo. Also when the application is shutting down, trigger a new storing early enough, before all the events are disconnected.

Comment 12

[Alex Chronopoulos [:achronop]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6d919217268f9d62285946abf3df5e666f63ebf9

Comment 13

[Alex Chronopoulos [:achronop]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=67e8da60999c96e7be5845a9fa7fc37a743917cc

Comment 14

[Alex Chronopoulos [:achronop]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=21150e21ab0dd24b70ae45842d02fd1122808ecc

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/763071ac85a2
Create a wrapper around kvstore module. r=jya
https://hg.mozilla.org/integration/autoland/rev/2dc1131a27e8
Create the IPC protocol to access the database from the content process. r=jya
https://hg.mozilla.org/integration/autoland/rev/112ead7965e3
Store the benchmark calculated from the frame statistics. r=jya
https://hg.mozilla.org/integration/autoland/rev/426b558f2f43
Use the benchmark class from MediaCapabilities. r=jya
https://hg.mozilla.org/integration/autoland/rev/eed72505cf9b
Use the new storage mechanism to save the current playback benchmark. r=jya

Comment 16

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/763071ac85a2
https://hg.mozilla.org/mozilla-central/rev/2dc1131a27e8
https://hg.mozilla.org/mozilla-central/rev/112ead7965e3
https://hg.mozilla.org/mozilla-central/rev/426b558f2f43
https://hg.mozilla.org/mozilla-central/rev/eed72505cf9b