VP9 benchmark always report that it's fast when using MediaCapabilities

RESOLVED FIXED in Firefox 66

Status

()

defect
P2
normal
Rank:
15
RESOLVED FIXED
5 months ago
Last month

People

(Reporter: jya, Assigned: jya)

Tracking

(Regressed 2 bugs, {regression})

unspecified
mozilla67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 unaffected, firefox65+ wontfix, firefox66+ fixed, firefox67+ verified)

Details

Attachments

(1 attachment)

Assignee

Description

5 months ago

From https://bugzilla.mozilla.org/show_bug.cgi?id=1474943#c8

Shouldn't "VP9Benchmark::MediaBenchmarkVp9Fps()" return "StaticPrefs::MediaBenchmarkVp9Fps()" instead of "StaticPrefs::MediaBenchmarkVp9Threshold()"?

Assignee

Comment 1

5 months ago

And not the threshold which is always 150.

Assignee

Updated

5 months ago
Assignee: nobody → jyavenard

Updated

5 months ago
Rank: 15
Priority: -- → P2

Comment 2

5 months ago

FWIW: I can confirm that this very simple fix works.

Try build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=731d88883ddb226d246f70664120914235b66e50

Assignee

Comment 3

5 months ago

Thank you for reporting this. It would have led to people getting poor playback performance with YouTube :(

Comment 4

5 months ago
Pushed by jyavenard@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9f49631d6536
Properly get VP9 benchmark value. r=bryce

Comment 5

5 months ago

(In reply to Jean-Yves Avenard [:jya] from comment #3)

Thank you for reporting this. It would have led to people getting poor playback performance with YouTube :(

No problem. Any chance this will be backported? or does it need to ride the train?

Assignee

Comment 6

5 months ago

(In reply to Kristian Klausen from comment #5)

(In reply to Jean-Yves Avenard [:jya] from comment #3)

Thank you for reporting this. It would have led to people getting poor playback performance with YouTube :(

No problem. Any chance this will be backported? or does it need to ride the train?

Will definitely ask for it to be backported, even to release if a dot release is planned.

Assignee

Comment 7

5 months ago

[Tracking Requested - why for this release]:
With this regression, we always report to YouTube that VP9 will play nice and smooth for most content. Even on slow machines. This will lead to bad playback performance, stuttering or frozen videos.

Comment 8

5 months ago
bugherder
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67

Yikes. Please nominate this for Beta & Release approval when you get a chance. Kristian, can you also please confirm that recent Nightly builds are working for you? Thanks!

Flags: qe-verify+
Flags: needinfo?(kristian)
Flags: needinfo?(jyavenard)
Assignee

Comment 10

4 months ago

Comment on attachment 9040296 [details]
Bug 1524119 - Properly get VP9 benchmark value. r?bryce

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

Bug 1474943

User impact if declined

YouTube will serve higher resolution than the machine can handle, or will use a codec that we definitely can't decode fast enough.
There's been a few reports that YT was no longer playing well. It was assumed to just be because we enabled vp9 on more machines than before.

Is this code covered by automated tests?

No

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

It can't be worse than before. Should the fix be wrong (and it isn't) we would continue being bad, which is report that the machine can always do the VP9 codec even if it can't.

String changes made/needed

none

Flags: needinfo?(jyavenard)
Attachment #9040296 - Flags: approval-mozilla-release?
Attachment #9040296 - Flags: approval-mozilla-beta?

Comment 11

4 months ago

(In reply to Ryan VanderMeulen [:RyanVM] from comment #9)

Yikes. Please nominate this for Beta & Release approval when you get a chance. Kristian, can you also please confirm that recent Nightly builds are working for you? Thanks!

I can confirm it works as expected with the 2019-02-04 Nightly, tested with https://beaufortfrancois.github.io/sandbox/media-capabilities/decoding-info-2.html and adjusting media.benchmark.vp9.fps.

I think the algorithm could be improved, as it says I cant play 720p30fps< smoothly although testing reveal I can play 1080p60fps smoothly.

Flags: needinfo?(kristian)
Whiteboard: [qa-triaged]
Assignee

Comment 12

4 months ago

(In reply to Kristian Klausen from comment #11)

I think the algorithm could be improved, as it says I cant play 720p30fps< smoothly although testing reveal I can play 1080p60fps smoothly.

It is extremely conservative on purpose.

Right now, it decodes a short 720p vp9 videos and measure the decoding speed (and only the decoding speed). We will report that it can play smoothly if we can decode at twice the speed requested. So if the test decoded at 130fps, then we'll report that 720p30 can be smooth.

What is your vp9 score right now? and what platform are you on?

Comment 13

4 months ago

(In reply to Jean-Yves Avenard [:jya] from comment #12)

(In reply to Kristian Klausen from comment #11)

I think the algorithm could be improved, as it says I cant play 720p30fps< smoothly although testing reveal I can play 1080p60fps smoothly.

It is extremely conservative on purpose.

Right now, it decodes a short 720p vp9 videos and measure the decoding speed (and only the decoding speed). We will report that it can play smoothly if we can decode at twice the speed requested. So if the test decoded at 130fps, then we'll report that 720p30 can be smooth.

What is your vp9 score right now? and what platform are you on?

114 fps with 2019-02-04 Nightly on Arch Linux (i5-4200U).

I've tried to reproduce the issue on Firefox 66.0a1 (2019-01-28), under Windows 10x64 without success.
I'm not sure where I can see what value is returned for VP9Benchmark::MediaBenchmarkVp9Fps().
While playing https://www.youtube.com/watch?v=f8reRUupmU4 on an affected build, I noticed in Browser console that StaticPrefs::MediaBenchmarkVp9Fps() is displayed.
Jean-Yves, can you please provide me some clear STR, so I can reproduce and verify this issue?
Also, is there a VPN connection mandatory?

Flags: needinfo?(jyavenard)

Comment 15

4 months ago

(In reply to Mihai Boldan, QA [:mboldan] from comment #14)

I've tried to reproduce the issue on Firefox 66.0a1 (2019-01-28), under Windows 10x64 without success.
I'm not sure where I can see what value is returned for VP9Benchmark::MediaBenchmarkVp9Fps().
While playing https://www.youtube.com/watch?v=f8reRUupmU4 on an affected build, I noticed in Browser console that StaticPrefs::MediaBenchmarkVp9Fps() is displayed.
Jean-Yves, can you please provide me some clear STR, so I can reproduce and verify this issue?
Also, is there a VPN connection mandatory?

The benchmark logic is only used if hardware acceleration isn't supported (all newer Intel cpus I think?).
You can use https://beaufortfrancois.github.io/sandbox/media-capabilities/decoding-info-2.html for a quick test, choose "vp9". For a affected build adjusting "media.benchmark.vp9.threshold" affect the result, with a fixed build "media.benchmark.vp9.fps" affect the result. If you get that working, you should be able to test with Youtube afterwards (I think you need to clear the cookies between test).

I've tried to reproduce this issue using our current hardware, and the link from comment 15 without success.
Since we don't have any computer with newer Intel cpu, I am not able to reproduce and verify this issue.
I am removing the qe-verify+ flag, but if there is any workaround in order for me to test this, please add the qe-verify+ flag again and a needinfo for me, along with the additional information.

Flags: qe-verify+

I'll take Kristian's verification here. Jean-Yves can you think of any reason why Mihai may not be seeing the bug?

Comment on attachment 9040296 [details]
Bug 1524119 - Properly get VP9 benchmark value. r?bryce

Fix for bad regression for video, let's uplift for beta 6.

Attachment #9040296 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Assignee

Comment 19

4 months ago

(In reply to Liz Henry (:lizzard) (use needinfo) from comment #17)

I'll take Kristian's verification here. Jean-Yves can you think of any reason why Mihai may not be seeing the bug?

I think there's been a misunderstanding.

You do not need a new intel CPU to test, quite the opposite. The issue is a problem on machine that are slow.

To test you go to https://beaufortfrancois.github.io/sandbox/media-capabilities/decoding-info-2.html with a Windows or Linux machine (on mac, the issue was bypassed as we always report it as not smooth unless a discrete GPU is used).

On that page select: VP09.01.20.08.01.01.01.01.00

Select all the resolutions in the 2nd column, the 4000kbp in the 3rd, and media-source in the 4th.

You'll see just under the list of supported resolution. You will notice that before the fix, this would be listing them all as supported, just not power-efficient
the list returned would always be:

7680x4320 (8K) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT

Regardless of the value stored in the preference media.benchmark.vp9.fps

You could change it to 300, or 5 and it would show the same. That is, we support all those resolutions.

Now after the fix:
in media.benchmark.vp9.fps store the value say 100.

reload the page.
Now what you will see in the list:
7680x4320 (8K) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 25 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 24 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
7680x4320 (8K) | 4000 kbps | 20 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 25 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 24 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
3840x2160 (4K) | 4000 kbps | 20 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 25 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 24 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
2560x1440 (HD) | 4000 kbps | 20 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 25 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 24 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1440 (HD) | 4000 kbps | 20 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 25 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 24 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1280 (HD) | 4000 kbps | 20 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 25 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 24 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x1080 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 25 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1920x960 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 30 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1920x854 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x960 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x854 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 50 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x720 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 60 fps | media-source | NOT SMOOTH & NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 50 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 30 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 25 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 24 fps | media-source | NOT POWER EFFICIENT
1280x640 (HD) | 4000 kbps | 20 fps | media-source | NOT POWER EFFICIENT

You can see that we now properly report the resolutions as "NOT SMOOTH" and anything > 30fps as not smooth too.

Flags: needinfo?(jyavenard)

I followed exactly the steps from Comment 19, using a Dell Laptop with Intel Core i5 4200U 1.60 GHz without being able to reproduce the issue.
If the value of media.benchmark.vp9.fps is over 150, the results are exactly how are described in Comment 19, but if it has a smaller value, than the results for the affected builds are different. On this test I've set the value of media.benchmark.vp9.fps to '5' on both profiles, affected an unaffected.
Here are the results: https://tinyurl.com/y75266bs
Note that the used builds are:

  • affected - Firefox 66.0a1(2019-01-28)
  • unaffected - Firefox 67.0a1(2019-02-08)

There are different results on the builds under test, but as far as I understood, only on the fixed builds should appear the NOT SMOOTH result. And also, from the comments above it appears that the result on affected build should be the same 'NOT POWER EFFICIENT', no matter what value does the media.benchmark.vp9.fps pref has.

Jean-Yves, can you please take a look over my results?

Flags: needinfo?(jyavenard)

Comment 22

4 months ago

(In reply to Mihai Boldan, QA [:mboldan] from comment #21)

I followed exactly the steps from Comment 19, using a Dell Laptop with Intel Core i5 4200U 1.60 GHz without being able to reproduce the issue.
If the value of media.benchmark.vp9.fps is over 150, the results are exactly how are described in Comment 19, but if it has a smaller value, than the results for the affected builds are different. On this test I've set the value of media.benchmark.vp9.fps to '5' on both profiles, affected an unaffected.
Here are the results: https://tinyurl.com/y75266bs
Note that the used builds are:

  • affected - Firefox 66.0a1(2019-01-28)
  • unaffected - Firefox 67.0a1(2019-02-08)

There are different results on the builds under test, but as far as I understood, only on the fixed builds should appear the NOT SMOOTH result. And also, from the comments above it appears that the result on affected build should be the same 'NOT POWER EFFICIENT', no matter what value does the media.benchmark.vp9.fps pref has.

Jean-Yves, can you please take a look over my results?

Your results look correct to me. You get a different result because your computer is slower.

The logic is implemented like this:

if HardwareAccelerated then {
    smooth = true;
    powerEfficent = true;
} else {
    if media.benchmark.vp9.fps >= media.benchmark.vp9.threshold {
        smooth = true;
        powerEfficent = false;
    } else {
        // In the following line media.benchmark.vp9.threshold is used instead of media.benchmark.vp9.fps for affected builds.
        if ((1280.0 * 720.0) / (videoWidth * videoHeight) * media.benchmark.vp9.fps) / videoFps) > 2 {
            smooth = true;
        } else {
            smooth = false;
        }
        powerEfficent = false;
    }
}

Comment on attachment 9040296 [details]
Bug 1524119 - Properly get VP9 benchmark value. r?bryce

[Triage Comment]
Fixes VP9 playback performance on lower-end systems. Verified by the reporter as working on Nightly & Beta. Approved for 65.0.1.

Attachment #9040296 - Flags: approval-mozilla-release? → approval-mozilla-release+
Flags: qe-verify+
Assignee

Comment 25

4 months ago

(In reply to Kristian Klausen from comment #22)

The logic is implemented like this:

if HardwareAccelerated then {
    smooth = true;
    powerEfficent = true;
} else {
    if media.benchmark.vp9.fps >= media.benchmark.vp9.threshold {
        smooth = true;
        powerEfficent = false;
    } else {
        // In the following line media.benchmark.vp9.threshold is used instead of media.benchmark.vp9.fps for affected builds.
        if ((1280.0 * 720.0) / (videoWidth * videoHeight) * media.benchmark.vp9.fps) / videoFps) > 2 {
            smooth = true;
        } else {
            smooth = false;
        }
        powerEfficent = false;
    }
}

we probably should remove the test IsVP9DecodeFast, was only kept to keep same behaviour as before.

it will lead to more accurate quality selection in YouTube

Flags: needinfo?(jyavenard)
Assignee

Comment 26

4 months ago

(In reply to Mihai Boldan, QA [:mboldan] from comment #21)

I followed exactly the steps from Comment 19, using a Dell Laptop with Intel Core i5 4200U 1.60 GHz without being able to reproduce the issue.
If the value of media.benchmark.vp9.fps is over 150, the results are exactly how are described in Comment 19, but if it has a smaller value, than the results for the affected builds are different. On this test I've set the value of media.benchmark.vp9.fps to '5' on both profiles, affected an unaffected.
Here are the results: https://tinyurl.com/y75266bs
Note that the used builds are:

  • affected - Firefox 66.0a1(2019-01-28)
  • unaffected - Firefox 67.0a1(2019-02-08)

There are different results on the builds under test, but as far as I understood, only on the fixed builds should appear the NOT SMOOTH result. And also, from the comments above it appears that the result on affected build should be the same 'NOT POWER EFFICIENT', no matter what value does the media.benchmark.vp9.fps pref has.

Jean-Yves, can you please take a look over my results?

what OS were you using?

otherwise, what I see here is that you had with a value of 5 (which really tell us this machine can’t play anything smoothly)

affected:
1280x720 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT

with fix:
1280x720 (HD) | 4000 kbps | 60 fps | file | NOT SMOOTH & NOT POWER EFFICIENT

so really, you’ve confirmed that it’s fixed.

For you to see the results you got, I can only assume that you tested on a Mac. on a Mac, if you’re using a machine with an Intel GPU, we always report things as being not fast, so it will do the comparison as if your test had returned 150fps.

Using Linux, windows or a Mac with an AMD or Nvidia connected to an external display, you would have got the result I mentioned.

Comment 27

4 months ago

(In reply to Jean-Yves Avenard [:jya] from comment #26)

Using Linux, windows or a Mac with an AMD or Nvidia connected to an external display, you would have got the result I mentioned.

I think you are incorrect. With a fast computer he will get the same result as you, with a slower computer he wont. It depends on what VP9Benchmark::IsVP9DecodeFast returns (I got similar results as him).

(In reply to Kristian Klausen from comment #22)

The logic is implemented like this:

if HardwareAccelerated then {
    smooth = true;
    powerEfficent = true;
} else {
    if media.benchmark.vp9.fps >= media.benchmark.vp9.threshold {
        smooth = true;
        powerEfficent = false;
    } else {
        // In the following line media.benchmark.vp9.threshold is used instead of media.benchmark.vp9.fps for affected builds.
        if ((1280.0 * 720.0) / (videoWidth * videoHeight) * media.benchmark.vp9.fps) / videoFps) > 2 {
            smooth = true;
        } else {
            smooth = false;
        }
        powerEfficent = false;
    }
}

typo:
((1280.0 * 720.0) / (videoWidth * videoHeight) * media.benchmark.vp9.fps) / videoFps) ->
((1280.0 * 720.0) / (videoWidth * videoHeight) * media.benchmark.vp9.fps) * videoFps)

Assignee

Comment 28

4 months ago

(In reply to Kristian Klausen from comment #27)

(In reply to Jean-Yves Avenard [:jya] from comment #26)

Using Linux, windows or a Mac with an AMD or Nvidia connected to an external display, you would have got the result I mentioned.

I think you are incorrect. With a fast computer he will get the same result as you, with a slower computer he wont. It depends on what VP9Benchmark::IsVP9DecodeFast returns (I got similar results as him).

I am not, on mac if the system is using an Intel GPU, IsVP9DecodeFast always return false
https://searchfox.org/mozilla-central/source/dom/media/Benchmark.cpp#79

and the FPS returned by VP9DecodeBenchmark will be 0

Also I was referring to results before the patch. VP9DecodeBenchmark was always returning 150 before which you spotted.

Comment 29

4 months ago

(In reply to Jean-Yves Avenard [:jya] from comment #28)

(In reply to Kristian Klausen from comment #27)

(In reply to Jean-Yves Avenard [:jya] from comment #26)

Using Linux, windows or a Mac with an AMD or Nvidia connected to an external display, you would have got the result I mentioned.

I think you are incorrect. With a fast computer he will get the same result as you, with a slower computer he wont. It depends on what VP9Benchmark::IsVP9DecodeFast returns (I got similar results as him).

I am not, on mac if the system is using an Intel GPU, IsVP9DecodeFast always return false
https://searchfox.org/mozilla-central/source/dom/media/Benchmark.cpp#79

and the FPS returned by VP9DecodeBenchmark will be 0

Also I was referring to results before the patch. VP9DecodeBenchmark was always returning 150 before which you spotted.

IsVP9DecodeFast was never broken, it don't use VP9Benchmark::MediaBenchmarkVp9Fps() but StaticPrefs::MediaBenchmarkVp9Fps() directly. I assume your first result was done with media.benchmark.vp9.fps=300(In reply to Jean-Yves Avenard [:jya] from comment #25)

(In reply to Kristian Klausen from comment #22)

The logic is implemented like this:

if HardwareAccelerated then {
    smooth = true;
    powerEfficent = true;
} else {
    if media.benchmark.vp9.fps >= media.benchmark.vp9.threshold {
        smooth = true;
        powerEfficent = false;
    } else {
        // In the following line media.benchmark.vp9.threshold is used instead of media.benchmark.vp9.fps for affected builds.
        if ((1280.0 * 720.0) / (videoWidth * videoHeight) * media.benchmark.vp9.fps) / videoFps) > 2 {
            smooth = true;
        } else {
            smooth = false;
        }
        powerEfficent = false;
    }
}

we probably should remove the test IsVP9DecodeFast, was only kept to keep same behaviour as before.

it will lead to more accurate quality selection in YouTube

The current behavior is definitely not ideal. I have opened a new bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1526684

Comment 30

4 months ago

(In reply to Jean-Yves Avenard [:jya] from comment #28)

(In reply to Kristian Klausen from comment #27)

(In reply to Jean-Yves Avenard [:jya] from comment #26)

Using Linux, windows or a Mac with an AMD or Nvidia connected to an external display, you would have got the result I mentioned.

I think you are incorrect. With a fast computer he will get the same result as you, with a slower computer he wont. It depends on what VP9Benchmark::IsVP9DecodeFast returns (I got similar results as him).

I am not, on mac if the system is using an Intel GPU, IsVP9DecodeFast always return false
https://searchfox.org/mozilla-central/source/dom/media/Benchmark.cpp#79

and the FPS returned by VP9DecodeBenchmark will be 0

Also I was referring to results before the patch. VP9DecodeBenchmark was always returning 150 before which you spotted.

Did you use the same media.benchmark.vp9.fps for both tests?
The only way to get the same result as you, is if IsVP9DecodeFast returns true and it was never broken (it use StaticPrefs::MediaBenchmarkVp9Fps() not VP9Benchmark::MediaBenchmarkVp9Fps()). So you must have been using a different media.benchmark.vp9.fps for each test as (((1280*720) / (7680*4320) * 150) / 60) > 2 isn't true.

(In reply to Jean-Yves Avenard [:jya] from comment #26)

(In reply to Mihai Boldan, QA [:mboldan] from comment #21)

I followed exactly the steps from Comment 19, using a Dell Laptop with Intel Core i5 4200U 1.60 GHz without being able to reproduce the issue.
If the value of media.benchmark.vp9.fps is over 150, the results are exactly how are described in Comment 19, but if it has a smaller value, than the results for the affected builds are different. On this test I've set the value of media.benchmark.vp9.fps to '5' on both profiles, affected an unaffected.
Here are the results: https://tinyurl.com/y75266bs
Note that the used builds are:

  • affected - Firefox 66.0a1(2019-01-28)
  • unaffected - Firefox 67.0a1(2019-02-08)

There are different results on the builds under test, but as far as I understood, only on the fixed builds should appear the NOT SMOOTH result. And also, from the comments above it appears that the result on affected build should be the same 'NOT POWER EFFICIENT', no matter what value does the media.benchmark.vp9.fps pref has.

Jean-Yves, can you please take a look over my results?

what OS were you using?

otherwise, what I see here is that you had with a value of 5 (which really
tell us this machine can’t play anything smoothly)

affected:
1280x720 (HD) | 4000 kbps | 60 fps | media-source | NOT POWER EFFICIENT

with fix:
1280x720 (HD) | 4000 kbps | 60 fps | file | NOT SMOOTH & NOT POWER
EFFICIENT

so really, you’ve confirmed that it’s fixed.

For you to see the results you got, I can only assume that you tested on a
Mac. on a Mac, if you’re using a machine with an Intel GPU, we always report
things as being not fast, so it will do the comparison as if your test had
returned 150fps.

Using Linux, windows or a Mac with an AMD or Nvidia connected to an external
display, you would have got the result I mentioned.

The tests were performed under Windows 10x64, using a Dell Laptop with Intel Core i5 4200U 1.60 GHz.
Please let me know if there is another way to test this, or if I am doing something wrong.

Comment on attachment 9040296 [details]
Bug 1524119 - Properly get VP9 benchmark value. r?bryce

Sorry, but after some further discussion, we decided not to take this patch after all for the 65.0.1 release. The fix will still be included in next month's Fx66 release. I apologize for the inconvenience.

Attachment #9040296 - Flags: approval-mozilla-release+ → approval-mozilla-release-

Comment 34

4 months ago

Hi, I'd like to ask about this:

if you’re using a machine with an Intel GPU, we always report things as being not fast

Could you please elaborate on what is the reasoning behind this and if there are plans to change the behaviour?

It seems that all Intel GPUs are detected as not power efficient.
I've tried to locate this part in the source code, but I haven't been able to track down the exact part where this happens.

Comment 35

4 months ago

(In reply to Lukas Havlicek from comment #34)

Hi, I'd like to ask about this:

if you’re using a machine with an Intel GPU, we always report things as being not fast

Could you please elaborate on what is the reasoning behind this and if there are plans to change the behaviour?

It seems that all Intel GPUs are detected as not power efficient.
I've tried to locate this part in the source code, but I haven't been able to track down the exact part where this happens.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1404042 and https://searchfox.org/mozilla-central/rev/93905b660fc99a5d52b683690dd26471daca08c8/dom/media/Benchmark.cpp#54-58

Comment 36

4 months ago

(In reply to Kristian Klausen from comment #35)

(In reply to Lukas Havlicek from comment #34)

Hi, I'd like to ask about this:

if you’re using a machine with an Intel GPU, we always report things as being not fast

Could you please elaborate on what is the reasoning behind this and if there are plans to change the behaviour?

It seems that all Intel GPUs are detected as not power efficient.
I've tried to locate this part in the source code, but I haven't been able to track down the exact part where this happens.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1404042 and https://searchfox.org/mozilla-central/rev/93905b660fc99a5d52b683690dd26471daca08c8/dom/media/Benchmark.cpp#54-58

Thank you for the answer.
I've actually come across this line in code, but it says that it skips Mac specifically, so I thought it must be something else, since I've confirmed this behaviour on all platforms, not just Mac.

Assignee

Comment 37

4 months ago

(In reply to Lukas Havlicek from comment #36)

Thank you for the answer.
I've actually come across this line in code, but it says that it skips Mac specifically, so I thought it must be something else, since I've confirmed this behaviour on all platforms, not just Mac.

you mean the bad performance occur on all intel devices on windows?

For VP8/VP9 , any intel 6th gen and later will do hardware decoding.

Comment 38

4 months ago

(In reply to Jean-Yves Avenard [:jya] from comment #37)

(In reply to Lukas Havlicek from comment #36)

Thank you for the answer.
I've actually come across this line in code, but it says that it skips Mac specifically, so I thought it must be something else, since I've confirmed this behaviour on all platforms, not just Mac.

you mean the bad performance occur on all intel devices on windows?

For VP8/VP9 , any intel 6th gen and later will do hardware decoding.

No, sorry, I haven't made myself clear.
I mean that using Decoding Info API, I've been getting powerEfficient: false on every platform (macOS, Windows, Linux) with Intel GPU.
Is this an expected behaviour?

Assignee

Comment 39

4 months ago

(In reply to Lukas Havlicek from comment #38)

No, sorry, I haven't made myself clear.
I mean that using Decoding Info API, I've been getting powerEfficient: false on every platform (macOS, Windows, Linux) with Intel GPU.
Is this an expected behaviour?

On Linux yes.
On mac certainly not for H264. Most of macs have HW decoder unless pre-2009.
On Windows same, unless the intel drivers have been blacklisted and HW decoding is disabled.

For FF, powerEfficient = true indicates HW decoding or resolution <= 480p

Comment 40

4 months ago

(In reply to Jean-Yves Avenard [:jya] from comment #39)

(In reply to Lukas Havlicek from comment #38)

No, sorry, I haven't made myself clear.
I mean that using Decoding Info API, I've been getting powerEfficient: false on every platform (macOS, Windows, Linux) with Intel GPU.
Is this an expected behaviour?

On Linux yes.
On mac certainly not for H264. Most of macs have HW decoder unless pre-2009.
On Windows same, unless the intel drivers have been blacklisted and HW decoding is disabled.

For FF, powerEfficient = true indicates HW decoding or resolution <= 480p

I'm actually testing VP9 codecs and this is the behaviour I experience.
The thing is, those devices support HW decoding and e.g. in Chrome I get powerEfficient: true.

Assignee

Comment 41

4 months ago

Please(In reply to Lukas Havlicek from comment #40)

I'm actually testing VP9 codecs and this is the behaviour I experience.
The thing is, those devices support HW decoding and e.g. in Chrome I get powerEfficient: true.

please open a new bug about this, this is totally unrelated to the problem described in this bug and we're getting out of topic in a closed bug.

Also provide your about:support output in that bug.

A copy of DXVAChecker for all graphic adapter would be good too.

thank you.

QA Whiteboard: [qa-triaged]
Whiteboard: [qa-triaged]

Updated

3 months ago
Depends on: 1537607

Jean-Yves are there any other tests that must be performed around this issue from my side? See Comment 31.
I'm removing the qe-verify+ flag for now.

Flags: qe-verify+ → needinfo?(jyavenard)
Assignee

Comment 43

3 months ago

no

Flags: needinfo?(jyavenard)
No longer depends on: 1537607
Regressions: 1537607

Updated

Last month
Regressions: 1540519
You need to log in before you can comment on or make changes to this bug.