1536041 - 14.61 - 18.94% basic_compositor_video (windows10-64, windows7-32) regression on pushe7f1772b47962f1038b749256929167416e5b06b (Fri Mar 15 2019)

Reporter

Description

•

6 years ago

Talos has detected a Firefox performance regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=e79f9f619f035cd2e70058eec48b064ed5c4e1eb&tochange=e7f1772b47962f1038b749256929167416e5b06b

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

19% basic_compositor_video windows7-32 pgo e10s stylo 2.71 -> 3.22
15% basic_compositor_video windows10-64 pgo e10s stylo 2.70 -> 3.10

You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=19935

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the Talos jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests

For information on reproducing and debugging the regression, either on try or locally, see: https://wiki.mozilla.org/Performance_sheriffing/Talos/Running

*** Please let us know your plans within 3 business days, or the offending patch(es) will be backed out! ***

Our wiki page outlines the common responses and expectations: https://wiki.mozilla.org/Performance_sheriffing/Talos/RegressionBugsHandling

Ionuț Goldan [:igoldan]

Reporter

Updated

•

6 years ago

Component: General → Audio/Video: Playback

Product: Testing → Core

Ionuț Goldan [:igoldan]

Reporter

Updated

•

6 years ago

Flags: needinfo?(jyavenard)

Ionuț Goldan [:igoldan]

Reporter

Comment 1

•

6 years ago

Here are the Gecko profiles for basic_compositor_video on Windows 10 (PGO builds):

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FY1iJ71K6R7qH-dV94_k9SQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FOJw7OcG5QcW-LD4B9wcCYg%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

For Windows 7 (PGO builds):

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FX7IB0EJZQ7OQ5Nq-BpwN3w%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FdnXIewg2QWaMkV5G_gGTjA%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

Chris Pearce [:cpearce (Not reading bugmail)]

Comment 2

•

6 years ago

Do the Windows 7 jobs run on Windows 10? The behaviour change from Bug 1305340 ( https://hg.mozilla.org/integration/autoland/rev/e7f1772b4796 ) is gated behind a IsWindows10OrLater() function call, so try as I might, I can't see how this change could cause such a regression on Windows 7, unless either IsWindows10OrLater() is incorrect, or the tests are actually running on Windows 10.

In fact,

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)

For Windows 7 (PGO builds):

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FX7IB0EJZQ7OQ5Nq-BpwN3w%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FdnXIewg2QWaMkV5G_gGTjA%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

Opening these profiles, I see in the top row of both profiles that they are running on Windows 10.

Are the jobs we think are running on Windows 7 are actually running on Windows 10?

Flags: needinfo?(igoldan)

Chris Pearce [:cpearce (Not reading bugmail)]

Comment 3

•

6 years ago

This reminds me of bug 1295630.

The test is trying to measure how many ms it takes to decode and paint a video frame. I think trading a regression of 0.4ms per frame overall in order to reduce latency of outputting individual frames is probably fine.

Jean-Yves Avenard [:jya]

Comment 4

•

6 years ago

Before commenting, I'd like to know what this test is actually testing, what it measures and how it does it.

The change is a Win10 only change, so getting regression on win7 is nonsensical or as :cpearce mentioned the test isn't doing what it's supposed to.

Lowering the latency of the decoder, can only speed up the time to paint the first frame, I can't think of an exception to that.

Flags: needinfo?(jyavenard)

Ionuț Goldan [:igoldan]

Reporter

Comment 5

•

6 years ago

(In reply to Chris Pearce [:cpearce (GMT+13)] from comment #2)

Do the Windows 7 jobs run on Windows 10? The behaviour change from Bug 1305340 ( https://hg.mozilla.org/integration/autoland/rev/e7f1772b4796 ) is gated behind a IsWindows10OrLater() function call, so try as I might, I can't see how this change could cause such a regression on Windows 7, unless either IsWindows10OrLater() is incorrect, or the tests are actually running on Windows 10.

In fact,

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #1)

For Windows 7 (PGO builds):

before: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FX7IB0EJZQ7OQ5Nq-BpwN3w%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

after: https://perf-html.io/from-url/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FdnXIewg2QWaMkV5G_gGTjA%2Fruns%2F0%2Fartifacts%2Fpublic%2Ftest_info%2Fprofile_basic_compositor_video.zip

Opening these profiles, I see in the top row of both profiles that they are running on Windows 10.

Are the jobs we think are running on Windows 7 are actually running on Windows 10?

This sounds like a Taskcluster/CI configuration issue. Actually, by looking at the logs from this Windows 7 pgo T-e10s(bcv) task, I can clearly see this is true:

[taskcluster 2019-03-16T05:33:05.691Z] Worker Type (gecko-t-win10-64-hw) settings:
[taskcluster 2019-03-16T05:33:05.691Z]   {
[taskcluster 2019-03-16T05:33:05.691Z]     "config": {
[taskcluster 2019-03-16T05:33:05.691Z]       "deploymentId": "",
[taskcluster 2019-03-16T05:33:05.691Z]       "runTasksAsCurrentUser": false
[taskcluster 2019-03-16T05:33:05.691Z]     },
[taskcluster 2019-03-16T05:33:05.691Z]     "generic-worker": {
[taskcluster 2019-03-16T05:33:05.691Z]       "go-arch": "amd64",
[taskcluster 2019-03-16T05:33:05.691Z]       "go-os": "windows",
[taskcluster 2019-03-16T05:33:05.691Z]       "go-version": "go1.10.3",
[taskcluster 2019-03-16T05:33:05.691Z]       "release": "https://github.com/taskcluster/generic-worker/releases/tag/v10.11.2",
[taskcluster 2019-03-16T05:33:05.691Z]       "revision": "4b47d5d9f45eabc283947da0a6b1c02eddd0a7fe",
[taskcluster 2019-03-16T05:33:05.691Z]       "source": "https://github.com/taskcluster/generic-worker/commits/4b47d5d9f45eabc283947da0a6b1c02eddd0a7fe",
[taskcluster 2019-03-16T05:33:05.691Z]       "version": "10.11.2"
[taskcluster 2019-03-16T05:33:05.691Z]     },
[taskcluster 2019-03-16T05:33:05.691Z]     "machine-setup": {
[taskcluster 2019-03-16T05:33:05.691Z]       "created": "20170404",
[taskcluster 2019-03-16T05:33:05.691Z]       "maintainer": "Markco \u003cmcornmesser@mozilla.com\u003e",
[taskcluster 2019-03-16T05:33:05.691Z]       "manifest": "https://github.com/markcor/OpenCloudConfig/blob/master/userdata/Manifest/gecko-t-win10-64-hw.json"
[taskcluster 2019-03-16T05:33:05.691Z]     }
[taskcluster 2019-03-16T05:33:05.691Z]   }

Joel, could you help us out here and correct the configs?

Flags: needinfo?(igoldan) → needinfo?(jmaher)

Ionuț Goldan [:igoldan]

Reporter

Comment 6

•

6 years ago

(In reply to Jean-Yves Avenard [:jya] from comment #4)

Before commenting, I'd like to know what this test is actually testing, what it measures and how it does it.

We have these docs presenting it.

Joel Maher ( :jmaher ) (UTC -8)

Comment 7

•

6 years ago

we run all our windows7 perf tests (not unit tests) on windows 10 OS's, but we are testing a 32 bit build on the machine. We do not have machines available that can run windows7 due to driver compatibility issues.

Flags: needinfo?(jmaher)

Michael Froman [:mjf]

Updated

•

6 years ago

Rank: 10

Priority: -- → P2

Ionuț Goldan [:igoldan]

Reporter

Updated

•

6 years ago

Keywords: backlog-deferred

Alexandru Ionescu (needinfo me) [:alexandrui]

Updated

•

6 years ago

status-firefox69: --- → affected

status-firefox70: --- → affected

Ryan VanderMeulen [:RyanVM]

Updated

•

6 years ago

No longer blocks: 1305340

status-firefox69: affected → wontfix

status-firefox70: affected → wontfix

status-firefox71: --- → fix-optional

status-firefox-esr60: --- → unaffected

status-firefox-esr68: --- → wontfix

Regressed by: 1305340

Dave Hunt [:davehunt] [he/him] ⌚BST

Updated

•

6 years ago

Keywords: perf-alert

Florin Strugariu [:Bebe]

Updated

•

6 years ago

Whiteboard: [qf]

Beth Rennie [:beth] (she/her)

Updated

•

6 years ago

Whiteboard: [qf] → [qf-]

Dave Hunt [:davehunt] [he/him] ⌚BST

Comment 8

•

5 years ago

It looks like this bug stalled and has been gathering dust for a year. Looking at the recent results for basic_compositor_video, the post-regression values still appear to be present.

There was some confusion as to why this regression was detected on Windows 7, which :jmaher addressed in comment 7:

we run all our windows7 perf tests (not unit tests) on windows 10 OS's, but we are testing a 32 bit build on the machine. We do not have machines available that can run windows7 due to driver compatibility issues.

:davidb as the documented owner of this test, can you confirm if this regression should be accepted, or if it's worth further investigation? If you're no longer the most suitable owner, could you suggest a new owner? Thanks.

Flags: needinfo?(dbolter)

David Bolter [:davidb] (NeedInfo me for attention)

Comment 9

•

5 years ago

Andrew it sounds like I might still be an owner for various rendering tests... should we move these to Jessie or Sean?
E.g. pgo e10s stylo

Flags: needinfo?(dbolter) → needinfo?(overholt)

Andrew Overholt [:overholt]

Comment 10

•

5 years ago

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #8)

:davidb as the documented owner of this test

Various tests on that wiki page need new owners so I'll work on that.

It looks like this bug stalled and has been gathering dust for a year. Looking at the recent results for basic_compositor_video, the post-regression values still appear to be present.

There was some confusion as to why this regression was detected on Windows 7, which :jmaher addressed in comment 7:

we run all our windows7 perf tests (not unit tests) on windows 10 OS's, but we are testing a 32 bit build on the machine. We do not have machines available that can run windows7 due to driver compatibility issues.

In the meantime, if :jya doesn't have time to look into things here, I suggest we give :jbauman some context and ask his thoughts.

Flags: needinfo?(overholt)

Dave Hunt [:davehunt] [he/him] ⌚BST

Comment 11

•

5 years ago

Given it's been almost two years since this regression was detected, I'm closing this as wontfix.

Status: NEW → RESOLVED

Closed: 5 years ago

Resolution: --- → WONTFIX

BugBot [:suhaib / :marco/ :calixte]

Updated

•

4 years ago

Has Regression Range: --- → yes

Dave Hunt [:davehunt] [he/him] ⌚BST

Updated

•

3 years ago

Performance Impact: --- → -

Whiteboard: [qf-]

Bugzilla

14.61 - 18.94% basic_compositor_video (windows10-64, windows7-32) regression on pushe7f1772b47962f1038b749256929167416e5b06b (Fri Mar 15 2019)

Categories

(Core :: Audio/Video: Playback, defect, P2)

Tracking

()

People

(Reporter: igoldan, Unassigned)

References

(Regression)

Details

(5 keywords)

Crash Data

Security

(public)

User Story

Description

Updated

Updated

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Updated

Updated

Updated

Updated

Updated

Updated

Updated

Comment 8

Comment 9

Comment 10

Comment 11

Updated

Updated