Closed Bug 1742431 Opened 2 years ago Closed 2 years ago

538.1 - 7.17% twitch FirstVisualChange / wikipedia ContentfulSpeedIndex + 54 more (Android, Linux, OSX, Windows) regression on Wed November 17 2021

Categories

(Testing :: Raptor, defect)

Product:
Testing
Component:
Raptor
Version:
Firefox 96
Type:
defect

Tracking

(firefox-esr91 unaffected, firefox94 unaffected, firefox95 unaffected, firefox96 fixed)

Status:
RESOLVED FIXED
Milestone:
96 Branch
Tracking Flags:
Tracking Status
firefox-esr91 --- unaffected
firefox94 --- unaffected
firefox95 --- unaffected
firefox96 --- fixed

People

(Reporter: alexandrui, Assigned: sparky)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression)

Attachments

(1 file)

Bug 1742431 - Don't do additional cropping if frame is already cropped. r?#perftest
48 bytes, text/x-phabricator-request
Details | Review

Perfherder has detected a browsertime performance regression from push 31353fc6ff8f891f3b8e9ad57d741efaa014137d. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Test Platform Options Absolute values (old vs new)
538% twitch FirstVisualChange macosx1014-64-shippable-qr cold webrender 140.00 -> 893.33
509% twitch FirstVisualChange macosx1014-64-shippable-qr cold webrender 141.67 -> 863.33
500% twitch FirstVisualChange macosx1015-64-shippable-qr warm webrender 80.00 -> 480.00
456% twitch FirstVisualChange macosx1015-64-shippable-qr cold webrender 106.67 -> 593.33
451% twitch FirstVisualChange linux1804-64-shippable-qr cold webrender 130.00 -> 716.67
444% twitch FirstVisualChange macosx1014-64-shippable-qr warm webrender 120.00 -> 653.33
418% twitch FirstVisualChange linux1804-64-shippable-qr warm webrender 110.00 -> 570.00
330% twitch FirstVisualChange windows10-64-shippable-qr warm webrender 100.83 -> 434.00
264% google-docs FirstVisualChange windows10-64-shippable-qr warm webrender 211.75 -> 771.17
236% google-docs FirstVisualChange linux1804-64-shippable-qr warm webrender 240.00 -> 806.67
... ... ... ... ...
9% nytimes FirstVisualChange windows10-64-shippable-qr cold webrender 518.38 -> 566.92
9% twitch ContentfulSpeedIndex macosx1015-64-shippable-qr cold webrender 916.67 -> 998.17
9% wikipedia FirstVisualChange android-hw-p2-8-0-android-aarch64-shippable-qr warm webrender 200.50 -> 218.33
8% google-docs ContentfulSpeedIndex windows10-64-shippable-qr cold webrender 1,666.00 -> 1,793.00
7% wikipedia ContentfulSpeedIndex android-hw-p2-8-0-android-aarch64-shippable-qr warm webrender 217.25 -> 232.83

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
18% fandom LastVisualChange macosx1014-64-shippable-qr fission warm webrender 928.33 -> 756.67
18% google-search LastVisualChange macosx1015-64-shippable-qr cold webrender 525.00 -> 430.00
17% google-search LastVisualChange windows10-64-shippable-qr cold webrender 364.83 -> 301.25
17% google LastVisualChange android-hw-g5-7-0-arm7-shippable-qr cold webrender 671.00 -> 554.50
17% fandom LastVisualChange macosx1015-64-shippable-qr fission warm webrender 696.67 -> 580.00
... ... ... ... ...
3% fandom ContentfulSpeedIndex linux1804-64-shippable-qr fission warm webrender 283.67 -> 276.58

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) will be backed out in accordance with our regression policy.

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(gmierz2)

Ah so this is because of a previous update to the vismet script which we didn't have yet: https://github.com/sitespeedio/browsertime/commit/cee2030d1ae5a5d354b41fdfa0f61b53ce36b25c#diff-9570d8f832d2eb067a17c15b66384c6515d215a4b1437c61534f6bfc5754bc9e

So it's unrelated to my changes in the vismet script but still related to my patch as I had updated the script. I'm going to look into this further - I don't like that we have a hardcoded value for what we should be ignoring for the url bar.

We can accept this change in perf numbers.

EDIT: Wait, I think something might be wrong.

Flags: needinfo?(gmierz2)

So the frames have already been cropped by this point so the url bar doesn't need to be removed (since cropping takes care of this). At this point, we are removing duplicates from the frames and this change to 40px makes us lose a portion of the test. See the following image comparison:

https://user-images.githubusercontent.com/10966989/142922604-5e572aee-bfbc-4904-be19-8494dc9e1489.png

For twitch, the issue is that the banner at the top of the page is the first thing to show up. With our previous behaviour, we were correctly catching it, but with 40px these frames get removed.

I have a patch to fix this here but I'm going to wait to see what Peter says in our github convo before making a PR: https://github.com/gmierz/browsertime/commit/dd1237f80279388d6d8f737f8690bb0e435b3e6e

The patch prevents the top from getting cut out if there's already been a cropping:

https://user-images.githubusercontent.com/10966989/142925940-23b0210b-8df4-463a-b15a-3410cac7e400.png

Set release status flags based on info from the regressing bug 1732956

status-firefox94: --- → unaffected
status-firefox95: --- → unaffected
status-firefox-esr91: --- → unaffected
Assignee: nobody → gmierz2
Status: NEW → ASSIGNED
Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/c32a2cf16642
Don't do additional cropping if frame is already cropped. r=perftest-reviewers,AlexandruIonescu

https://hg.mozilla.org/mozilla-central/rev/c32a2cf16642

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox96: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch

(In reply to Pulsebot from comment #8)

Pushed by gmierz2@outlook.com:
https://hg.mozilla.org/integration/autoland/rev/c32a2cf16642
Don't do additional cropping if frame is already cropped.
r=perftest-reviewers,AlexandruIonescu

== Change summary for alert #32469 (as of Thu, 25 Nov 2021 04:52:43 GMT) ==

Regressions:

Ratio Test Platform Options Absolute values (old vs new)
11% yahoo-mail ContentfulSpeedIndex linux1804-64-shippable-qr fission warm webrender 332.50 -> 368.08
10% google-slides LastVisualChange macosx1014-64-shippable-qr cold webrender 4,658.33 -> 5,143.33

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
30% cnn-ampstories PerceptualSpeedIndex android-hw-p2-8-0-android-aarch64-shippable-qr cold webrender 1,500.88 -> 1,053.00
10% office SpeedIndex windows10-64-shippable-qr warm webrender 540.54 -> 483.92
10% office PerceptualSpeedIndex windows10-64-shippable-qr warm webrender 563.79 -> 507.00
4% stackoverflow ContentfulSpeedIndex android-hw-g5-7-0-arm7-shippable-qr warm webrender 646.17 -> 617.83
3% imdb PerceptualSpeedIndex android-hw-g5-7-0-arm7-shippable-qr cold webrender 5,159.46 -> 5,006.50

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=32469

Regressions: 1745200
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: