Closed Bug 1832246 Opened 2 years ago Closed 2 years ago

624.02 - 6.27% reddit-billgates-post-1.posts LastVisualChange / reddit-billgates-post-2.hot LastVisualChange + 6 more (OSX, Windows) regression on Thu May 4 2023

Categories

(Testing :: Raptor, defect, P5)

Product:
Testing
Component:
Raptor
Platform:
Desktop
All
Type:
defect

Tracking

(firefox-esr102 unaffected, firefox113 unaffected, firefox114 fixed, firefox115 fixed)

Status:
RESOLVED FIXED
Milestone:
115 Branch
Tracking Flags:
Tracking Status
firefox-esr102 --- unaffected
firefox113 --- unaffected
firefox114 --- fixed
firefox115 --- fixed

People

(Reporter: afinder, Assigned: kshampur)

References

(Regression)

Details

(Keywords: perf, perf-alert, regression)

Attachments

(1 file)

Bug 1832246 - Set beforePageCompleteWaitTime to 2 seconds. r?#perftest
48 bytes, patch
pascalc
: approval-mozilla-beta+
Details | Diff | Splinter Review

Perfherder has detected a browsertime performance regression from push 7bdbd5815ea7dccfbb38623001ff92258d639804. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
624% reddit-billgates-post-1.posts LastVisualChange windows10-64-shippable-qr cold fission webrender 555.33 -> 4,020.71
567% reddit-billgates-post-1.posts LastVisualChange macosx1015-64-shippable-qr cold fission webrender 608.33 -> 4,056.67
471% reddit-billgates-ama.members LastVisualChange macosx1015-64-shippable-qr cold fission webrender 728.33 -> 4,160.00
452% reddit-billgates-post-1.posts LastVisualChange linux1804-64-shippable-qr cold fission webrender 756.67 -> 4,173.33
386% reddit-billgates-ama.members LastVisualChange linux1804-64-shippable-qr cold fission webrender 893.33 -> 4,343.33
369% reddit-billgates-ama.members LastVisualChange windows10-64-shippable-qr cold fission webrender 930.46 -> 4,362.29
32% reddit-billgates-post-1.posts SpeedIndex linux1804-64-shippable-qr cold fission webrender 137.46 -> 181.62
24% reddit-billgates-post-1.posts SpeedIndex windows10-64-shippable-qr cold fission webrender 118.23 -> 146.17
14% reddit-billgates-ama.members SpeedIndex windows10-64-shippable-qr cold fission webrender 381.40 -> 432.88
10% reddit-billgates-ama.members SpeedIndex linux1804-64-shippable-qr cold fission webrender 324.96 -> 355.92
9% reddit-billgates-ama.members SpeedIndex macosx1015-64-shippable-qr cold fission webrender 345.08 -> 375.29
6% reddit-billgates-post-2.hot LastVisualChange windows10-64-shippable-qr cold fission webrender 473.08 -> 502.75
4% office loadtime linux1804-64-shippable-qr fission warm webrender 428.54 -> 446.17

Improvements:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
19% twitch fcp linux1804-64-shippable-qr fission warm webrender 64.33 -> 51.88
7% facebook LastVisualChange linux1804-64-shippable-qr fission warm webrender 1,061.67 -> 983.33
3% facebook LastVisualChange windows10-64-shippable-qr fission warm webrender 923.75 -> 900.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

Flags: needinfo?(kshampur)

Set release status flags based on info from the regressing bug 1823730

status-firefox113: --- → unaffected
status-firefox114: --- → affected
status-firefox115: --- → affected
status-firefox-esr102: --- → unaffected

to summarize matrix discussion, this is very likely the culprit https://github.com/sitespeedio/browsertime/commit/ee96cfaa12aaa21b4e835014c9b2b7d3dadf9a7a

have not decided if we should use that flag and set it to 2k or leave as is the new default

Severity: -- → S4
Priority: -- → P5

(In reply to Alex Finder|PTO on May 12, returning May 15th from comment #0)

Perfherder has detected a browsertime performance regression from push 7bdbd5815ea7dccfbb38623001ff92258d639804. As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
624% reddit-billgates-post-1.posts LastVisualChange windows10-64-shippable-qr cold fission webrender 555.33 -> 4,020.71
567% reddit-billgates-post-1.posts LastVisualChange macosx1015-64-shippable-qr cold fission webrender 608.33 -> 4,056.67
471% reddit-billgates-ama.members LastVisualChange macosx1015-64-shippable-qr cold fission webrender 728.33 -> 4,160.00
452% reddit-billgates-post-1.posts LastVisualChange linux1804-64-shippable-qr cold fission webrender 756.67 -> 4,173.33
386% reddit-billgates-ama.members LastVisualChange linux1804-64-shippable-qr cold fission webrender 893.33 -> 4,343.33
369% reddit-billgates-ama.members LastVisualChange windows10-64-shippable-qr cold fission webrender 930.46 -> 4,362.29
32% reddit-billgates-post-1.posts SpeedIndex linux1804-64-shippable-qr cold fission webrender 137.46 -> 181.62
24% reddit-billgates-post-1.posts SpeedIndex windows10-64-shippable-qr cold fission webrender 118.23 -> 146.17
14% reddit-billgates-ama.members SpeedIndex windows10-64-shippable-qr cold fission webrender 381.40 -> 432.88
10% reddit-billgates-ama.members SpeedIndex linux1804-64-shippable-qr cold fission webrender 324.96 -> 355.92
9% reddit-billgates-ama.members SpeedIndex macosx1015-64-shippable-qr cold fission webrender 345.08 -> 375.29
6% reddit-billgates-post-2.hot LastVisualChange windows10-64-shippable-qr cold fission webrender 473.08 -> 502.75
4% office loadtime linux1804-64-shippable-qr fission warm webrender 428.54 -> 446.17

Improvements:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
19% twitch fcp linux1804-64-shippable-qr fission warm webrender 64.33 -> 51.88
7% facebook LastVisualChange linux1804-64-shippable-qr fission warm webrender 1,061.67 -> 983.33
3% facebook LastVisualChange windows10-64-shippable-qr fission warm webrender 923.75 -> 900.00

Details of the alert can be found in the alert summary, including links to graphs and comparisons for each of the affected tests. Please follow our guide to handling regression bugs and let us know your plans within 3 business days, or the offending patch(es) may be backed out in accordance with our regression policy.

If you need the profiling jobs you can trigger them yourself from treeherder job view or ask a sheriff to do that for you.

For more information on performance sheriffing please see our FAQ.

== Change summary for alert #38296 (as of Mon, 08 May 2023 05:46:30 GMT) ==

Regressions:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
4% office fcp linux1804-64-shippable-qr fission warm webrender 406.75 -> 424.88 Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=38296

A recent change in browsertime changed the default value of beforePageCompleteWaitTime from 2s to 5s which caused some regressions in our interactive tests.
This patch now explicitly sets the value to 2s for consistency and also taking into consideration that if we instead continued with the new default of 5s,
we may miss regressions in the future due to our 2% alerting threshold

Assignee: nobody → kshampur
Status: NEW → ASSIGNED
Flags: needinfo?(kshampur)
Pushed by kshampur@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/9935d6652455 Set beforePageCompleteWaitTime to 2 seconds. r=perftest-reviewers,AlexandruIonescu

https://hg.mozilla.org/mozilla-central/rev/9935d6652455

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
status-firefox115: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → 115 Branch

The patch landed in nightly and beta is affected.
:kshampur, is this bug important enough to require an uplift?

  • If yes, please nominate the patch for beta approval.
  • If no, please set status-firefox114 to wontfix.

For more information, please visit BugBot documentation.

Flags: needinfo?(kshampur)

Comment on attachment 9333119 [details] [diff] [review]
Bug 1832246 - Set beforePageCompleteWaitTime to 2 seconds. r?#perftest

Beta/Release Uplift Approval Request

  • User impact if declined: none
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): does not affect user, only perf tests
  • String changes made/needed:
  • Is Android affected?: No
Attachment #9333119 - Attachment is patch: true
Attachment #9333119 - Attachment mime type: text/x-phabricator-request → text/plain
Flags: needinfo?(kshampur)
Attachment #9333119 - Flags: approval-mozilla-beta?

Comment on attachment 9333119 [details] [diff] [review]
Bug 1832246 - Set beforePageCompleteWaitTime to 2 seconds. r?#perftest

Approved for 114 beta 5, thanks.

Attachment #9333119 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

(In reply to Pulsebot from comment #5)

Pushed by kshampur@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9935d6652455
Set beforePageCompleteWaitTime to 2 seconds.
r=perftest-reviewers,AlexandruIonescu

== Change summary for alert #38340 (as of Sun, 14 May 2023 22:42:59 GMT) ==

Regressions:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
16% twitch fcp linux1804-64-shippable-qr fission warm webrender 52.23 -> 60.38
6% youtube fcp linux1804-64-shippable-qr fission warm webrender 215.21 -> 227.12 Before/After
3% youtube fcp windows10-64-shippable-qr fission warm webrender 194.79 -> 199.83 Before/After

Improvements:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
86% reddit-billgates-post-1.posts LastVisualChange windows10-64-shippable-qr cold fission webrender 4,020.71 -> 558.00
85% reddit-billgates-post-1.posts LastVisualChange macosx1015-64-shippable-qr cold fission webrender 4,046.67 -> 601.67
83% reddit-billgates-ama.members LastVisualChange macosx1015-64-shippable-qr cold fission webrender 4,170.00 -> 715.00
82% reddit-billgates-post-1.posts LastVisualChange linux1804-64-shippable-qr cold fission webrender 4,195.00 -> 765.00
79% reddit-billgates-ama.members LastVisualChange linux1804-64-shippable-qr cold fission webrender 4,348.33 -> 895.00
... ... ... ... ... ...
2% facebook loadtime macosx1015-64-shippable-qr fission warm webrender 201.81 -> 197.58

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=38340

== Change summary for alert #38379 (as of Fri, 19 May 2023 11:15:37 GMT) ==

Regressions:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
27% twitch fcp linux1804-64-shippable-qr fission warm webrender 49.26 -> 62.79 Before/After

Improvements:

Ratio Test Platform Options Absolute values (old vs new) Performance Profiles
21% google-slides loadtime linux1804-64-shippable-qr fission warm webrender 1,893.81 -> 1,503.50 Before/After
6% google-slides LastVisualChange linux1804-64-shippable-qr fission warm webrender 2,463.33 -> 2,310.00 Before/After
4% google-slides ContentfulSpeedIndex linux1804-64-shippable-qr fission warm webrender 758.08 -> 727.92 Before/After
3% imgur fcp macosx1015-64-shippable-qr fission warm webrender 396.47 -> 385.67 Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=38379

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: