Closed Bug 1721640 Opened 2 months ago Closed 2 months ago

Re-record tp6 test for ebay

Categories

(Testing :: Performance, task, P1)

Default
task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kimberlythegeek, Assigned: kimberlythegeek)

References

(Blocks 1 open bug)

Details

(Keywords: perf-alert)

Attachments

(2 files, 2 obsolete files)

ebay tp6 is one of the failing tests in intermittent failure Bug 1680214 -- generating a new page recording may resolve this failure

Priority: -- → P1
Attachment #9232434 - Attachment description: Bug 1721640: Re-record tp6 test for ebay → WIP: Bug 1721640: Re-record tp6 test for ebay
Attachment #9232434 - Attachment description: WIP: Bug 1721640: Re-record tp6 test for ebay → Bug 1721640: Re-record tp6 test for ebay
Pushed by ksereduck@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/dc28d43a2814
Re-record tp6 test for ebay r=perftest-reviewers,Bebe

Backed out changeset dc28d43a2814 (Bug 1721640) for causing ebay Btime raptor failures.
Backout link
Push with failures
Failure Log

Flags: needinfo?(ksereduck)
Attachment #9232434 - Attachment description: Bug 1721640: Re-record tp6 test for ebay → WIP: Bug 1721640: Re-record tp6 test for ebay
Flags: needinfo?(ksereduck)
Pushed by ksereduck@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5d2f7222ca3f
WIP: Bug 1721640: Re-record tp6 test for ebay r=perftest-reviewers,Bebe
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED

== Change summary for alert #30714 (as of Fri, 30 Jul 2021 09:02:56 GMT) ==

Regressions:

Ratio Suite Test Platform Options Absolute values (old vs new)
407% ebay SpeedIndex windows10-64-shippable-qr warm webrender 451.42 -> 2,286.75
293% ebay LastVisualChange windows10-64-shippable-qr warm webrender 1,105.58 -> 4,349.08
234% ebay PerceptualSpeedIndex windows10-64-shippable-qr warm webrender 443.83 -> 1,483.00
20% ebay SpeedIndex macosx1015-64-shippable-qr warm webrender 2,047.83 -> 2,453.08
18% ebay SpeedIndex macosx1014-64-shippable-qr warm webrender 2,204.04 -> 2,593.67
17% ebay SpeedIndex linux1804-64-shippable-qr warm webrender 2,011.08 -> 2,356.83
17% ebay SpeedIndex linux1804-64-shippable-qr warm webrender 2,010.33 -> 2,352.33
13% ebay PerceptualSpeedIndex macosx1015-64-shippable-qr warm webrender 1,411.83 -> 1,598.83
11% ebay SpeedIndex linux1804-64-shippable-qr cold webrender 2,521.08 -> 2,797.67
11% ebay SpeedIndex macosx1015-64-shippable-qr cold webrender 2,607.25 -> 2,887.00
10% ebay PerceptualSpeedIndex linux1804-64-shippable-qr warm webrender 1,379.92 -> 1,515.50

Improvements:

Ratio Suite Test Platform Options Absolute values (old vs new)
35% ebay ContentfulSpeedIndex macosx1014-64-shippable-qr warm webrender 668.25 -> 432.50
31% ebay ContentfulSpeedIndex macosx1015-64-shippable-qr warm webrender 584.83 -> 406.08
26% ebay ContentfulSpeedIndex linux1804-64-shippable-qr warm webrender 534.08 -> 393.08
21% ebay ContentfulSpeedIndex macosx1015-64-shippable-qr cold webrender 1,089.92 -> 863.42
20% ebay ContentfulSpeedIndex linux1804-64-shippable-qr cold webrender 976.75 -> 782.75
... ... ... ... ... ...
3% ebay loadtime windows10-64-shippable-qr warm webrender 358.62 -> 347.17

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=30714

Flags: needinfo?(ksereduck)
Flags: needinfo?(fstrugariu)
Attachment #9233968 - Attachment is obsolete: true
Status: RESOLVED → REOPENED
Flags: needinfo?(ksereduck)
Resolution: FIXED → ---
Attachment #9234239 - Attachment is obsolete: true

This is a re-recording, alert should be marked as wontfix and we should accept these measurements as the new baseline

Flags: needinfo?(fstrugariu)
Status: REOPENED → RESOLVED
Closed: 2 months ago2 months ago
Resolution: --- → FIXED
Attachment #9232434 - Attachment description: WIP: Bug 1721640: Re-record tp6 test for ebay → Bug 1721640: Re-record tp6 test for ebay
You need to log in before you can comment on or make changes to this bug.