Closed Bug 1727780 Opened 3 years ago Closed 3 years ago

upgrade version of browsertime and imagemagick

Categories

(Testing :: Performance, task, P1)

Default
task

Tracking

(firefox93 fixed)

RESOLVED FIXED
93 Branch
Tracking Status
firefox93 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

References

Details

(Keywords: perf-alert)

Attachments

(1 file)

No description provided.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Blocks: 1713815

as a note, the upgrade was to use features for responsiveness testing, then there was a small issue with our benchmarks that do not have paintTiming metrics and that was fixed in:
https://github.com/sitespeedio/browsertime/pull/1627

Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2c24c285ed0a
bump version of browsertime and imagemagick. r=perftest-reviewers,AlexandruIonescu
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 93 Branch

Backed out changeset 2c24c285ed0a (Bug 1727780) for breaking browsertime tasks (new yargs version requires node 12 but version 10 is used).
Backout link
Failure Log

Flags: needinfo?(jmaher)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: 93 Branch → ---
Priority: -- → P1

The browsertime tasks still fail for fenix - they might pull artifacts which got built when this change landed and didn't get rebuilt for the backout because previous artifacts could be reused.

I have submitted:
https://github.com/mozilla-mobile/fenix/pull/21064

ideally that will get reviewed and merged in today, then we can see if it fixes stuff. I will be on PTO until Friday September 3rd.

If this doesn't resolve the problem I see 2 choices:

  1. live with no fenix results for a few more days or possibly another week (if I cannot fix this Friday)
  2. somebody else fixes what is broken

For the future, we need this documented better- how to run the tests, what binaries are used and how they are updated. After spending ~4 hours on this, I haven't found anybody online today that knows how this works- some great pointers and tips. I had asked almost a week ago in #perftest what is needed for upgrading browsertime and the general consensus was run everything on try- it appears we need a bit more than that- and running 2000+ jobs on try takes a lot of time from our limited pool.

:davehunt - ni you for documentation and finding someone to fix this if it isn't fixed whilst i am on PTO. Ideally if fenix is working we can reland this patch which would be a prerequisite to responsiveness tests.

Flags: needinfo?(jmaher) → needinfo?(dave.hunt)
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c3499484e459
bump version of browsertime and imagemagick. r=perftest-reviewers,AlexandruIonescu
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 93 Branch
Regressions: 1728739

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #7)

:davehunt - ni you for documentation and finding someone to fix this if it isn't fixed whilst i am on PTO. Ideally if fenix is working we can reland this patch which would be a prerequisite to responsiveness tests.

Thanks :jmaher, I've opened bug 1729272 to improvement our documentation.

Flags: needinfo?(dave.hunt)

(In reply to Marian-Vasile Laza from comment #5)

Backed out changeset 2c24c285ed0a (Bug 1727780) for breaking browsertime tasks (new yargs version requires node 12 but version 10 is used).
Backout link
Failure Log

== Change summary for alert #31246 (as of Fri, 10 Sep 2021 08:26:37 GMT) ==

Regressions:

Ratio Test Platform Options Absolute values (old vs new)
7% google-slides ContentfulSpeedIndex macosx1014-64-shippable-qr cold webrender 1,781.58 -> 1,912.42
7% google-slides ContentfulSpeedIndex macosx1014-64-shippable-qr cold webrender 1,787.21 -> 1,911.33
6% google-slides loadtime macosx1014-64-shippable-qr cold webrender 6,140.72 -> 6,517.12

Improvements:

Ratio Test Platform Options Absolute values (old vs new)
4% google-slides loadtime macosx1014-64-shippable-qr warm webrender 1,452.17 -> 1,395.21

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=31246

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: