1600838 - Make conditioned profiles for desktop Firefox the default in Raptor and Browsertime

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Let's make conditioned profiles the default for Firefox, for Raptor and Browsertime.

GeckoView-based Android support is still being worked on in bug 1597711, and might still block this bug, partially (depending on how cleanly/permanently we wish to switch conditioned profiles to the default).

This will involve, among other things (which will be filed separately, and set to either blocking this bug or bug 1596491), changes to the following:

taskcluster/ci/test/raptor.yml

• remove condprof: True lines for raptor-tp6-1-firefox - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/taskcluster/ci/test/raptor.yml#50
• remove condprof: True lines for raptor-tp6-1-firefox-cold - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/taskcluster/ci/test/raptor.yml#690
• same for browsertime-tp6-1 (warm and cold variants) - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/taskcluster/ci/test/raptor.yml#1008

taskcluster/taskgraph/transforms/raptor.py

• remove build_condprof_tests - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/taskcluster/taskgraph/transforms/raptor.py#171-196

testing/mozharness/mozharness/mozilla/testing/raptor.py

• remove with-conditioned-profile flag from config_options - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/mozharness/mozharness/mozilla/testing/raptor.py#136-140
• remove the same for options_extend - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/mozharness/mozharness/mozilla/testing/raptor.py#499-500

testing/raptor/raptor/cmdline.py

• remove arg from create_parser - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/cmdline.py#143-145

testing/raptor/raptor/raptor.py

• remove with_conditioned_profile variable in init - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/raptor.py#109
• remove with_conditioned_profile mapping/assignment in self.config - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/raptor.py#135
• hard-code self.post_startup_delay (remove conditional which listens for with_conditioned_profile cmd-line arg) - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/raptor.py#172-175
• excepting win64-aarch64 (Windows 10), remove conditional in build_browser_profile - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/raptor.py#210
• remove with_conditioned_profile args from raptor_class - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/raptor.py#1777
• remove with_conditioned_profile conditional check for Browsertime - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/raptor.py#694-695

testing/raptor/raptor/results.py

• remove "condprof" annotations from RaptorResultsHandler - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/results.py#178-179
• remove appending "condprof" to extra_options in new_pageload_results - https://searchfox.org/mozilla-central/rev/04d8e7629354bab9e6a285183e763410860c5006/testing/raptor/raptor/results.py#178-179

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

The current patch approach (https://searchfox.org/mozilla-central/rev/8bc24752246aeac8a9aed566cf1caccf88d97d11/taskcluster/taskgraph/transforms/raptor.py#171-199) still breaks an Android 7 power unittest , aarch64, and a unittest for desktop Firefox.

This is because it removes build_condprof_tests (which keyed off of the condprof: True flag, and thus, gracefully-degraded on both platforms, before. Judging by an earlier test push to try, Chrome might be OK?

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

Latest try push here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=62ce6450e928821591b7340f88ba68a54801b430; still need to work out Android and better Windows 10 support/fixes for https://treeherder.mozilla.org/#/jobs?repo=try&revision=62ce6450e928821591b7340f88ba68a54801b430&selectedJob=279508523.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Bug 1600838 - Make conditioned profiles for desktop Firefox the default, in Raptor and Browsertime. r?tarek,#perftest

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

Latest push to try here with "only a couple" unittests failures I've introduced; will fix next: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=e3d045bd0422fb246a0c1bca73244566cc0ce967

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

Latest try push -- which incorporates feedback from :sparky (thanks!) -- still has failing unittests, but that's a separate issue, which I hope will be fixed by bug 1601759, so we can land the missing-piece rename in bug 1599471 can successfully use the updated API's ROOT_URL, etc.: from bug 1601197:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0005269cab40f5bff0dbe1323a9bd1f9bbfe8ec7

Currently, there is a mismatch in the conditioned-profile's tarball name and location, like so:

• unittests fail with "cold"; they want "settled": https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&author=sdonner%40mozilla.com&selectedJob=280048755
  • [task 2019-12-06T22:59:06.280Z] 0:27.01 [2019-12-06T22:59:05.265880] Downloading https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-linux64/artifacts/public/condprof/profile-linux64-cold-default.tgz
  • ../../../condprofile/condprof/client.py:94: ProfileNotFoundError; with "settled", we're fine again: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.firefox.condprof-linux64/artifacts/public/condprof/profile-linux64-settled-default.tgz
• raptor-tp6-1-cold on try server (and locally) still uses https://index.taskcluster.net/v1/task/gecko.v2.try.latest.firefox.condprof-macosx64/artifacts/public/condprof/profile-macosx64-cold-default.tgz which returns a 200 OK (and thus, "works")
• (bug 1599471) we need to rename it to "settled" in our get_conditioned_profile caller in raptor.py, so we're using the correct, fresh set of conditioned profiles
• as of yet, "settled" doesn't exist in TaskCluster: [task 2019-12-06T11:14:16.976Z] 11:14:16 INFO - Getting https://index.taskcluster.net/v1/task/gecko.v2.try.latest.firefox.condprof-macosx64/artifacts/public/condprof/profile-macosx64-settled-default.tgz
  • but we want it to either be https://index.taskcluster.net/v1/task/gecko.v2.try.latest.firefox.condprof-macosx64/artifacts/public/condprof/profile-macosx64-cold-default.tgz (which works) or the platform-specific firefox-ci-tc.services.mozilla.com URL?

Comment 8

[Stephen Donner [:stephend] Not actively reading bugmail]

Try pushes:

• desktop Raptor and Browsertime, plus Raptor mobile: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=faa1a850c2ba4e315c4f0fff59b704f97158763d
• Browsertime on mobile, here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=c334f2b058a1fbc24e6a13e3794c94a7d4c167d7

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/86faf1cf452a
Make conditioned profiles for desktop Firefox the default, in Raptor and Browsertime.  r=tarek,perftest-reviewers,sparky

Comment 10

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/86faf1cf452a