1708760 - Use in-task prebuilt profiles in our browsertime tests

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Greg Mierzwinski [:sparky]]

This bug is for building a method in raptor for making a prebuilt conditioned profile during the test (or in-task) rather than using the m-c built conditioned profiles.

Comment 1

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1708760 - Build conditioned profiles during the tests. r?#perftest

This patch adds the ability to build conditioned profiles during a test (i.e. in-test, prebuilt, or in-task for CI). Using the test-built: prefix in --conditioned-profile will cause the conditioned profile to be produced locally rather than downloaded from mozilla-central.

The condprof package was also modified to prevent it from building and saving logs that are not very useful in local runs, preventing useless downloads or archiving, and to also be able to obtain the location of the test-built profile.

Comment 2

[Greg Mierzwinski [:sparky]]

Attachment: Bug 1708760 - Add a pageload test for about:welcome. r?#perftest

Depends on D116520

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5e7be02a8487
Build conditioned profiles during the tests. r=perftest-reviewers,AlexandruIonescu
https://hg.mozilla.org/integration/autoland/rev/828dd9f79646
Add a pageload test for about:welcome. r=perftest-reviewers,AlexandruIonescu

Comment 4

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/5e7be02a8487
https://hg.mozilla.org/mozilla-central/rev/828dd9f79646

Comment 5

[Jim Mathies [:jimm]]

Hey Dave, we noticed the profile change helped our numbers for GeckoView, but didn't impact Fenix. Any idea why we didn't get the improvement there as well?

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=fenix,3491467,1,13&series=autoland,3388673,1,13&timerange=5184000&zoom=1622994230493,1624041026588,729.9385257995207,1663.3725127276252

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Jim Mathies [:jimm] from comment #5)

Hey Dave, we noticed the profile change helped our numbers for GeckoView, but didn't impact Fenix. Any idea why we didn't get the improvement there as well?

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=fenix,3491467,1,13&series=autoland,3388673,1,13&timerange=5184000&zoom=1622994230493,1624041026588,729.9385257995207,1663.3725127276252

Looking at a log for a recent Fenix job it doesn't appear that we're building a conditioned profile. This could be because the Fenix project hasn't been updated to use one.

I imagine we would just need to add --conditioned-profile=settled to https://github.com/mozilla-mobile/fenix/blob/main/taskcluster/fenix_taskgraph/transforms/browsertime.py. Perhaps :mcomella could help with this?

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Looking at a log for a recent Fenix job it doesn't appear that we're building a conditioned profile. This could be because the Fenix project hasn't been updated to use one.

I imagine we would just need to add --conditioned-profile=settled to https://github.com/mozilla-mobile/fenix/blob/main/taskcluster/fenix_taskgraph/transforms/browsertime.py. Perhaps :mcomella could help with this?

We intentionally disabled condition profiles in bug 1665440 for fenix perf start up tasks because it was causing an unexplained large slowdown in the results that I was concerned was introducing noise and making the tests less reliable. I'm not sure what we're doing outside of the start up tests.

I think reintroducing conditioned profiles would require an analysis to ensure it's actually benefiting the results. I'm hesitant to say we should enable conditioned profiles because we don't have anyone to invest the time in the effort.

Does that seem reasonable, Jim? I don't know too much about the benefits of conditioned profiles or the other tests that could be using them to feel entirely confident in my thinking.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Michael Comella (:mcomella) [needinfo or I won't see it] from comment #7)

Looking at a log for a recent Fenix job it doesn't appear that we're building a conditioned profile. This could be because the Fenix project hasn't been updated to use one.

I imagine we would just need to add --conditioned-profile=settled to https://github.com/mozilla-mobile/fenix/blob/main/taskcluster/fenix_taskgraph/transforms/browsertime.py. Perhaps :mcomella could help with this?

We intentionally disabled condition profiles in bug 1665440 for fenix perf start up tasks because it was causing an unexplained large slowdown in the results that I was concerned was introducing noise and making the tests less reliable. I'm not sure what we're doing outside of the start up tests.

I think reintroducing conditioned profiles would require an analysis to ensure it's actually benefiting the results. I'm hesitant to say we should enable conditioned profiles because we don't have anyone to invest the time in the effort.

Does that seem reasonable, Jim? I don't know too much about the benefits of conditioned profiles or the other tests that could be using them to feel entirely confident in my thinking.

I wonder if building the profiles in the same task will help, but agree that we should hold off until we have time to perform further analysis.

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I think reintroducing conditioned profiles would require an analysis to ensure it's actually benefiting the results. I'm hesitant to say we should enable conditioned profiles because we don't have anyone to invest the time in the effort.

We're enabling conditioned profiles in fenix#21290 because it's difficult to compare perf with geckoview_example and GVE can be used as an important baseline to know if fenix is doing things we can optimize out. See fenix#20936 for details.

There may be some side effects (noise?) but I think it's worth pursuing for the optimization identification benefits.