1594433 - Compare performance of Fenix with Glean enabled/disabled

Status: RESOLVED WONTFIX

(Data Platform and Tools :: Glean: SDK, task, P3)

Description

[Michael Droettboom [:mdroettboom]]

As a rough measurement of the impact of telemetry on performance, it would be worthwhile to measure the impact on memory/bandwidth/battery life when Glean upload is enabled vs. disabled.

Maybe Nimbledroid can be used for this purpose.

Comment 1

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Like https://bugzilla.mozilla.org/show_bug.cgi?id=1594431#c3, my FE performance team would love to help you with this. Please let me know how we can be of assistance. :)

Comment 2

[Alessio Placitelli [:Dexter]]

(In reply to Michael Comella (:mcomella) [needinfo or I won't see it] from comment #1)

Like https://bugzilla.mozilla.org/show_bug.cgi?id=1594431#c3, my FE performance team would love to help you with this. Please let me know how we can be of assistance. :)

Thanks (I also responded in the other bug :) )!

For this specific bug it might be a bit trickier in Nimbledroid: I imagine that Fenix builds uploaded there have upload enabled. In order to get some sensible measures out, for the purpose of this bug, we'd need to have some uploded builds with Glean upload off.

Is there any alternative way to run these tests that would not require us to upload custom builds?

With respect to Nimbledroid, how does it lunch the Fenix app? How often does it test? Since upload is enabled there, pings will show up in the Fenix tables, probably... unless Nimbledroid blocks network.

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Is there any alternative way to run these tests that would not require us to upload custom builds?

how does it lunch the Fenix app?

Custom performance tests in Nimbledroid are just Android UI tests (e.g. Espresso) that have a "start" and "end" log statement: therefore, you can write a custom startup test that starts the Activity with an Intent that disables upload. It's not perfect, given the impact on perf for extra conditional code, but maybe it's good enough? Here are their docs on writing custom tests.

How often does it test?

It runs the test suite once per upload. We're currently uploading every Nightly build and intend to upload every Release and Beta build. Depending on your goals, we can add your tests to the test suite – to prevent regressions – or run it once off with a manual upload – to do a once off comparison. The downside to adding it to the test suite is that it increases the overall runtime but, afaik, it will not otherwise effect our capacity to run tests on ND.

Since upload is enabled there, pings will show up in the Fenix tables, probably... unless Nimbledroid blocks network.

They do not block the network – we previously ran page load tests on ND.

I believe each test is run 10 times during testing: this seems like a problem for our telemetry numbers. Do you agree? We should be able to send an Intent when starting the app to disable, or modify, the Glean upload: is this desirable? Caveat: we'll be unable to use Nimbledroid's default cold startup test but that might be okay.

it would be worthwhile to measure the impact on memory/bandwidth/battery life when Glean upload is enabled vs. disabled.

Out of curiosity, is there a particular code path you'd want to test, for example startup or page load?

Comment 4

[Alessio Placitelli [:Dexter]]

(In reply to Michael Comella (:mcomella) [needinfo or I won't see it] from comment #3)

Is there any alternative way to run these tests that would not require us to upload custom builds?

how does it lunch the Fenix app?

Custom performance tests in Nimbledroid are just Android UI tests (e.g. Espresso) that have a "start" and "end" log statement: therefore, you can write a custom startup test that starts the Activity with an Intent that disables upload. It's not perfect, given the impact on perf for extra conditional code, but maybe it's good enough? Here are their docs on writing custom tests.

What do you mean with given the impact on perf for extra conditional code? Are you referring to the check in glean for "is upload enabled"?

That sounds like a good way to check upload enabled vs upload disabled, which is the subject of this bug. We could flip the preference that Fenix uses to call Glean.setUploadEnabled(true|false) before launching the Fenix activity.

Where do current Fenix performance tests live?

How often does it test?

It runs the test suite once per upload. We're currently uploading every Nightly build and intend to upload every Release and Beta build. Depending on your goals, we can add your tests to the test suite – to prevent regressions – or run it once off with a manual upload – to do a once off comparison. The downside to adding it to the test suite is that it increases the overall runtime but, afaik, it will not otherwise effect our capacity to run tests on ND.

Since upload is enabled there, pings will show up in the Fenix tables, probably... unless Nimbledroid blocks network.

They do not block the network – we previously ran page load tests on ND.

I believe each test is run 10 times during testing: this seems like a problem for our telemetry numbers. Do you agree? We should be able to send an Intent when starting the app to disable, or modify, the Glean upload: is this desirable? Caveat: we'll be unable to use Nimbledroid's default cold startup test but that might be okay.

That depends. How many tests are run each time? Is storage cleared between the tests?

There are two approaches, I think, we could follow:

• Best option: Is the Fenix build used for ND using the same application id as the one on the play store, i.e. org.mozilla.fenix? If so, would you consider using a build with a different application id? e.g. org.mozilla.fenix.perf.nd? This would guarantee, without changing anything in the code, that the telemetry goes to a different place, without polluting other data. Moreover, this would allow to analyse the performance telemetry that we get from both ND and Glean.

• The debug activity could be used to launch Fenix, tagging pings so that they go to the debug view and can be told apart from normal pings. This is quirky as it pollutes the dashboard used by QA. The first option is cleaner and does not require any hack at all.

it would be worthwhile to measure the impact on memory/bandwidth/battery life when Glean upload is enabled vs. disabled.

Out of curiosity, is there a particular code path you'd want to test, for example startup or page load?

I'd be interested in seeing what's the impact of performing telemetry measurements during a normal user workflow, e.g. open a website, navigate, close it, do it again. On Desktop there was talos, which opened N local copies of websites (so that tests were not network dependent) and you could control for other factors, including telemetry.

Gecko, in Fenix, performs some measurements on page loads, so the above scenario would be interesting.

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Alessio Placitelli [:Dexter] from comment #4)

What do you mean with given the impact on perf for extra conditional code? Are you referring to the check in glean for "is upload enabled"?

Yes.

Where do current Fenix performance tests live?

As far as I'm aware, we currently run:

• The default Nimbledroid cold startup test. Since ND creates it themselves, we haven't figured out where to store the code for custom tests yet. We'll do that in https://github.com/mozilla-mobile/fenix/issues/6508 (a high priority issue)
• Page load tests on Raptor. The test infrastructure is owned by dhunt's team and tbh I'm not sure where the code lives

That depends. How many tests are run each time? Is storage cleared between the tests?
There are two approaches, I think, we could follow:

Currently, just one test. I'm not sure if storage is cleared between tests. I filed https://github.com/mozilla-mobile/fenix/issues/6727 for disabling telemetry uploads during performance testing: let's continue the discussion there.

Out of curiosity, is there a particular code path you'd want to test, for example startup or page load?

I'd be interested in seeing...

I filed https://github.com/mozilla-mobile/fenix/issues/6728 to track this for our own prioritization. It'll get triaged on Monday: I'll let you know how it goes.

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

FYI: the Android FE perf group is currently focused on improving Fenix startup performance so we may not be able to get to this soon. "As a rough measurement of the impact of telemetry on performance," is a valuable idea so, after we've made improvements to essential use cases like startup, this potentially sounds like something for us to prioritize.

NI Alessio for context but no action needed.

Comment 7

[Alessio Placitelli [:Dexter]]

(In reply to Michael Comella (:mcomella) [needinfo or I won't see it] from comment #6)

FYI: the Android FE perf group is currently focused on improving Fenix startup performance so we may not be able to get to this soon. "As a rough measurement of the impact of telemetry on performance," is a valuable idea so, after we've made improvements to essential use cases like startup, this potentially sounds like something for us to prioritize.

NI Alessio for context but no action needed.

Hey Michael,

thank you for following up and helping with the other analyses. I think that's fair, and there's no urgency on this.