1670948 - Restore nightly android build optimization to -Oz (currently -O2)

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, task)

Description

[Andrew Creskey [:acreskey]]

In Bug 1591725 we considered at optimization at -O2 instead of -Oz.

The conclusion of the experiment was that we did not have sufficient evidence of an improvement in the wild to justify the increase in binary size.

Currently we still build nightly android at -O2.

This bug is to revert the nightly back to -Oz

https://searchfox.org/mozilla-central/rev/d25eb00ab4e90cc0130cd18f303a04cc2a2f8409/old-configure.in#544-564

Comment 1

[Andrew Creskey [:acreskey]]

Attachment: Bug 1670948 - Restore nightly android build optimization to -Oz (currently -O2) r=dmajor

In Bug 1591725 we looked at trade offs between the faster -O2 and the increased binary size.
We have decided to not ship this, so restoring -Oz level optimization to nightly android.

Comment 2

[Andrew Creskey [:acreskey]]

Reverting this will regress some tests including speedometer and some of the pageload tests.

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=fb54f5b49ec4383a1499d5edd547b3734a1e41db&newProject=try&newRevision=a72674c61c42ee26161a89cf6b92a40986f98ec3&framework=10

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9a93d87ab951
Restore nightly android build optimization to -Oz (currently -O2) r=dmajor

Comment 4

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/9a93d87ab951

Comment 5

[Florin Strugariu [:Bebe]]

Comment moved to regression bug Bug 1671153 - 3.16 - 3.59% compiler_metrics num_static_constructors (android-4-0-armv7-api16, android-5-0-aarch64, android-5-0-x86_64) regression on push 9a93d87ab951182265d06fcbc59fa835efd43ad5 (Wed October 14 2020)

Comment 6

[Jeff Muizelaar [:jrmuizel]]

Andrew, do you mind elaborating on what evidence we gathered and what changes we saw or didn't see?

Comment 7

[Andrew Creskey [:acreskey]]

Jeff, yes - let me find the data science analysis of the experiment.
The tl;dr is:
We ran an A/B experiment on large Fenix Preview populations. It was tricky to do A/B with different binaries, but a lot of care and effort when into this execution.

We have data from google that says that larger APK sizes reduce usage (one reason being that people are distracted during the longer download time and never run the app...)
So that's what we're balancing against.

In the results (we looked at performance and also engagement and retention metrics) we did not see any improvements with the faster, slightly larger -O2 binary.
And I believe certain metrics were slightly in the negative territory (but not quite statistically significantly so) with this optimized binary.

Comment 8

[Andrew Creskey [:acreskey]]

This doesn't go into the decision making, but it describes the changes observed:

https://metrics.mozilla.com/~sguha/mz/ds-349/report.1.html

I forgot that we did see gfx_content_paint_time improved ~12%

Comment 9

[Andrew Creskey [:acreskey]]

Jeff. let me know if you want more details on the build flag decision making, I may be able to find more documentation with more effort.

But we were in agreement at the time, just before Fenix launch, I believe, that this increase in binary size was not worth the risk.

I don't think this should close to door to future work in this area.
In particular, the -Os optimization, for size instead of aggressively for size, showed almost the same performance improvements with less of an apk bloat.
And of course we could find other ways to reduce the binary size.

Comment 10

[Andrew Creskey [:acreskey]]

But note that it's currently quite time-consuming to experiment on user populations with different binaries.

Another issue we discovered is that our observed performance gains from users did not match the improvements that we recorded in CI or in local testing.
This is an ongoing concern, in my opinion.