Closed Bug 1353812 Opened 3 years ago Closed 3 years ago

find owner for installer size regressions in build_metrics, or stop uploading/alerts on the data

Categories

(Tree Management :: Perfherder, enhancement)

enhancement
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jmaher, Assigned: jmaher)

Details

Attachments

(1 file)

there are a lot of installer size alerts in perfherder- who specifically cares about these?

:wlach, do you have historical data that might help out?
Flags: needinfo?(wlachance)
if now owner, I would prefer to reduce our data and alerts :)
We most definitely care about this, especially on Android. :snorp would know about Android. On the desktop side I am not sure, maybe a question for firefox-dev.

Even still, we probably shouldn't alert on non-release configurations (opt on windows/linux, debug on all platforms)
Flags: needinfo?(wlachance)
:snorp, for android what is the threshold that you would like alerts (and related bugs) to be generated for?  a certain byte size or a percentage?
Flags: needinfo?(snorp)
:lmandel, do you know who on the firefox desktop side of the world cares about the installer size?
Flags: needinfo?(lmandel)
(In reply to Joel Maher ( :jmaher) from comment #3)
> :snorp, for android what is the threshold that you would like alerts (and
> related bugs) to be generated for?  a certain byte size or a percentage?

I think maybe 200k or so?
Flags: needinfo?(snorp)
(In reply to Joel Maher ( :jmaher) from comment #4)
> :lmandel, do you know who on the firefox desktop side of the world cares
> about the installer size?

Jeff - Do you have guidance on whether the product team has any requirements for installer size? (Assuming this is Firefox binary size not the stub installer.)
Flags: needinfo?(lmandel) → needinfo?(jgriffiths)
FYI the perfherder data is generated here:

http://searchfox.org/mozilla-central/rev/b8cce081200129a1307e2a682f121135b5ca1d19/testing/mozharness/mozharness/mozilla/building/buildbase.py#1990

It should be fairly easy to customize this (including whether it is emitted at all) based on platform and options.
(In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #6)
> Jeff - Do you have guidance on whether the product team has any requirements
> for installer size? (Assuming this is Firefox binary size not the stub
> installer.)

Yeah, this is for Firefox binary size, not stub installer.
(In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #6)
> (In reply to Joel Maher ( :jmaher) from comment #4)
> > :lmandel, do you know who on the firefox desktop side of the world cares
> > about the installer size?
> 
> Jeff - Do you have guidance on whether the product team has any requirements
> for installer size? (Assuming this is Firefox binary size not the stub
> installer.)

For desktop, I do not know of any specific concerns personally, needinfo'ing RT who works more directly with the installer team.

Generally, PMs who might care about this are Joe Cheng for Android ( and I agree regressions for Android are a big deal ) and Romain Testard for Desktop.
Flags: needinfo?(jgriffiths) → needinfo?(rtestard)
(In reply to Jeff Griffiths (:canuckistani) (:⚡︎) from comment #9)
> (In reply to Lawrence Mandel [:lmandel] (use needinfo) from comment #6)
> > (In reply to Joel Maher ( :jmaher) from comment #4)
> > > :lmandel, do you know who on the firefox desktop side of the world cares
> > > about the installer size?
> > 
> > Jeff - Do you have guidance on whether the product team has any requirements
> > for installer size? (Assuming this is Firefox binary size not the stub
> > installer.)
> 
> For desktop, I do not know of any specific concerns personally, needinfo'ing
> RT who works more directly with the installer team.
> 
> Generally, PMs who might care about this are Joe Cheng for Android ( and I
> agree regressions for Android are a big deal ) and Romain Testard for
> Desktop.

I've been focussing on the stub installer so far (monitoring download time during stub installation process) but my understanding is that the Firefox binary size will impact the full installer size as well as download time for stub installs - both would affect new user's onboarding. I'm happy to own this but can you point me to the place where I can get data on the binary sizes per OS (historical data would be great)? I'll follow-up with Matt (stub installer engineering owner) to find out about the following:
- Engineering owner for full installer
- Size benchmarks per OS and alert levels to be defined
Flags: needinfo?(rtestard)
Matt Howell is the engineering owner for this and he pointed me to this historical view of the installer size: https://treeherder.mozilla.org/perf.html#/graphs?timerange=31536000&series=%5Bmozilla-release,e8cc44207bb0bad4643184ce5d031b6f48a4c23f,1,2%5D

The installer although does not really have control over its own size since it really depends on the size of the build so here is what I propose:
- We do need to monitor the installer size although this feels more like a release management business as usual task to have this as a ship success criteria. I NI Sylvestre to find out if this gets monitored as part of the release process (installer alerts in perherder should maybe go to someone in relman group if he agrees)
- We need a boundary. Since historical variations in the past year have been of about 2Mb and did not seem to impact the Stub installer download cancellation rate or overall stub installer install success rate (see all stub indicators here: https://sql.telemetry.mozilla.org/dashboard/stub-installer---key-indicators), I propose we use 60Mb as upper limit (hitting that limit would probably require a decision on whether it's OK to increase the limit or make the build smaller somehow).
Flags: needinfo?(sledru)
Yeah, this should be part of the release criteria.
I contacted the relevant persons in the PI org.
Flags: needinfo?(sledru)
Android and Desktop installer sizes will be added to the list of PI monitors to track for releases. Will work with the owners to set ranges/boundaries. Thanks!
ok, so the action items for perfherder are:
Android: 200K change for an alert (about .6% currently, maybe we set it at .5% if we need percentage)
Desktop: I say 1% for alerts, that way if we do cross the 60MB threshold we can pinpoint the larger changes.

:wlach, is this something you can advise on or handle changing?
Flags: needinfo?(wlachance)
(In reply to Joel Maher ( :jmaher) from comment #14)
> ok, so the action items for perfherder are:
> Android: 200K change for an alert (about .6% currently, maybe we set it at
> .5% if we need percentage)
> Desktop: I say 1% for alerts, that way if we do cross the 60MB threshold we
> can pinpoint the larger changes.
> 
> :wlach, is this something you can advise on or handle changing?

Code that needs changing is here:

http://searchfox.org/mozilla-central/rev/4bd7a206dea5382c97a8a0c30beef668cc449f5b/testing/mozharness/mozharness/mozilla/building/buildbase.py#1991

For the desktop case, we can just alter the value. For Android, we need to change the alert type to absolute, as done here:

http://searchfox.org/mozilla-central/rev/4bd7a206dea5382c97a8a0c30beef668cc449f5b/build/util/count_ctors.py#60

We can make the output conditional on whether the packageName variable ends with .apk (which indicates Android)
Flags: needinfo?(wlachance)
I believe this should do the trick :)
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8859307 - Flags: review?(wlachance)
Comment on attachment 8859307 [details] [diff] [review]
adjust thresholds for installer_alerts

Review of attachment 8859307 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with nit addressed + a try run

While we're here, do we want to stop generating this data for debug configurations?

::: testing/mozharness/mozharness/mozilla/building/buildbase.py
@@ +2019,5 @@
>              },
>              "suites": [],
>          }
>          if installer_size or size_measurements:
> +            if installer.endswith('.apk'):

Change to:

if installer.endswith('.apk'):  # Android
Attachment #8859307 - Flags: review?(wlachance) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b9f406319da
adjust alert thresholds for android/desktop installer size. r=wlach
the only actions remaining here is to document these type of regressions and who is responsible for making decisions.  I have :snorp for Android, and possibly :sylvestre for desktop?

:sylvestre, can you confirm we should cc you on desktop installer size regressions as the point of contact?  If not, maybe you could recommend who we talk to?
Flags: needinfo?(sledru)
https://hg.mozilla.org/mozilla-central/rev/8b9f406319da
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Joel, I think the release owner (release manager) should make the call. 
Ritu and I can help to make a call if needed.
(if  you need just one person, you can put Ritu or me)
Flags: needinfo?(sledru)
You need to log in before you can comment on or make changes to this bug.