Closed Bug 1313408 Opened 8 years ago Closed 8 years ago

Performance Telemetry for SDK patches to Beta 50

Categories

(Firefox :: Extension Compatibility, defect, P1)

Product:
Firefox
Component:
Extension Compatibility
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: shell, Assigned: dzeber)

References

Details

(Whiteboard: triaged)

Why: ~information to decide if the performance improvement from 2 patches is big enough to merit delaying 50 to include. ~need more than anecdotal evidence (andy’s vidyo of VHD and many, notes in bug) ~how much worse is it in e10s ~how broadly the impact is seen in users without add-ons, even users without any add-ons saw some improvement Targets: ~Beta 9 & 10 (from Oct 21-26) versus newest Beta Comparing what: ~both e10s and non-e10s, for profiles with add-ons ~both e10s and non-e10s, for profiles without add-ons expect performance improvement to increase across all areas in the later Beta. Expect to be bigger with e10s and bigger with the more add-ons installed Gotchya: the patches do not benefit every add-on - so need to determine if we look at just MPC=True add-ons or a specific list of add-ons.
Would this bug be better placed in an extension related bugzilla component?
Flags: needinfo?(sescalante)
Yeah, this is not the right component for this work. Shell, will the addons engineering team be doing this? Somebody needs to do some serious work to translate the generic product ask into specific metrics that we want to compare, if there is in fact any telemetry to cover what you care about at all.
i was debating where to put this - Dave zeber is working on it now. I'll put under firefox::extensions
Component: Telemetry → Extension Compatibility
Flags: needinfo?(sescalante)
Product: Toolkit → Firefox
Whiteboard: triaged
talked through likely target oct 31. Dave validating notebook against nightly and then porting to Beta. RC1 released 11/1 (with MPC=true patch bug 1312779) - Before Betas are 9,&10 (oct 21-26). https://docs.google.com/document/d/1OwXjbcGWf_CpqqO-8rV_-xTaPT5dl1xF1z-faoeUZlw/edit#heading=h.9yjkwzxf2fkw
Assignee: nobody → dzeber
dzeber validated hang metrics against nightly - https://metrics.mozilla.com/protected/dzeber/reports/addon-sdk-fix-analysis.html Will port to Beta and run Nov 8th if there is enough data. on Beta the dates for analysis are: * Before patches: Oct 21-27 covers beta 9 and 10. If not could include from Beta 8 on the 18th. * After patches: Beta release candidate from Oct 31st and later
more info from davez: Another clear improvement can be seen on the evolution dashboard for MEMORY_JS_COMPARTMENTS_SYSTEM. The median appears to have dropped by half. Also, most notable is the 95-th percentile (an indication of worst-case scenarios) which was also cut in half. This is one of the metrics that was mentioned at our discussion on Monday, and should be directly affected by the add-on SDK fixes. https://telemetry.mozilla.org/new-pipeline/evo.html#!aggregates=median!5th-percentile!25th-percentile!75th-percentile!95th-percentile&cumulative=0&end_date=null&keys=&max_channel_version=nightly%252F52&measure=MEMORY_JS_COMPARTMENTS_SYSTEM&min_channel_version=nightly%252F52&product=Firefox&sanitize=1&sort_keys=submissions&start_date=null&trim=1&use_submission_date=0 adding some startup time analysis to that report. I'm seeing some improvement, but not as clearly as for hangs (which I'm assuming is because startup is influenced by many other factors). This will be something we can look at on Beta as well.
David and Ben completed the telemetry that validated only improvements looking at hang number and length
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.