1199018 - [e10s] automate extracting slow addon data from telemetry

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[Jim Mathies [:jimm]]

We have two probes currently that need analysis work to extract useful data. The method of extraction should be automated in some way for easy access.

MISBEHAVING_ADDONS_CPOW_TIME_MS
Measures time spent waiting for CPOW response. This is a keyed histogram, so the data is per-addon.
MISBEHAVING_ADDONS_JANK_LEVEL
Measures jank that happens when add-on code is on the JS stack. Not sure how it’s measured (Yoric knows). Also keyed on add-on ID.

Comment 1

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

This is kind of already happening. You can see the results here:
https://s3-us-west-2.amazonaws.com/telemetry-public-analysis/e10s-addon-perf/data/cpow-data.json

I think we mostly just need someone to look over the code and make sure it's sensible. I haven't looked at it in a while.

Comment 2

[Vladan Djeric (:vladan)]

Yoric's misbehaving addons analysis (updates live):

http://nbviewer.ipython.org/urls/s3-us-west-2.amazonaws.com/telemetry-public-analysis/Misbehaving%20addons/data/Misbehaving%20Add-ons.ipynb

Comment 3

[Vladan Djeric (:vladan)]

Actually I learned the analysis script above was turned off recently since this data is also available in the main Telemetry dash:

http://mzl.la/1FIIzfd
http://mzl.la/1FIIfNF

Comment 4

[Vladan Djeric (:vladan)]

Cumulative views, e.g. take a look at % of CPOW operations that cause jank of 10ms or more (ranked worst to best):

MISBEHAVING_ADDONS_CPOW_TIME_MS http://mzl.la/1FIJyMw
MISBEHAVING_ADDONS_JANK_LEVEL http://mzl.la/1FIJBI6

Comment 5

[David Teller [:Yoric] - still alive but not very active]

I would wait until bug 1189799 has landed to actually look at data, though.

Comment 6

[Vladan Djeric (:vladan)]

This should be an experiment report. Yoric, is this Telemetry ready to be looked at?

Comment 7

[David Teller [:Yoric] - still alive but not very active]

Indeed.
The histograms will change again once bug 1186491 has landed, though.

Comment 8

[David Teller [:Yoric] - still alive but not very active]

The new histograms are named PERF_MONITORING_SLOW_ADDON_JANK_US and PERF_MONITORING_SLOW_ADDON_CPOW_US.

Comment 9

[David Teller [:Yoric] - still alive but not very active]

What needs to be done at this stage?

Comment 10

[Vladan Djeric (:vladan)]

(In reply to David Rajchenbach-Teller [:Yoric] (please use "needinfo") from comment #9)
> What needs to be done at this stage?

Roberto or Birunthan should do an analysis of the gathered data from the Beta 44 experiment. The data won't be trustworthy because of bug 1236754 but we should still have the analysis ready before the Beta 45 experiment runs.

Comment 11

[Biru [:poiru]]

I'll take this.

Comment 12

[Biru [:poiru]]

(In reply to David Rajchenbach-Teller [:Yoric] (please use "needinfo") from comment #9)
> What needs to be done at this stage?

Bug 1186491 is not present on Beta 44 (i.e. PERF_MONITORING_SLOW_ADDON_{CPOW,JANK}_US are not available). I checked Aurora 45 and all pings from Jan 17 had empty payload/keyedHistograms/PERF_MONITORING_SLOW_ADDON_{CPOW,JANK}_US fields. Is this data actually being collected?

Comment 13

[David Teller [:Yoric] - still alive but not very active]

It's only in Nightly. I'm waiting for the authorization to let it ride the trains, but I guess that will only happen once we have released e10s.

Comment 14

[Biru [:poiru]]

(In reply to David Rajchenbach-Teller [:Yoric] (please use "needinfo") from comment #13)
> It's only in Nightly. I'm waiting for the authorization to let it ride the
> trains, but I guess that will only happen once we have released e10s.

OK, unassigning then.

Comment 15

[Chris Peterson [:cpeterson]]

Collecting add-on data is not an e10s release blocker.

Comment 16

[:shell escalante]

probes for webextension timing added - histograms and alerting set up for changes.  http://alerts.telemetry.mozilla.org/index.html#/detectors/1/metrics/2442/alerts/?from=2017-07-12&to=2017-07-12&metrics-filter=