1502489 - Iterate on the WebRender experiment dashboard

Status: RESOLVED FIXED

(Data Science :: Dashboard, task, P1)

Description

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Revise the prototype dashboard at https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/38270/dashboard/39142/present based on the review at https://docs.google.com/document/d/1AXQ7syCSUwC59Sf5lz3ofvoEfYkbSEpLQPaeWDDO0Nk.

The requester for this work is Maire Reavy.

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

A v2 prototype was stood up at https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/45053/dashboard/45125.

Feedback received on the v2 prototype:
* Usage hours -> usage khours ✅
* Add a high-level summary reflecting progress-to-goal for ride-to-beta and ride-to-nightly
* Add a display for FX_TAB_SWITCH_COMPOSITE_E10S_MS

Additional work in progress:
* Move plot rendering to hala in order to expose a dashboard on metrics.mozilla.com
* Add core product metrics
* Add experiment health metrics

Blockers to completion (but not the next iteration) are:
* Define a high-resolution replacement for FX_PAGE_LOAD_MS
* Define presentations for the CHECKERBOARD metrics

Related work:
* Mozanalysis bug to add histogram-quantile UDFs https://github.com/mozilla/mozanalysis/issues/10
* Databricks bug for bogus "undefined function" failures https://help.databricks.com/hc/en-us/requests/18277

Comment 2

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

A version of the dashboard is up at https://metrics.mozilla.com/webrender/dashboard.html. It incorporates the feedback and additional work described above.

Some remaining work on this iteration:
* Add bug burndowns
* Improve aesthetics
* Improve significance display for top-level metrics
* Set up cron jobs for the Databricks ETL notebook and the rendering script

Comment 3

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

The dashboard is now set to update itself twice a day, at midnight and noon UTC.

Work to do next week includes:
* Add plots for fractions of COMPOSITE_TIME, CONTENT_PAINT_TIME ≥ 16ms
* Count pings excluded by first-session and null-pref rules
* Work represented in blocking bugs

Comment 4

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

I think this is done for now. I'll keep the CHECKERBOARD bug open.

The threshold plots for COMPOSITE_ and PAINT_TIMEs are up, and FX_PAGE_LOAD_MS_2 is represented.

In lieu of a dropped-ping count, I'm counting the number of users we actually count data from.

This will need more attention before the WR release experiment.

Comment 5

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Adding a link to a `moztelemetry` library bug that was uncovered by this work.

Comment 6

[Experimenter Bot]

[Experiment]WebRender performance tracking study: nightly status has been changed to: Complete
            url:https://experimenter.services.mozilla.com/experiments/webrender-performance-tracking-study-nightly/

Comment 7

[Experimenter Bot]

[Experiment]WebRender performance tracking study: beta status has been changed to: Complete
            url:https://experimenter.services.mozilla.com/experiments/webrender-performance-tracking-study-beta/