1501470 - Review the first strawman WebRender dashboard

Status: RESOLVED FIXED

(Data Science :: General, task)

Description

[Maire Reavy [:mreavy]]

Brief description of the request:
The WebRender team is putting together a dashboard. We need assistance from the Data Science team to assess and confirm that we are choosing the right statistics from the field to look at and analyzing the data properly. We need to assess stability, performance, and correctness in order to determine readiness to ship.

Link to any assets:
The WIP dashboard is here:
https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/38270/dashboard/39142/present

Is there a specific data scientist you would like or someone who has helped to triage this request:
I believe Tim Smith worked with the WR team before I was involved

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

I made contact with Maire in #desktop-perf-data-pod and invited akomar to a meeting for tomorrow to discuss

* goals for the dashboard,
* scope of the review,
* timeline, and
* next steps

Comment 2

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

PR to add webrender feature status environment fields to main_summary: https://github.com/mozilla/telemetry-batch-view/pull/493

Comment 3

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

I've summarized the first part of the review here: https://docs.google.com/document/d/1AXQ7syCSUwC59Sf5lz3ofvoEfYkbSEpLQPaeWDDO0Nk

Work completed to date:
* Review the methodology for selecting user sessions to include in the dashboards
* Review the presentation of the CHECKERBOARD_SEVERITY probe data and collect feedback from Kartikaya

Work still pending:
* Understand and evaluate the presentation of crash rate metrics
* Evaluate the presentation of CONTENT_FRAME_TIME, CONTENT_PAINT_TIME, and COMPOSITE_TIME
* Understand concerns about retention and propose a presentation for a retention metric
* Iterate on the dashboard to respond to the review, including moving the dashboard to a public URL; I'll open a new bug for this work.

Comment 4

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Dashboard development is being tracked in bug 1502489.

Additional work completed:
* Understand and evaluate the presentation of crash rate metrics
* Evaluate the presentation of CONTENT_FRAME_TIME, CONTENT_PAINT_TIME, and COMPOSITE_TIME
* Evaluate the suitability of FX_PAGE_LOAD_MS [1] → this probe doesn't have enough resolution to comment meaningfully on webrender performance; filed bug 1504247 to define a replacement

Work currently in progress:
* Evaluate presentation of FX_TAB_SWITCH_COMPOSITE_E10S_MS [1]

Work still pending:
* Understand concerns about retention and propose a presentation for a retention metric
* Propose a presentation for the CHECKERBOARD metrics; the small number of counts per user and the strongly bimodal nature of the _SEVERITY metric make it resistant to aggregation to a single scalar. 

[1] These were present in the release criteria document but not in the reviewed v1 prototype. https://docs.google.com/document/d/1HSe0XCyPsgxdJpQV9HRpdiLdAd4CnXalyw92sWxKkNM

Comment 5

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Closing out the review and spinning remaining work out into their own bugs.

The final review document is here: https://docs.google.com/document/d/1AXQ7syCSUwC59Sf5lz3ofvoEfYkbSEpLQPaeWDDO0Nk

Work on the dashboard itself is ongoing.