1527034 - Regression Testing Dashboard

Status: RESOLVED FIXED

(Data Science :: General, task)

Description

[Corey Dow-Hygelund [:ccd]]

Brief description of the request: Implement a Nightly regression testing dashboard utilizing page load and graphics histogram probes of Nightly pings.

Link to any assets:

Is there a specific data scientist you would like or someone who has helped to triage this request:

Requestor: Eric Smyth

Comment 1

[Corey Dow-Hygelund [:ccd]]

Utilize the relative difference estimator:
https://drive.google.com/file/d/1Xj9N6IgNaLctMO2tlLgLnQV8QndfU9fo/view

Methods to calculate dataset for dashboard: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/76530/command/76534

Comment 2

[Corey Dow-Hygelund [:ccd]]

Dashboard repo: https://github.com/godelstheory/ff-perf-regression-dashboard

Comment 3

[Corey Dow-Hygelund [:ccd]]

Initial dashboard implementation:
https://metrics.mozilla.com/protected/shiny/cdowhygelund/ff-perf-regression-dashboard/

Page load histograms included using small subset of data.
Graphics histograms not included.

Comment 4

[Corey Dow-Hygelund [:ccd]]

Implemented scheduled Databricks and hala cron jobs for daily updates (i.e., each new Nightly build).

Comment 5

[Corey Dow-Hygelund [:ccd]]

To be added once in telemetry: Raptor/Talos data.

Comment 6

[Corey Dow-Hygelund [:ccd]]

Job was failing due to 4G driver limit error. After much debugging, determined this was an issue with both a data frame toPandas versus collect call, in addition to utilizing the createOrReplaceTempView + SQL path, versus the pyspark API. This resulted in a few days worth of missing data in the dashboard. A batch filling job is in order.

https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/76530/command/80487

Comment 7

[Corey Dow-Hygelund [:ccd]]

Current dashboard for use in determining Firefox health.

Comment 8

[Corey Dow-Hygelund [:ccd]]

Regression detection PRD

Comment 9

[Corey Dow-Hygelund [:ccd]]

Closing. New bug(s) will be created regarding PRD.