961810 - Port SlowSQL analysis to new telemetry backend

Status: RESOLVED FIXED

(Webtools Graveyard :: Telemetry Server, defect)

Description

[Mark Reid [:mreid]]

We want to continue to provide SlowSQL data for analysis, so we'll need to port the Hadoop based export over to the AWS backend.

The script that runs the export on the old system is here:
https://github.com/mozilla-metrics/telemetry-toolbox/blob/master/src/main/sh/daily-telemetry-apc_slowsql.sh

Comment 1

[Vladan Djeric (:vladan)]

This is a feature wishlist, I'm guessing it might not be possible to implement all these features. 

1) As a first cut, I'd like two tables showing slowSQL stats for the past X days: one for main-thread slow sql and another for "other thread" slowsql

Table columns: 
a) slow SQL string
b) database name (ping me for details)
c) median execution time
d) sum of all execution times
e) the total # of occurrences of this SQL in pings (some pings will report multiple occurrences)
f) the # of pings reporting at least 1 instance of this slow sql string
g) the % of pings reporting at least 1 instance of this slow sql string

By default, these stats should be based on pings from the previous 14 days.
By default, the table should be sorted by column G above, in descending order.

User should be able to filter on:
- idle-daily pings vs saved-session pings
- application name (Firefox, Fennec, Thunderbird)
- submission date range
- build ID range
- application version
- update channel

It should be possible to sort the data in the table by each column.

2) In the next set of features, we'd like to be able to track the change in slow SQL stats week-to-week. I think we'd want to base it on changes in rank (column G again), e.g. slowSQL string "foo" is new since date X, string "bar" has disappeared since date Y, string "baz" has become twice more common since date Z. We can discuss this over IRC.

Comment 2

[Mark Reid [:mreid]]

V1 of the SlowSQL dashboard has been deployed at http://telemetry.mozilla.org/slowsql/

Please file follow-up bugs for changes and enhancements.