1771567 - Investigate Firefox Desktop Nightly Glean client count pattern change starting 2022-05-03

Status: RESOLVED WONTFIX

(Data Platform and Tools :: General, task)

Description

[Sean Rose [:srose]]

The client count is generally a bit higher suddenly, and oddly we didn't see the normal significant weekend dip for May 7-8.

https://mozilla.cloud.looker.com/dashboards/387

Comment 1

[Sean Rose [:srose]]

:chutten do you have any theories?

Comment 2

[Chris H-C :chutten]

Okay, let's see... "metrics" pings and client counts start going weird on the 4th of May then only get some semblance of their usual seasonality back on the 15th. Ish. query

In "baseline" pings we see a very similar pattern for client counts, but the ping counts remain mostly unperturbed. Perhaps the much higher ping volumes hide the noise a bit better? query

Definitely seems like a population effect, and it's ongoing. If it escapes Nightly to Beta we should start seeing a similar effect starting with builds from yesterday (merge day).

Keeping the ni? to look at this again tomorrow.

Comment 3

[Chris H-C :chutten]

Holy jumpin' jiminey, it disappeared!: https://sql.telemetry.mozilla.org/queries/86212/source#213484

Hey Klukas, did anything reprocess firefox_desktop data from May3 onwards in the last 24h?

Comment 4

[Jeff Klukas [:klukas] (UTC-4)]

I think this is shredder-related. A new run of the shredder_all task started running yesterday (log). It starts from the present and works backwards. For example, the firefox_desktop_stable.metrics_v1$20220517 partition was processed at 19:32:45. We can rewind to that time in a query like: https://sql.telemetry.mozilla.org/queries/86246/source#213571

That shows data for the 17th onward looks normal, but older dates still look wacky. So that strongly suggests the shredder processing got things back to normal.

What are these clients that are getting cleaned up? I have no idea. But I've saved off the affected period (since time travel window will make this inaccessible in ~5 days) for further investigation:

CREATE TABLE mozdata.analysis.klukas_bug1771567 AS
SELECT
*
FROM `moz-fx-data-shared-prod.firefox_desktop_stable.metrics_v1`
FOR SYSTEM_TIME AS OF '2022-05-30 12:00:00'
WHERE
    submission_timestamp >= '2022-05-01'
    AND normalized_channel = 'nightly'

I don't expect to have bandwidth to delve into the question of what these clients are, so leave that for others to pick up.