1341369 - Aurora 51's showing some weird re-adoption rates on tmo

Status: RESOLVED DUPLICATE of bug 1342111

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Chris H-C :chutten]

:mconley noticed that a particular Aurora 51 buildid (20161015) is showing 5 orders of magnitude more submissions than all other Aurora buildids across history: https://mzl.la/2kVPM1W

When we look at it by submission date we see that Aurora 51 went to sleep as per usual, but then suddenly around Dec 27 we started seeing an uptick in submissions: https://mzl.la/2kImVTQ These submissions have only gotten more populous and, from the first link, seem to come from one particular Aurora 51 build.

Something looks to be causing a re-adoption of Aurora 51 build 20161015.

Comment 1

[Frank Bertsch [:frank]]

https://mzl.la/2lrzcZX shows the problem is not as bad as we thought. This is a flag histogram, so looking at submissions, shows exactly how many pings were submitted (since they record exactly once per ping). Still requires investigation.

Comment 2

[Chris H-C :chutten]

That's odd, I could've sworn it was 20161015 when last I looked at it. Now it's looking like 20161014. Sorry.

Comment 3

[Chris H-C :chutten]

Alrighty, so evidence is mounting that some rogue actors are sending us a whole heckuva lot of "main" pings from Aurora 51 for no discernible reason: https://sql.telemetry.mozilla.org/queries/3085/source#5872 

The client count is decreasing as expected, but the ping count has gotten rather ridiculous and is showing no particular pattern, not even a seven-day loop.

Comment 4

[Chris H-C :chutten]

This is mostly just one particular client_id who, instead of submitting <10 main pings per day, is sending tens of thousands of main pings per day. I'm running an analysis to explore the character of this client_id.

Comment 5

[Mark Reid [:mreid]]

We're going to handle this by detecting and removing duplicate submissions per Bug 1342111.