:mconley noticed that a particular Aurora 51 buildid (20161015) is showing 5 orders of magnitude more submissions than all other Aurora buildids across history: https://mzl.la/2kVPM1W When we look at it by submission date we see that Aurora 51 went to sleep as per usual, but then suddenly around Dec 27 we started seeing an uptick in submissions: https://mzl.la/2kImVTQ These submissions have only gotten more populous and, from the first link, seem to come from one particular Aurora 51 build. Something looks to be causing a re-adoption of Aurora 51 build 20161015.
https://mzl.la/2lrzcZX shows the problem is not as bad as we thought. This is a flag histogram, so looking at submissions, shows exactly how many pings were submitted (since they record exactly once per ping). Still requires investigation.
That's odd, I could've sworn it was 20161015 when last I looked at it. Now it's looking like 20161014. Sorry.
Alrighty, so evidence is mounting that some rogue actors are sending us a whole heckuva lot of "main" pings from Aurora 51 for no discernible reason: https://sql.telemetry.mozilla.org/queries/3085/source#5872 The client count is decreasing as expected, but the ping count has gotten rather ridiculous and is showing no particular pattern, not even a seven-day loop.
2 months ago
This is mostly just one particular client_id who, instead of submitting <10 main pings per day, is sending tens of thousands of main pings per day. I'm running an analysis to explore the character of this client_id.
We're going to handle this by detecting and removing duplicate submissions per Bug 1342111.