1142543 - Many submissions from a few clientIds

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P5)

Description

[Mark Reid [:mreid]]

I don't have a lot of information yet, but in the course of preparing a data export for bug 1140094, I noticed that there were two clients who each submitted more than 500MB of data (thousands of submissions).

One client submitted a bit more than 500 times per day from Feb. 27 to Mar. 3rd

Another submitted up to 2877 times per day between Mar 6 and Mar 8.

Comment 1

[Mark Reid [:mreid]]

Just a quick update that I checked the "payload.info.reason" field for these two clients and for both clients, all but three submissions were "environment-change", and the others were "idle-daily".

Comment 2

[Georg Fritzsche [:gfritzsche]]

Could we sample these reports on what the environment changes are?
Changes in addons or pref changes?
Maybe this is some automation setup?

Comment 3

[Mark Reid [:mreid]]

In a lot of cases, there are consecutive pings with exactly the same environment.

One possibly interesting thing is that many of the pings have "subsessionLength":0

I will get you some sample data to look at.

Comment 4

[Georg Fritzsche [:gfritzsche]]

Mark, the environment changes landed (after some bouncing) on 2015-04-03.
Can you check whether the situation improved since then?

Comment 5

[Mark Reid [:mreid]]

Waiting on bug 1151839 for this. Please ni? after that is fixed.

Comment 6

[Georg Fritzsche [:gfritzsche]]

(In reply to Mark Reid [:mreid] from comment #5)
> Waiting on bug 1151839 for this. Please ni? after that is fixed.

Comment 7

[Mark Reid [:mreid]]

There have not been any clients submitting thousands of times per day in the past several days.  It still appears that a few clients are maxing out the 5-minute threshold (~280 submissions per day), but the throttling seems to have worked.

Comment 8

[Benjamin Smedberg]

We should still follow up with the clients that are hitting the threshold all the time. Can you send me a couple samples?

Comment 9

[Mark Reid [:mreid]]

I've uploaded daily samples for one such client to gdrive (shared "telemetry_data_validation" dir). The file is called client24.tar.gz, and the data format is the same as bug 1149666.

It contains that user's submissions for April 5th, 10th, and 15th.  The throttling fix appears to have kicked in either on Apr. 6 or 7.

Let me know if you'd like more / different data.

Comment 10

[Georg Fritzsche [:gfritzsche]]

Benjamin, did you have a chance to look at the samples?

Comment 11

[Benjamin Smedberg]

No, I have not looked at this.

Comment 12

[brendan c]

Just want to put a note in this bug for posterity: the issue of "many submissions from a few clients" may actually be masking many clients sharing a clientId (from copied machine images, etc).

Comment 13

[Georg Fritzsche [:gfritzsche]]

We should be able to differentiate here by session chaining and activity date overlaps?

Comment 14

[brendan c]

(In reply to Georg Fritzsche [:gfritzsche] from comment #13)
> We should be able to differentiate here by session chaining and activity
> date overlaps?

yep, that is correct.

Comment 15

[Wesley Dawson [:whd]]

We have code to de-dupe in multiple places now.