1157734 - Wrong "type" submitted for aborted-session-ping payloads

Status: RESOLVED WORKSFORME

(Toolkit :: Telemetry, defect, P3)

Description

[Mark Reid [:mreid]]

The "type" in the ping appears to include the full path to the ping file.

Comment 1

[Georg Fritzsche [:gfritzsche]]

Example from Mark:

{"type":"C:\\Users\\**snip**.default\\datareporting\\aborted-session-ping","id":"abe7eddd-cb3e-43ca-b74a-488fc11e941c","creationDate":"2015-04-21T09:12:45.764Z","version":4,"application":{"architecture":"x86-64","buildId":"20150420030204","name":"Firefox","version":"40.0a1","vendor":"Mozilla","platformVersion":"40.0a1","xpcomAbi":"x86_64-msvc","channel":"nightly"},"payload":true}
EnvVersion: 1
Severity: 7
Fields: [name:"submissionDate" value_string:"20150422"  name:"appVersion" value_string:"40.0a1"  name:"Host" value_string:"incoming.telemetry.mozilla.org"  name:"appUpdateChannel" value_string:"nightly"  name:"sourceVersion" value_string:"4"  name:"creationTimestamp" value_type:DOUBLE value_double:1.429607565764e+18  name:"docType" value_string:"C:\\Users\\**snip**.default\\datareporting\\aborted-session-ping"  name:"appBuildId" value_string:"20150420030204"  name:"geoCountry" value_string:"SE"  name:"appVendor" value_string:"Mozilla"  name:"documentId" value_string:"abe7eddd-cb3e-43ca-b74a-488fc11e941c"  name:"sourceName" value_string:"telemetry"  name:"appName" value_string:"Firefox" ]

Comment 2

[Alessio Placitelli [:Dexter]]

Does this only happen for Windows users?

Comment 3

[Benjamin Smedberg]

Does it matter? Sounds like we can fix it regardless.

Comment 4

[Georg Fritzsche [:gfritzsche]]

Indeed.

Comment 5

[Alessio Placitelli [:Dexter]]

Mark, does this still happen on latest nightlies?

Comment 6

[Mark Reid [:mreid]]

:Dexter, here is a monitor that's counting doc types for nightly builds on/after May 1st:

https://pipeline-prototype-cep.prod.mozaws.net/#plugins/filters/PrototypeSandbox-mreid_CountRecentByDocType

Comment 7

[Mark Reid [:mreid]]

Looks like this is still happening (one bad type so far since I added the monitoring yesterday).

Comment 8

[Mark Reid [:mreid]]

I suspect the one remaining bad type is from a custom build, which (despite a recent buildid) probably does not include your recent telemetry changes.

Should we be seeing actual submissions with type = "aborted-session-ping"?

Comment 9

[Alessio Placitelli [:Dexter]]

No, we shouldn't be seeing submissions with an "aborted-session-ping" type. Aborted session pings must have a "main" ping type with an "aborted-session" reason.

Comment 10

[Alessio Placitelli [:Dexter]]

Mark, would you kindly extend the monitor to consider a 7 days history? Thanks!

Comment 11

[Mark Reid [:mreid]]

Done!

Comment 13

[Georg Fritzsche [:gfritzsche]]

The dupe linked to some example data:
https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/temp/AU2cvvGLAMAkpwMF7NLQ

Comment 14

[Alessio Placitelli [:Dexter]]

Mark, would you kindly provide us with some sample pings affected by this issue?

Comment 15

[Alessio Placitelli [:Dexter]]

Mark, could you give us the source to this monitor? Or filter out the v4 payloads that don't have the |environment.settings.defaultSearchEngine| property?

Comment 16

[Georg Fritzsche [:gfritzsche]]

(In reply to Alessio Placitelli [:Dexter] from comment #15)
> Mark, could you give us the source to this monitor? Or filter out the v4
> payloads that don't have the |environment.settings.defaultSearchEngine|
> property?

The idea being that we want to confirm whether this is really old source builds that are submitting that data.

Comment 17

[Georg Fritzsche [:gfritzsche]]

(In reply to Mark Reid [:mreid] from comment #8)
> I suspect the one remaining bad type is from a custom build, which (despite
> a recent buildid) probably does not include your recent telemetry changes.

Does this monitor filter on the "nightly" channel too?
Per the discussion here: https://mail.mozilla.org/pipermail/fhr-dev/2015-June/000483.html
... that should probably be sufficient.

Comment 18

[Mark Reid [:mreid]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #17)
> (In reply to Mark Reid [:mreid] from comment #8)
> > I suspect the one remaining bad type is from a custom build, which (despite
> > a recent buildid) probably does not include your recent telemetry changes.
> 
> Does this monitor filter on the "nightly" channel too?
> Per the discussion here:
> https://mail.mozilla.org/pipermail/fhr-dev/2015-June/000483.html
> ... that should probably be sufficient.
That monitor is filtering for only "nightly".

Comment 19

[Mark Reid [:mreid]]

Attachment: mreid_count_recent_by_type.toml

Configuration for the heka monitor

Comment 20

[Mark Reid [:mreid]]

Attachment: mreid_count_by_type.lua

Code for the monitor

Comment 21

[Mark Reid [:mreid]]

Some recent examples sent via email.

Comment 22

[Georg Fritzsche [:gfritzsche]]

Checking the monitor we don't see any recent submissions with this issue.

I also checked on Kibana with this filter to see if there was recent submissions:
-clientId:* AND appName:Firefox AND appUpdateChannel:nightly AND appBuildId:201505*
... but we don't seem to have much data (yet?) for this month, so that doesn't seem useful.

Comment 23

[Georg Fritzsche [:gfritzsche]]

We haven't seen any recent submissions coming up, so closing this for now.

Comment 24

[Vladan Djeric (:vladan)]

We're still seeing wacky data in Telemetry fields, like integer clientIDs when they should be GUID strings. This could be a sign of a JS engine bug or GC bug.

I think the first step is to try to reproduce this in a controlled environment. 

Georg, what do you think about "stress-testing" telemetry by changing the thresholds to create pings every 5 mins, leaving it running for a day, then validating the resulting pings? Maybe Sam could help with this?