1223269 - Histogram data for custom metrics sent to server when metric unchanged since last transmission

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Metrics, defect)

Description

[Iris Hsiao [:ihsiao]]

This bug is opened based on bug 12077992 comment #6 and comment #7.

###Steps###
Here I set Telemetry Report_interval to 4 minutes for the following scenario.

Time        | User Activity              | Expected Result                         
------------+----------------------------+----------------------------------------
15:00:00 PM | An user installed the test | The metrics will be collected and sent
     ~      | app, and is using it.      | to server every 4 minutes.(i.e. 
15:10:00 PM |                            | 15:04:00 PM and 15:08:00 PM)
------------+----------------------------+----------------------------------------
15:10:10 PM | The user deleted the test  |
            | app.                       |
------------+----------------------------+----------------------------------------
15:12:00 PM | N/A                        | (Report_interval arrived) The metrics 
            |                            | will be collected and sent to
            |                            | server.
------------+----------------------------+----------------------------------------
15:16:00 PM | N/A                        | (Report_interval arrived) The metrics 
            |                            | of the test app will NOT send to server anymore.

However the actual result is not the expected. After I deleted test app over 1 hour, I still can see the custom metrics of the test app existed in Telemetry payload, and the metrics' counts are all zero.
For example:
"addonHistograms": {
	"DEVTOOLS_HUD_CUSTOM_{APP_ID}/{Telemetry_name}":{...}
}

###Expected result###
The metrics won't send to server after deleted the test app.

###Actual result###
The metrics keep sending to server until restart device.

Comment 1

[Iris Hsiao [:ihsiao]]

Correct the typo in the description.
> This bug is opened based on "bug 1207992" comment #6 and comment #7.

Comment 2

[Russ Nicoletti [:russn]]

See [1] for more information regarding this issue. For the reasons described in [1], IMO this is a low priority issue.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1207992#c7

Comment 3

[Russ Nicoletti [:russn]]

Further investigation has shown that this issue, unlike what I described here [1], affects only custom metrics. Histogram data for engineering metrics (reflows, jank, startup time, etc.) will be empty, that is, will contain no data -- no zeroes, nothing -- when there hasn't been any data collected since the last time data was sent to the server. Here is an example of 'empty' data sent for engineering metrics and custom metrics [2]. This shows 

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1207992#c7

[2] Console(27640): Content JS LOG: [AdvancedTelemetry]{"type":"advancedtelemetry","id":"a08fa2d6-7b4f-46e7-9be9-2aa64d0049a9","creationDate":"2015-11-17T23:07:03.328Z","version":4,"application":{"architecture":"arm","buildId":"20151112122114","name":"FirefoxOS","version":"45.0a1","vendor":"Mozilla","platformVersion":"45.0a1","xpcomAbi":"arm-gcc3","channel":"default"},"clientId":"f9bee611-24cc-4daf-9757-332293911eee","payload":{"keyedHistograms":{"DEVTOOLS_HUD_REFLOW_DURATION":{"system":{"min":1,"max":1000,"histogram_type":0,"sum":7,"log_sum":3.465735912322998,"log_sum_squares":4.324077129364014,"ranges":[0,1,2,5,12,29,70,170,412,1000],"counts":[0,1,2,0,0,0,0,0,0,0]}},"DEVTOOLS_HUD_REFLOWS":{"system":{"min":1,"max":2,"histogram_type":4,"sum":3,"sum_squares_lo":3,"sum_squares_hi":0,"ranges":[0,1,2],"counts":[3,0,0]}},"DEVTOOLS_HUD_ERRORS":{},"DEVTOOLS_HUD_APP_STARTUP_TIME_NAVIGATIONLOADED":{},"DEVTOOLS_HUD_APP_STARTUP_TIME_NAVIGATIONINTERACTIVE":{},"DEVTOOLS_HUD_APP_MEMORY_NAVIGATIONLOADED_V2":{},"DEVTOOLS_HUD_APP_MEMORY_NAVIGATION_INTERACTIVE_V2":{},"DEVTOOLS_HUD_APP_STARTUP_TIME_VISUALLYLOADED":{},"DEVTOOLS_HUD_APP_STARTUP_TIME_CONTENTINTERACTIVE":{},"DEVTOOLS_HUD_APP_MEMORY_VISUALLYLOADED_V2":{},"DEVTOOLS_HUD_APP_MEMORY_CONTENTINTERACTIVE_V2":{},"DEVTOOLS_HUD_WARNINGS":{},"DEVTOOLS_HUD_APP_STARTUP_TIME_FULLYLOADED":{},"DEVTOOLS_HUD_APP_STARTUP_TIME_SCANEND":{},"DEVTOOLS_HUD_APP_MEMORY_FULLYLOADED_V2":{},"DEVTOOLS_HUD_APP_MEMORY_SCANEND_V2":{},"DEVTOOLS_HUD_APP_STARTUP_TIME_MEDIAENUMERATED":{},"DEVTOOLS_HUD_APP_MEMORY_MEDIAENUMERATED_V2":{}},"addonHistograms":{"DEVTOOLS_HUD_CUSTOM_VIDEO_SHOWPLAYERCOUNT":{"min":1,"max":2,"histogram_type":4,"sum":0,"sum_squares_lo":0,"sum_squares_hi":0,"ranges":[0,1,2],"counts":[0,0,0]}}}}

Comment 4

[Russ Nicoletti [:russn]]

Following up on comment 3, the histogram data in [2] above shows how the histogram data for engineering metrics that have not changed since the last transmission is truly empty while histogram data for custom metrics that haven't changed gets sent as zeroes. I have changed the summary of this bug to reflect the new information.

Comment 5

[Russ Nicoletti [:russn]]

Iris, can you confirm it is only the custom metrics that have this issue? By that I mean, are you seeing that we're not sending histogram data for engineering metrics that haven't changed (as shown in comment 3)?

Comment 6

[Iris Hsiao [:ihsiao]]

Hi Russ,
Yes, what I'm seeing just like the histogram data [2] in comment 3. For engineering metrics, they are empty if that haven't changed since last transmission; but for custom metrics, they are all zeroes.

Comment 7

[GitHub Autolander Bot]

Attachment: [gaia] tamarahills:bugfix/1223269-Remove-Histograms-With-No-Data > mozilla-b2g:master

Comment 8

[Tamara Hills [:thills]]

Comment on attachment 8714810 [details] [review]
[gaia] tamarahills:bugfix/1223269-Remove-Histograms-With-No-Data > mozilla-b2g:master

Hi Russ,

This removes any extraneous entries so we can save bytes on the payload.

Thanks,
-tamara

Comment 9

[Russ Nicoletti [:russn]]

Comment on attachment 8714810 [details] [review]
[gaia] tamarahills:bugfix/1223269-Remove-Histograms-With-No-Data > mozilla-b2g:master

Looks good to me, a couple of nits -- you can thank Jan :)

Comment 10

[Tamara Hills [:thills]]

Comment on attachment 8714810 [details] [review]
[gaia] tamarahills:bugfix/1223269-Remove-Histograms-With-No-Data > mozilla-b2g:master

Hi Russ,

I changed it to bailout using continue since it's a double loop, but also created a helper function since they are exactly the same anyways.

-tamara

Comment 11

[Russ Nicoletti [:russn]]

Comment on attachment 8714810 [details] [review]
[gaia] tamarahills:bugfix/1223269-Remove-Histograms-With-No-Data > mozilla-b2g:master

Nice refactoring, I left a couple of nits in the PR.

Comment 12

[BMO Automation]

Firefox OS is not being worked on