1207992 - The metrics of app will still be recorded and sent even after the app is deleted

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Gaia::System, defect)

Description

[Gerry Chang [:gchang]]

### Steps:
1. Modify the interval to 1 minute.
AT.REPORT_INTERVAL = 1 * 60 * 1000;

1. Install app in device via Web IDE.
The app id is app://5e19fab8-6b47-ae4a-ae34-c4abf48fda8a/manifest.webapp
2. Perform some actions and gather logs for 1 minute.
3. Remove the app

Actual result:
I found that after deleting the app from homescreen, the metrics of the app are still sent and recorded.

Expected result:
The metrics should not be recorded.

Build ID               20150923010616
Gaia Revision          98bc7efb7c79b180bde067b3a1b000106c356f13
Gaia Date              2015-09-24 02:32:55
Gecko Revision         https://hg.mozilla.org/mozilla-central/rev/19b4265d0d568d232fb3a34705f947b6db7496dc
Gecko Version          44.0a1
Device Name            aries
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.worker.20150923.002655
Firmware Date          Wed Sep 23 00:27:03 UTC 2015
Bootloader             s1

Comment 1

[Gerry Chang [:gchang]]

Attachment: telemetry_app_deleted.log

Comment 2

[Gerry Chang [:gchang]]

Hi Russ,
Not sure if the data should still be recorded and sent after the app is removed.
Can you help to check this?

Comment 3

[Russ Nicoletti [:russn]]

I don't know how metrics would be recorded after an app was killed. I assume the app is being killed when it is removed from the homescreen. Can you attach the app?

Comment 4

[Gerry Chang [:gchang]]

I put the test app in https://github.com/gerrycfchang/firefoxos-quick-start.git
The app is in ./app/www folder.

Comment 5

[Russ Nicoletti [:russn]]

Hi Gerry, I think I'm only now understanding the issue you're asking about. At first I thought you were saying metrics were still being collected when an app is removed (and of course the metrics will not continue to be collected as the app will no longer be running). 

But now I understand your question to be, after metrics have been collected for an app and then the app is deleted, should the metrics that have been collected be sent to the telemetry server? Yes, developers will be interested in how the app was performing before it was deleted, so we should send the metrics we collected while it was running.

I'm going to mark this WORKSFORME as it is working as expected.

Comment 6

[Iris Hsiao [:ihsiao]]

Hi Russ,
I have a question about Telemetry keep sending metrics after deleted the test app.
Here I set Telemetry Report_interval to 4 minutes for the following scenario.

Time        | User Activity              | Expected Result                         
------------+----------------------------+----------------------------------------
15:00:00 PM | An user installed the test | The metrics will be collected and sent
     ~      | app, and is using it.      | to server every 4 minutes.(i.e. 
15:10:00 PM |                            | 15:04:00 PM and 15:08:00 PM)
------------+----------------------------+----------------------------------------
15:10:10 PM | The user deleted the test  |
            | app.                       |
------------+----------------------------+----------------------------------------
15:12:00 PM | N/A                        | (Report_interval arrived) The metrics 
            |                            | will be collected and sent to
            |                            | server.
------------+----------------------------+----------------------------------------
15:16:00 PM | N/A                        | (Report_interval arrived) The metrics 
            |                            | of the test app will NOT send to server anymore.

However the actual result is not the expected. After I deleted test app over 1 hour, I still can see the custom metrics of the test app existed in Telemetry payload, and the metrics' counts are all zero.
For example:
"addonHistograms": {
	"DEVTOOLS_HUD_CUSTOM_{APP_ID}/{Telemetry_name}":{...}
}

Comment 7

[Russ Nicoletti [:russn]]

Hi Iris, currently the Advanced Telemetry module does not support removing a metric from the set of metrics it is recording within the current B2G session. Every time B2G starts, the Advanced Telemetry module's list of metrics being collected gets reset. But while the current B2G session is active, the Advanced Telemetry module will report the current state of all the metrics that it has been collecting for the current session each time it transmits metric data to the telemetry server.

This will naturally result in some "empty" histogram data. Because when the reporting interval is reached and the histogram data for the metrics is sent to the server, the histogram data for all the metrics is cleared. The next time the report interval is reached, the histogram data for metrics for which no new information was collected will be sent to the server as "empty" histogram data.

For example, if app A generates a custom metric each time a certain page is opened, the histogram data for that metric will be collected and then sent when the reporting interval is reached. The next time the report interval is reached, if that page of that app was not opened, the histogram for that metric will be sent as "empty".

For the most common use case -- collecting "engineering" metrics (i.e., app startup time, app memory, reflows, event loop lag, etc.) -- this is not an issue because those metrics cannot be removed except by a code update and the report interval is such that it would be unlikely there won't be new data to be sent at each reporting interval.

For custom metrics, I agree it is not ideal to continue to send empty histogram data for custom metrics. However, the size of this "empty" data should be quite small relative to other telemetry data being sent and therefore it's affect on the size of the transmission to the server should be minimal.

With all that said, I consider the issue you're raising to be different enough from the original issue of this bug that it could have its own bug. You can create a new bug, something like "empty histogram data sent to server when metric not updated". For the reasons I mentioned above, I would argue it's a "wontfix" but I think it's fine to have a record of the issue.

Comment 8

[Iris Hsiao [:ihsiao]]

Hi Russ,
Thanks for replying me, I opened bug 1223269 to have a record of the issue.