756152 - disable persistent telemetry to prevent telemetry data loss

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect)

Description

[Nathan Froyd [:froydnj]]

For persistent telemetry, we only send histogram data; we don't send other information that we collect during normal telemetry operation (amount of RAM, various startup times, etc.).  Due to my poor assumptions and the way telemetry data is aggregated on the server side, this approach discards (on the server side) that "extra" non-histogram information.

Loss of telemetry data is bad.  Given that restructuring the server side would be rather involved, switching persistent telemetry off for the short term is a better option.  We have open bugs (bug 748517 and related) for restructuring persistent telemetry to make it easier to save this extra information.  While those bugs can be fixed relatively quickly, shoving through the train process on such short notice doesn't seem advisable.

Comment 1

[Nathan Froyd [:froydnj]]

Setting tracking flags.  I'm new to tracking flags, so hopefully I'm not screwing it up!

Comment 2

[Nathan Froyd [:froydnj]]

Attachment: patch

Here's a patch that just disables the sending, but leaves everything else intact.  Tests still pass, thanks to the observer hook.

Comment 3

[(dormant account)]

Comment on attachment 624821 [details] [diff] [review]
patch

r-, please just change the check for persistence data existence

Comment 4

[Nathan Froyd [:froydnj]]

Attachment: patch

Disabling by simply not loading data.

Comment 5

[Nathan Froyd [:froydnj]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/82154680bc6a

Comment 6

[Alex Keybl [:akeybl]]

(In reply to Nathan Froyd (:froydnj) from comment #5)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/82154680bc6a

Please nominate for aurora/beta approval tomorrow (5/22) and plan to land tomorrow as well so that this change makes it into FF13 beta 5.

Comment 7

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/82154680bc6a

Comment 8

[Nathan Froyd [:froydnj]]

Comment on attachment 624972 [details] [diff] [review]
patch

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 707320 and followups.
User impact if declined: None, this feature has no user interaction.
Testing completed (on m-c, etc.): Tests in the test suite.
Risk to taking this patch (and alternatives if risky): Epsilon risk that something will go wrong and telemetry won't get sent in, period.  Unwilling to say zero risk. :)
String or UUID changes made by this patch: None.

Comment 9

[Alex Keybl [:akeybl]]

Comment on attachment 624972 [details] [diff] [review]
patch

[Triage Comment]
Understood on risk. Can you make sure to watch aurora/beta channel telemetry volume it's as expected?

Approved for Aurora 14 and Beta 13. Pleas land asap to make it into beta 5.

Comment 10

[Nathan Froyd [:froydnj]]

http://hg.mozilla.org/releases/mozilla-aurora/rev/7bb1691988a5
http://hg.mozilla.org/releases/mozilla-beta/rev/85406204d3ca

Comment 11

[u279076]

Is there something QA can do to verify this fix?

Comment 12

[Nathan Froyd [:froydnj]]

Nothing for QA here.

Comment 13

[u279076]

Thanks Nathan.

Comment 14

[Nathan Froyd [:froydnj]]

(In reply to Alex Keybl [:akeybl] from comment #9)
> Understood on risk. Can you make sure to watch aurora/beta channel telemetry
> volume it's as expected?

Aurora and Beta telemetry volume seem as expected so far, save for a brief dip (~30-40%) on the 24th and 25th, respectively.  I don't claim to understand this, seeing as how telemetry volume went up when we enabled persistent telemetry (since we were sending roughly 2x the number of pings).  One would therefore expect it to decrease by the same volume...I suppose it's possible people are not updating quickly?

In any event, Nightly telemetry volume has held more-or-less steady, so I think we're OK on the telemetry front.