1169037 - UT: Validate the correct behavior when turning prefs off and on

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Thomas Huelbert]

My understanding is these prefs can be flipped via add-on hot fix. Need to test this before r39.

Comment 1

[Mark Reid [:mreid]]

One thing to note - at one point, disabling FHR on the client resulted in a delete request being sent to the server. Something to check if/when we modify the relevant prefs with a hotfix.

Comment 2

[Thomas Huelbert]

whats the pref to turn on/off each? needs more info and off to Mr Reid it goes...

Comment 3

[Mark Reid [:mreid]]

The FHR-related prefs are:
datareporting.healthreport.service.enabled
datareporting.healthreport.uploadEnabled


Georg, can you comment on the UT pref(s)?

Comment 4

[Georg Fritzsche [:gfritzsche]]

(In reply to Mark Reid [:mreid] from comment #3)
> The FHR-related prefs are:
> datareporting.healthreport.service.enabled
> datareporting.healthreport.uploadEnabled
> 
> 
> Georg, can you comment on the UT pref(s)?

What are we looking for?
Prefs to flip of either of Unified Telemetry & FHR on/off?
Please file bugs with some more detail here.

Assuming that:
* FHR (which we plan to obsolete) should be controlled by "datareporting.healthreport.service.enabled"
* Unified telemetry behavior should be controlled by "toolkit.telemetry.unified"

datareporting.healthreport.uploadEnabled controls whether
* we can send FHR data
* if unified is enabled, whether we can send Telemetry data (otherwise controlled by "toolkit.telemetry.enabled")

See also: https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/toolkit/components/telemetry/telemetry/preferences.html

Comment 5

[Georg Fritzsche [:gfritzsche]]

Setting the title to what i think the intent is, please fix if that's not right.

Comment 6

[Georg Fritzsche [:gfritzsche]]

(In reply to Mark Reid [:mreid] from comment #1)
> One thing to note - at one point, disabling FHR on the client resulted in a
> delete request being sent to the server. Something to check if/when we
> modify the relevant prefs with a hotfix.

datareporting.healthreport.uploadEnabled triggers that, we shouldn't change that pref for any work here.

Comment 7

[Georg Fritzsche [:gfritzsche]]

Note that for 39 we don't have unified behavior for people that haven't opted into Telemetry, so we should be fine.
I don't think we need to do anything for 39 except verify that things behave as expected.

Comment 8

[Thomas Huelbert]

updated description to reflect actual needed tests

Comment 9

[Stuart Philp :sphilp]

Notes
---------------
- Everything refers to FHR, old telemetry, and unified telemetry collectively, with the prefs:
toolkit.telemetry.enabled, toolkit.telemetry.unified and datareporting.healthreport.service.enabled
- OT refers to just old telemetry and the pref toolkit.telemetry.enabled
- UT refers to unified telemetry and the pref toolkit.telemetry.unified
- FHR refers to firefox health reporting and the pref datareporting.healthreport.service.enabled
- Upload disabled refers to the pref datareporting.healthreport.uploadEnabled for FHR
- toggling FHR in the UI seems to control the upload pref but not the FHR service, which is confusing. on disabling the upload pref a DELETE request is sent

Test Cases
---------------
1. Everything disabled | PASS
State: clean start up with everything enabled, upload enabled
Expected result: nothing is sent
Actual result: nothing is sent

2. Everything enabled | PASS
State: clean start up with everything enabled, upload enabled
Expected result: data is sent to UT and FHR, but not OT
Actual result: data is submitted to https://incoming.telemetry.mozilla.org/submit/?v=4 and https://fhr.data.mozilla.com/1.0/submit/

3. FHR and OT enabled, UT disabled | PASS
State: clean start up with FHR and OT enabled, upload enabled
Expected result: data is sent to FHR and OT
Actual result: data is submitted to https://fhr.data.mozilla.com/1.0/submit/ and https://incoming.telemetry.mozilla.org/submit/

4. FHR enabled, but upload disabled | PASS
State: clean start up with FHR enabled, upload disabled
Expected result: no data sent
Actual result: no data sent

5. Everything enabled, but upload disabled | PASS
State: clean start up with everything enabled, upload disabled
Expected result: no data sent to FHR or OT but UT continues to send
Actual result: no data sent to FHR or OT but UT continues to send

6. Everything enabled, confirm UT is sending, then disable UT | FAIL?
Initial State: Active instance with everything enabled, upload enabled
Expected result: UT stops sending, OT starts sending again
Actual result: data still being sent to UT (v=4), but a restart seems to clear it up (see test case 3)

7. Everything enabled, confirm UT is sending, then disable upload | PASS
Initial State: Active instance with everything enabled, upload enabled
Expected result: UT continues sending but FHR stops
Actual result: UT continues sending with v=4, and fhr does not send data

8. FHR and OT enabled, confirm OT is sending, then enable UT | FAIL?
Initial State: Active instance with FHR and OT enabled, UT disabled, upload enabled
Expected result: FHR and UT send data, OT stops sending data
Actual result: FHR works, but UT does not seem to send data, a restart fixes it similar to test case 6

9. Disabling FHR results in a delete request being sent | PASS
expected result: delete request is sent to https://fhr.data.mozilla.com/1.0/submit/
result: delete request was sent to https://fhr.data.mozilla.com/1.0/submit/

Summary
---------------
Everything in general seems to work correctly, however a restart seems to be required when enabling/disabling UT. Is that expected and okay going to release? Seems like part of what we wanted to achieve was to be able to pref off in a hurry if needed.

Comment 10

[Georg Fritzsche [:gfritzsche]]

Thanks for putting this together.
Was this tested on 40 or 41?
Maybe we can gather QA information (required setup, triggering pings, relevant prefs, ...) in a wiki page for future reference?

(In reply to Stuart Philp :sphilp from comment #9)
> Summary
> ---------------
> Everything in general seems to work correctly, however a restart seems to be
> required when enabling/disabling UT. Is that expected and okay going to
> release? Seems like part of what we wanted to achieve was to be able to pref
> off in a hurry if needed.

This is expected, handling switching things off and on during a browser session would introduce a lot of additional complexity.
It should be sufficient to have clients switch over on restart.

> - toggling FHR in the UI seems to control the upload pref but not the FHR service, which is confusing. on disabling the upload pref a DELETE request is sent

We always record at least the "base" set of data. This is required to power selfsupport, about:healthreport, ...

---------------

For the sending of v4 data - we should always send v4 style of data independent of the UT pref:
* the UT pref controls if we send "main" pings (on shutdown, midnight or environment change)
* the OT pref controls - independently - if we send "saved-session" pings (on shutdown)

We should always submit to https://incoming.telemetry.mozilla.org/submit/?v=4 unless you happen to have old pings on disk that still need to be sent.
Does the above mean that we are not always submitting new pings to "?v=4" URLs?

---------------

> 2. Everything enabled | PASS
> State: clean start up with everything enabled, upload enabled
> Expected result: data is sent to UT and FHR, but not OT

Expected: data is sent to FHR and Telemetry, both "main" & "saved-session" pings

> 3. FHR and OT enabled, UT disabled | PASS
> State: clean start up with FHR and OT enabled, upload enabled
> Expected result: data is sent to FHR and OT

Expect: Expected: data is sent to FHR and Telemetry, only "saved-session" pings

> 5. Everything enabled, but upload disabled | PASS
> State: clean start up with everything enabled, upload disabled
> Expected result: no data sent to FHR or OT but UT continues to send
> Actual result: no data sent to FHR or OT but UT continues to send

Expected: no data sent
Note that the upload pref & the OT pref are coupled, disabling upload in the preferences also turns off OT.
I can't confirm this with normal behavior:
* have everything enabled
* go to preferences
* disable upload
* trigger a new ping (e.g. by disabling a plugin or calling TelemetryController.submitExternalPing("foobar", {}))

I think we need more details here to understand whether what you are seeing is expected behavior or not.

> 6. Everything enabled, confirm UT is sending, then disable UT | FAIL?
> Initial State: Active instance with everything enabled, upload enabled
> Expected result: UT stops sending, OT starts sending again

Expected: no further "main" pings sent, "saved-session" still being sent

> 7. Everything enabled, confirm UT is sending, then disable upload | PASS
> Initial State: Active instance with everything enabled, upload enabled
> Expected result: UT continues sending but FHR stops
> Actual result: UT continues sending with v=4, and fhr does not send data

Expected: no data sent after disabling upload

> 8. FHR and OT enabled, confirm OT is sending, then enable UT | FAIL?
> Initial State: Active instance with FHR and OT enabled, UT disabled, upload enabled
> Expected result: FHR and UT send data, OT stops sending data

Expected: FHR and "saved-session" pings sent, after enabling UT & restart "main" pings sent too

> 9. Disabling FHR results in a delete request being sent | PASS
> expected result: delete request is sent to https://fhr.data.mozilla.com/1.0/submit/
> result: delete request was sent to https://fhr.data.mozilla.com/1.0/submit/

expected on 41: delete request is sent to https://fhr.data.mozilla.com/1.0/submit/ & "deletion" ping sent to Telemetry
same on 40 when/if bug 1120379 gets uplifted

Comment 11

[Georg Fritzsche [:gfritzsche]]

Thinking a bit about other features that can be affected here:
* experiments should keep working: https://wiki.mozilla.org/QA/Desktop_Firefox/Telemetry_Experiments
* about:healtreport should keep working with FHR off when bug 1150115 is fixed
* selfsupport features like Heartbeat should keep working, bug 1179226
* about:telemetry should keep working

Comment 12

[Stuart Philp :sphilp]

Thanks Georg! I've been keeping notes on how to test, so I can dump that into a wiki after some basic formatting. This was all done on 40, I will check the expected results and other things you mentioned in a follow up.

Comment 13

[Stuart Philp :sphilp]

Started a wiki https://mana.mozilla.org/wiki/display/~[e-mail]/Verifying+FHR+and+UT+pings

From the revised expected results, it seems like the main issue was UT pings still sending after disabling upload, however I tried on nightly (42) and it looks to be resolved there.

I was confused with the v=4 stuff, I thought v=4 was only for UT and that OT would use the an older version num (or no version). To clarify: main and saved-session both send to v=4, and they look to be correct in that main pings are sent when UT is enabled, and saved-session for OT.

Confirmed the deletion ping is sent to UT as well when disabling upload.

Everything looks good on 42

Comment 14

[Georg Fritzsche [:gfritzsche]]

(In reply to Stuart Philp :sphilp from comment #13)
> Started a wiki
> https://mana.mozilla.org/wiki/display/~[e-mail]/
> Verifying+FHR+and+UT+pings

Great - can we move this to the public wiki so people without LDAP are not locked out? :)
(see e.g. https://wiki.mozilla.org/QA/Telemetry)

> From the revised expected results, it seems like the main issue was UT pings
> still sending after disabling upload, however I tried on nightly (42) and it
> looks to be resolved there.

Ok, that's unexpected on 40 - can you maybe file a bug with details so we can make sure there is nothing strange going on?
 
> I was confused with the v=4 stuff, I thought v=4 was only for UT and that OT
> would use the an older version num (or no version). To clarify: main and
> saved-session both send to v=4, and they look to be correct in that main
> pings are sent when UT is enabled, and saved-session for OT.
> 
> Confirmed the deletion ping is sent to UT as well when disabling upload.
> 
> Everything looks good on 42

Great to hear!

Comment 15

[Mark Reid [:mreid]]

This looks good to me, but can we please update the docs at
 https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/toolkit/components/telemetry/telemetry/preferences.html
to call out prefs that require a restart to take effect?

Comment 16

[Georg Fritzsche [:gfritzsche]]

(In reply to Mark Reid [:mreid] from comment #15)
> This looks good to me, but can we please update the docs at
>  https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/toolkit/
> components/telemetry/telemetry/preferences.html
> to call out prefs that require a restart to take effect?

Good point, filed bug 1180738.