Closed Bug 1169037 Opened 5 years ago Closed 4 years ago

UT: Validate the correct behavior when turning prefs off and on

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED
mozilla39

People

(Reporter: thuelbert, Unassigned)

References

Details

(Whiteboard: [b5][unifiedTelemtry])

My understanding is these prefs can be flipped via add-on hot fix. Need to test this before r39.
Priority: -- → P3
One thing to note - at one point, disabling FHR on the client resulted in a delete request being sent to the server. Something to check if/when we modify the relevant prefs with a hotfix.
Blocks: 1173438
whats the pref to turn on/off each? needs more info and off to Mr Reid it goes...
Flags: needinfo?(mreid)
The FHR-related prefs are:
datareporting.healthreport.service.enabled
datareporting.healthreport.uploadEnabled


Georg, can you comment on the UT pref(s)?
Flags: needinfo?(mreid) → needinfo?(gfritzsche)
(In reply to Mark Reid [:mreid] from comment #3)
> The FHR-related prefs are:
> datareporting.healthreport.service.enabled
> datareporting.healthreport.uploadEnabled
> 
> 
> Georg, can you comment on the UT pref(s)?

What are we looking for?
Prefs to flip of either of Unified Telemetry & FHR on/off?
Please file bugs with some more detail here.

Assuming that:
* FHR (which we plan to obsolete) should be controlled by "datareporting.healthreport.service.enabled"
* Unified telemetry behavior should be controlled by "toolkit.telemetry.unified"

datareporting.healthreport.uploadEnabled controls whether
* we can send FHR data
* if unified is enabled, whether we can send Telemetry data (otherwise controlled by "toolkit.telemetry.enabled")

See also: https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/toolkit/components/telemetry/telemetry/preferences.html
Flags: needinfo?(gfritzsche)
Setting the title to what i think the intent is, please fix if that's not right.
Summary: Validate we can turn UT/FHR off via add-on hot fix mechanism → Validate we can turn Unified Telemetry off via add-on hot fix mechanism
(In reply to Mark Reid [:mreid] from comment #1)
> One thing to note - at one point, disabling FHR on the client resulted in a
> delete request being sent to the server. Something to check if/when we
> modify the relevant prefs with a hotfix.

datareporting.healthreport.uploadEnabled triggers that, we shouldn't change that pref for any work here.
Note that for 39 we don't have unified behavior for people that haven't opted into Telemetry, so we should be fine.
I don't think we need to do anything for 39 except verify that things behave as expected.
Priority: P3 → P2
Summary: Validate we can turn Unified Telemetry off via add-on hot fix mechanism → UT: Validate the correct behavior when turning prefs off and on
Whiteboard: [b5][unifiedTelemtry]
updated description to reflect actual needed tests
Notes
---------------
- Everything refers to FHR, old telemetry, and unified telemetry collectively, with the prefs:
toolkit.telemetry.enabled, toolkit.telemetry.unified and datareporting.healthreport.service.enabled
- OT refers to just old telemetry and the pref toolkit.telemetry.enabled
- UT refers to unified telemetry and the pref toolkit.telemetry.unified
- FHR refers to firefox health reporting and the pref datareporting.healthreport.service.enabled
- Upload disabled refers to the pref datareporting.healthreport.uploadEnabled for FHR
- toggling FHR in the UI seems to control the upload pref but not the FHR service, which is confusing. on disabling the upload pref a DELETE request is sent

Test Cases
---------------
1. Everything disabled | PASS
State: clean start up with everything enabled, upload enabled
Expected result: nothing is sent
Actual result: nothing is sent

2. Everything enabled | PASS
State: clean start up with everything enabled, upload enabled
Expected result: data is sent to UT and FHR, but not OT
Actual result: data is submitted to https://incoming.telemetry.mozilla.org/submit/?v=4 and https://fhr.data.mozilla.com/1.0/submit/

3. FHR and OT enabled, UT disabled | PASS
State: clean start up with FHR and OT enabled, upload enabled
Expected result: data is sent to FHR and OT
Actual result: data is submitted to https://fhr.data.mozilla.com/1.0/submit/ and https://incoming.telemetry.mozilla.org/submit/

4. FHR enabled, but upload disabled | PASS
State: clean start up with FHR enabled, upload disabled
Expected result: no data sent
Actual result: no data sent

5. Everything enabled, but upload disabled | PASS
State: clean start up with everything enabled, upload disabled
Expected result: no data sent to FHR or OT but UT continues to send
Actual result: no data sent to FHR or OT but UT continues to send

6. Everything enabled, confirm UT is sending, then disable UT | FAIL?
Initial State: Active instance with everything enabled, upload enabled
Expected result: UT stops sending, OT starts sending again
Actual result: data still being sent to UT (v=4), but a restart seems to clear it up (see test case 3)

7. Everything enabled, confirm UT is sending, then disable upload | PASS
Initial State: Active instance with everything enabled, upload enabled
Expected result: UT continues sending but FHR stops
Actual result: UT continues sending with v=4, and fhr does not send data

8. FHR and OT enabled, confirm OT is sending, then enable UT | FAIL?
Initial State: Active instance with FHR and OT enabled, UT disabled, upload enabled
Expected result: FHR and UT send data, OT stops sending data
Actual result: FHR works, but UT does not seem to send data, a restart fixes it similar to test case 6

9. Disabling FHR results in a delete request being sent | PASS
expected result: delete request is sent to https://fhr.data.mozilla.com/1.0/submit/
result: delete request was sent to https://fhr.data.mozilla.com/1.0/submit/

Summary
---------------
Everything in general seems to work correctly, however a restart seems to be required when enabling/disabling UT. Is that expected and okay going to release? Seems like part of what we wanted to achieve was to be able to pref off in a hurry if needed.
Flags: needinfo?(mreid)
Thanks for putting this together.
Was this tested on 40 or 41?
Maybe we can gather QA information (required setup, triggering pings, relevant prefs, ...) in a wiki page for future reference?

(In reply to Stuart Philp :sphilp from comment #9)
> Summary
> ---------------
> Everything in general seems to work correctly, however a restart seems to be
> required when enabling/disabling UT. Is that expected and okay going to
> release? Seems like part of what we wanted to achieve was to be able to pref
> off in a hurry if needed.

This is expected, handling switching things off and on during a browser session would introduce a lot of additional complexity.
It should be sufficient to have clients switch over on restart.

> - toggling FHR in the UI seems to control the upload pref but not the FHR service, which is confusing. on disabling the upload pref a DELETE request is sent

We always record at least the "base" set of data. This is required to power selfsupport, about:healthreport, ...

---------------

For the sending of v4 data - we should always send v4 style of data independent of the UT pref:
* the UT pref controls if we send "main" pings (on shutdown, midnight or environment change)
* the OT pref controls - independently - if we send "saved-session" pings (on shutdown)

We should always submit to https://incoming.telemetry.mozilla.org/submit/?v=4 unless you happen to have old pings on disk that still need to be sent.
Does the above mean that we are not always submitting new pings to "?v=4" URLs?

---------------

> 2. Everything enabled | PASS
> State: clean start up with everything enabled, upload enabled
> Expected result: data is sent to UT and FHR, but not OT

Expected: data is sent to FHR and Telemetry, both "main" & "saved-session" pings

> 3. FHR and OT enabled, UT disabled | PASS
> State: clean start up with FHR and OT enabled, upload enabled
> Expected result: data is sent to FHR and OT

Expect: Expected: data is sent to FHR and Telemetry, only "saved-session" pings

> 5. Everything enabled, but upload disabled | PASS
> State: clean start up with everything enabled, upload disabled
> Expected result: no data sent to FHR or OT but UT continues to send
> Actual result: no data sent to FHR or OT but UT continues to send

Expected: no data sent
Note that the upload pref & the OT pref are coupled, disabling upload in the preferences also turns off OT.
I can't confirm this with normal behavior:
* have everything enabled
* go to preferences
* disable upload
* trigger a new ping (e.g. by disabling a plugin or calling TelemetryController.submitExternalPing("foobar", {}))

I think we need more details here to understand whether what you are seeing is expected behavior or not.

> 6. Everything enabled, confirm UT is sending, then disable UT | FAIL?
> Initial State: Active instance with everything enabled, upload enabled
> Expected result: UT stops sending, OT starts sending again

Expected: no further "main" pings sent, "saved-session" still being sent

> 7. Everything enabled, confirm UT is sending, then disable upload | PASS
> Initial State: Active instance with everything enabled, upload enabled
> Expected result: UT continues sending but FHR stops
> Actual result: UT continues sending with v=4, and fhr does not send data

Expected: no data sent after disabling upload

> 8. FHR and OT enabled, confirm OT is sending, then enable UT | FAIL?
> Initial State: Active instance with FHR and OT enabled, UT disabled, upload enabled
> Expected result: FHR and UT send data, OT stops sending data

Expected: FHR and "saved-session" pings sent, after enabling UT & restart "main" pings sent too

> 9. Disabling FHR results in a delete request being sent | PASS
> expected result: delete request is sent to https://fhr.data.mozilla.com/1.0/submit/
> result: delete request was sent to https://fhr.data.mozilla.com/1.0/submit/

expected on 41: delete request is sent to https://fhr.data.mozilla.com/1.0/submit/ & "deletion" ping sent to Telemetry
same on 40 when/if bug 1120379 gets uplifted
Thinking a bit about other features that can be affected here:
* experiments should keep working: https://wiki.mozilla.org/QA/Desktop_Firefox/Telemetry_Experiments
* about:healtreport should keep working with FHR off when bug 1150115 is fixed
* selfsupport features like Heartbeat should keep working, bug 1179226
* about:telemetry should keep working
Thanks Georg! I've been keeping notes on how to test, so I can dump that into a wiki after some basic formatting. This was all done on 40, I will check the expected results and other things you mentioned in a follow up.
Started a wiki https://mana.mozilla.org/wiki/display/~sphilp@mozilla.com/Verifying+FHR+and+UT+pings

From the revised expected results, it seems like the main issue was UT pings still sending after disabling upload, however I tried on nightly (42) and it looks to be resolved there.

I was confused with the v=4 stuff, I thought v=4 was only for UT and that OT would use the an older version num (or no version). To clarify: main and saved-session both send to v=4, and they look to be correct in that main pings are sent when UT is enabled, and saved-session for OT.

Confirmed the deletion ping is sent to UT as well when disabling upload.

Everything looks good on 42
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
(In reply to Stuart Philp :sphilp from comment #13)
> Started a wiki
> https://mana.mozilla.org/wiki/display/~sphilp@mozilla.com/
> Verifying+FHR+and+UT+pings

Great - can we move this to the public wiki so people without LDAP are not locked out? :)
(see e.g. https://wiki.mozilla.org/QA/Telemetry)

> From the revised expected results, it seems like the main issue was UT pings
> still sending after disabling upload, however I tried on nightly (42) and it
> looks to be resolved there.

Ok, that's unexpected on 40 - can you maybe file a bug with details so we can make sure there is nothing strange going on?
 
> I was confused with the v=4 stuff, I thought v=4 was only for UT and that OT
> would use the an older version num (or no version). To clarify: main and
> saved-session both send to v=4, and they look to be correct in that main
> pings are sent when UT is enabled, and saved-session for OT.
> 
> Confirmed the deletion ping is sent to UT as well when disabling upload.
> 
> Everything looks good on 42

Great to hear!
This looks good to me, but can we please update the docs at
 https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/toolkit/components/telemetry/telemetry/preferences.html
to call out prefs that require a restart to take effect?
Flags: needinfo?(mreid)
(In reply to Mark Reid [:mreid] from comment #15)
> This looks good to me, but can we please update the docs at
>  https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/toolkit/
> components/telemetry/telemetry/preferences.html
> to call out prefs that require a restart to take effect?

Good point, filed bug 1180738.
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.