1550718 - Set preferences in telemetry-tests-client tests that persist across browser restarts

Status: RESOLVED FIXED

(Toolkit :: Telemetry, enhancement, P1)

Description

[Raphael Aurich [:raphael] UTC+02:00]

While working on bug 1539830 we have discovered that we don't seem to have support for setting browser preferences in automation that persist across browser restarts. In order to thoroughly test the optout ping, we need a way to do this.

Attempts to use marionette.enforce_gecko_prefs() for this have been unsuccessful.

Comment 1

[Chris H-C :chutten]

Specifically, if I have a test of the form

self.marionette.set_pref("datareporting.healthreport.uploadEnabled", False)
# perform some tests of behaviour that is triggered by the preference change
self.restart_browser()
# perform some tests of behaviour before the pref is changed back
self.marionette.set_pref("datareporting.healthreport.uploadEnabled", True);
# perform some tests of behaviour in response to the preference change

The second set_pref call is a no-op because the browser that started had already reset the pref to True.

I attempted to use enforce_gecko_prefs like so

self.marionette.set_pref("datareporting.healthreport.uploadEnabled", False)
# perform some tests of behaviour that is triggered by the preference change
self.marionette.enforce_gecko_prefs({"datareporting.healthreport.uploadEnabled": False})
self.restart_browser()
# perform some tests of behaviour before the pref is changed back
self.marionette.set_pref("datareporting.healthreport.uploadEnabled", True);
# perform some tests of behaviour in response to the preference change

But since the pref was already set earlier, enforce_gecko_prefs is a no-op and the pref is reset to True as before.

(( Also, ideally I'd be able to close the browser and then perform some tests before starting it up again (essentially splitting restart_browser into two pieces) but I couldn't find a way to do that properly. Probably not too relevant here except for discussion of things I tried. ))

Comment 2

[Chris H-C :chutten]

Raphael, would you happen to know what component this should go to? Is there a particular marionette or harness BMO product?

Comment 3

[Raphael Aurich [:raphael] UTC+02:00]

There's a Testing :: Marionette BMO product.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

No, this is not Marionette. telemetry-client is it's own harness. And enforce_gecko_prefs() as listed above works as expected. If it shouldn't be a no-op, you cannot set a preference to a user value before. So remove the first call to set_pref().

Comment 5

[Chris H-C :chutten]

Unfortunately in this case we need to run some tests after the pref is set so enforce_gecko_prefs() won't get the job done (it'll restart the instance). Is there a Marionette-puppeting-Firefox idiomatic way to set a pref in a running Firefox, test some stuff, quit firefox normally, test some stuff, start firefox normally (with that aforementioned pref still set), test some stuff, flip the pref back, test some stuff, quit firefox normally, and test some stuff?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's complicated logic. So in regards the pref it will be tricky, and once set enforce_gecko_prefs() will indeed not change it nor restart. Also this method needs an overhaul (bug 1260502). Feel free to add this use-case.

In the interim you could mangle with the profile directly, and use this method:

https://searchfox.org/mozilla-central/rev/cc280c4be94ff8cf64a27cc9b3d6831ffa49fa45/testing/mozbase/mozprofile/mozprofile/profile.py#296

This might work.

Also you can use self.marionette.quit() to quit the browser. Then you can run tests and start it again. See for example this unit test:

https://searchfox.org/mozilla-central/rev/cc280c4be94ff8cf64a27cc9b3d6831ffa49fa45/testing/marionette/harness/marionette_harness/tests/unit/test_profile_management.py#81-86

Comment 7

[Chris H-C :chutten]

Thank you :whimboo, those links look promising. I'll give it another go and see if I can write the test using those methods.

Comment 8

[Chris H-C :chutten]

Attachment: Bug 1550718 - Test Telemetry opt out across sessions r?raphael

This supercedes the previous test and ensures the optout persists across
sessions. It would also ideally ensure there's no Pending Pings present,
but the tests appear to be currently using any available app data dir
instead of a test-specific one so we should implement that part later.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fb0a19250082
Test Telemetry opt out across sessions r=raphael

Comment 10

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/fb0a19250082