Set default testing preferences (e.g. disabling FHR and other metric pings)

RESOLVED FIXED in mozilla38

Status

defect
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: whimboo, Assigned: galgeek)

Tracking

unspecified
mozilla38
Points:
---
Dependency tree / graph
Bug Flags:
in-testsuite -

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 2 obsolete attachments)

For Mozmill tests we have been asked to disable certain preferences which exist to collect metrics and send them to the production server. This is interfering with their statistics and should not be done.

That said we should figure out which of the preferences have to be set by Marionette directly.

Currently there are already a lot of those preferences listed in mozprofile, but it might be better to get them out of this package and move them into the test harness. Reason is that also mozrunner is affected, and it could be used by non test harnesses, whereby testing prefs do not apply.

http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprofile/mozprofile/profile.py#330

The reason why so many entries exist here is that we added them directly here in the past instead of Mozmill. Lately added preferences for Mozmill can be found here:

https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L138

There was bug 905400 to rework those preferences for Mozmill, which is unfinished.

Testing preferences for Mochitest can be found here:
http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js

Those might want to give us a good indication what's needed.

So we should investigate which of those preferences are necessary and need to be added. I will ask Barbara if she can get her hands on that.
is there anyway we can use the preferences from in-tree, maybe point at the mochitest prefs_general.js or something like that?
I assume those would need to be packaged together with the marionette client, so you can run tests outside the tree. 

Keep in mind that this would also add a dependency to marionette-client. Changes to this file could cause damage to customers of marionette-client. Also backporting to older branches would not be that easy, in case of preferences getting removed due to new features but older branches still have those.
Assignee

Comment 3

4 years ago
Does it make sense to read in http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js (per tree) as well as a second marionette-specific or firefox-puppeteer-specific (also likely per tree) prefs set?

Or does it make sense for marionette to maintain its own equivalent of http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js (per tree)? Maybe still with a second firefox-puppeteer-specific prefs set?
WebDriver has created the following list of prefs that need to be disabled. The frozen ones are not allowed to be changed by the user through webdriver APIs but mutable ones can be changed as we see fit.

https://code.google.com/p/selenium/source/browse/javascript/firefox-driver/webdriver.json
Assignee

Comment 6

4 years ago
Comment on attachment 8552052 [details]
prefs set by Mozmill but not by webdriver, compiled 2015-01-20

This attachment shows prefs set by Mozmill but not by Webdriver. I don't see any frozen Webdriver prefs among them.

Mozmill prefs from these two sources:

https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L138

http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprofile/mozprofile/profile.py#330

Webdriver prefs from this source:

https://code.google.com/p/selenium/source/browse/javascript/firefox-driver/webdriver.json
Attachment #8552052 - Attachment description: Mozmill preferences compiled 2015-01-20 → prefs set by Mozmill but not by webdriver, compiled 2015-01-20
Flags: needinfo?(hskupin)
Assignee

Comment 7

4 years ago
Another option for setting preferences is a lock file.

More here: http://kb.mozillazine.org/Locking_preferences
Assignee

Updated

4 years ago
Assignee: nobody → galgeek
Assignee

Comment 8

4 years ago
This attachment shows prefs set by Mozmill but not by prefs_general, quite similar to the comparison to webdriver.

prefs_general source:

http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js

(Mozmill prefs sources as for webdriver comparison.)
Assignee

Comment 9

4 years ago
A couple of other notes from today's prefs review:

All of the prefs settings added by mozmill's __init__.py are present only there, not in webdriver prefs nor in prefs_general.

'datareporting.healthreport.documentServerURI', 'http://%(server)s/healthreport/' is NOT mozmill-only; the colon after the http got caught by my prefs-set-massaging.

The first 30+ lines of prefs_general is labelled "// Base preferences file used by most test harnesses."

I noticed several prefs settings in the webdriver prefs and in prefs_general that looked like alternatives to some "mozmill-only" settings, including...

53 // Set a future policy version to avoid the telemetry prompt.    (prefs_general)
54 user_pref("toolkit.telemetry.prompted", 999);
55 user_pref("toolkit.telemetry.notifiedOptOut", 999); 

I could come up with a more complete list of these Wednesday.
Assignee

Comment 10

4 years ago
We’re collecting more information in this etherpad:

https://etherpad.mozilla.org/automation-firefox-ui-tests
Flags: needinfo?(hskupin)
Assignee

Updated

4 years ago
Attachment #8552052 - Attachment is obsolete: true
Assignee

Updated

4 years ago
Attachment #8552199 - Attachment is obsolete: true
Assignee

Comment 11

4 years ago
I've updated the etherpad ( https://etherpad.mozilla.org/automation-firefox-ui-tests ) with draft lists of "necessary and wanted" prefs, and "questions" after reviewing with Henrik earlier today current marionette client settings, current mozmill settings, and the set of webdriver prefs noted above.

We haven't yet reviewed the prefs_general set.

I'll be following up on the "questions" list Tuesday.
Flags: needinfo?(hskupin)
Assignee

Comment 12

4 years ago
I've added on the etherpad, from yesterday's "questions" list, a list of "likely additions."

There are some remaining "questions" that show up on the prefs_general list, too, which Henrik and I have not yet reviewed together.
I find the etherpad way confusing to read. Not sure where I can find the draft list you proposed. I would say you add that at the beginning and prefs which are not clear or not going to be used at the end. The mix-up between all the different frameworks is not relevant anymore i think.
Flags: needinfo?(hskupin)
Assignee

Comment 14

4 years ago
The first draft is at the top of this etherpad: https://etherpad.mozilla.org/automation-firefox-ui-tests
now more clearly marked and sorted alphabetically.

I could post diffs against current mozmill prefs or one of the other sets we've looked at, if that would be helpful.
Assignee

Comment 15

4 years ago
Attachment #8556935 - Flags: feedback?(hskupin)
Assignee

Comment 16

4 years ago
I've updated the github PR per Henrik's feedback there.
Given that we want to get those prefs added to our Firefox UI tests harness, lets move the bug to the right component.
Status: NEW → ASSIGNED
Component: Marionette → Firefox UI Tests
Product: Testing → Mozilla QA
Comment on attachment 8556935 [details] [review]
github_pull_request.txt

Those additions look fine to me. So lets get them added. We can fine-tune later in case it's necessary.
Attachment #8556935 - Flags: feedback?(hskupin) → review+
PR has been merged as:
https://github.com/mozilla/firefox-ui-tests/commit/16a2a3a8dd49c1e5854b2e7cf506b4876b59fa7e
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Flags: in-testsuite-
Resolution: --- → FIXED
Summary: Marionette has to set default testing preferences (e.g. disabling FHR and other metric pings) → Set default testing preferences (e.g. disabling FHR and other metric pings)
Target Milestone: --- → Firefox 38
Product: Mozilla QA → Testing
You need to log in before you can comment on or make changes to this bug.