Closed Bug 1123683 Opened 10 years ago Closed 10 years ago

Set default testing preferences (e.g. disabling FHR and other metric pings)

Categories

(Testing :: Firefox UI Tests, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla38

People

(Reporter: whimboo, Assigned: galgeek)

References

Details

Attachments

(1 file, 2 obsolete files)

For Mozmill tests we have been asked to disable certain preferences which exist to collect metrics and send them to the production server. This is interfering with their statistics and should not be done. That said we should figure out which of the preferences have to be set by Marionette directly. Currently there are already a lot of those preferences listed in mozprofile, but it might be better to get them out of this package and move them into the test harness. Reason is that also mozrunner is affected, and it could be used by non test harnesses, whereby testing prefs do not apply. http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprofile/mozprofile/profile.py#330 The reason why so many entries exist here is that we added them directly here in the past instead of Mozmill. Lately added preferences for Mozmill can be found here: https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L138 There was bug 905400 to rework those preferences for Mozmill, which is unfinished. Testing preferences for Mochitest can be found here: http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js Those might want to give us a good indication what's needed. So we should investigate which of those preferences are necessary and need to be added. I will ask Barbara if she can get her hands on that.
is there anyway we can use the preferences from in-tree, maybe point at the mochitest prefs_general.js or something like that?
I assume those would need to be packaged together with the marionette client, so you can run tests outside the tree. Keep in mind that this would also add a dependency to marionette-client. Changes to this file could cause damage to customers of marionette-client. Also backporting to older branches would not be that easy, in case of preferences getting removed due to new features but older branches still have those.
Does it make sense to read in http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js (per tree) as well as a second marionette-specific or firefox-puppeteer-specific (also likely per tree) prefs set? Or does it make sense for marionette to maintain its own equivalent of http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js (per tree)? Maybe still with a second firefox-puppeteer-specific prefs set?
WebDriver has created the following list of prefs that need to be disabled. The frozen ones are not allowed to be changed by the user through webdriver APIs but mutable ones can be changed as we see fit. https://code.google.com/p/selenium/source/browse/javascript/firefox-driver/webdriver.json
Comment on attachment 8552052 [details] prefs set by Mozmill but not by webdriver, compiled 2015-01-20 This attachment shows prefs set by Mozmill but not by Webdriver. I don't see any frozen Webdriver prefs among them. Mozmill prefs from these two sources: https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L138 http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprofile/mozprofile/profile.py#330 Webdriver prefs from this source: https://code.google.com/p/selenium/source/browse/javascript/firefox-driver/webdriver.json
Attachment #8552052 - Attachment description: Mozmill preferences compiled 2015-01-20 → prefs set by Mozmill but not by webdriver, compiled 2015-01-20
Flags: needinfo?(hskupin)
Another option for setting preferences is a lock file. More here: http://kb.mozillazine.org/Locking_preferences
Assignee: nobody → galgeek
This attachment shows prefs set by Mozmill but not by prefs_general, quite similar to the comparison to webdriver. prefs_general source: http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js (Mozmill prefs sources as for webdriver comparison.)
A couple of other notes from today's prefs review: All of the prefs settings added by mozmill's __init__.py are present only there, not in webdriver prefs nor in prefs_general. 'datareporting.healthreport.documentServerURI', 'http://%(server)s/healthreport/' is NOT mozmill-only; the colon after the http got caught by my prefs-set-massaging. The first 30+ lines of prefs_general is labelled "// Base preferences file used by most test harnesses." I noticed several prefs settings in the webdriver prefs and in prefs_general that looked like alternatives to some "mozmill-only" settings, including... 53 // Set a future policy version to avoid the telemetry prompt. (prefs_general) 54 user_pref("toolkit.telemetry.prompted", 999); 55 user_pref("toolkit.telemetry.notifiedOptOut", 999); I could come up with a more complete list of these Wednesday.
We’re collecting more information in this etherpad: https://etherpad.mozilla.org/automation-firefox-ui-tests
Flags: needinfo?(hskupin)
Attachment #8552052 - Attachment is obsolete: true
Attachment #8552199 - Attachment is obsolete: true
I've updated the etherpad ( https://etherpad.mozilla.org/automation-firefox-ui-tests ) with draft lists of "necessary and wanted" prefs, and "questions" after reviewing with Henrik earlier today current marionette client settings, current mozmill settings, and the set of webdriver prefs noted above. We haven't yet reviewed the prefs_general set. I'll be following up on the "questions" list Tuesday.
Flags: needinfo?(hskupin)
I've added on the etherpad, from yesterday's "questions" list, a list of "likely additions." There are some remaining "questions" that show up on the prefs_general list, too, which Henrik and I have not yet reviewed together.
I find the etherpad way confusing to read. Not sure where I can find the draft list you proposed. I would say you add that at the beginning and prefs which are not clear or not going to be used at the end. The mix-up between all the different frameworks is not relevant anymore i think.
Flags: needinfo?(hskupin)
The first draft is at the top of this etherpad: https://etherpad.mozilla.org/automation-firefox-ui-tests now more clearly marked and sorted alphabetically. I could post diffs against current mozmill prefs or one of the other sets we've looked at, if that would be helpful.
Attachment #8556935 - Flags: feedback?(hskupin)
I've updated the github PR per Henrik's feedback there.
Given that we want to get those prefs added to our Firefox UI tests harness, lets move the bug to the right component.
Status: NEW → ASSIGNED
Component: Marionette → Firefox UI Tests
Product: Testing → Mozilla QA
Comment on attachment 8556935 [details] [review] github_pull_request.txt Those additions look fine to me. So lets get them added. We can fine-tune later in case it's necessary.
Attachment #8556935 - Flags: feedback?(hskupin) → review+
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Flags: in-testsuite-
Resolution: --- → FIXED
Summary: Marionette has to set default testing preferences (e.g. disabling FHR and other metric pings) → Set default testing preferences (e.g. disabling FHR and other metric pings)
Target Milestone: --- → Firefox 38
Product: Mozilla QA → Testing
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: