Closed
Bug 1123683
Opened 10 years ago
Closed 10 years ago
Set default testing preferences (e.g. disabling FHR and other metric pings)
Categories
(Testing :: Firefox UI Tests, defect)
Testing
Firefox UI Tests
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla38
People
(Reporter: whimboo, Assigned: galgeek)
References
Details
Attachments
(1 file, 2 obsolete files)
For Mozmill tests we have been asked to disable certain preferences which exist to collect metrics and send them to the production server. This is interfering with their statistics and should not be done.
That said we should figure out which of the preferences have to be set by Marionette directly.
Currently there are already a lot of those preferences listed in mozprofile, but it might be better to get them out of this package and move them into the test harness. Reason is that also mozrunner is affected, and it could be used by non test harnesses, whereby testing prefs do not apply.
http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprofile/mozprofile/profile.py#330
The reason why so many entries exist here is that we added them directly here in the past instead of Mozmill. Lately added preferences for Mozmill can be found here:
https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L138
There was bug 905400 to rework those preferences for Mozmill, which is unfinished.
Testing preferences for Mochitest can be found here:
http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js
Those might want to give us a good indication what's needed.
So we should investigate which of those preferences are necessary and need to be added. I will ask Barbara if she can get her hands on that.
Comment 1•10 years ago
|
||
is there anyway we can use the preferences from in-tree, maybe point at the mochitest prefs_general.js or something like that?
Reporter | ||
Comment 2•10 years ago
|
||
I assume those would need to be packaged together with the marionette client, so you can run tests outside the tree.
Keep in mind that this would also add a dependency to marionette-client. Changes to this file could cause damage to customers of marionette-client. Also backporting to older branches would not be that easy, in case of preferences getting removed due to new features but older branches still have those.
Assignee | ||
Comment 3•10 years ago
|
||
Does it make sense to read in http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js (per tree) as well as a second marionette-specific or firefox-puppeteer-specific (also likely per tree) prefs set?
Or does it make sense for marionette to maintain its own equivalent of http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js (per tree)? Maybe still with a second firefox-puppeteer-specific prefs set?
Comment 4•10 years ago
|
||
WebDriver has created the following list of prefs that need to be disabled. The frozen ones are not allowed to be changed by the user through webdriver APIs but mutable ones can be changed as we see fit.
https://code.google.com/p/selenium/source/browse/javascript/firefox-driver/webdriver.json
Assignee | ||
Comment 5•10 years ago
|
||
Assignee | ||
Comment 6•10 years ago
|
||
Comment on attachment 8552052 [details]
prefs set by Mozmill but not by webdriver, compiled 2015-01-20
This attachment shows prefs set by Mozmill but not by Webdriver. I don't see any frozen Webdriver prefs among them.
Mozmill prefs from these two sources:
https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L138
http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprofile/mozprofile/profile.py#330
Webdriver prefs from this source:
https://code.google.com/p/selenium/source/browse/javascript/firefox-driver/webdriver.json
Attachment #8552052 -
Attachment description: Mozmill preferences compiled 2015-01-20 → prefs set by Mozmill but not by webdriver, compiled 2015-01-20
Flags: needinfo?(hskupin)
Assignee | ||
Comment 7•10 years ago
|
||
Another option for setting preferences is a lock file.
More here: http://kb.mozillazine.org/Locking_preferences
Assignee | ||
Updated•10 years ago
|
Assignee: nobody → galgeek
Assignee | ||
Comment 8•10 years ago
|
||
This attachment shows prefs set by Mozmill but not by prefs_general, quite similar to the comparison to webdriver.
prefs_general source:
http://mxr.mozilla.org/mozilla-central/source/testing/profiles/prefs_general.js
(Mozmill prefs sources as for webdriver comparison.)
Assignee | ||
Comment 9•10 years ago
|
||
A couple of other notes from today's prefs review:
All of the prefs settings added by mozmill's __init__.py are present only there, not in webdriver prefs nor in prefs_general.
'datareporting.healthreport.documentServerURI', 'http://%(server)s/healthreport/' is NOT mozmill-only; the colon after the http got caught by my prefs-set-massaging.
The first 30+ lines of prefs_general is labelled "// Base preferences file used by most test harnesses."
I noticed several prefs settings in the webdriver prefs and in prefs_general that looked like alternatives to some "mozmill-only" settings, including...
53 // Set a future policy version to avoid the telemetry prompt. (prefs_general)
54 user_pref("toolkit.telemetry.prompted", 999);
55 user_pref("toolkit.telemetry.notifiedOptOut", 999);
I could come up with a more complete list of these Wednesday.
Assignee | ||
Comment 10•10 years ago
|
||
We’re collecting more information in this etherpad:
https://etherpad.mozilla.org/automation-firefox-ui-tests
Flags: needinfo?(hskupin)
Assignee | ||
Updated•10 years ago
|
Attachment #8552052 -
Attachment is obsolete: true
Assignee | ||
Updated•10 years ago
|
Attachment #8552199 -
Attachment is obsolete: true
Updated•10 years ago
|
Keywords: ateam-marionette-client,
ateam-marionette-runner
Assignee | ||
Comment 11•10 years ago
|
||
I've updated the etherpad ( https://etherpad.mozilla.org/automation-firefox-ui-tests ) with draft lists of "necessary and wanted" prefs, and "questions" after reviewing with Henrik earlier today current marionette client settings, current mozmill settings, and the set of webdriver prefs noted above.
We haven't yet reviewed the prefs_general set.
I'll be following up on the "questions" list Tuesday.
Flags: needinfo?(hskupin)
Assignee | ||
Comment 12•10 years ago
|
||
I've added on the etherpad, from yesterday's "questions" list, a list of "likely additions."
There are some remaining "questions" that show up on the prefs_general list, too, which Henrik and I have not yet reviewed together.
Reporter | ||
Comment 13•10 years ago
|
||
I find the etherpad way confusing to read. Not sure where I can find the draft list you proposed. I would say you add that at the beginning and prefs which are not clear or not going to be used at the end. The mix-up between all the different frameworks is not relevant anymore i think.
Flags: needinfo?(hskupin)
Assignee | ||
Comment 14•10 years ago
|
||
The first draft is at the top of this etherpad: https://etherpad.mozilla.org/automation-firefox-ui-tests
now more clearly marked and sorted alphabetically.
I could post diffs against current mozmill prefs or one of the other sets we've looked at, if that would be helpful.
Assignee | ||
Comment 15•10 years ago
|
||
Attachment #8556935 -
Flags: feedback?(hskupin)
Assignee | ||
Comment 16•10 years ago
|
||
I've updated the github PR per Henrik's feedback there.
Reporter | ||
Comment 17•10 years ago
|
||
Given that we want to get those prefs added to our Firefox UI tests harness, lets move the bug to the right component.
Status: NEW → ASSIGNED
Component: Marionette → Firefox UI Tests
Product: Testing → Mozilla QA
Reporter | ||
Comment 18•10 years ago
|
||
Comment on attachment 8556935 [details] [review]
github_pull_request.txt
Those additions look fine to me. So lets get them added. We can fine-tune later in case it's necessary.
Attachment #8556935 -
Flags: feedback?(hskupin) → review+
Reporter | ||
Comment 19•10 years ago
|
||
PR has been merged as:
https://github.com/mozilla/firefox-ui-tests/commit/16a2a3a8dd49c1e5854b2e7cf506b4876b59fa7e
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Flags: in-testsuite-
Keywords: ateam-marionette-client,
ateam-marionette-runner
Resolution: --- → FIXED
Summary: Marionette has to set default testing preferences (e.g. disabling FHR and other metric pings) → Set default testing preferences (e.g. disabling FHR and other metric pings)
Target Milestone: --- → Firefox 38
Updated•9 years ago
|
Product: Mozilla QA → Testing
You need to log in
before you can comment on or make changes to this bug.
Description
•