Closed Bug 1528236 Opened 5 years ago Closed 5 years ago

Improve profile and preference handling

Categories

(Testing :: Raptor, task, P1)

Version 3
task

Tracking

(firefox68 fixed)

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: Bebe, Assigned: whimboo)

References

Details

Attachments

(9 files, 1 obsolete file)

8.49 KB, text/plain
Details
11.54 KB, patch
Details | Diff | Splinter Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review

Raptor is not setting preferences if the app is geckoview
We can set these preferences when creating the profile

Add this capability to raptor

https://searchfox.org/mozilla-central/source/testing/raptor/raptor/raptor.py#210

Assignee: nobody → fstrugariu
Status: NEW → ASSIGNED
Priority: -- → P2

Good find :bebe! Yes so turns out no Raptor tests running on geckoview currently use custom prefs, that's why this hasn't been spotted before now. I thought 'ugl' used prefs but they don't - only some 'wasm' variations on Raptor desktop do, i.e. [0].

No rush to fix this but if any tests coming online on geckoview need custom prefs in the Raptor test INI then this should be fixed for that.

[0] https://searchfox.org/mozilla-central/rev/a7315d78417179b151fef6108f2bce14786ba64d/testing/raptor/raptor/tests/raptor-wasm-godot-baseline.ini#21

This blocks my work for streaming tests. So taking this bug.

Assignee: fstrugariu → hskupin
Blocks: 1539111
Type: enhancement → defect
Priority: P2 → P1

Note, that this problem is not only for custom prefs in an INI file! Also some preferences as specified in testing/profiles/common/user.js which should be used for Raptor, aren't present when geckoview_example is running. The profile at /sdcard/raptor/profile contains all the user preferences.

This is actually a diff between the user.js file as generated by Raptor, and the prefs.js file as used by geckoview_example app. As you can see a lot of user preferences are not getting saved to prefs.js, and as such might not be used.

The problem here is actually not in Raptor, or how we use mozprofile and such, but the GeckoRuntimeSettings are resetting preferences even user defined ones. See bug 1547717. As such there is nothing we can do on that end.

But nevertheless while working on this bug, and trying to investigate the problem I found a couple of bugs, and also improvements Raptor could be happy with. So I will address those here.

Summary: Raptor does not set preferences when running on on geckoview → Improve profile and preference handling
Depends on: 1547933

We cannot land this patch before bug 1547933 has been fixed.

Priority: P1 → P2

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #14)

We cannot land this patch before bug 1547933 has been fixed.

Well that bug is actually a regression from my patch on this bug. We have to mock the ADBDevice in the tests to not require the adb command which isn't available on those workers.

Maybe I get to it next week.

Due to missing class abstractions for Gecko- and non-Gecko based
browsers it's currently sub-optimal to define when preferences
as defined by tests will be set.

Given that by default we run Gecko-based applications other
browsers should opt-out from setting test preferences.

Depends on D30922

I will revert parts of the patch, so that we no longer will initialize the ADBDevice immediately, and as such don't run into those failures for Raptor python-tests.

There is so much to fix in the Raptor code, which actually makes it hard for me to not blow-up this patch series even more. I hope that the upcoming remaining patches are a good compromise for all of us. More improvements can always be done at a later point.

Priority: P2 → P1
Blocks: 1539090
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c6cccc4ebfcc
[raptor] Refactor handling of remote profiles. r=perftest-reviewers,rwood
https://hg.mozilla.org/integration/autoland/rev/9eb4d019ee37
[raptor] Remove test root folder during clean-up. r=perftest-reviewers,rwood
https://hg.mozilla.org/integration/autoland/rev/e8d6d453337d
[raptor] Allow Raptor to run the tests all at once. r=perftest-reviewers,stephendonner,rwood
https://hg.mozilla.org/integration/autoland/rev/09eea8c87cd5
[raptor] Use constructor for initialization of raptor instances. r=perftest-reviewers,rwood
https://hg.mozilla.org/integration/autoland/rev/ed679c1e266d
[raptor] Update profile building code to align with mochitests. r=perftest-reviewers,rwood
https://hg.mozilla.org/integration/autoland/rev/90a51dbd6ee1
[raptor] Environment variables should not be set via extra args. r=perftest-reviewers,rwood
https://hg.mozilla.org/integration/autoland/rev/d12eb6d840fc
[raptor] Fix root permission failures for device tuning. r=perftest-reviewers,stephendonner
https://hg.mozilla.org/integration/autoland/rev/0d0a4250902b
[raptor] Use opt-out mechanism for setting test specific preferences. r=perftest-reviewers,stephendonner

Backed out changeset 90a51dbd6ee1 (Bug 1528236) for causing permafailures in raptor tests CLOSED TREE

Failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&fromchange=e1437ba34a4afff6cee82968503144f15e9e1949&selectedJob=246473621

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=246473621&repo=autoland&lineNumber=1293

21:11:51 INFO - raptor-main Exception: TypeError launch_application() got an unexpected keyword argument 'moz_env'
21:11:51 INFO - raptor-main removing reverse socket connections
21:11:51 INFO - adb command_output: adb -s ZY3222K8ND wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
21:11:51 INFO - raptor-main No crash directory (/sdcard/raptor/profile/minidumps) found on remote device
21:11:51 INFO - mozproxy Stopping mitmproxy playback, killing process 676
21:11:51 INFO - mozproxy Successfully killed the mitmproxy playback process
21:11:51 INFO - raptor-main removing webext /builds/task_1557868152/workspace/build/tests/raptor/raptor/../webext/raptor
21:11:51 INFO - raptor-main removing test folder for raptor: /sdcard/raptor
21:11:51 INFO - adb shell_output: adb -s ZY3222K8ND wait-for-device shell rm -r /sdcard/raptor; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
21:11:51 INFO - raptor-control-server shutting down control server
21:11:52 INFO - raptor-main finished
21:11:52 ERROR - Traceback (most recent call last):
21:11:52 INFO - File "/builds/task_1557868152/workspace/build/tests/raptor/raptor/raptor.py", line 1225, in <module>
21:11:52 INFO - main()
21:11:52 INFO - File "/builds/task_1557868152/workspace/build/tests/raptor/raptor/raptor.py", line 1193, in main
21:11:52 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
21:11:52 INFO - File "/builds/task_1557868152/workspace/build/tests/raptor/raptor/raptor.py", line 955, in run_tests
21:11:52 INFO - return super(RaptorAndroid, self).run_tests(tests, test_names)
21:11:52 INFO - File "/builds/task_1557868152/workspace/build/tests/raptor/raptor/raptor.py", line 186, in run_tests
21:11:52 INFO - self.run_test(test, timeout=int(test['page_timeout']))
21:11:52 INFO - File "/builds/task_1557868152/workspace/build/tests/raptor/raptor/raptor.py", line 974, in run_test
21:11:52 INFO - self.run_test_cold(test, timeout)
21:11:52 INFO - File "/builds/task_1557868152/workspace/build/tests/raptor/raptor/raptor.py", line 1062, in run_test_cold
21:11:52 INFO - self.launch_firefox_android_app(test['name'])
21:11:52 INFO - File "/builds/task_1557868152/workspace/build/tests/raptor/raptor/raptor.py", line 923, in launch_firefox_android_app
21:11:52 INFO - fail_if_running=False)
21:11:52 ERROR - TypeError: launch_application() got an unexpected keyword argument 'moz_env'
21:11:52 ERROR - Return code: 1
21:11:52 WARNING - setting return code to 1
21:11:52 INFO - Killing logcat pid 466.
21:11:52 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
21:11:52 INFO - copying raptor results to upload dir:
21:11:52 INFO - /builds/task_1557868152/workspace/build/blobber_upload_dir/perfherder-data.json
21:11:52 INFO - copying raptor results from /builds/task_1557868152/workspace/build/raptor.json to /builds/task_1557868152/workspace/build/blobber_upload_dir/perfherder-data.json
21:11:52 CRITICAL - Error copying results /builds/task_1557868152/workspace/build/raptor.json to upload dir /builds/task_1557868152/workspace/build/blobber_upload_dir/perfherder-data.json
21:11:52 INFO - [Errno 2] No such file or directory: u'/builds/task_1557868152/workspace/build/raptor.json'
21:11:52 INFO - Running post-action listener: _package_coverage_data
21:11:52 INFO - Running post-action listener: _resource_record_post_action
21:11:52 INFO - Running post-action listener: process_java_coverage_data
21:11:52 INFO - Running post-action listener: stop_device
21:12:01 INFO - /data/tombstones/dsps deleted
21:12:03 INFO - /data/tombstones/lpass deleted

Flags: needinfo?(rwood)
Backout by shindli@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/304607e12723
Backed out changeset 90a51dbd6ee1 for causing permafailures in raptor tests CLOSED TREE

I will have a look at this when I'm back on Friday. Most likely will push this to a different bug if necessary at all.

Flags: needinfo?(rwood) → needinfo?(hskupin)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #25)

I will have a look at this when I'm back on Friday. Most likely will push this to a different bug if necessary at all.

I filed bug 1552296 as a follow-up bug. So no further work is needed on this bug.

Flags: needinfo?(hskupin)
Attachment #9061600 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: