Local Speedometer3 browsertime run fails with an "System access is required" error
Categories
(Testing :: Raptor, defect, P1)
Tracking
(firefox161 fixed)
| Tracking | Status | |
|---|---|---|
| firefox161 | --- | fixed |
People
(Reporter: canova, Assigned: aglavic)
References
Details
Attachments
(1 external file)
I was trying to run speedometer3 test locally with this command:
./mach raptor -t speedometer3 --browser-cycles 1 --page-cycles 1 --post-startup-delay 0
but I get this error:
(snip)
10:28:42 INFO - raptor-browsertime Info: Mark run as failure with message: System access is required. Start Firefox with the "MOZ_REMOTE_ALLOW_SYSTEM_ACCESS=1" environment variable set to enable it.
10:28:42 ERROR - raptor-browsertime Error: Browsertime failed to run
10:28:42 CRITICAL - raptor-browsertime Critical: UnsupportedOperationError: System access is required. Start Firefox with the "MOZ_REMOTE_ALLOW_SYSTEM_ACCESS=1" environment variable set to enable it.
10:28:43 INFO - raptor-browsertime Info: Test support class is cleaning up...
10:28:43 INFO - raptor-perftest Info: Removing temporary directory: /var/folders/ld/5vn2pljj0mx6qxcq_g8v_h9m0000gn/T/tmp209xx34n
10:28:43 ERROR - Traceback (most recent call last):
10:28:43 INFO - File "/Volumes/mozilla-source/firefox-2/testing/raptor/raptor/raptor.py", line 196, in <module>
10:28:43 INFO - main()
10:28:43 INFO - ~~~~^^
10:28:43 INFO - File "/Volumes/mozilla-source/firefox-2/testing/raptor/raptor/raptor.py", line 141, in main
10:28:43 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
10:28:43 INFO - File "/Volumes/mozilla-source/firefox-2/testing/raptor/raptor/perftest.py", line 590, in run_tests
10:28:43 INFO - self.run_test(test, timeout=int(test.get("page_timeout")))
10:28:43 INFO - ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10:28:43 INFO - File "/Volumes/mozilla-source/firefox-2/testing/raptor/raptor/browsertime/base.py", line 1248, in run_test
10:28:43 INFO - raise Exception(self.browsertime_failure)
10:28:43 INFO - Exception: UnsupportedOperationError: System access is required. Start Firefox with the "MOZ_REMOTE_ALLOW_SYSTEM_ACCESS=1" environment variable set to enable it.
10:28:43 INFO - Return code: 1
10:28:43 WARNING - setting return code to 1
10:28:43 INFO - Running post-action listener: _package_coverage_data
10:28:43 INFO - Running post-action listener: _resource_record_post_action
10:28:43 INFO - Running post-action listener: stop_device
10:28:43 INFO - [mozharness: 2026-10-19 10:28:43.495730Z] Finished run-tests step (success)
10:28:43 INFO - Running post-run listener: _resource_record_post_run
10:28:53 INFO - Total resource usage - Wall time: 19s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
10:28:53 INFO - TinderboxPrint: I/O read bytes / time<br/>0 / 0
10:28:53 INFO - TinderboxPrint: I/O write bytes / time<br/>0 / 0
10:28:53 INFO - TinderboxPrint: Swap in / out<br/>0 / 0
10:28:53 INFO - install-chromium-distribution - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
10:28:53 INFO - run-tests - Wall time: 9s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
10:28:53 INFO - Collecting usage
10:28:53 INFO - Processing ping at: /Users/naltinova/.mozbuild/glean/pending_pings/2a1ef1ce-f8fd-4b2d-b57a-5d434755d8fa
10:28:53 INFO - The ping 'usage' was submitted and will be sent as soon as possible
Sentry is attempting to send 1 pending events
Waiting up to 2 seconds
Press Ctrl-C to quit
The most relevant line is:
Exception: UnsupportedOperationError: System access is required. Start Firefox with the "MOZ_REMOTE_ALLOW_SYSTEM_ACCESS=1" environment variable set to enable it.
When I pass MOZ_REMOTE_ALLOW_SYSTEM_ACCESS=1 I can now run it locally. But ideally ./mach raptor command should just work without me needing to do anything special. I think there are multiple solutions, but the easiest would be to pass this env variable while starting Firefox.
Comment 1•28 days ago
|
||
hey :canova, I believe this is because by default ./mach raptor uses the geckodriver bundled with our browsertime (0.36.0 from Feb 2025), while CI passes because it uses the geckodriver built from the tree. That old version still passes --remote-allow-system-access to Firefox, but Firefox 159 only supports the MOZ_REMOTE_ALLOW_SYSTEM_ACCESS env var now. The tree's geckodriver sets that env var, which is why CI works.
You could grab the latest geckodriver from CI and pass it with --browsertime-geckodriver, but that's kinda cumbersome and not really better than just setting the env var like you did, so that's the easiest workaround for now.
a proper fix is probably updating our browsertime, since upstream is on geckodriver 0.37.1 now, which sets the env var. That update has a couple of breaking changes upstream so it'll need some testing; I'll file a bug for it and link it here.
Longer term I'll have to think about it a bit more, since relying on the bundled geckodriver will probably bite us again.
I am thinking maybe have raptor use the objdir geckodriver when there is one, or pull the latest one from a mozilla-central build like CI does....
| Reporter | ||
Comment 2•28 days ago
|
||
Oh I see. Yeah it looks like Feb 2025 is a bit outdated. I see that full mach build locally also builds the geckodriver. For example, I could find it in objdir/dist/host/bin/geckodriver in my local checkout. I think artifact builds don't compile it though. But I think it would be nice to use that latest built geckodriver if we are also using that locally built Firefox for mach raptor for non-artifact builds. That would definitely solve my issue.
I'm not so sure what's the backwards compatibility rules of geckodriver though. For example, if browsertime works with 0.37.1 does it mean that it will always work with 0.38.0 and the following versions? If not that could be a problem too, but it seems like the CI already has this, so it might make sense to do the same.
| Assignee | ||
Comment 3•26 days ago
|
||
Local runs fall back to the geckodriver bundled with browsertime, which
doesn't set the variable Firefox now requires for system access.
Updated•26 days ago
|
Comment 4•26 days ago
|
||
(In reply to Nazım Can Altınova [:canova] from comment #2)
Oh I see. Yeah it looks like Feb 2025 is a bit outdated. I see that full
mach buildlocally also builds the geckodriver. For example, I could find it inobjdir/dist/host/bin/geckodriverin my local checkout. I think artifact builds don't compile it though. But I think it would be nice to use that latest built geckodriver if we are also using that locally built Firefox formach raptorfor non-artifact builds. That would definitely solve my issue.
good to know! so if user is not running artifact it would make sense to point to the locally built gecko driver... i'll file a bug for that.
I'm not so sure what's the backwards compatibility rules of geckodriver though. For example, if browsertime works with
0.37.1does it mean that it will always work with 0.38.0 and the following versions? If not that could be a problem too, but it seems like the CI already has this, so it might make sense to do the same.
yeah i am not sure. At the very least i appreciate it is not strict on versions like chromedriver seems to be :P
Comment 5•26 days ago
|
||
andrej's patch is a good stopgap for now while we think of a better long term solution for this...
| Assignee | ||
Updated•25 days ago
|
Comment 7•24 days ago
|
||
| bugherder | ||
Description
•