geckodriver/mozbase unit tests are no longer run as part of rusttests
Categories
(Testing :: geckodriver, defect, P1)
Tracking
(firefox-esr68 unaffected, firefox-esr78 wontfix, firefox78 wontfix, firefox79 wontfix, firefox80 fixed)
Tracking | Status | |
---|---|---|
firefox-esr68 | --- | unaffected |
firefox-esr78 | --- | wontfix |
firefox78 | --- | wontfix |
firefox79 | --- | wontfix |
firefox80 | --- | fixed |
People
(Reporter: whimboo, Assigned: whimboo)
References
(Regression)
Details
(Keywords: regression)
Attachments
(1 file)
Since we stopped building the geckodriver binaries via the build job (bug 1557741) we no longer run any of its Rust unittests as part of the rusttests
jobs.
Chris, do we have to run these tests as part of the toolchain job instead, or make rusttests dependent on the geckodriver toolchain build job?
Assignee | ||
Comment 2•5 years ago
|
||
See here for the last time these tests were run:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=rusttests&revision=8f68705097b4bf88cd61b43b14401cde98ac75b6
[task 2020-05-19T16:16:31.063Z] 16:16:31 INFO - Finished release [optimized] target(s) in 54.59s
[task 2020-05-19T16:16:31.063Z] 16:16:31 INFO - Running `/builds/worker/workspace/obj-build/testing/geckodriver/./i686-unknown-linux-gnu/release/deps/geckodriver-9d768cd946fbada3`
[task 2020-05-19T16:16:31.063Z] 16:16:31 INFO - running 53 tests
[task 2020-05-19T16:16:31.064Z] 16:16:31 INFO - test capabilities::tests::fx_options_android_device_serial ... ok
[..]
[task 2020-05-19T16:16:31.091Z] 16:16:31 INFO - test response::tests::test_json_window_rect_response ... ok
[task 2020-05-19T16:16:31.091Z] 16:16:31 INFO - test result: ok. 200 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
[task 2020-05-19T16:16:31.091Z] 16:16:31 INFO - make[1]: Leaving directory '/builds/worker/workspace/obj-build/testing/geckodriver'
So it's all done within a second.
Assignee | ||
Comment 3•5 years ago
|
||
The Rust tests are listed here:
https://searchfox.org/mozilla-central/source/testing/geckodriver/moz.build#9-19
Comment 4•5 years ago
|
||
In that case, maybe it makes sense to do as part of the toolchain job?
Assignee | ||
Comment 5•5 years ago
|
||
Personally I would be happy with that. It would directly show us if something fails and no need to wait for some other jobs.
Assignee | ||
Comment 6•5 years ago
|
||
Chris, can you please help us with that? We are going to have a geckodriver release today/tomorrow. So having final test results would be great.
Updated•5 years ago
|
Assignee | ||
Comment 7•5 years ago
|
||
Chris submitted a patch to try but that fails on MacOS only:
[task 2020-07-08T15:21:37.705Z] error: linking with `/builds/worker/workspace/build/src/taskcluster/scripts/misc/osx-cross-linker` failed: exit code: 1
[task 2020-07-08T15:21:37.705Z] |
[task 2020-07-08T15:21:37.706Z] = note: "/builds/worker/workspace/build/src/taskcluster/scripts/misc/osx-cross-linker" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.0.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.1.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.2.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.3.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.4.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.5.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.6.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.7.rcgu.o" "-o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.aheq60pvfpub4me.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/builds/worker/workspace/build/src/target/debug/deps" "-L" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7c5e456310a1373c.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-b981d9b2a408308f.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-43d0ea1b5ae34d0d.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-09e7f22e773899cd.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace-aa74f166651adf6e.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-22c386707b639611.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-db04c9c5cd3bcf45.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-bb27492f721492e8.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6b95245dbf686e20.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-287409d75db2ecd3.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-a93f70ee2006b6e3.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-566cdfbcc94b4360.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-7bb8dddc7ce34e92.rlib" "-Wl,--end-group" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b117658e17259aa6.rlib" "-Wl,-Bdynamic" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
[task 2020-07-08T15:21:37.706Z] = note: clang version 9.0.1
[task 2020-07-08T15:21:37.706Z] Target: x86_64-apple-darwin
[task 2020-07-08T15:21:37.706Z] Thread model: posix
[task 2020-07-08T15:21:37.706Z] InstalledDir: /builds/worker/fetches/clang/bin
[task 2020-07-08T15:21:37.706Z] clang-9: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument]
[task 2020-07-08T15:21:37.706Z] "/builds/worker/fetches/cctools/bin/x86_64-apple-darwin-ld" -dynamic -arch x86_64 -macosx_version_min 10.11.0 -syslibroot /builds/worker/fetches/MacOSX10.11.sdk -o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159 -L/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib -L/builds/worker/workspace/build/src/target/debug/deps -L/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib --as-needed -z noexecstack /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.0.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.1.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.2.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.3.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.4.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.5.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.6.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.7.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.aheq60pvfpub4me.rcgu.o --gc-sections -zrelro -znow --start-group -Bstatic /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7c5e456310a1373c.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-b981d9b2a408308f.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-43d0ea1b5ae34d0d.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-09e7f22e773899cd.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace-aa74f166651adf6e.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-22c386707b639611.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-db04c9c5cd3bcf45.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-bb27492f721492e8.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6b95245dbf686e20.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-287409d75db2ecd3.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-a93f70ee2006b6e3.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-566cdfbcc94b4360.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-7bb8dddc7ce34e92.rlib --end-group /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b117658e17259aa6.rlib -Bdynamic -ldl -lrt -lpthread -lgcc_s -lc -lm -lrt -lpthread -lutil -lutil
[task 2020-07-08T15:21:37.706Z] ld: unknown option: --as-needed
[task 2020-07-08T15:21:37.706Z] clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
Nathan, any idea what this could be? It's only happening for cargo test
but not cargo build
.
![]() |
||
Comment 8•5 years ago
|
||
It looks like what's happening is that the OS X linker is getting used for the host (i.e. Linux) binary links, due to this line:
https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/build-geckodriver.sh#42
I'm not entirely sure how this works with the build generally -- maybe we just get very very lucky and don't have any build scripts or something? Or RUSTFLAGS
is applied inconsistently within cargo itself?
I think according to https://doc.rust-lang.org/cargo/reference/environment-variables.html we want to try setting CARGO_TARGET_x86_64_apple_darwin_LINKER
instead.
Assignee | ||
Comment 9•5 years ago
|
||
Oh wait. That's a good call. While the build has to produce binaries for the target platform, the unit tests would be run only on Linux that way because that's the platform we use to cross-compile for each supported platform. That also means we won't be able to run the tests on Windows and MacOS.
Looks like we would have to set a dependency to the rusttests
task, or would we only have to include the testing/geckodriver/
folder to run the included tests?
![]() |
||
Comment 10•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #9)
Oh wait. That's a good call. While the build has to produce binaries for the target platform, the unit tests would be run only on Linux that way because that's the platform we use to cross-compile for each supported platform. That also means we won't be able to run the tests on Windows and MacOS.
Under the current scheme, yes.
Looks like we would have to set a dependency to the
rusttests
task, or would we only have to include thetesting/geckodriver/
folder to run the included tests?
Running the geckodriver tests as part of the rusttests
tasks is probably the way to go if you want cross-platform coverage, yes. I don't know if you want rusttests to block the geckodriver build, exactly; it seems like that would have negative effects on automation builds generally? But maybe there's something I'm missing there.
Assignee | ||
Comment 11•5 years ago
|
||
(In reply to Nathan Froyd [:froydnj] from comment #10)
Running the geckodriver tests as part of the
rusttests
tasks is probably the way to go if you want cross-platform coverage, yes. I don't know if you want rusttests to block the geckodriver build, exactly; it seems like that would have negative effects on automation builds generally? But maybe there's something I'm missing there.
Right, we don't want to have them blocked. So just let them run in parallel. So what would be necessary for rusttests
to get the geckodriver/webdriver/mozbase tests running again? By checking the task definition it's not clear to me.
![]() |
||
Comment 12•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #11)
(In reply to Nathan Froyd [:froydnj] from comment #10)
Running the geckodriver tests as part of the
rusttests
tasks is probably the way to go if you want cross-platform coverage, yes. I don't know if you want rusttests to block the geckodriver build, exactly; it seems like that would have negative effects on automation builds generally? But maybe there's something I'm missing there.Right, we don't want to have them blocked. So just let them run in parallel. So what would be necessary for
rusttests
to get the geckodriver/webdriver/mozbase tests running again? By checking the task definition it's not clear to me.
You should be able to add RUST_TESTS += ['geckodriver']
in testing/geckodriver
.
Assignee | ||
Comment 13•5 years ago
|
||
That is actually what we have:
https://searchfox.org/mozilla-central/source/testing/geckodriver/moz.build#9
But tests aren't run.
![]() |
||
Comment 14•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #13)
That is actually what we have:
https://searchfox.org/mozilla-central/source/testing/geckodriver/moz.build#9
But tests aren't run.
I guess that's because geckodriver isn't getting built during the rusttests build since we moved it out to be built in its own task, and therefore the build system never sees testing/geckodriver/moz.build
?
If so, then we'd need to --enable-geckodriver
for the rusttests build?
Assignee | ||
Comment 15•5 years ago
|
||
I assume that the following lines stop doing anything for geckodriver in build and rust test jobs?
Would I have to combine this condition with enable_tests
to get the geckodriver tests running?
Assignee | ||
Comment 16•5 years ago
|
||
Assignee | ||
Comment 17•5 years ago
|
||
Assignee | ||
Comment 18•5 years ago
|
||
Hm, but maybe this also enables building geckodriver as part of the build job again? If that is the case is there a way to only run the tests?
![]() |
||
Comment 19•5 years ago
|
||
(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #18)
Hm, but maybe this also enables building geckodriver as part of the build job again? If that is the case is there a way to only run the tests?
Just --enable-geckodriver
in the configure options for the rusttests build. The lines in comment 12 are only for computing the default, and passing an explicit option will bypass that code.
Assignee | ||
Comment 20•5 years ago
|
||
Ah ok. I just did that and here is the try build:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a31f9670404ec56bafcfaa96fa2e24672d8116ce
Updated•5 years ago
|
Assignee | ||
Updated•5 years ago
|
Comment 21•5 years ago
|
||
Comment 22•5 years ago
|
||
bugherder |
Updated•5 years ago
|
Description
•