Closed Bug 1650907 Opened 2 months ago Closed 2 months ago

geckodriver/mozbase unit tests are no longer run as part of rusttests

Categories

(Testing :: geckodriver, defect, P1)

Product:
Testing
Component:
geckodriver
Type:
defect

Tracking

(firefox-esr68 unaffected, firefox-esr78 wontfix, firefox78 wontfix, firefox79 wontfix, firefox80 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla80
Tracking Flags:
Tracking Status
firefox-esr68 --- unaffected
firefox-esr78 --- wontfix
firefox78 --- wontfix
firefox79 --- wontfix
firefox80 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file)

Bug 1650907 - Re-enable mozbase, webdriver, and geckodriver rust tests in CI.
47 bytes, text/x-phabricator-request
Details | Review

Since we stopped building the geckodriver binaries via the build job (bug 1557741) we no longer run any of its Rust unittests as part of the rusttests jobs.

Chris, do we have to run these tests as part of the toolchain job instead, or make rusttests dependent on the geckodriver toolchain build job?

Flags: needinfo?(catlee)

either way could work. how long do the tests take?

Flags: needinfo?(catlee)

See here for the last time these tests were run:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=rusttests&revision=8f68705097b4bf88cd61b43b14401cde98ac75b6

[task 2020-05-19T16:16:31.063Z] 16:16:31     INFO -      Finished release [optimized] target(s) in 54.59s
[task 2020-05-19T16:16:31.063Z] 16:16:31     INFO -       Running `/builds/worker/workspace/obj-build/testing/geckodriver/./i686-unknown-linux-gnu/release/deps/geckodriver-9d768cd946fbada3`
[task 2020-05-19T16:16:31.063Z] 16:16:31     INFO -  running 53 tests
[task 2020-05-19T16:16:31.064Z] 16:16:31     INFO -  test capabilities::tests::fx_options_android_device_serial ... ok
[..]
[task 2020-05-19T16:16:31.091Z] 16:16:31     INFO -  test response::tests::test_json_window_rect_response ... ok
[task 2020-05-19T16:16:31.091Z] 16:16:31     INFO -  test result: ok. 200 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
[task 2020-05-19T16:16:31.091Z] 16:16:31     INFO -  make[1]: Leaving directory '/builds/worker/workspace/obj-build/testing/geckodriver'

So it's all done within a second.

In that case, maybe it makes sense to do as part of the toolchain job?

Personally I would be happy with that. It would directly show us if something fails and no need to wait for some other jobs.

Chris, can you please help us with that? We are going to have a geckodriver release today/tomorrow. So having final test results would be great.

Flags: needinfo?(catlee)

Chris submitted a patch to try but that fails on MacOS only:

https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=UYtc360DS_KkgPBKKoljVQ.0&revision=46b34b85c3b69fa9684b7e6a5780d9e3fbf77001

[task 2020-07-08T15:21:37.705Z] error: linking with `/builds/worker/workspace/build/src/taskcluster/scripts/misc/osx-cross-linker` failed: exit code: 1
[task 2020-07-08T15:21:37.705Z]   |
[task 2020-07-08T15:21:37.706Z]   = note: "/builds/worker/workspace/build/src/taskcluster/scripts/misc/osx-cross-linker" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.0.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.1.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.2.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.3.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.4.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.5.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.6.rcgu.o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.7.rcgu.o" "-o" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159" "/builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.aheq60pvfpub4me.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/builds/worker/workspace/build/src/target/debug/deps" "-L" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7c5e456310a1373c.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-b981d9b2a408308f.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-43d0ea1b5ae34d0d.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-09e7f22e773899cd.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace-aa74f166651adf6e.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-22c386707b639611.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-db04c9c5cd3bcf45.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-bb27492f721492e8.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6b95245dbf686e20.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-287409d75db2ecd3.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-a93f70ee2006b6e3.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-566cdfbcc94b4360.rlib" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-7bb8dddc7ce34e92.rlib" "-Wl,--end-group" "/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b117658e17259aa6.rlib" "-Wl,-Bdynamic" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
[task 2020-07-08T15:21:37.706Z]   = note: clang version 9.0.1 
[task 2020-07-08T15:21:37.706Z]           Target: x86_64-apple-darwin
[task 2020-07-08T15:21:37.706Z]           Thread model: posix
[task 2020-07-08T15:21:37.706Z]           InstalledDir: /builds/worker/fetches/clang/bin
[task 2020-07-08T15:21:37.706Z]           clang-9: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument]
[task 2020-07-08T15:21:37.706Z]            "/builds/worker/fetches/cctools/bin/x86_64-apple-darwin-ld" -dynamic -arch x86_64 -macosx_version_min 10.11.0 -syslibroot /builds/worker/fetches/MacOSX10.11.sdk -o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159 -L/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib -L/builds/worker/workspace/build/src/target/debug/deps -L/builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib --as-needed -z noexecstack /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.0.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.1.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.2.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.3.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.4.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.5.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.6.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.build_script_build.6hb1pcqp-cgu.7.rcgu.o /builds/worker/workspace/build/src/target/debug/build/log-8cf15826fc019159/build_script_build-8cf15826fc019159.aheq60pvfpub4me.rcgu.o --gc-sections -zrelro -znow --start-group -Bstatic /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7c5e456310a1373c.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-b981d9b2a408308f.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-43d0ea1b5ae34d0d.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-09e7f22e773899cd.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace-aa74f166651adf6e.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-22c386707b639611.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-db04c9c5cd3bcf45.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-bb27492f721492e8.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-6b95245dbf686e20.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-287409d75db2ecd3.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-a93f70ee2006b6e3.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-566cdfbcc94b4360.rlib /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-7bb8dddc7ce34e92.rlib --end-group /builds/worker/fetches/rustc/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b117658e17259aa6.rlib -Bdynamic -ldl -lrt -lpthread -lgcc_s -lc -lm -lrt -lpthread -lutil -lutil
[task 2020-07-08T15:21:37.706Z]           ld: unknown option: --as-needed
[task 2020-07-08T15:21:37.706Z]           clang-9: error: linker command failed with exit code 1 (use -v to see invocation)

Nathan, any idea what this could be? It's only happening for cargo test but not cargo build.

Flags: needinfo?(catlee) → needinfo?(nfroyd)

It looks like what's happening is that the OS X linker is getting used for the host (i.e. Linux) binary links, due to this line:

https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/build-geckodriver.sh#42

I'm not entirely sure how this works with the build generally -- maybe we just get very very lucky and don't have any build scripts or something? Or RUSTFLAGS is applied inconsistently within cargo itself?

I think according to https://doc.rust-lang.org/cargo/reference/environment-variables.html we want to try setting CARGO_TARGET_x86_64_apple_darwin_LINKER instead.

Flags: needinfo?(nfroyd)

Oh wait. That's a good call. While the build has to produce binaries for the target platform, the unit tests would be run only on Linux that way because that's the platform we use to cross-compile for each supported platform. That also means we won't be able to run the tests on Windows and MacOS.

Looks like we would have to set a dependency to the rusttests task, or would we only have to include the testing/geckodriver/ folder to run the included tests?

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #9)

Oh wait. That's a good call. While the build has to produce binaries for the target platform, the unit tests would be run only on Linux that way because that's the platform we use to cross-compile for each supported platform. That also means we won't be able to run the tests on Windows and MacOS.

Under the current scheme, yes.

Looks like we would have to set a dependency to the rusttests task, or would we only have to include the testing/geckodriver/ folder to run the included tests?

Running the geckodriver tests as part of the rusttests tasks is probably the way to go if you want cross-platform coverage, yes. I don't know if you want rusttests to block the geckodriver build, exactly; it seems like that would have negative effects on automation builds generally? But maybe there's something I'm missing there.

(In reply to Nathan Froyd [:froydnj] from comment #10)

Running the geckodriver tests as part of the rusttests tasks is probably the way to go if you want cross-platform coverage, yes. I don't know if you want rusttests to block the geckodriver build, exactly; it seems like that would have negative effects on automation builds generally? But maybe there's something I'm missing there.

Right, we don't want to have them blocked. So just let them run in parallel. So what would be necessary for rusttests to get the geckodriver/webdriver/mozbase tests running again? By checking the task definition it's not clear to me.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #11)

(In reply to Nathan Froyd [:froydnj] from comment #10)

Running the geckodriver tests as part of the rusttests tasks is probably the way to go if you want cross-platform coverage, yes. I don't know if you want rusttests to block the geckodriver build, exactly; it seems like that would have negative effects on automation builds generally? But maybe there's something I'm missing there.

Right, we don't want to have them blocked. So just let them run in parallel. So what would be necessary for rusttests to get the geckodriver/webdriver/mozbase tests running again? By checking the task definition it's not clear to me.

You should be able to add RUST_TESTS += ['geckodriver'] in testing/geckodriver.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #13)

That is actually what we have:

https://searchfox.org/mozilla-central/source/testing/geckodriver/moz.build#9

But tests aren't run.

I guess that's because geckodriver isn't getting built during the rusttests build since we moved it out to be built in its own task, and therefore the build system never sees testing/geckodriver/moz.build?

If so, then we'd need to --enable-geckodriver for the rusttests build?

I assume that the following lines stop doing anything for geckodriver in build and rust test jobs?

https://searchfox.org/mozilla-central/rev/8d55e18875b89cdf2a22a7cba60dc40999c18356/toolkit/moz.configure#984-985

Would I have to combine this condition with enable_tests to get the geckodriver tests running?

Hm, but maybe this also enables building geckodriver as part of the build job again? If that is the case is there a way to only run the tests?

Flags: needinfo?(nfroyd)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #18)

Hm, but maybe this also enables building geckodriver as part of the build job again? If that is the case is there a way to only run the tests?

Just --enable-geckodriver in the configure options for the rusttests build. The lines in comment 12 are only for computing the default, and passing an explicit option will bypass that code.

Flags: needinfo?(nfroyd)
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Priority: P3 → P1
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e3d0b3f05852
Re-enable mozbase, webdriver, and geckodriver rust tests in CI. r=froydnj

https://hg.mozilla.org/mozilla-central/rev/e3d0b3f05852

Status: ASSIGNED → RESOLVED
Closed: 2 months ago
status-firefox80: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla80
You need to log in before you can comment on or make changes to this bug.