1395933 - Execute in-tree unit tests for geckodriver

Status: RESOLVED FIXED

(Testing :: geckodriver, enhancement, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Since geckodriver landed in-tree and we build it via Taskcluster on various platforms the integrated Rust unit tests are no longer executed. I just noticed that myself by running `cargo test` locally.

Andreas, looks like we missed to add the execution of those beside just building geckodriver.

Comment 1

[Andreas Tolfsen ❲:ato❳]

We didn’t miss it.  It needs to be done.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I see. So this should now apply to both geckodriver and webdriver-rust.

Comment 3

[Andreas Tolfsen ❲:ato❳]

I’m going to split the work of getting the webdriver crate Rust
tests running out to a separate bug because it is a problem to mach
that it is not a separately vendored crate in m-c.  I’ve filed bug
https://bugzil.la/1424369 about this.

Comment 4

[Andreas Tolfsen ❲:ato❳]

Attachment: Bug 1395933 - Run geckodriver tests in automation.

Review commit: https://reviewboard.mozilla.org/r/206760/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206760/

Comment 5

[Andreas Tolfsen ❲:ato❳]

Attachment: Bug 1395933 - Expand geckodriver mach command with test and build targets.

The current "./mach geckodriver" commands runs the geckodriver
binary, but this patch introduces a geckodriver namespace to mach
and renames that command to "./mach geckodriver run".

It also introduces two new commands to build geckodriver and run
its Rust unit tests through "./mach geckodriver build" and "./mach
geckodriver test", respectively.

Review commit: https://reviewboard.mozilla.org/r/206762/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206762/

Comment 6

[Ralph Giles (:rillian)]

Comment on attachment 8935891 [details]
Bug 1395933 - Run geckodriver tests in automation.

https://reviewboard.mozilla.org/r/206760/#review212574

r=me assuming it passes tests.

Comment 7

[Ralph Giles (:rillian)]

Comment on attachment 8935892 [details]
Bug 1395933 - Expand geckodriver mach command with test and build targets.

https://reviewboard.mozilla.org/r/206762/#review212580

::: testing/geckodriver/CONTRIBUTING.md:157
(Diff revision 1)
>  a set of conformance tests with other browser vendors through the
>  [Web Platform Tests] (WPT) initiative.  This lets us ensure web
>  compatibility between _different_ WebDriver implementations for
>  different browsers.
>  
> -In addition to the WPT tests, geckodriver and libwebdriver has
> +In addition to the WPT tests, geckodriver and webdriver has unit tests.

`geckodriver and webdriver *have* unit tests` for correct number agreement.

::: testing/geckodriver/CONTRIBUTING.md:162
(Diff revision 1)
> -you need to kick off a separate build using [cargo]:
>  
> -	% cd testing/geckodriver
> +	% ./mach geckodriver test
> -	   Compiling geckodriver v0.19.0 (file:///home/ato/src/gecko/testing/geckodriver)
>  	…
> -	test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
> +	 0:02.39 test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Did you mean to add the runtime to the documented output here?

Comment 8

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 8935891 [details]
Bug 1395933 - Run geckodriver tests in automation.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206760/diff/1-2/

Comment 9

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 8935892 [details]
Bug 1395933 - Expand geckodriver mach command with test and build targets.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206762/diff/1-2/

Comment 10

[Andreas Tolfsen ❲:ato❳]

rillian: Thanks for all your useful input.  I’ve addressed your
concerns about testing/geckodriver/CONTRIBUTING.md.

There are some compile failures with winapi on windows2012, but
rebased and triggered a new try build to make sure they are real.

Comment 11

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8935892 [details]
Bug 1395933 - Expand geckodriver mach command with test and build targets.

https://reviewboard.mozilla.org/r/206762/#review213036

Comment 12

[Andreas Tolfsen ❲:ato❳]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=35d8193e484ceb6f995469c8d4258a535e97ba7a

Comment 13

[Andreas Tolfsen ❲:ato❳]

The instability of the Windows jobs seems to an unrelated issue to
the changes I am making in this changeset.  There is a intermittent
filed in https://bugzilla.mozilla.org/show_bug.cgi?id=1420472.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7827a7002f94
Run geckodriver tests in automation. r=rillian
https://hg.mozilla.org/integration/autoland/rev/81228d9e87e5
Expand geckodriver mach command with test and build targets. r=maja_zf,rillian

Comment 15

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 8935891 [details]
Bug 1395933 - Run geckodriver tests in automation.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206760/diff/2-3/

Comment 16

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 8935892 [details]
Bug 1395933 - Expand geckodriver mach command with test and build targets.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206762/diff/2-3/

Comment 17

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 8935892 [details]
Bug 1395933 - Expand geckodriver mach command with test and build targets.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206762/diff/3-4/

Comment 18

[Cosmin Sabou [:CosminS]]

Backed out for build bustages on js/src/jit-test/tests/typedobject/bug-1415313.js 

https://treeherder.mozilla.org/logviewer.html#?job_id=151467469&repo=autoland&lineNumber=326

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=81228d9e87e5a110ed42580e73c65c0f5fabefc1&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception

https://hg.mozilla.org/integration/autoland/rev/5b4ca3085e743cb2c45309b652f96f5741033f82

Comment 19

[Andreas Tolfsen ❲:ato❳]

Comment on attachment 8935892 [details]
Bug 1395933 - Expand geckodriver mach command with test and build targets.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206762/diff/4-5/

Comment 20

[Andreas Tolfsen ❲:ato❳]

I forgot to update the patches that were on mozreview.  The problems
had been fixed in my local branch.

New try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f7a06fa1f65c49e769b619574abc910ea4613511

Comment 21

[Andreas Tolfsen ❲:ato❳]

Even newer try run without artifact builds:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=18199899918112e3d72c2de8f66390f8b6bc2501

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7c394ade418a
Run geckodriver tests in automation. r=rillian
https://hg.mozilla.org/integration/autoland/rev/dfba797e7a1f
Expand geckodriver mach command with test and build targets. r=maja_zf,rillian

Comment 23

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/7c394ade418a
https://hg.mozilla.org/mozilla-central/rev/dfba797e7a1f