Enable Wdspec tests on Android
Categories
(Testing :: web-platform-tests, task, P2)
Tracking
(firefox100 fixed)
Tracking | Status | |
---|---|---|
firefox100 | --- | fixed |
People
(Reporter: whimboo, Assigned: whimboo)
References
(Blocks 5 open bugs)
Details
(Whiteboard: [bidi-m3-mvp], [wptsync upstream])
Attachments
(4 files)
Trying to run Wdspec jobs for Android in CI gives the following failure:
https://treeherder.mozilla.org/logviewer?job_id=363672422&repo=try&lineNumber=1461
[task 2022-01-11T05:38:18.502Z] 05:38:18 INFO - Running main action method: start_emulator
[task 2022-01-11T05:38:18.504Z] 05:38:18 FATAL - The adb binary '/builds/worker/fetches/android-sdk-linux/platform-tools/adb' is not a valid file!
[task 2022-01-11T05:38:18.504Z] 05:38:18 FATAL - Running post_fatal callback...
[task 2022-01-11T05:38:18.504Z] 05:38:18 FATAL - Exiting -1
James could you maybe have a look at this? As it looks like we fail to start the emulator and I wonder what's different to other wpt jobs here. Thanks.
Comment 1•2 years ago
|
||
It looks like we're setting the wrong toolchain
property in the wdspec job definitions. Because we need geckodriver, we endup overwriting the defaults from https://searchfox.org/mozilla-central/source/taskcluster/ci/test/kind.yml#71-76. I think you need to update https://searchfox.org/mozilla-central/source/taskcluster/ci/test/web-platform.yml#243 (and elsewhere in that file) to include all the android requirements in the android case.
Assignee | ||
Comment 2•2 years ago
|
||
Oh I see! That's indeed a good explanation. Thanks!
Given that this work is closely related to enabling the wdspec jobs on Android lets actually have it all in this bug. Once a try build runs fine we will have to update the meta data whereby a lot of tests will be marked as expected fail.
Assignee | ||
Comment 3•2 years ago
|
||
That actually brings us a bit forward and we can start the emulator, but then sadly fail in running certutil
:
https://treeherder.mozilla.org/jobs?repo=try&revision=e2f70f7747d344e5dda134f822b9d3179664d393
[task 2022-01-11T21:42:14.543Z] 21:42:14 CRITICAL - subprocess.CalledProcessError: Command '['/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil', '-N', '-d', '/tmp/tmprfyom2sb.mozrunner', '-f', '/tmp/tmprfyom2sb.mozrunner/.crtdbpw']' returned non-zero exit status 1.
[task 2022-01-11T21:42:14.745Z] 21:42:14 ERROR - Return code: 1
Comparing the path to certutil with other wpt jobs I cannot find a difference. So not sure what exactly is causing it to fail and what the exit code of 1 actually means here. I may have to add some logging to mozharness to figure that out given that interactive tasks for this kind of job do not work.
Assignee | ||
Comment 4•2 years ago
|
||
With an interactive task I can see that launching certutil
fails with the following failure dumped to the console:
workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil: /usr/lib/x86_64-linux-gnu/libnss3.so: version `NSS_3
.39' not found (required by workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil)
Here more details about this binary:
(venv) worker@466a5ae7f537:~$ file workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil
workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil: ELF 64-bit LSB shared object, x86-64, version 1 (SYS
V), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=b29d6711ffb91163c1ed23f2
0d02456ec1c14579, with debug_info, not stripped
Maybe I missed to specify some other dependency for the android jobs?
https://hg.mozilla.org/try/rev/80a54faaeae329ff8788ba868f8e1c3bd21845b2
Mike do you have an idea? Thanks
Assignee | ||
Comment 5•2 years ago
|
||
Note that this is the exact same path as wpt jobs in general also make use of and which gets passed into runtests.py
:
--certutil-binary=/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil
So why it's only failing for our wdspec jobs?
Comment 6•2 years ago
|
||
Something is probably going wrong with this test:
https://searchfox.org/mozilla-central/rev/afbff3f3ae087181d9988fe19e09cc15927fb7ff/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py#738-743
Assignee | ||
Comment 7•2 years ago
|
||
That was a good hint. Thanks Mike!
I pushed another try build to see which path we actually set for certutils and here is the difference between a normal wpt and wdspec job:
wpt:
https://treeherder.mozilla.org/logviewer?job_id=364602584&repo=try&lineNumber=1847-1850
[task 2022-01-19T09:09:50.749Z] 09:09:50 INFO - STDOUT: *** self.binary=None
[task 2022-01-19T09:09:50.750Z] 09:09:50 INFO - STDOUT: *** self.certutil_binary=/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil
[task 2022-01-19T09:09:50.750Z] 09:09:50 INFO - STDOUT: *** certutil dir: /builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64
[task 2022-01-19T09:09:50.751Z] 09:09:50 INFO - STDOUT: *** certutil env_var: /builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64
wdspec:
https://treeherder.mozilla.org/logviewer?job_id=364602588&repo=try&lineNumber=1707-1710
[task 2022-01-19T09:12:20.774Z] 09:12:20 INFO - STDOUT: *** self.binary=/tmp
[task 2022-01-19T09:12:20.777Z] 09:12:20 INFO - STDOUT: *** self.certutil_binary=/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil
[task 2022-01-19T09:12:20.778Z] 09:12:20 INFO - STDOUT: *** certutil dir: /
[task 2022-01-19T09:12:20.778Z] 09:12:20 INFO - STDOUT: *** certutil env_var: /
So for wdspec the binary is not None and as such the following code takes the directory of the binary:
certutil_dir = os.path.dirname(self.binary or self.certutil_binary)
I think that if a specific certutil binary has been passed as argument we should use its own path and if not specified fallback to the binary.
Assignee | ||
Comment 8•2 years ago
|
||
It's all working now. And with additional fixes for certutil (see above), Marionette (assert for desktop for window manipulation commands), and wdspec fixtures (don't resize the window if not supported on that platform when resetting the session) I only have to update the manifest data.
A try build without the update of manifest data has been just started:
https://treeherder.mozilla.org/jobs?repo=try&revision=c32ad6ccfcaacca55528a78b783b3cfd44df54cb
Comment 10•2 years ago
|
||
The severity field is not set for this bug.
:jgraham, could you have a look please?
For more information, please visit auto_nag documentation.
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Comment 11•2 years ago
|
||
The problem with the binary
and androidPackage
capability has actually been fixed by bug 1753083.
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Comment 12•2 years ago
|
||
Assignee | ||
Comment 13•2 years ago
•
|
||
Due to a lot of timeouts due to missing support for the new window command and handling modal dialogs the wdspec job on Android gets force killed. As such we do not upload any artifacts:
[taskcluster:error] Task timeout after 5400 seconds. Force killing container.
[taskcluster 2022-02-17 10:10:01.460Z] === Task Finished ===
I might go ahead and disable all the user_prompts.py
tests that are going to fail right now.
Assignee | ||
Comment 14•2 years ago
|
||
Actually this is a huge amount of manifests that would have to be created. Maybe it will be better to get bug 1708105 fixed. I'll work with Olivia so that we can get this done soon.
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Updated•2 years ago
|
Assignee | ||
Comment 15•2 years ago
|
||
Now that user prompts are supported for alert, confirm and prompt the tests successfully run through and artifacts are provided. I updated a lot of meta data files to disable all the tests that make use of the WebDriver:NewWindow
command and others for intermittent failures.
The following try push should hopefully be green now so that I will push the patches for review soon:
https://treeherder.mozilla.org/jobs?repo=try&revision=92687673f3bcbe379464f44cb34ddd1b120538e6
Assignee | ||
Comment 16•2 years ago
|
||
Depends on D138996
Assignee | ||
Comment 17•2 years ago
|
||
Depends on D142530
Assignee | ||
Comment 18•2 years ago
|
||
Depends on D142531
Comment 19•2 years ago
|
||
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b8b21732202e [wdspec] Enable tests on Android x86_64. r=jmaher https://hg.mozilla.org/integration/autoland/rev/8c346bdbdd59 [wdspec] Use timestamps from the remote end for comparison. r=webdriver-reviewers,jdescottes https://hg.mozilla.org/integration/autoland/rev/4a784f34b8a8 [wdspec] Move file protocol tests for "get current url" and "navigate to" into its own test module. r=webdriver-reviewers,jgraham https://hg.mozilla.org/integration/autoland/rev/496bfcc6f908 [wdspec] Update wpt meta data for Android jobs. r=webdriver-reviewers,jgraham
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/33453 for changes under testing/web-platform/tests
Comment 21•2 years ago
|
||
Backed out for causing wpt failures on printcmd.py
- Backout link
- Push with failures
- Failure Log
- Failure line: TEST-UNEXPECTED-FAIL | /webdriver/tests/print/printcmd.py | test_html_document - AssertionError: unknown error (500): Failed to decode response from marionette
Assignee | ||
Updated•2 years ago
|
Upstream PR was closed without merging
Assignee | ||
Comment 23•2 years ago
|
||
I'm going to reland the patches with the print to PDF tests disabled for now on Android debug jobs because these are the ones failing on autoland but not on try. This is very confusing! Once tests are running in CI I will have a look at bug 1762428 to get these tests re-enabled.
Comment 24•2 years ago
|
||
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/75fd101c4574 [wdspec] Enable tests on Android x86_64. r=jmaher https://hg.mozilla.org/integration/autoland/rev/7be929c7cc88 [wdspec] Use timestamps from the remote end for comparison. r=webdriver-reviewers,jdescottes https://hg.mozilla.org/integration/autoland/rev/86259f6ad100 [wdspec] Move file protocol tests for "get current url" and "navigate to" into its own test module. r=webdriver-reviewers,jgraham https://hg.mozilla.org/integration/autoland/rev/acbd3772f0c9 [wdspec] Update wpt meta data for Android jobs. r=webdriver-reviewers,jgraham
Comment 25•2 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/75fd101c4574
https://hg.mozilla.org/mozilla-central/rev/7be929c7cc88
https://hg.mozilla.org/mozilla-central/rev/86259f6ad100
https://hg.mozilla.org/mozilla-central/rev/acbd3772f0c9
Upstream PR merged by moz-wptsync-bot
Description
•