Closed Bug 1749444 Opened 2 years ago Closed 2 years ago

Enable Wdspec tests on Android

Categories

(Testing :: web-platform-tests, task, P2)

Default
task
Points:
2

Tracking

(firefox100 fixed)

RESOLVED FIXED
100 Branch
Tracking Status
firefox100 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

(Blocks 5 open bugs)

Details

(Whiteboard: [bidi-m3-mvp], [wptsync upstream])

Attachments

(4 files)

Trying to run Wdspec jobs for Android in CI gives the following failure:

https://treeherder.mozilla.org/logviewer?job_id=363672422&repo=try&lineNumber=1461

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=eca79aa48424a7ae493aaafbdf62426ebb2ef004

[task 2022-01-11T05:38:18.502Z] 05:38:18     INFO - Running main action method: start_emulator
[task 2022-01-11T05:38:18.504Z] 05:38:18    FATAL - The adb binary '/builds/worker/fetches/android-sdk-linux/platform-tools/adb' is not a valid file!
[task 2022-01-11T05:38:18.504Z] 05:38:18    FATAL - Running post_fatal callback...
[task 2022-01-11T05:38:18.504Z] 05:38:18    FATAL - Exiting -1

James could you maybe have a look at this? As it looks like we fail to start the emulator and I wonder what's different to other wpt jobs here. Thanks.

Flags: needinfo?(james)

It looks like we're setting the wrong toolchain property in the wdspec job definitions. Because we need geckodriver, we endup overwriting the defaults from https://searchfox.org/mozilla-central/source/taskcluster/ci/test/kind.yml#71-76. I think you need to update https://searchfox.org/mozilla-central/source/taskcluster/ci/test/web-platform.yml#243 (and elsewhere in that file) to include all the android requirements in the android case.

Flags: needinfo?(james)

Oh I see! That's indeed a good explanation. Thanks!

Given that this work is closely related to enabling the wdspec jobs on Android lets actually have it all in this bug. Once a try build runs fine we will have to update the meta data whereby a lot of tests will be marked as expected fail.

Summary: The adb binary '/builds/worker/fetches/android-sdk-linux/platform-tools/adb' is not a valid file! → Enable Wdspec tests on Android

That actually brings us a bit forward and we can start the emulator, but then sadly fail in running certutil:

https://treeherder.mozilla.org/jobs?repo=try&revision=e2f70f7747d344e5dda134f822b9d3179664d393

[task 2022-01-11T21:42:14.543Z] 21:42:14 CRITICAL - subprocess.CalledProcessError: Command '['/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil', '-N', '-d', '/tmp/tmprfyom2sb.mozrunner', '-f', '/tmp/tmprfyom2sb.mozrunner/.crtdbpw']' returned non-zero exit status 1.
[task 2022-01-11T21:42:14.745Z] 21:42:14    ERROR - Return code: 1

Comparing the path to certutil with other wpt jobs I cannot find a difference. So not sure what exactly is causing it to fail and what the exit code of 1 actually means here. I may have to add some logging to mozharness to figure that out given that interactive tasks for this kind of job do not work.

With an interactive task I can see that launching certutil fails with the following failure dumped to the console:

workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil: /usr/lib/x86_64-linux-gnu/libnss3.so: version `NSS_3
.39' not found (required by workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil)

Here more details about this binary:

(venv) worker@466a5ae7f537:~$ file workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil 
workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil: ELF 64-bit LSB shared object, x86-64, version 1 (SYS
V), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=b29d6711ffb91163c1ed23f2
0d02456ec1c14579, with debug_info, not stripped

Maybe I missed to specify some other dependency for the android jobs?
https://hg.mozilla.org/try/rev/80a54faaeae329ff8788ba868f8e1c3bd21845b2

Mike do you have an idea? Thanks

Flags: needinfo?(mh+mozilla)

Note that this is the exact same path as wpt jobs in general also make use of and which gets passed into runtests.py:

--certutil-binary=/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil

So why it's only failing for our wdspec jobs?

That was a good hint. Thanks Mike!

I pushed another try build to see which path we actually set for certutils and here is the difference between a normal wpt and wdspec job:

wpt:
https://treeherder.mozilla.org/logviewer?job_id=364602584&repo=try&lineNumber=1847-1850

[task 2022-01-19T09:09:50.749Z] 09:09:50     INFO - STDOUT: *** self.binary=None
[task 2022-01-19T09:09:50.750Z] 09:09:50     INFO - STDOUT: *** self.certutil_binary=/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil
[task 2022-01-19T09:09:50.750Z] 09:09:50     INFO - STDOUT: *** certutil dir: /builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64
[task 2022-01-19T09:09:50.751Z] 09:09:50     INFO - STDOUT: *** certutil env_var: /builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64

wdspec:
https://treeherder.mozilla.org/logviewer?job_id=364602588&repo=try&lineNumber=1707-1710

[task 2022-01-19T09:12:20.774Z] 09:12:20     INFO - STDOUT: *** self.binary=/tmp
[task 2022-01-19T09:12:20.777Z] 09:12:20     INFO - STDOUT: *** self.certutil_binary=/builds/worker/workspace/build/hostutils/host-utils-93.0a1.en-US.linux-x86_64/certutil
[task 2022-01-19T09:12:20.778Z] 09:12:20     INFO - STDOUT: *** certutil dir: /
[task 2022-01-19T09:12:20.778Z] 09:12:20     INFO - STDOUT: *** certutil env_var: /

So for wdspec the binary is not None and as such the following code takes the directory of the binary:

certutil_dir = os.path.dirname(self.binary or self.certutil_binary)

I think that if a specific certutil binary has been passed as argument we should use its own path and if not specified fallback to the binary.

It's all working now. And with additional fixes for certutil (see above), Marionette (assert for desktop for window manipulation commands), and wdspec fixtures (don't resize the window if not supported on that platform when resetting the session) I only have to update the manifest data.

A try build without the update of manifest data has been just started:
https://treeherder.mozilla.org/jobs?repo=try&revision=c32ad6ccfcaacca55528a78b783b3cfd44df54cb

Blocks: 1563334
Blocks: 1559120
Blocks: 1560181
Blocks: 1578424
Depends on: 1751361
Depends on: 1751364
Depends on: 1751369
Depends on: 1751357

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)
Type: defect → task
Priority: -- → P3
Flags: needinfo?(james)
Depends on: 1753083
Blocks: 1753991

The problem with the binary and androidPackage capability has actually been fixed by bug 1753083.

No longer depends on: 1751369
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Depends on: 1754735
Depends on: 1723919
Depends on: 1506782
Depends on: 1754698

Due to a lot of timeouts due to missing support for the new window command and handling modal dialogs the wdspec job on Android gets force killed. As such we do not upload any artifacts:

[taskcluster:error] Task timeout after 5400 seconds. Force killing container.
[taskcluster 2022-02-17 10:10:01.460Z] === Task Finished ===

I might go ahead and disable all the user_prompts.py tests that are going to fail right now.

Actually this is a huge amount of manifests that would have to be created. Maybe it will be better to get bug 1708105 fixed. I'll work with Olivia so that we can get this done soon.

Depends on: 1708105
Blocks: 1761480
Blocks: 1506782
No longer depends on: 1506782
Points: --- → 2
Priority: P3 → P2
Whiteboard: [bidi-m3-mvp]

Now that user prompts are supported for alert, confirm and prompt the tests successfully run through and artifacts are provided. I updated a lot of meta data files to disable all the tests that make use of the WebDriver:NewWindow command and others for intermittent failures.

The following try push should hopefully be green now so that I will push the patches for review soon:
https://treeherder.mozilla.org/jobs?repo=try&revision=92687673f3bcbe379464f44cb34ddd1b120538e6

No longer depends on: 1754698
Blocks: 1762386
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b8b21732202e
[wdspec] Enable tests on Android x86_64. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/8c346bdbdd59
[wdspec] Use timestamps from the remote end for comparison. r=webdriver-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/4a784f34b8a8
[wdspec] Move file protocol tests for "get current url" and "navigate to" into its own test module. r=webdriver-reviewers,jgraham
https://hg.mozilla.org/integration/autoland/rev/496bfcc6f908
[wdspec] Update wpt meta data for Android jobs. r=webdriver-reviewers,jgraham
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/33453 for changes under testing/web-platform/tests
Whiteboard: [bidi-m3-mvp] → [bidi-m3-mvp], [wptsync upstream]
Regressions: 1762428

Backed out for causing wpt failures on printcmd.py

  • Backout link
  • Push with failures
  • Failure Log
  • Failure line: TEST-UNEXPECTED-FAIL | /webdriver/tests/print/printcmd.py | test_html_document - AssertionError: unknown error (500): Failed to decode response from marionette
Flags: needinfo?(hskupin)
Blocks: 1762428
No longer regressions: 1762428
Upstream PR was closed without merging

I'm going to reland the patches with the print to PDF tests disabled for now on Android debug jobs because these are the ones failing on autoland but not on try. This is very confusing! Once tests are running in CI I will have a look at bug 1762428 to get these tests re-enabled.

Flags: needinfo?(hskupin)
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/75fd101c4574
[wdspec] Enable tests on Android x86_64. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/7be929c7cc88
[wdspec] Use timestamps from the remote end for comparison. r=webdriver-reviewers,jdescottes
https://hg.mozilla.org/integration/autoland/rev/86259f6ad100
[wdspec] Move file protocol tests for "get current url" and "navigate to" into its own test module. r=webdriver-reviewers,jgraham
https://hg.mozilla.org/integration/autoland/rev/acbd3772f0c9
[wdspec] Update wpt meta data for Android jobs. r=webdriver-reviewers,jgraham
Upstream PR merged by moz-wptsync-bot
Regressions: 1764620
No longer regressions: 1764620
See Also: → 1607210
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: