windows/aarch64 - testing/web-platform/tests/encrypted-media/clearkey-mp4

RESOLVED FIXED in Firefox 67

Status

()

defect
P4
normal
RESOLVED FIXED
5 months ago
4 months ago

People

(Reporter: egao, Assigned: egao)

Tracking

(Blocks 1 bug)

unspecified
mozilla67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox67 fixed)

Details

Attachments

(1 attachment)

Assignee

Description

5 months ago

Treeherder: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f2edbc029097dc7ca3c14f2ae488e4d7275d147&selectedJob=226236514

Taskcluster: https://tools.taskcluster.net/groups/efMBjsn0Q66A2tvOdHEHDw/tasks/fUBUdoEPTo2TmrwjRgGbng/details

Multiple failures reported relating to the media keys in Web Platform Tests for win10/aarch64. Decided to file one issue to cover all new failures as the alternative was to file 20+ similar issues.

TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-keystatuses.https.html | org.w3.clearkey, temporary, keystatuses - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-keystatuses.https.html | expected OK
...
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-at-same-time.https.html | org.w3.clearkey, setmediakeys at same time - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-at-same-time.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-multiple-times-with-different-mediakeys.https.html | org.w3.clearkey, setmediakeys multiple times with different mediakeys - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-multiple-times-with-different-mediakeys.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-multiple-times-with-the-same-mediakeys.https.html | org.w3.clearkey, setmediakeys multiple times with the same mediakeys - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-multiple-times-with-the-same-mediakeys.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-to-multiple-video-elements.https.html | org.w3.clearkey, setMediaKeys to multiple video elements - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys-to-multiple-video-elements.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys.https.html | org.w3.clearkey, setMediaKeys - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-setmediakeys.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-syntax-mediakeys.https.html | org.w3.clearkey test MediaKeys attribute syntax - Test timed out
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeys.https.html | org.w3.clearkey test MediaKeys setServerCertificate() exceptions. - expected PASS
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeys.https.html | org.w3.clearkey test MediaKeys setServerCertificate() syntax with non-empty certificate. - expected PASS
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-syntax-mediakeys.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | org.w3.clearkey test MediaKeySession generateRequest() exceptions. - Test timed out
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | org.w3.clearkey test MediaKeySession load() exceptions. - expected PASS
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | org.w3.clearkey test if MediaKeySession generateRequest() resolves for various sessions - expected PASS
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | org.w3.clearkey test MediaKeySession update() exceptions. - expected PASS
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | org.w3.clearkey test MediaKeySession close() exceptions. - expected PASS
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | org.w3.clearkey test MediaKeySession remove() exceptions. - expected PASS
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | org.w3.clearkey test MediaKeySession attribute syntax. - expected PASS
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-syntax-mediakeysession.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-unique-origin.https.html | Unique origin is unable to create MediaKeys - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-unique-origin.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html | update() with invalid response (longer than 64Kb characters) should fail. - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-mp4-update-disallowed-input.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-not-callable-after-createsession.https.html | org.w3.clearkey, temporary, update() immediately after createSession() - Test timed out
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-not-callable-after-createsession.https.html | org.w3.clearkey, temporary, close() immediately after createSession() - expected PASS
TEST-UNEXPECTED-NOTRUN | /encrypted-media/clearkey-not-callable-after-createsession.https.html | org.w3.clearkey, temporary, remove() immediately after createSession() - expected PASS
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-not-callable-after-createsession.https.html | expected OK
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-update-non-ascii-input.https.html | org.w3.clearkey test handling of non-ASCII responses for update() - Test timed out
TEST-UNEXPECTED-TIMEOUT | /encrypted-media/clearkey-update-non-ascii-input.https.html | expected OK
TEST-UNEXPECTED-FAIL | /encrypted-media/drm-mp4-playback-temporary-playduration-keystatus.html | drm, temporary, mp4, playback with limited playduration, check keystatus, single key - assert_unreached: NotSupportedError: Operation is not supported Reached unreachable code
TEST-UNEXPECTED-OK | /encrypted-media/drm-mp4-playback-temporary-playduration-keystatus.html | expected TIMEOUT
TEST-UNEXPECTED-FAIL | /encrypted-media/drm-mp4-playback-temporary-playduration.html | drm, temporary, mp4, playback with limited playduration, single key - assert_unreached: NotSupportedError: Operation is not supported Reached unreachable code
TEST-UNEXPECTED-OK | /encrypted-media/drm-mp4-playback-temporary-playduration.html | expected TIMEOUT

:jimm- do you think these failing tests should be failing on win/arm64 due to missing drm, etc. ?

Flags: needinfo?(jmathies)

Not sure, I know little about GMP implementation. Maybe Chris knows.

Flags: needinfo?(jmathies) → needinfo?(cpearce)

EME doesn't work on aarch64 yet, and that includes the ClearKey implementation. So we can expect these tests to fail for a while longer. I'm working on making EME work on aarch, tracking bug is bug 1515208 (aarch64-eme).

Blocks: aarch64-eme
Flags: needinfo?(cpearce)

thanks :cpearce, we will annotate the test as expected-fail, when bug 1515208 is fixed these tests should fail as unexpected-pass, and we can always double check the bugs for what we changed in manifests as this bug (and probably others will be annotated)

:egao, I assume you can do something like:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/encrypted-media/clearkey-events-session-closed-event.https.html.ini

but in the dir.ini file:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/encrypted-media/__dir__.ini

that I assume the expected: if os == 'win' && processor == 'arm': FAIL

we would need to ensure that processor == 'arm' is valid. If we mark it as fail as opposed to disabled, then when EME is working these tests will start passing and have TEST-UNEXPECTED-PASS and fail the job.

Flags: needinfo?(egao)
Assignee

Comment 6

4 months ago

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #5)

:egao, I assume you can do something like:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/encrypted-media/clearkey-events-session-closed-event.https.html.ini

but in the dir.ini file:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/encrypted-media/__dir__.ini

that I assume the expected: if os == 'win' && processor == 'arm': FAIL

we would need to ensure that processor == 'arm' is valid. If we mark it as fail as opposed to disabled, then when EME is working these tests will start passing and have TEST-UNEXPECTED-PASS and fail the job.

If the pattern used is taken from the ENV portion of the task setup, then I would imagine it is the following:

if 'OS' == 'Windows_NT' and 'PROCESSOR_ARCHITECTURE' == 'x86' and 'PROCESSOR_ARCHITEW6432' == 'ARM64':
   FAIL

The PROCESSOR_ARCHITEW6432 field does not appear to be present on non-ARM64 builds. PROCESSOR_ARCHITECTURE has value of 'amd64' for Windows_10_64/opt builds.

Update:

Using local macOS machine running x86_64 processor, the following produces a UNEXPECTED-OK outcome:

if not (processor == "arm"): FAIL

So it would follow that the following should work to expect failure for ARM64 version of the test:
if os == "win" and processor == "arm": FAIL

Treeherder push that has the files above mass-edited to expect either TIMEOUT or FAIL: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d6361838d38af905daad05811dfa4d6629c2599

Flags: needinfo?(egao)

sadly we cannot use random data, it needs to exist in some format in our manifest parsing logic, but that is hackable and the fields you see could get included in there :)

Summary: TEST-UNEXPECTED-FAIL | testing/web-platform/tests/encrypted-media/clearkey-mp4 → windows/aarch64 - testing/web-platform/tests/encrypted-media/clearkey-mp4
Comment hidden (Intermittent Failures Robot)
Assignee

Comment 9

4 months ago

I'm a bit confused here.

I tried these combinations:

  • if (os == "win") and (processor == "arm64"): TIMEOUT
  • if (os == "win") and (processor == "arm"): TIMEOUT

And the non-paranthesis version of both;

Yet the test clearkey-keystatuses.https.html continues to fail as it expects OK.

If other processor flags are any indication, ARM64 should read as arm64 but that doesn't seem to work.

:cpearce do these tests use arm64 or arm?

Some test suites, like mochitests, use mozinfo for this sort of thing; I don't know if that applies to wpt.

Build jobs (not test jobs) have mozinfo artifacts. Those for aarch64 builds have processor == "aarch64":

https://queue.taskcluster.net/v1/task/P4qsyV2AQfGhoC14AJUk9A/runs/0/artifacts/public/build/target.mozinfo.json

(In reply to Geoff Brown [:gbrown] from comment #10)

Some test suites, like mochitests, use mozinfo for this sort of thing; I don't know if that applies to wpt.

It looks like mozinfo is used by wpt too:

https://searchfox.org/mozilla-central/rev/00c0d068ece99717bea7475f7dc07e61f7f35984/testing/web-platform/tests/tools/wptrunner/docs/expectation.rst#210-218

Assignee

Comment 12

4 months ago

(In reply to Geoff Brown [:gbrown] from comment #11)

(In reply to Geoff Brown [:gbrown] from comment #10)

Some test suites, like mochitests, use mozinfo for this sort of thing; I don't know if that applies to wpt.

It looks like mozinfo is used by wpt too:

https://searchfox.org/mozilla-central/rev/00c0d068ece99717bea7475f7dc07e61f7f35984/testing/web-platform/tests/tools/wptrunner/docs/expectation.rst#210-218

Thanks, one more thing learned.

Priority: -- → P4
Assignee

Comment 13

4 months ago
  • drm-check-encryption-scheme.https.html.ini
  • drm-check-status-for-hdcp.https.html.ini
  • drm-mp4-playback-temporary-playduration-keystatus.html.ini
  • drm-mp4-playback-temporary-playduration.html.ini
Assignee

Comment 14

4 months ago

Try: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=3ca5af208c4a5a4f62115bdecca4c3c5e270ee28

Log: https://taskcluster-artifacts.net/K4yVBmgDTEK6EAHSyQgqEQ/0/public/logs/live_backing.log

The list of failures have shortened since the bug was initially filed. Differential covers the remaining errors in opt-web-platform-test-7.

Assignee

Comment 15

4 months ago

Try run seems to confirm integrity of the additional conditions implemented in the patch.

Comment 16

4 months ago
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6927cfe9f9ca
insert expected outcomes for encrypted-media tests in wpt for windows10-aarch64 r=jmaher,gbrown

Comment 17

4 months ago
bugherder
Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.