1499871 - Green up wpt audio-output directory on fennec

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://docs.google.com/spreadsheets/d/11LE-ITakYnH9XIiKzSgFntOazXPxDrx9jtZ-tk71Kls/edit?usp=sharing

setSinkId seems to not be defined, despite the wpt metadata setting media.setsinkid.enabled to true.

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

This seems similar to the failures I see in bug 1499866, where things that should be enabled aren't getting enabled, maybe it's something in the harness?

The log output includes >> 1:04.72 INFO Setting pref media.setsinkid.enabled (true) << but maybe that isn't actually making it into the profile on Fennec?

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

If it matters, these are being run on builds from fairly current mozilla-central. I assume these builds get the EARLY_BETA_OR_EARLIER treatment, but I haven't looked to confirm that. Don't know if that could make a difference for some of these features...

Comment 3

[David Burns :automatedtester]

(In reply to Wes Kocher (:KWierso) from comment #1)
> This seems similar to the failures I see in bug 1499866, where things that
> should be enabled aren't getting enabled, maybe it's something in the
> harness?
> 
> The log output includes >> 1:04.72 INFO Setting pref media.setsinkid.enabled
> (true) << but maybe that isn't actually making it into the profile on Fennec?

Can you run 1 test and then when it is finished and the emulator is left open can you have a look at the about:config for that preferece to make sure it is set. Part of me wonders if we are setting the profile on the host but not actually making sure we do it for mobile...

Comment 4

[James Graham [:jgraham]]

The preferences are set using Chrome js executed via marionette [1], so in theory it ought to work on mobile. It's possible that it doesn't of course. You could also try reading back the pref after setting it and logging the value you get. The pref setting is happening at [2], so adding something like 

logging.info("%s has value " % (name, self.prefs.get(name)))

After that should allow us to check consistency.

[1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py#209
[2] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py#209

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to David Burns :automatedtester from comment #3)
> Can you run 1 test and then when it is finished and the emulator is left
> open can you have a look at the about:config for that preferece to make sure
> it is set. Part of me wonders if we are setting the profile on the host but
> not actually making sure we do it for mobile...

When I do that, after a test that does NOT explicitly set that pref, media.setsinkid.enabled is 'false', but when the test explicitly DOES set the pref, it is 'true'. So the preference IS getting set correctly. There goes that theory...

Alex, is this expected to not work on Android, maybe?

Comment 6

[Alex Chronopoulos [:achronop]]

The setSinkId method should be available but the enumerate devices does not work on android so it return 0 devices, so:

These 2 failures are not expected. I am testing on my android device and the method is there. It must be a failure of the test harness.

  FAIL setSinkId on default audio output should always work - audio.setSinkId is not a function
@http://web-platform.test:8000/audio-output/setSinkId.html:12:19
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1561:20
promise_test/tests.promise_tests</<@http://web-platform.test:8000/resources/testharness.js:591:31
promise_test/tests.promise_tests<@http://web-platform.test:8000/resources/testharness.js:590:20

  FAIL setSinkId fails with NotFoundError on made up deviceid - audio.setSinkId is not a function
@http://web-platform.test:8000/audio-output/setSinkId.html:14:55
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1561:20
promise_test/tests.promise_tests</<@http://web-platform.test:8000/resources/testharness.js:591:31
promise_test/tests.promise_tests<@http://web-platform.test:8000/resources/testharness.js:590:20

The following error is expected, so over all, we expect that the test will fail in android with that error.

  FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0
@http://web-platform.test:8000/audio-output/setSinkId.html:20:3

The same for the other test.

Comment 7

[Alex Chronopoulos [:achronop]]

The feature landed one week ago in case you don't test the latest. Before that the _function does not exist_ error is expected. See also Bug 934425.

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Okay, so running on a build from a recent (last few hours) pull from autoland, I get these errors:

Unexpected Results
------------------
/audio-output/setSinkId.html
  FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0
@http://web-platform.test:8000/audio-output/setSinkId.html:20:3
/audio-output/setSinkId.https.html
  FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0
@https://web-platform.test:8443/audio-output/setSinkId.https.html:22:3

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1499871 - Fix spelling of "NotAllowedError" r?achronop

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Wes Kocher (:KWierso) from comment #9)
> Created attachment 9019564 [details]
> Bug 1499871 - Fix spelling of "NotAllowedError" r?achronop

(This doesn't fix the failures, but the misspelling annoyed me enough to correct it.)

Comment 11

[Alex Chronopoulos [:achronop]]

(In reply to Wes Kocher (:KWierso) from comment #8)
> Okay, so running on a build from a recent (last few hours) pull from
> autoland, I get these errors:
> 
> Unexpected Results
> ------------------
> /audio-output/setSinkId.html
>   FAIL List device, setSinkId should on the default, the rest of the devices
> will get a NotAlowedError - assert_not_equals: media device list includes at
> least one audio output device got disallowed value 0
> @http://web-platform.test:8000/audio-output/setSinkId.html:20:3
> /audio-output/setSinkId.https.html
>   FAIL List device, setSinkId should on the default, the rest of the devices
> will get a NotAlowedError - assert_not_equals: media device list includes at
> least one audio output device got disallowed value 0
> @https://web-platform.test:8443/audio-output/setSinkId.https.html:22:3

That looks good. It's the expected failure on android (for now).

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/121ccc3d4437
Fix spelling of "NotAllowedError" r=achronop

Comment 13

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/13711 for changes under testing/web-platform/tests

Comment 14

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/121ccc3d4437

Comment 15

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1499871 - Update test metadata for wpt audio-output on fennec

Comment 16

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ca744888101e
Update test metadata for wpt audio-output on fennec r=achronop

Comment 18

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/ca744888101e

Comment 19

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

These tests started passing again on try, and started doing the same for me locally after I force-updated the android avd. Patch incoming to update the expectation data.

Comment 20

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Bug 1499871 - Mark the wpt audio-output expectations as passing everywhere r?jgraham

At some point, these started passing again on CI. Force-updating the Android AVD makes these tests start passing locally as well. Let's just remove the android-specific failing expectations since they don't seem necessary any more.

Comment 21

[Alex Chronopoulos [:achronop]]

It's Bug 1482150 that has just been landed. Those tests are not on try yet, right? I did not see a change there.

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Correct, nothing runs without https://hg.mozilla.org/try/rev/29da107e44d72adf184b5cbcce9a53a157f3cc41 applied, and then only on try with you specifically asking for them to run.
Sample failure: https://treeherder.mozilla.org/logviewer.html#?job_id=210768951&repo=try

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/91ad57be0886
Mark the wpt audio-output expectations as passing everywhere r=jgraham

Comment 24

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/91ad57be0886