Green up wpt audio-output directory on fennec

RESOLVED FIXED in Firefox 65

Status

defect
RESOLVED FIXED
10 months ago
9 months ago

People

(Reporter: KWierso, Assigned: KWierso)

Tracking

Version 3
mozilla65
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox65 fixed)

Details

Attachments

(3 attachments)

https://docs.google.com/spreadsheets/d/11LE-ITakYnH9XIiKzSgFntOazXPxDrx9jtZ-tk71Kls/edit?usp=sharing

setSinkId seems to not be defined, despite the wpt metadata setting media.setsinkid.enabled to true.
This seems similar to the failures I see in bug 1499866, where things that should be enabled aren't getting enabled, maybe it's something in the harness?

The log output includes >> 1:04.72 INFO Setting pref media.setsinkid.enabled (true) << but maybe that isn't actually making it into the profile on Fennec?
Flags: needinfo?(james)
Flags: needinfo?(dburns)
If it matters, these are being run on builds from fairly current mozilla-central. I assume these builds get the EARLY_BETA_OR_EARLIER treatment, but I haven't looked to confirm that. Don't know if that could make a difference for some of these features...
(In reply to Wes Kocher (:KWierso) from comment #1)
> This seems similar to the failures I see in bug 1499866, where things that
> should be enabled aren't getting enabled, maybe it's something in the
> harness?
> 
> The log output includes >> 1:04.72 INFO Setting pref media.setsinkid.enabled
> (true) << but maybe that isn't actually making it into the profile on Fennec?

Can you run 1 test and then when it is finished and the emulator is left open can you have a look at the about:config for that preferece to make sure it is set. Part of me wonders if we are setting the profile on the host but not actually making sure we do it for mobile...
Flags: needinfo?(dburns)
The preferences are set using Chrome js executed via marionette [1], so in theory it ought to work on mobile. It's possible that it doesn't of course. You could also try reading back the pref after setting it and logging the value you get. The pref setting is happening at [2], so adding something like 

logging.info("%s has value " % (name, self.prefs.get(name)))

After that should allow us to check consistency.

[1] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py#209
[2] https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py#209
Flags: needinfo?(james)
(In reply to David Burns :automatedtester from comment #3)
> Can you run 1 test and then when it is finished and the emulator is left
> open can you have a look at the about:config for that preferece to make sure
> it is set. Part of me wonders if we are setting the profile on the host but
> not actually making sure we do it for mobile...

When I do that, after a test that does NOT explicitly set that pref, media.setsinkid.enabled is 'false', but when the test explicitly DOES set the pref, it is 'true'. So the preference IS getting set correctly. There goes that theory...

Alex, is this expected to not work on Android, maybe?
Flags: needinfo?(achronop)
The setSinkId method should be available but the enumerate devices does not work on android so it return 0 devices, so:

These 2 failures are not expected. I am testing on my android device and the method is there. It must be a failure of the test harness.

  FAIL setSinkId on default audio output should always work - audio.setSinkId is not a function
@http://web-platform.test:8000/audio-output/setSinkId.html:12:19
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1561:20
promise_test/tests.promise_tests</<@http://web-platform.test:8000/resources/testharness.js:591:31
promise_test/tests.promise_tests<@http://web-platform.test:8000/resources/testharness.js:590:20

  FAIL setSinkId fails with NotFoundError on made up deviceid - audio.setSinkId is not a function
@http://web-platform.test:8000/audio-output/setSinkId.html:14:55
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:1561:20
promise_test/tests.promise_tests</<@http://web-platform.test:8000/resources/testharness.js:591:31
promise_test/tests.promise_tests<@http://web-platform.test:8000/resources/testharness.js:590:20

The following error is expected, so over all, we expect that the test will fail in android with that error.

  FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0
@http://web-platform.test:8000/audio-output/setSinkId.html:20:3

The same for the other test.
Flags: needinfo?(achronop)
The feature landed one week ago in case you don't test the latest. Before that the _function does not exist_ error is expected. See also Bug 934425.
Okay, so running on a build from a recent (last few hours) pull from autoland, I get these errors:

Unexpected Results
------------------
/audio-output/setSinkId.html
  FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0
@http://web-platform.test:8000/audio-output/setSinkId.html:20:3
/audio-output/setSinkId.https.html
  FAIL List device, setSinkId should on the default, the rest of the devices will get a NotAlowedError - assert_not_equals: media device list includes at least one audio output device got disallowed value 0
@https://web-platform.test:8443/audio-output/setSinkId.https.html:22:3
Flags: needinfo?(achronop)
(In reply to Wes Kocher (:KWierso) from comment #9)
> Created attachment 9019564 [details]
> Bug 1499871 - Fix spelling of "NotAllowedError" r?achronop

(This doesn't fix the failures, but the misspelling annoyed me enough to correct it.)
(In reply to Wes Kocher (:KWierso) from comment #8)
> Okay, so running on a build from a recent (last few hours) pull from
> autoland, I get these errors:
> 
> Unexpected Results
> ------------------
> /audio-output/setSinkId.html
>   FAIL List device, setSinkId should on the default, the rest of the devices
> will get a NotAlowedError - assert_not_equals: media device list includes at
> least one audio output device got disallowed value 0
> @http://web-platform.test:8000/audio-output/setSinkId.html:20:3
> /audio-output/setSinkId.https.html
>   FAIL List device, setSinkId should on the default, the rest of the devices
> will get a NotAlowedError - assert_not_equals: media device list includes at
> least one audio output device got disallowed value 0
> @https://web-platform.test:8443/audio-output/setSinkId.https.html:22:3

That looks good. It's the expected failure on android (for now).
Flags: needinfo?(achronop)
Keywords: leave-open
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/121ccc3d4437
Fix spelling of "NotAllowedError" r=achronop
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/13711 for changes under testing/web-platform/tests
Keywords: leave-open
Upstream PR merged
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ca744888101e
Update test metadata for wpt audio-output on fennec r=achronop
https://hg.mozilla.org/mozilla-central/rev/ca744888101e
Status: NEW → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
These tests started passing again on try, and started doing the same for me locally after I force-updated the android avd. Patch incoming to update the expectation data.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
At some point, these started passing again on CI. Force-updating the Android AVD makes these tests start passing locally as well. Let's just remove the android-specific failing expectations since they don't seem necessary any more.
It's Bug 1482150 that has just been landed. Those tests are not on try yet, right? I did not see a change there.
See Also: → 1482150
Correct, nothing runs without https://hg.mozilla.org/try/rev/29da107e44d72adf184b5cbcce9a53a157f3cc41 applied, and then only on try with you specifically asking for them to run.
Sample failure: https://treeherder.mozilla.org/logviewer.html#?job_id=210768951&repo=try
Pushed by wkocher@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/91ad57be0886
Mark the wpt audio-output expectations as passing everywhere r=jgraham
https://hg.mozilla.org/mozilla-central/rev/91ad57be0886
Status: REOPENED → RESOLVED
Closed: 10 months ago9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.