Open Bug 1516441 Opened 5 years ago Updated 2 years ago

Perma Tier 2 dom/media/tests/mochitest/identity/test_fingerprints.html | error in test: Promise rejection value is a non-unwrappable cross-compartment wrapper.

Categories

(Core :: WebRTC, defect, P5)

Product:
Core
Component:
WebRTC
Type:
defect

Tracking

()

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure, leave-open, Whiteboard: [stockwell disabled][retriggered])

Attachments

(1 file)

skip identity tests on android
789 bytes, patch
bc
: review+
Details | Diff | Splinter Review 
Filed by: rgurzau [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=218847132&repo=autoland

https://queue.taskcluster.net/v1/task/dnCo50ytRA-lOagP5whnDA/runs/0/artifacts/public/logs/live_backing.log

16:08:38     INFO -  1 INFO TEST-START | dom/media/tests/mochitest/identity/test_fingerprints.html
16:08:38     INFO -  Buffered messages logged at 16:08:32
16:08:38     INFO -  2 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | A valid string reason is expected
16:08:38     INFO -  3 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | Reason cannot be empty
16:08:38     INFO -  4 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | A valid string reason is expected
16:08:38     INFO -  5 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | Reason cannot be empty
16:08:38     INFO -  Buffered messages logged at 16:08:33
16:08:38     INFO -  6 INFO Skipping ICE Server for this test
16:08:38     INFO -  7 INFO Call getUserMedia for {"video":true}
16:08:38     INFO -  8 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | No audio tracks shall be present
16:08:38     INFO -  9 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | One video track shall be present
16:08:38     INFO -  10 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | Track kind should be video
16:08:38     INFO -  11 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | Track id should be defined
16:08:38     INFO -  12 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | Track should not be muted
16:08:38     INFO -  13 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | Got test stream
16:08:38     INFO -  Buffered messages logged at 16:08:34
16:08:38     INFO -  14 INFO TEST-PASS | dom/media/tests/mochitest/identity/test_fingerprints.html | Got offer
16:08:38     INFO -  Buffered messages finished
16:08:38     INFO -  15 INFO TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/identity/test_fingerprints.html | error in test: Promise rejection value is a non-unwrappable cross-compartment wrapper.
16:08:38     INFO -      SimpleTest.ok@SimpleTest/SimpleTest.js:275:7
16:08:38     INFO -      @dom/media/tests/mochitest/identity/test_fingerprints.html:81:5
16:08:38     INFO -  16 INFO TEST-OK | dom/media/tests/mochitest/identity/test_fingerprints.html | took 3788ms
This seemed to start with the host-utils update in bug 1514075. We can back out 1514075 if needed.
Blocks: 1514075
I don't know how that would cause an "Promise rejection value is a non-unwrappable cross-compartment wrapper" error.
If I understand correctly, Android hostutils updates pick up every change that has occurred since the last such update.  Not just the UTF-8 bit of changing that was the impetus for bug 1514075 updates.  And I think there has been some promises implementation changing quasi-recently, by jorendorff and maybe others, that *could* be the cause of that.
Hi! Since bug 1514075 landed besides this newly created bug we have permafailing bug 1392946. This happens only on tier 2 tests on Android 8.0 Pixel2. 
https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&searchStr=android%2Cmda&group_state=expanded&fromchange=792bc67640ca3156a647a00f2bdfc4fb87b53cf3&selectedJob=218846498
Geoff, please let the sheriffs know if you still want bug 1514075 backed out. Thank you.
Flags: needinfo?(gbrown)
Something appears to be wrong with *all* of the dom/media/tests/mochitest/identity tests as a whole.  Scroll further down in logs and it looks like most every test in that directory fails.  Given that one of the failures (several, really) manifest as stuff like

1505 INFO TEST-UNEXPECTED-FAIL | dom/media/tests/mochitest/identity/test_setIdentityProvider.html | Error in test execution: IdpError: Load failed: 2152398920

where that number is really NS_ERROR_PROXY_CONNECTION_REFUSED, and given that Android's test setup has the hostutils thing and quite possibly some sort of other proxy in the mix, I suspect an Android-specific issue in the layers between httpd.js and this code when it runs.  But I don't know what/where the problem would be exactly.

(There are also additional problems with things, some clearly being the wrapper-related stuff of this bug's summary, but my guess is this proxy-connection-refused thing is close to the heart of most to all of it.)
So if you look at bug 1361325 which is for other tests in this area being disabled on some Android, it sounds like this exact sort of issue in this identity-code is very much not new.  :-|

Tho, that bug *maybe* suggests that some error is being reported, then unwrapping is failing somehow when *that* error is attempting to be reported -- which is to say unwrapping may be a red herring here.  Not sure.

Anyway, an IRC braindump of whatever epiphanies led to this point:

	Waldo	so actually https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=894e36ecd784a8d25448e32722f0e129f673d9fb&selectedJob=218912172 having a whole mess'o'wrapped-promise failures is worse than I thought it was
	I thought it was just the one test
[22:47]	so hum, actually
	Error in test execution: IdpError: Load failed: 2152398920
	that's NS_ERROR_PROXY_CONNECTION_REFUSED
[22:48]	I did change some SJS things
	it is mildly conceivable it hitting some of that
[22:49]	although something that hits only Android is more weirder
	CosminS|sheriffduty	actually that mda shows up there because it was merged around now
	Waldo	wait what
	CosminS|sheriffduty	since no response on the bug and only tier 2
	dom/media/test/test_webvtt_positionalign.html | application timed out after 370 seconds with no output
	Waldo	CosminS|sheriffduty: wait, so what exactly has happened since a week ago?
[22:50]	we updated hostutils, then...we undid the update, or did nothing, or did what?
	CosminS|sheriffduty	this is where it first started permafailling https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&searchStr=android%2Cmda&group_state=expanded&selectedJob=218846498&revision=792bc67640ca3156a647a00f2bdfc4fb87b53cf3
[22:51]	I know nothing about updoing the update
	Bug 1514075 is still around and impacting bug 1392946
	firebot	https://bugzil.la/1514075 — NEW, egao@mozilla.com — Update hostutils for Android tests, to pick up bug 1492937 and bug 1513152 changes when both have la
[22:53]	Waldo	proxy-connection-refused, tho, seems very likely illuminating here
[22:58]	so https://searchfox.org/mozilla-central/source/dom/media/IdpSandbox.jsm has that "Load failed" string in it
[23:02]	I wonder if Android test runs have some extra layer of proxying involved in them, between the hostutils server and what the mobile setup sees, and that's coming into play here
[23:05]	CosminS|sheriffduty	unfortunately I don't have any kind of expertise in this, was just observing a behavior after 1514075 landed and what kind of failures it caused doing so
[23:12]	that indexedDB is intermittent
[23:21]	Waldo	good, good
[23:33]	ooh, ugh
	so https://searchfox.org/mozilla-central/source/dom/media/PeerConnection.js#39
	that full list of CIDs looks dangerously fragile, and the sort of thing that could easily go sideways with a hostutils update
[23:35]	it would also be highly difficult to figure out exactly what broke that, just from looking at that list
	because the breaking change would be some *other* file's use of that uuid being changed, but this file not being changed
[23:36]	and then if these dom/media tests are maybe only being run on certain platforms or something...
[23:37]	CosminS|sheriffduty	they are run on all platforms but these failures are only on Android 8.0 Pixel2 debug & opt
	Waldo	so https://searchfox.org/mozilla-central/source/dom/media/tests/mochitest/identity/mochitest.ini lists a bunch of version-specificities
	CosminS|sheriffduty	I think this info above would be helpful if you put it on the bug
	Waldo	is it possible those need "expanding" to newer Android?
[23:38]	yeah, I'll stream-of-conscious this into the bug
	CosminS|sheriffduty	thank you
	maybe Bob Clary I think can answer that Android related question
[23:39]	Waldo	so https://bugzilla.mozilla.org/show_bug.cgi?id=1361325 has some errors/complaints being discussed in it that sound *exactly* like the ones currently up for discussion here
	firebot	Bug 1361325 — NEW, nobody@mozilla.org — Autophone Mdm4 perma fail on Android 5.1/Nexus 6
	Waldo	https://bugzilla.mozilla.org/show_bug.cgi?id=1361325#c7 for example
	so whatever issue there is here, it seems pre-existing somehow
[23:42]	Waldo	maybe
Whiteboard: [stockwell needswork:owner]
Flags: needinfo?(bob)
See Also: → 1392946
Skipping test_fingerprints.html and test_webvtt_positionalign.html on android-hw yields a green run, for a possible short-term solution:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=1a389129b1b4500dddbc4626f97c24a133cb43cb
gbrown: I say we do that.
Flags: needinfo?(bob)
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f2dc090551f0
Skip test_fingerprints.html on android-hw; r=me,a=test-only
Flags: needinfo?(gbrown)
Keywords: leave-open
Sigh. That's not effective actually.

I see now that my try push (comment 11) did not include the host-utils update, so was not relevant.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8d7a5d1ab4d8
backout f2dc090551f076387b984a670fe9eb7be9c8d460 since it was not effective
I'm inching toward a new skip-if solution on try; here's my latest:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=bb0fb8c87aab0b277dc4b62d426943e2b17a6ddf

I may be able to finish this up tonight; otherwise, I'll get back to it first thing Wednesday.
(In reply to Jeff Walden [:Waldo] from comment #6)
> So if you look at bug 1361325 which is for other tests in this area being
> disabled on some Android, it sounds like this exact sort of issue in this
> identity-code is very much not new.  :-|

We've had various problems with mochitest-media on android over time; these days we only run that test suite on real hardware - the Android 8.0 Pixel2 platform that is giving us trouble here.

> [22:53]	Waldo	proxy-connection-refused, tho, seems very likely illuminating
> here
> [22:58]	so
> https://searchfox.org/mozilla-central/source/dom/media/IdpSandbox.jsm has
> that "Load failed" string in it
> [23:02]	I wonder if Android test runs have some extra layer of proxying
> involved in them, between the hostutils server and what the mobile setup
> sees, and that's coming into play here

hostutils gives us host-side binaries used by various test harnesses. The host is a linux host, and the binaries are taken from a linux64 firefox build. The binaries are xpcshell, ssltunnel, certutil, etc. Notably xpcshell is used to run the web server (httpd.js). Desktop mochitests use xpcshell (etc) also, but take them from the build under test (eg. linux tests just use xpcshell from the build under test); Android can't do that because we need a linux xpcshell but we only have an android (arm) build available. Desktop mochitests run httpd.js in xpcshell, but a notable difference is that in the desktop case httpd.js is running on the same host as firefox; on Android, firefox is running on a remote device.

From my perspective, that only makes this situation more puzzling: if the only structural difference is that the web server is on a remote host, how could that make a difference? Perhaps there's something else? (This is fennec, notably non-e10s.) Or perhaps the local/remote difference is significant?
(In reply to Geoff Brown [:gbrown] from comment #16)
> I'm inching toward a new skip-if solution on try; here's my latest:

That didn't work out: I kept skipping the test reported as timing out, and the timeout just moved to the previous test, at least in dom/media/test/mochitest.ini.
The dom/media/test issue is better discussed in bug 1392946 -- see https://bugzilla.mozilla.org/show_bug.cgi?id=1392946#c20, etc.
Skip identity tests to restore android-hw mochitest-media to green runs.

I still don't know why these started failing on android-hw with the host-utils update. It seems suspicious that they were already skipped on android 4.3.
Attachment #9033978 - Flags: review?(bob)
Comment on attachment 9033978 [details] [diff] [review]
skip identity tests on android

Review of attachment 9033978 [details] [diff] [review]:
-----------------------------------------------------------------

r+
Attachment #9033978 - Flags: review?(bob) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/753a63f6aa54
Skip dom/media/tests/mochitest/identity on android; r=bc
Whiteboard: [stockwell disable-recommended] → [stockwell disabled]
Flags: needinfo?(gbrown)
Flags: needinfo?(gbrown)
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: