1216551 - osx 10.10.5 error for test file browser/components/loop/test/shared/test_shared_all.py

Status: RESOLVED FIXED

(Hello (Loop) :: General, defect, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we are close to getting osx 10.10.5 up and running, unfortunately the Mn job fails on debug and we need to get it fixed.

here is a clip from the log:
09:55:09     INFO -  TEST-START | test_shared_all.py TestSharedUnits.test_units
09:55:30    ERROR -  TEST-UNEXPECTED-FAIL | test_shared_all.py TestSharedUnits.test_units | AssertionError: build/tests/marionette/tests/browser/components/loop/test/shared/index.html: 1 failure(s) encountered:
09:55:30    ERROR -  TEST-UNEXPECTED-FAIL | index.html | should long only the warnings we expect - AssertionError: expected 2 to deeply equal 0
09:55:30     INFO -  AssertionError: expected 2 to deeply equal 0
09:55:30     INFO -  Traceback (most recent call last):
09:55:30     INFO -    File "/builds/slave/test/build/venv/lib/python2.7/site-packages/marionette/marionette_test.py", line 296, in run
09:55:30     INFO -      testMethod()
09:55:30     INFO -    File "/builds/slave/test/build/tests/marionette/tests/browser/components/loop/test/shared/test_shared_all.py", line 16, in test_units
09:55:30     INFO -      self.check_page("index.html")
09:55:30     INFO -    File "/builds/slave/test/build/tests/marionette/tests/browser/components/loop/test/shared/frontend_tester.py", line 128, in check_page
09:55:30     INFO -      raise AssertionError(self.get_failure_details(page))
09:55:30     INFO -  TEST-INFO took 20530ms

there is no screenshots available (might be harness specific), or other info in the debug log.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:standard8, can you look at this and help us figure out if we can fix the test, the os configuration, or disable the test in the manifest?

Comment 2

[Mark Banner (:standard8)]

Ok, I've just done a try push to attempt to add some more debug info. If it works, its likely something we'll want to add to the test setup anyway (or modify slightly) so useful to see:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cf2f915db9e4

Comment 3

[Mark Banner (:standard8)]

Ok, the try run failed, but I don't understand why. The patch I want to try is:

https://hg.mozilla.org/try/rev/9e9b0cb8a2d0

Joel, can you help here please?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this looks like infra structure errors or build errors, not related to the test.  any chance the parent revision you pushed with was bad?

Comment 5

[Mark Banner (:standard8)]

Ah, yes it was a bad one - forgot to look. I've pushed afresh:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4be066010010

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

pushed to try with updated revisions:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f69624997f77

lets see if 3rd time is the charm.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Mn failed on try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f69624997f77

Comment 8

[Mark Banner (:standard8)]

Ok, I've pushed something that should get us a screenshot url out, not perfect, but it'll do for now:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9dc7c75ec206

Comment 9

[Mark Banner (:standard8)]

I forgot that a screenshot isn't going to solve this issue, since primarily these are warnings that go to the console. I've found a possible change to the test that might now give us more information and pushed again:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c83192b2ae9e

Comment 10

[Mark Banner (:standard8)]

Ok, we now have something reasonable as an output:

TEST-UNEXPECTED-FAIL | index.html | should long only the warnings we expect - Error: No stores registered for event type ,connectionFailure,No stores registered for event type ,connectionFailure

That's a bit weird though as these failures appear to be part of the test code, and we don't see them normally. Could be timing, but its strange that afaik we haven't seen this normally.

I'll dig into it a bit today.

Comment 11

[Mark Banner (:standard8)]

Now with a stack output to see who's trying to dispatch the connectionFailure action:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=334ff430d321

Comment 12

[Mark Banner (:standard8)]

Green Mn: https://treeherder.mozilla.org/#/jobs?repo=try&revision=572aab8ad03b

So either the builder probably doesn't have any mic/camera devices set up whereas our other boxes have at least one *Or* there's an async timing issue that's revealed on this machine.

I'll tidy the patch up tomorrow and get it ready for review.

Comment 13

[Mark Banner (:standard8)]

Attachment: Fix an issue with Loop's unit tests failing when no devices are installed, due to bad stubbing.

This fixes the stubbing for hasAudioOrVideoDevices when doing the join room tests, so that hasAudioOrVideoDevices never actually gets called.

Comment 14

[Mark Banner (:standard8)]

Comment on attachment 8678068 [details] [diff] [review]
Fix an issue with Loop's unit tests failing when no devices are installed, due to bad stubbing.

Just remembered that Mike is out for a couple of days, passing to Ed or Dan to get this reviewed.

Comment 15

[Ed Lee :Mardak]

Comment on attachment 8678068 [details] [diff] [review]
Fix an issue with Loop's unit tests failing when no devices are installed, due to bad stubbing.

Took me a while to figure out it's the other tests not in the patch that call joinRoom triggering the unstubbed function.

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/4278e19ea239

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4278e19ea239

Comment 18

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr140/rev/bb5500506525
https://hg.mozilla.org/releases/mozilla-esr140/rev/6f355c6e9e17