1411348 - Permafail test_crash.py TestCrash.test_crash_chrome_process | AssertionError: None unexpectedly found in (None, 0)

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This started to permafail on Android 4.3 API 16+ debug when the patch for bug 1410355 landed: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=363bf0516c8938219c67553522f9e6ac69b98fad&selectedJob=138945618

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=138945618&repo=autoland
[task 2017-10-23T17:34:46.691Z] 17:34:46     INFO -  TEST-START | test_crash.py TestCrash.test_crash_chrome_process
[task 2017-10-23T17:34:52.582Z] 17:34:52  WARNING -  TEST-UNEXPECTED-FAIL | test_crash.py TestCrash.test_crash_chrome_process | AssertionError: None unexpectedly found in (None, 0)
[task 2017-10-23T17:34:52.582Z] 17:34:52     INFO -  Traceback (most recent call last):
[task 2017-10-23T17:34:52.583Z] 17:34:52     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/marionette_test/testcases.py", line 156, in run
[task 2017-10-23T17:34:52.583Z] 17:34:52     INFO -      testMethod()
[task 2017-10-23T17:34:52.585Z] 17:34:52     INFO -    File "/builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/test_crash.py", line 123, in test_crash_chrome_process
[task 2017-10-23T17:34:52.585Z] 17:34:52     INFO -      self.assertNotIn(self.marionette.instance.runner.returncode, (None, 0))
[task 2017-10-23T17:34:52.585Z] 17:34:52     INFO -  TEST-INFO took 5891ms
[task 2017-10-23T17:34:53.377Z] 17:34:53     INFO -  mozdevice Non-zero return code (1) from ['/builds/worker/workspace/build/android-sdk-linux/platform-tools/adb', '-s', 'emulator-5554', 'forward', '--remove', 'tcp:2828']
[task 2017-10-23T17:34:53.378Z] 17:34:53     INFO -  mozdevice Output: ["error: listener 'tcp:2828' not found"]
[task 2017-10-23T17:34:53.380Z] 17:34:53 CRITICAL -  test_end for test_crash.py TestCrash.test_crash_chrome_process logged while not in progress. Logged with data: {"status": "ERROR", "extra": {"class_name": "test_crash.TestCrash", "method_name": "test_crash_chrome_process"}, "expected": "PASS", "test": "test_crash.py TestCrash.test_crash_chrome_process", "message": "DMError: Failed to remove connection forwarding.", "stack": "Traceback (most recent call last):\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/marionette_test/testcases.py\", line 187, in run\n    self.tearDown()\n  File \"/builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/test_crash.py\", line 87, in tearDown\n    super(BaseCrashTestCase, self).tearDown()\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_harness/marionette_test/testcases.py\", line 322, in tearDown\n    self.marionette.start_session()\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py\", line 28, in _\n    m._handle_socket_failure()\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 818, in _handle_socket_failure\n    self.cleanup()\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py\", line 655, in cleanup\n    self.instance.close(clean=True)\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/geckoinstance.py\", line 407, in close\n    \"tcp:{}\".format(self.marionette_port))\n  File \"/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/devicemanagerADB.py\", line 211, in remove_forward\n    raise DMError(\"Failed to remove connection forwarding.\")\n"}

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The job gets always retried 5 times and seems to fail like that when it hits the retry limit. See https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=363bf0516c8938219c67553522f9e6ac69b98fad&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This test should not be run on Android. Maybe the skip logic is wrong. I will have a look at it tomorrow.

Comment 3

[Geoff Brown [:gbrown]]

Attachment: fix test annotation

I think the test for fennec is okay, but there's just a single '|' earlier in the annotation.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8922366 [details] [diff] [review]
fix test annotation

Review of attachment 8922366 [details] [diff] [review]:
-----------------------------------------------------------------

Ups! Sorry for that mistake. Thanks for fixing it.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/77cbe7b85ee2
Fix manifest annotation for test_crash.py; r=whimboo

Comment 6

[Intermittent Failures Robot]

25 failures in 196 pushes (0.128 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* try: 10
* autoland: 9
* mozilla-inbound: 6

Platform breakdown:
* android-4-3-armv7-api16: 25

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411348&startday=2017-10-26&endday=2017-10-26&tree=all

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/77cbe7b85ee2

Comment 8

[Intermittent Failures Robot]

89 failures in 912 pushes (0.098 failures/push) were associated with this bug in the last 7 days. 

This is the #12 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* autoland: 39
* mozilla-inbound: 29
* try: 11
* mozilla-central: 10

Platform breakdown:
* android-4-3-armv7-api16: 89

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411348&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel, this test is not disabled per-se. But only for those platforms it cannot be run on. As such I remove your whiteboard entry again.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

:whimboo, how would you categorize the fix?  If this doesn't run on Android, we disabled the test from running there to fix the problem, correct?  I do not see a fix to the product or a fix to the test in order to see this resolved.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This test cannot be run on Android because the application is not controlled by Marionette. The fix here was only a regression from by patch on bug 1410355, and didn't change anything in regards of where the tests are getting run.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

given this the problem of a perma failing test was resolved by disabling it on the platform where it was perma failing.  We have marked many tests as disabled for turning off jobs where they are not wanted anymore, etc.

Comment 13

[BMO Automation]

Moving bug to Testing::Marionette Client and Harness component per bug 1815831.