1558177 - Intermittent raptor-main Error: No crash directory (/sdcard/raptor/profile/minidumps) found on remote device

Status: RESOLVED INCOMPLETE

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=250908832&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/Avft_R_IQtCIVivRPVDR8A/runs/0/artifacts/public/logs/live_backing.log

---

11:56:43 ERROR - raptor-main Error: No crash directory (/sdcard/raptor/profile/minidumps) found on remote device
11:56:43 INFO - raptor-main Info: removing webext /builds/task_1560167707/workspace/build/tests/raptor/raptor/../webext/raptor
11:56:43 INFO - raptor-main Info: removing test folder for raptor: /sdcard/raptor
11:56:43 INFO - adb shell_output: adb -s FA83T1A00347 wait-for-device shell rm -r /sdcard/raptor; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
11:56:43 INFO - raptor-control-server Info: shutting down control server
11:56:43 INFO - raptor-main Info: finished
11:56:43 ERROR - Traceback (most recent call last):
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/tests/raptor/raptor/raptor.py", line 1244, in <module>
11:56:44 INFO - main()
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/tests/raptor/raptor/raptor.py", line 1212, in main
11:56:44 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/tests/raptor/raptor/raptor.py", line 975, in run_tests
11:56:44 INFO - return super(RaptorAndroid, self).run_tests(tests, test_names)
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/tests/raptor/raptor/raptor.py", line 182, in run_tests
11:56:44 INFO - self.run_test(test, timeout=int(test['page_timeout']))
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/tests/raptor/raptor/raptor.py", line 994, in run_test
11:56:44 INFO - self.run_test_cold(test, timeout)
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/tests/raptor/raptor/raptor.py", line 1049, in run_test_cold
11:56:44 INFO - self.start_playback(test)
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/tests/raptor/raptor/raptor.py", line 372, in start_playback
11:56:44 INFO - self.playback.start()
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 125, in start
11:56:44 INFO - self.setup()
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 384, in setup
11:56:44 INFO - self.install_mitmproxy_cert(self.browser_path)
11:56:44 INFO - File "/builds/task_1560167707/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 457, in install_mitmproxy_cert
11:56:44 INFO - )[0]
11:56:44 INFO - IndexError: list index out of range
11:56:44 ERROR - Return code: 1
11:56:44 WARNING - setting return code to 1

Comment 1

[Greg Mierzwinski [:sparky]]

This seems to be a failure to find the host-utils archive when it is not in the cache (missing from the excerpt above):

11:56:39     INFO -  mozproxy INFO - File host-utils-68.0a1.en-US.linux-x86_64.tar.gz not present in local cache folder /builds/tooltool_cache
11:56:39     INFO -  mozproxy INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
11:56:43     INFO -  mozproxy INFO - ...failed to fetch 'host-utils-68.0a1.en-US.linux-x86_64.tar.gz' from https://tooltool.mozilla-releng.net/
11:56:43    ERROR -  mozproxy Traceback (most recent call last):
11:56:43     INFO -  mozproxy   File "/builds/task_1560167685/workspace/mozharness/external_tools/tooltool.py", line 694, in fetch_file
11:56:43     INFO -  mozproxy     f = urllib2.urlopen(req)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
11:56:43     INFO -  mozproxy     return opener.open(url, data, timeout)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 435, in open
11:56:43     INFO -  mozproxy     response = meth(req, response)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
11:56:43     INFO -  mozproxy     'http', request, response, code, msg, hdrs)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 467, in error
11:56:43     INFO -  mozproxy     result = self._call_chain(*args)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
11:56:43     INFO -  mozproxy     result = func(*args)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
11:56:43     INFO -  mozproxy     return self.parent.open(new, timeout=req.timeout)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 429, in open
11:56:43     INFO -  mozproxy     response = self._open(req, data)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
11:56:43     INFO -  mozproxy     '_open', req)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
11:56:43     INFO -  mozproxy     result = func(*args)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
11:56:43     INFO -  mozproxy     context=self._context)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
11:56:43     INFO -  mozproxy     raise URLError(err)
11:56:43     INFO -  mozproxy URLError: <urlopen error [Errno -2] Name or service not known>
11:56:43     INFO -  mozproxy ERROR - The following files failed: 'host-utils-68.0a1.en-US.linux-x86_64.tar.gz'
11:56:43     INFO -  mozproxy Stopping mitmproxy playback, killing process 707
11:56:43     INFO -  mozproxy Successfully killed the mitmproxy playback process
11:56:43     INFO -  raptor-main Info: removing reverse socket connections
11:56:43     INFO -  adb command_output: adb -s FA83T1A00347 wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
11:56:43    ERROR -  raptor-main Error: No crash directory (/sdcard/raptor/profile/minidumps) found on remote device

Comment 2

[Intermittent Failures Robot]

18 failures in 859 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 4
• mozilla-inbound: 1

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 6
• android-hw-p2-8-0-android-aarch64: 7
• android-hw-p2-8-0-arm7-api-16: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-14&endday=2019-06-14&tree=all

Comment 3

[Intermittent Failures Robot]

21 failures in 228 pushes (0.092 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-central: 8
• mozilla-inbound: 1
• try: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 10
• android-hw-g5-7-0-arm7-api-16: 7
• android-hw-p2-8-0-arm7-api-16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-15&endday=2019-06-15&tree=all

Comment 4

[Intermittent Failures Robot]

82 failures in 4753 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 28
• mozilla-inbound: 9
• autoland: 44
• try: 1

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 30
• android-hw-p2-8-0-android-aarch64: 29
• android-hw-p2-8-0-arm7-api-16: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-10&endday=2019-06-16&tree=all

Comment 5

[Cosmin Sabou [:CosminS]]

Since this was filed 8 days ago there are 94 failures across android-hw-7-0 and android-hw-8-0 pgo & opt builds: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-05-19&endday=2019-06-18&tree=trunk&bug=1558177
Robert, could you please take a look over this? Thank you.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252269883&repo=autoland&lineNumber=1269
07:42:35 INFO - raptor-main Info: removing reverse socket connections
07:42:35 INFO - adb command_output: adb -s ZY3223F92P wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
07:42:35 ERROR - raptor-main Error: No crash directory (/sdcard/raptor/profile/minidumps) found on remote device
07:42:35 INFO - raptor-main Info: removing webext /builds/task_1560843603/workspace/build/tests/raptor/raptor/../webext/raptor
07:42:35 INFO - raptor-main Info: removing test folder for raptor: /sdcard/raptor
07:42:35 INFO - adb shell_output: adb -s ZY3223F92P wait-for-device shell rm -r /sdcard/raptor; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
07:42:36 INFO - raptor-control-server Info: shutting down control server
07:42:36 INFO - raptor-main Info: finished

Comment 6

[Robert Wood [:rwood]]

(In reply to Greg Mierzwinski [:sparky] from comment #1)

This seems to be a failure to find the host-utils archive when it is not in the cache (missing from the excerpt above):

11:56:39     INFO -  mozproxy INFO - File host-utils-68.0a1.en-US.linux-x86_64.tar.gz not present in local cache folder /builds/tooltool_cache
11:56:39     INFO -  mozproxy INFO - Attempting to fetch from 'https://tooltool.mozilla-releng.net/'...
11:56:43     INFO -  mozproxy INFO - ...failed to fetch 'host-utils-68.0a1.en-US.linux-x86_64.tar.gz' from https://tooltool.mozilla-releng.net/
11:56:43    ERROR -  mozproxy Traceback (most recent call last):
11:56:43     INFO -  mozproxy   File "/builds/task_1560167685/workspace/mozharness/external_tools/tooltool.py", line 694, in fetch_file
11:56:43     INFO -  mozproxy     f = urllib2.urlopen(req)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
11:56:43     INFO -  mozproxy     return opener.open(url, data, timeout)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 435, in open
11:56:43     INFO -  mozproxy     response = meth(req, response)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
11:56:43     INFO -  mozproxy     'http', request, response, code, msg, hdrs)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 467, in error
11:56:43     INFO -  mozproxy     result = self._call_chain(*args)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
11:56:43     INFO -  mozproxy     result = func(*args)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
11:56:43     INFO -  mozproxy     return self.parent.open(new, timeout=req.timeout)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 429, in open
11:56:43     INFO -  mozproxy     response = self._open(req, data)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 447, in _open
11:56:43     INFO -  mozproxy     '_open', req)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
11:56:43     INFO -  mozproxy     result = func(*args)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
11:56:43     INFO -  mozproxy     context=self._context)
11:56:43     INFO -  mozproxy   File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
11:56:43     INFO -  mozproxy     raise URLError(err)
11:56:43     INFO -  mozproxy URLError: <urlopen error [Errno -2] Name or service not known>
11:56:43     INFO -  mozproxy ERROR - The following files failed: 'host-utils-68.0a1.en-US.linux-x86_64.tar.gz'
11:56:43     INFO -  mozproxy Stopping mitmproxy playback, killing process 707
11:56:43     INFO -  mozproxy Successfully killed the mitmproxy playback process
11:56:43     INFO -  raptor-main Info: removing reverse socket connections
11:56:43     INFO -  adb command_output: adb -s FA83T1A00347 wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
11:56:43    ERROR -  raptor-main Error: No crash directory (/sdcard/raptor/profile/minidumps) found on remote device

:gbrown, any ideas here?

Comment 7

[Geoff Brown [:gbrown]]

The url from comment 8 looks correct to me. Possibly a temporary failure? I have a feeling that bc and/or aerickson looked at similar issues in the past. Let's see if Andrew recalls seeing something like this before....

Comment 8

[Geoff Brown [:gbrown]]

Most of the recent failures look different: the host-utils download completes successfully, but then other things fail, like:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=252516100&repo=mozilla-central&lineNumber=1236

22:38:44 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
...
22:39:15 ERROR - Traceback (most recent call last):
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/tests/raptor/raptor/raptor.py", line 1281, in <module>
22:39:15 INFO - main()
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/tests/raptor/raptor/raptor.py", line 1249, in main
22:39:15 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/tests/raptor/raptor/raptor.py", line 1004, in run_tests
22:39:15 INFO - return super(RaptorAndroid, self).run_tests(tests, test_names)
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/tests/raptor/raptor/raptor.py", line 183, in run_tests
22:39:15 INFO - self.run_test(test, timeout=int(test['page_timeout']))
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/tests/raptor/raptor/raptor.py", line 1025, in run_test
22:39:15 INFO - self.run_test_warm(test, timeout)
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/tests/raptor/raptor/raptor.py", line 1139, in run_test_warm
22:39:15 INFO - self.start_playback(test)
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/tests/raptor/raptor/raptor.py", line 369, in start_playback
22:39:15 INFO - self.playback.start()
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 125, in start
22:39:15 INFO - self.setup()
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 384, in setup
22:39:15 INFO - self.install_mitmproxy_cert(self.browser_path)
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 535, in install_mitmproxy_cert
22:39:15 INFO - if not self.is_mitmproxy_cert_installed():
22:39:15 INFO - File "/builds/task_1560983832/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 552, in is_mitmproxy_cert_installed
22:39:15 INFO - cmd_output = subprocess.check_output(command, env=os.environ.copy())
22:39:15 INFO - File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
22:39:15 INFO - raise CalledProcessError(retcode, cmd, output=output)
22:39:15 INFO - subprocess.CalledProcessError: Command '['/builds/task_1560983832/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpo3HV6W.mozrunner/', '-L']' returned non-zero exit status 255
22:39:15 INFO - certutil: could not add certificate to token or database: SEC_ERROR_ADDING_CERT: Error adding certificate to database.

Comment 9

[Geoff Brown [:gbrown]]

SEC_ERROR_BAD_DATABASE is present in many/all passing tests too...but that still seems wrong!

Comment 10

[Geoff Brown [:gbrown]]

Regarding the more general issue of this bug: Why are we reporting "No crash directory..."? The harness checks that the minidumps directory is present on teardown; the directory should have been created by the test app. In most/all recent failures reported here, the test app has not been launched. Successful raptor runs have something like:

23:13:51     INFO -  raptor-main Info: starting geckoview
...
23:13:53     INFO -  adb launch_application: am start -W -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -a android.intent.action.MAIN --ez use_multiprocess True --es args "-profile /sdcard/raptor/profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6" -d about:blank
23:13:53     INFO -  adb shell_output: adb -s HT86B1A00744 wait-for-device shell am start -W -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -a android.intent.action.MAIN --ez use_multiprocess True --es args "-profile /sdcard/raptor/profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6" -d about:blank; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: Starting: Intent { act=android.intent.action.MAIN dat=about:blank cmp=org.mozilla.geckoview_example/.GeckoViewActivity (has extras) }
23:13:53     INFO -  Status: ok
23:13:53     INFO -  Activity: org.mozilla.geckoview_example/.GeckoViewActivity

The failures here do not. Possibly launch_firefox_android_app():
https://searchfox.org/mozilla-central/rev/da14c413ef663eb1ba246799e94a240f81c42488/testing/raptor/raptor/raptor.py#940

should set a flag "android_app_launched" and check_for_crashes should no-op if !android_app_launched.

Comment 11

[Intermittent Failures Robot]

79 failures in 2581 pushes (0.031 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 41
• mozilla-central: 25
• mozilla-inbound: 12
• try: 1

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 19
• android-hw-p2-8-0-android-aarch64: 37
• android-hw-p2-8-0-arm7-api-16: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-17&endday=2019-06-23&tree=all

Comment 12

[Robert Wood [:rwood]]

(In reply to Geoff Brown [:gbrown] (pto until June 26) from comment #9)

SEC_ERROR_BAD_DATABASE is present in many/all passing tests too...but that still seems wrong!

Yes - that error message is expected the first time Raptor (mozproxy actually) checks to see if the cert db exists already and it doesn't so then it is created. I have filed follow-up: Bug 1560974 - [mozproxy] Indicate in logging that the SEC_ERROR_BAD_DATABASE error is expected the first time mozproxy checks for the nss ca db.

Comment 13

[Robert Wood [:rwood]]

So looks to me that there are a few issues here being grouped into this one bug description.

Let's take this bug to clean-up the situation as noted by :gbrown in comment 10. :bebe can someone on SV take this, thanks!

Comment 14

[Andrew Erickson [:aerickson]]

I remember having issues finding certutil when my upgrade of tooltool caused a glob failure (Bug 1548008), but not "SEC_ERROR_BAD_DATABASE" exactly.

Comment 15

[Intermittent Failures Robot]

16 failures in 625 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 11
• mozilla-inbound: 2
• try: 1
• mozilla-central: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 7
• android-hw-p2-8-0-android-aarch64: 6
• android-hw-g5-7-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-24&endday=2019-06-24&tree=all

Comment 16

[Intermittent Failures Robot]

15 failures in 847 pushes (0.018 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 6
• android-hw-g5-7-0-arm7-api-16: 7
• android-hw-p2-8-0-arm7-api-16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-25&endday=2019-06-25&tree=all

Comment 17

[Intermittent Failures Robot]

22 failures in 844 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 9
• mozilla-inbound: 2
• try: 2

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 6
• android-hw-g5-7-0-arm7-api-16: 7
• android-hw-p2-8-0-android-aarch64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-26&endday=2019-06-26&tree=all

Comment 18

[Intermittent Failures Robot]

17 failures in 792 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 6
• mozilla-inbound: 6
• mozilla-central: 5

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 7
• android-hw-p2-8-0-android-aarch64: 6
• android-hw-p2-8-0-arm7-api-16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-27&endday=2019-06-27&tree=all

Comment 19

[Cosmin Sabou [:CosminS]]

This has failed 111 in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-21&endday=2019-06-28&tree=all&bug=1558177
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=253863545&repo=mozilla-inbound&lineNumber=1249
Robert, could you take over this as Florin has not responded yet?

Comment 20

[Intermittent Failures Robot]

97 failures in 4409 pushes (0.022 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 57
• mozilla-inbound: 14
• try: 3
• mozilla-central: 23

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 25
• android-hw-p2-8-0-android-aarch64: 37
• android-hw-g5-7-0-arm7-api-16: 35

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-06-24&endday=2019-06-30&tree=all

Comment 21

[Intermittent Failures Robot]

16 failures in 509 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 10
• mozilla-central: 6

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 8
• android-hw-p2-8-0-android-aarch64: 5
• android-hw-p2-8-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-01&endday=2019-07-01&tree=all

Comment 22

[Andreea Pavel [:apavel]]

Joel any thoughts here?

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

this is primarily on geckoview-tp6-cold tests- I suspect with cold we do many more profile pushes and browser shutdowns, therefore increasing the chance of failing to pull minidumps.

I am glad to see that Alexandru picked this up, please keep this bug updated with progress.

Comment 24

[Geoff Brown [:gbrown]]

Bug 1563229 will avoid "No crash directory" in most of these cases, which may help to focus this bug on the root cause.

It looks like most of these failures originate in tooltool failures, especially network failures during tooltool downloads originated from mozproxy. One issue then is, why do we see these network failures on bitbar? Another perspective, perhaps mozproxy should retry failed tooltool downloads? Consider https://searchfox.org/mozilla-central/rev/11712bd3ce7454923e5931fa92eaf9c01ef35a0a/testing/mozharness/mozharness/mozilla/tooltool.py#106 - tooltool retry logic in mozharness.

Comment 25

[Intermittent Failures Robot]

23 failures in 848 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 7
• mozilla-inbound: 3

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 9
• android-hw-p2-8-0-arm7-api-16: 2
• windows10-64-ccov: 1
• android-hw-g5-7-0-arm7-api-16: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-03&endday=2019-07-03&tree=all

Comment 26

[Intermittent Failures Robot]

16 failures in 648 pushes (0.025 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 7
• mozilla-inbound: 2
• mozilla-central: 7

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 3
• android-hw-g5-7-0-arm7-api-16: 6
• android-hw-p2-8-0-android-aarch64: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-04&endday=2019-07-04&tree=all

Comment 27

[Andreea Pavel [:apavel]]

There are 113 total failures in the last 7 days on android-hw-p2-8-0-arm7-api-16 and android-hw-p2-8-0-android-aarch64 opt and pgo

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255168134&repo=autoland&lineNumber=1267

4:58:57 INFO - mozproxy checking if the nss cert db already exists in the android browser profile
14:58:57 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
14:58:57 INFO - mozproxy nss cert db doesn't exist yet
14:59:07 INFO - mozproxy creating nss cert database using command: /builds/task_1562511419/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil -N -v -d sql:/tmp/tmpqrZibN.mozrunner/ --empty-password
14:59:07 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
14:59:17 INFO - mozproxy importing mitmproxy cert into db using command: /builds/task_1562511419/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil -A -d sql:/tmp/tmpqrZibN.mozrunner/ -n mitmproxy-cert -t TC,, -a -i /builds/worker/.mitmproxy/mitmproxy-ca-cert.cer
14:59:27 INFO - mozproxy command to import mitmproxy cert into cert db failed to complete
14:59:27 INFO - mozproxy verifying that the mitmproxy ca cert is installed on android
14:59:27 INFO - mozproxy getting the list of certs in the nss cert db in the android browser profile
14:59:28 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
14:59:28 INFO - mozproxy certutil command failed
14:59:28 INFO - mozproxy Stopping mitmproxy playback, killing process 710
14:59:28 INFO - mozproxy Successfully killed the mitmproxy playback process
14:59:28 INFO - raptor-main Info: removing reverse socket connections
14:59:29 INFO - adb command_output: adb -s ZY322BVC2F wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
14:59:29 ERROR - raptor-main Error: No crash directory (/sdcard/raptor/profile/minidumps) found on remote device
14:59:29 INFO - raptor-main Info: removing webext /builds/task_1562511419/workspace/build/tests/raptor/raptor/../webext/raptor
14:59:29 INFO - raptor-main Info: removing test folder for raptor: /sdcard/raptor
14:59:29 INFO - adb shell_output: adb -s ZY322BVC2F wait-for-device shell rm -r /sdcard/raptor; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
14:59:29 INFO - raptor-control-server Info: shutting down control server
14:59:29 INFO - raptor-main Info: finished

Comment 28

[Intermittent Failures Robot]

104 failures in 3501 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 53
• mozilla-central: 37
• mozilla-inbound: 14

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 49
• android-hw-p2-8-0-android-aarch64: 36
• android-hw-p2-8-0-arm7-api-16: 18
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 29

[Robert Wood [:rwood]]

(In reply to Cosmin Sabou [:CosminS] from comment #19)

This has failed 111 in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-06-21&endday=2019-06-28&tree=all&bug=1558177
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=253863545&repo=mozilla-inbound&lineNumber=1249
Robert, could you take over this as Florin has not responded yet?

:gbrown is taking care of the issue in comment 10 (thanks!)

Comment 30

[Intermittent Failures Robot]

26 failures in 4710 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 4
• mozilla-central: 13
• mozilla-beta: 6
• mozilla-inbound: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 13
• android-hw-g5-7-0-arm7-api-16: 8
• android-hw-p2-8-0-arm7-api-16: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 31

[Intermittent Failures Robot]

10 failures in 4298 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 9
• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 7
• android-hw-g5-7-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 32

[Intermittent Failures Robot]

5 failures in 4608 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 5

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 4
• android-hw-g5-7-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 33

[Intermittent Failures Robot]

2 failures in 4439 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 34

[Intermittent Failures Robot]

1 failures in 4666 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 35

[Intermittent Failures Robot]

6 failures in 4839 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 6

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 3
• android-hw-g5-7-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 36

[Intermittent Failures Robot]

16 failures in 5073 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 15
• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 8
• android-hw-g5-7-0-arm7-api-16: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 37

[Intermittent Failures Robot]

1 failures in 5047 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 38

[Intermittent Failures Robot]

1 failures in 4350 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558177&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 39

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.