1549209 - Intermitent raptor certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.

Status: RESOLVED INCOMPLETE

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: dvarga [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=244722361&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/UTv49654SsOvKxXYxoh2ag/runs/0/artifacts/public/logs/live_backing.log

---

10:20:59 INFO - mozproxy getting the list of certs in the nss cert db in the android browser profile
10:21:01 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
10:21:01 INFO - mozproxy certutil command failed
10:21:01 INFO - mozproxy Stopping mitmproxy playback, killing process 557
10:21:01 INFO - mozproxy Successfully killed the mitmproxy playback process
10:21:01 INFO - raptor-main removing reverse socket connections
10:21:01 INFO - adb command_output: adb -s HT86C1A02550 wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
10:21:01 INFO - raptor-control-server shutting down control server
10:21:02 INFO - raptor-main finished
10:21:02 ERROR - Traceback (most recent call last):
10:21:02 INFO - File "/builds/task_1557051532/workspace/build/tests/raptor/raptor/raptor.py", line 1027, in <module>
10:21:02 INFO - main()
10:21:02 INFO - File "/builds/task_1557051532/workspace/build/tests/raptor/raptor/raptor.py", line 990, in main
10:21:02 INFO - raptor.run_test(next_test, timeout=int(next_test['page_timeout']))
10:21:02 INFO - File "/builds/task_1557051532/workspace/build/tests/raptor/raptor/raptor.py", line 777, in run_test
10:21:02 INFO - self.run_test_teardown()
10:21:02 INFO - File "/builds/task_1557051532/workspace/build/tests/raptor/raptor/raptor.py", line 180, in run_test_teardown
10:21:02 INFO - self.check_for_crashes()
10:21:02 INFO - File "/builds/task_1557051532/workspace/build/tests/raptor/raptor/raptor.py", line 913, in check_for_crashes
10:21:02 INFO - remote_dir = posixpath.join(self.device_profile, 'minidumps')
10:21:02 INFO - AttributeError: 'RaptorAndroid' object has no attribute 'device_profile'
10:21:07 INFO - certutil: could not change trust on certificate: SEC_ERROR_PKCS11_DEVICE_ERROR: A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot.
10:21:07 ERROR - Return code: 1
10:21:07 WARNING - setting return code to 1
10:21:07 INFO - Killing logcat pid 463.
10:21:07 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
10:21:07 INFO - copying raptor results to upload dir:
10:21:07 INFO - /builds/task_1557051532/workspace/build/blobber_upload_dir/perfherder-data.json
10:21:07 INFO - copying raptor results from /builds/task_1557051532/workspace/build/raptor.json to /builds/task_1557051532/workspace/build/blobber_upload_dir/perfherder-data.json
10:21:07 CRITICAL - Error copying results /builds/task_1557051532/workspace/build/raptor.json to upload dir /builds/task_1557051532/workspace/build/blobber_upload_dir/perfherder-data.json
10:21:07 INFO - [Errno 2] No such file or directory: u'/builds/task_1557051532/workspace/build/raptor.json'
10:21:07 INFO - Running post-action listener: _package_coverage_data
10:21:07 INFO - Running post-action listener: _resource_record_post_action
10:21:07 INFO - Running post-action listener: process_java_coverage_data
10:21:07 INFO - Running post-action listener: stop_device
10:21:11 INFO - Killing logcat pid 463.

Comment 1

[Intermittent Failures Robot]

4 failures in 4131 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 3
• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1549209&startday=2019-04-29&endday=2019-05-05&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 4705 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1549209&startday=2019-05-06&endday=2019-05-12&tree=all

Comment 3

[Andreea Pavel [:apavel]]

Recent occurences here have been classified with bug 1543933.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=247177459&repo=mozilla-central&lineNumber=1283

11:10:47 INFO - raptor-main starting raptor test: raptor-tp6m-instagram-geckoview

11:11:08 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
11:11:18 INFO - mozproxy importing mitmproxy cert into db using command: /builds/task_1558177734/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil -A -d sql:/tmp/tmpbRLk0F.mozrunner/ -n mitmproxy-cert -t TC,, -a -i /builds/worker/.mitmproxy/mitmproxy-ca-cert.cer
11:11:28 INFO - mozproxy command to import mitmproxy cert into cert db failed to complete
11:11:28 INFO - mozproxy verifying that the mitmproxy ca cert is installed on android
11:11:28 INFO - mozproxy getting the list of certs in the nss cert db in the android browser profile
11:11:29 INFO - certutil: function failed: SEC_ERROR_BAD_DATABASE: security library: bad database.
11:11:29 INFO - mozproxy certutil command failed
11:11:29 INFO - mozproxy Stopping mitmproxy playback, killing process 678
11:11:30 INFO - mozproxy Successfully killed the mitmproxy playback process
11:11:30 INFO - raptor-main removing reverse socket connections
11:11:30 INFO - adb command_output: adb -s ZY3223F92P wait-for-device reverse --remove-all, timeout: None, timedout: None, exitcode: 0, output:
11:11:30 INFO - raptor-main No crash directory (/sdcard/raptor/profile/minidumps) found on remote device
11:11:30 INFO - raptor-main removing webext /builds/task_1558177734/workspace/build/tests/raptor/raptor/../webext/raptor
11:11:30 INFO - raptor-main removing test folder for raptor: /sdcard/raptor
11:11:30 INFO - adb shell_output: adb -s ZY3223F92P wait-for-device shell rm -r /sdcard/raptor; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
11:11:30 INFO - raptor-control-server shutting down control server
11:11:30 INFO - raptor-main finished
11:11:30 ERROR - Traceback (most recent call last):
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/tests/raptor/raptor/raptor.py", line 1222, in <module>
11:11:30 INFO - main()
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/tests/raptor/raptor/raptor.py", line 1190, in main
11:11:30 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/tests/raptor/raptor/raptor.py", line 952, in run_tests
11:11:30 INFO - return super(RaptorAndroid, self).run_tests(tests, test_names)
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/tests/raptor/raptor/raptor.py", line 186, in run_tests
11:11:30 INFO - self.run_test(test, timeout=int(test['page_timeout']))
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/tests/raptor/raptor/raptor.py", line 973, in run_test
11:11:30 INFO - self.run_test_warm(test, timeout)
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/tests/raptor/raptor/raptor.py", line 1084, in run_test_warm
11:11:30 INFO - self.start_playback(test)
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/tests/raptor/raptor/raptor.py", line 357, in start_playback
11:11:30 INFO - self.playback.start()
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 125, in start
11:11:30 INFO - self.setup()
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 384, in setup
11:11:30 INFO - self.install_mitmproxy_cert(self.browser_path)
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 535, in install_mitmproxy_cert
11:11:30 INFO - if not self.is_mitmproxy_cert_installed():
11:11:30 INFO - File "/builds/task_1558177734/workspace/build/venv/lib/python2.7/site-packages/mozproxy/backends/mitm.py", line 552, in is_mitmproxy_cert_installed
11:11:30 INFO - cmd_output = subprocess.check_output(command, env=os.environ.copy())
11:11:30 INFO - File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
11:11:30 INFO - raise CalledProcessError(retcode, cmd, output=output)
11:11:30 INFO - subprocess.CalledProcessError: Command '['/builds/task_1558177734/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpbRLk0F.mozrunner/', '-L']' returned non-zero exit status 255
11:11:33 INFO - certutil: Could not set password for the slot: SEC_ERROR_BAD_PASSWORD: The security password entered is incorrect.
11:11:33 ERROR - Return code: 1

Robert can you assign someone to take a look?

Comment 4

[Intermittent Failures Robot]

4 failures in 4658 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform breakdown:

• android-hw-g5-7-0-arm7-api-16: 1
• android-hw-p2-8-0-android-aarch64: 1
• android-hw-p2-8-0-arm7-api-16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1549209&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 5

[Robert Wood [:rwood]]

Thanks, '4 failures in 4658 pushes', not a high priority for now.

Comment 6

[Andreea Pavel [:apavel]]

(In reply to Robert Wood [:rwood] from comment #5)

Thanks, '4 failures in 4658 pushes', not a high priority for now.

Other occurrences have been classified under https://bugzilla.mozilla.org/show_bug.cgi?id=1543933#c19

Comment 7

[Intermittent Failures Robot]

8 failures in 4730 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 5
• autoland: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 5
• android-hw-p2-8-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1549209&startday=2019-05-20&endday=2019-05-26&tree=all

Comment 8

[Intermittent Failures Robot]

4 failures in 5197 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 1
• android-hw-g5-7-0-arm7-api-16: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1549209&startday=2019-06-03&endday=2019-06-09&tree=all

Comment 9

[Intermittent Failures Robot]

1 failures in 4753 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1549209&startday=2019-06-10&endday=2019-06-16&tree=all

Comment 10

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup