1581345 - Intermittent No tests run or test summary not found

Status: RESOLVED FIXED

(Testing :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=266716016&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/J5QCh_4aQDiDwGTw0byifA/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/J5QCh_4aQDiDwGTw0byifA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2019-09-14T22:04:37.289Z] 22:04:37 INFO - Running main action method: run_tests
[task 2019-09-14T22:04:37.289Z] 22:04:37 INFO - Per-test suites: {'mochitest-chrome': 'mochitest-chrome', 'mochitest-plain': 'mochitest-plain'}
[task 2019-09-14T22:04:37.289Z] 22:04:37 INFO - Per-test suites: {}
[task 2019-09-14T22:04:37.289Z] 22:04:37 INFO - Per-test suites: {'xpcshell': 'xpcshell'}
[task 2019-09-14T22:04:37.290Z] 22:04:37 INFO - Minidump tooltool manifest unknown. Determining based upon platform and architecture.
[task 2019-09-14T22:04:37.290Z] 22:04:37 INFO - grabbing minidump binary from tooltool
[task 2019-09-14T22:04:37.290Z] 22:04:37 INFO - retry: Calling run_command with args: (['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/tooltool.mozilla-releng.net/', 'fetch', '-m', '/builds/worker/workspace/build/tests/config/tooltool-manifests/linux64/releng.manifest', '-o', '-c', '/builds/worker/tooltool-cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f10b8eeb408>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f10b9b50af0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7f10b9b368b8>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/builds/worker/workspace/build', 'privileged': False}, attempt #1
[task 2019-09-14T22:04:37.290Z] 22:04:37 INFO - Running command: ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/tooltool.mozilla-releng.net/', 'fetch', '-m', '/builds/worker/workspace/build/tests/config/tooltool-manifests/linux64/releng.manifest', '-o', '-c', '/builds/worker/tooltool-cache'] in /builds/worker/workspace/build
[task 2019-09-14T22:04:37.290Z] 22:04:37 INFO - Copy/paste: /usr/bin/python2.7 -u /builds/worker/workspace/mozharness/external_tools/tooltool.py --url http://taskcluster/tooltool.mozilla-releng.net/ fetch -m /builds/worker/workspace/build/tests/config/tooltool-manifests/linux64/releng.manifest -o -c /builds/worker/tooltool-cache
[task 2019-09-14T22:04:37.290Z] 22:04:37 INFO - Calling ['/usr/bin/python2.7', '-u', '/builds/worker/workspace/mozharness/external_tools/tooltool.py', '--url', 'http://taskcluster/tooltool.mozilla-releng.net/', 'fetch', '-m', '/builds/worker/workspace/build/tests/config/tooltool-manifests/linux64/releng.manifest', '-o', '-c', '/builds/worker/tooltool-cache'] with output_timeout 600
[task 2019-09-14T22:04:37.345Z] 22:04:37 INFO - INFO - File linux64-minidump_stackwalk retrieved from local cache /builds/worker/tooltool-cache
[task 2019-09-14T22:04:37.411Z] 22:04:37 INFO - Return code: 0
[task 2019-09-14T22:04:37.411Z] 22:04:37 INFO - Chmoding /builds/worker/workspace/build/linux64-minidump_stackwalk to 0755
[task 2019-09-14T22:04:37.412Z] 22:04:37 INFO - Per-test file(s) for 'mochitest-chrome': [u'toolkit/components/extensions/test/mochitest/test_chrome_ext_contentscript_unrecognizedprop_warning.html', u'toolkit/components/extensions/test/mochitest/test_chrome_ext_webnavigation_resolved_urls.html', u'toolkit/modules/tests/chrome/test_findCssSelector.html']
[task 2019-09-14T22:04:37.412Z] 22:04:37 INFO - Running the command /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --flavor=chrome --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/components/extensions/test/mochitest/test_chrome_ext_contentscript_unrecognizedprop_warning.html
[task 2019-09-14T22:04:37.413Z] 22:04:37 INFO - ##### mochitest-chrome log begins
[task 2019-09-14T22:04:37.413Z] 22:04:37 INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtestsremote.py', '--app=org.mozilla.geckoview.test', '--remote-webserver=10.0.2.2', '--xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--http-port=8854', '--ssl-port=4454', '--certificate-path=/builds/worker/workspace/build/tests/certs', u'--symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip', '--quiet', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log', '--log-raw-level=info', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log', '--log-tbpl-level=info', '--extra-profile-file=fonts', '--extra-profile-file=hyphenation', '--screenshot-on-fail', '--flavor=chrome', '--deviceSerial=emulator-5554', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--verify-max-time=900', '--verify', u'toolkit/components/extensions/test/mochitest/test_chrome_ext_contentscript_unrecognizedprop_warning.html'] in /builds/worker/workspace/build/tests/mochitest
[task 2019-09-14T22:04:37.413Z] 22:04:37 INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --flavor=chrome --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/components/extensions/test/mochitest/test_chrome_ext_contentscript_unrecognizedprop_warning.html
[task 2019-09-14T22:04:37.413Z] 22:04:37 INFO - Using env: (same as previous command)
[task 2019-09-14T22:04:37.575Z] 22:04:37 INFO - usage: runtestsremote.py [options] [test paths]
[task 2019-09-14T22:04:37.575Z] 22:04:37 INFO - runtestsremote.py: error: mochitest-chrome does not support e10s, try again with --disable-e10s.
[task 2019-09-14T22:04:37.584Z] 22:04:37 ERROR - Return code: 2
[task 2019-09-14T22:04:37.584Z] 22:04:37 ERROR - No tests run or test summary not found
[task 2019-09-14T22:04:37.584Z] 22:04:37 INFO - TinderboxPrint: mochitest-chrome<br/><em class="testfail">T-FAIL</em>
[task 2019-09-14T22:04:37.584Z] 22:04:37 INFO - ##### mochitest-chrome log ends
[task 2019-09-14T22:04:37.584Z] 22:04:37 WARNING - # TBPL WARNING #
[task 2019-09-14T22:04:37.585Z] 22:04:37 WARNING - setting return code to 1
[task 2019-09-14T22:04:37.585Z] 22:04:37 WARNING - TinderboxPrint: Per-test run of .../test_chrome_ext_contentscript_unrecognizedprop_warning.html<br/>: WARNING
[task 2019-09-14T22:04:37.585Z] 22:04:37 INFO - Running the command /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --flavor=chrome --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/components/extensions/test/mochitest/test_chrome_ext_webnavigation_resolved_urls.html
[task 2019-09-14T22:04:37.585Z] 22:04:37 INFO - ##### mochitest-chrome log begins
[task 2019-09-14T22:04:37.585Z] 22:04:37 INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtestsremote.py', '--app=org.mozilla.geckoview.test', '--remote-webserver=10.0.2.2', '--xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--http-port=8854', '--ssl-port=4454', '--certificate-path=/builds/worker/workspace/build/tests/certs', u'--symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip', '--quiet', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log', '--log-raw-level=info', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log', '--log-tbpl-level=info', '--extra-profile-file=fonts', '--extra-profile-file=hyphenation', '--screenshot-on-fail', '--flavor=chrome', '--deviceSerial=emulator-5554', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--verify-max-time=900', '--verify', u'toolkit/components/extensions/test/mochitest/test_chrome_ext_webnavigation_resolved_urls.html'] in /builds/worker/workspace/build/tests/mochitest
[task 2019-09-14T22:04:37.585Z] 22:04:37 INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --flavor=chrome --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/components/extensions/test/mochitest/test_chrome_ext_webnavigation_resolved_urls.html
[task 2019-09-14T22:04:37.586Z] 22:04:37 INFO - Using env: (same as previous command)
[task 2019-09-14T22:04:37.742Z] 22:04:37 INFO - usage: runtestsremote.py [options] [test paths]
[task 2019-09-14T22:04:37.742Z] 22:04:37 INFO - runtestsremote.py: error: mochitest-chrome does not support e10s, try again with --disable-e10s.
[task 2019-09-14T22:04:37.751Z] 22:04:37 ERROR - Return code: 2
[task 2019-09-14T22:04:37.751Z] 22:04:37 ERROR - No tests run or test summary not found
[task 2019-09-14T22:04:37.751Z] 22:04:37 INFO - TinderboxPrint: mochitest-chrome<br/><em class="testfail">T-FAIL</em>
[task 2019-09-14T22:04:37.751Z] 22:04:37 INFO - ##### mochitest-chrome log ends
[task 2019-09-14T22:04:37.751Z] 22:04:37 WARNING - # TBPL WARNING #
[task 2019-09-14T22:04:37.751Z] 22:04:37 WARNING - TinderboxPrint: Per-test run of .../test_chrome_ext_webnavigation_resolved_urls.html<br/>: WARNING
[task 2019-09-14T22:04:37.752Z] 22:04:37 INFO - Running the command /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --flavor=chrome --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/modules/tests/chrome/test_findCssSelector.html
[task 2019-09-14T22:04:37.752Z] 22:04:37 INFO - ##### mochitest-chrome log begins
[task 2019-09-14T22:04:37.752Z] 22:04:37 INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtestsremote.py', '--app=org.mozilla.geckoview.test', '--remote-webserver=10.0.2.2', '--xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--http-port=8854', '--ssl-port=4454', '--certificate-path=/builds/worker/workspace/build/tests/certs', u'--symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip', '--quiet', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log', '--log-raw-level=info', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log', '--log-tbpl-level=info', '--extra-profile-file=fonts', '--extra-profile-file=hyphenation', '--screenshot-on-fail', '--flavor=chrome', '--deviceSerial=emulator-5554', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--verify-max-time=900', '--verify', u'toolkit/modules/tests/chrome/test_findCssSelector.html'] in /builds/worker/workspace/build/tests/mochitest
[task 2019-09-14T22:04:37.752Z] 22:04:37 INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-chrome_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --flavor=chrome --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/modules/tests/chrome/test_findCssSelector.html
[task 2019-09-14T22:04:37.752Z] 22:04:37 INFO - Using env: (same as previous command)
[task 2019-09-14T22:04:37.887Z] 22:04:37 INFO - usage: runtestsremote.py [options] [test paths]
[task 2019-09-14T22:04:37.887Z] 22:04:37 INFO - runtestsremote.py: error: mochitest-chrome does not support e10s, try again with --disable-e10s.
[task 2019-09-14T22:04:37.896Z] 22:04:37 ERROR - Return code: 2
[task 2019-09-14T22:04:37.896Z] 22:04:37 ERROR - No tests run or test summary not found
[task 2019-09-14T22:04:37.897Z] 22:04:37 INFO - TinderboxPrint: mochitest-chrome<br/><em class="testfail">T-FAIL</em>
[task 2019-09-14T22:04:37.897Z] 22:04:37 INFO - ##### mochitest-chrome log ends
[task 2019-09-14T22:04:37.897Z] 22:04:37 WARNING - # TBPL WARNING #
[task 2019-09-14T22:04:37.897Z] 22:04:37 WARNING - TinderboxPrint: Per-test run of .../chrome/test_findCssSelector.html<br/>: WARNING
[task 2019-09-14T22:04:37.897Z] 22:04:37 INFO - Per-test file(s) for 'mochitest-plain': [u'toolkit/components/url-classifier/tests/mochitest/test_classifier.html']
[task 2019-09-14T22:04:37.897Z] 22:04:37 INFO - Running the command /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/components/url-classifier/tests/mochitest/test_classifier.html
[task 2019-09-14T22:04:37.897Z] 22:04:37 INFO - ##### mochitest-plain log begins
[task 2019-09-14T22:04:37.897Z] 22:04:37 INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtestsremote.py', '--app=org.mozilla.geckoview.test', '--remote-webserver=10.0.2.2', '--xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64', '--http-port=8854', '--ssl-port=4454', '--certificate-path=/builds/worker/workspace/build/tests/certs', u'--symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip', '--quiet', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_raw.log', '--log-raw-level=info', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_errorsummary.log', '--log-tbpl-level=info', '--extra-profile-file=fonts', '--extra-profile-file=hyphenation', '--screenshot-on-fail', '--deviceSerial=emulator-5554', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--verify-max-time=900', '--verify', u'toolkit/components/url-classifier/tests/mochitest/test_classifier.html'] in /builds/worker/workspace/build/tests/mochitest
[task 2019-09-14T22:04:37.898Z] 22:04:37 INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-68.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/SgwSs-3yRAG84ltL8QPcKg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --verify-max-time=900 --verify toolkit/components/url-classifier/tests/mochitest/test_classifier.html
[task 2019-09-14T22:04:37.898Z] 22:04:37 INFO - Using env: (same as previous command)
[task 2019-09-14T22:04:38.031Z] 22:04:38 INFO - adb Using adb 1.0.41
[task 2019-09-14T22:04:38.338Z] 22:04:38 INFO - adb adbd running as root
[task 2019-09-14T22:04:38.440Z] 22:04:38 INFO - adb su -c setenforce 0 exitcode 0, stdout: None
[task 2019-09-14T22:04:38.746Z] 22:04:38 INFO - adb su 0 supported
[task 2019-09-14T22:04:38.950Z] 22:04:38 INFO - adb /system/bin/ls -1A supported
[task 2019-09-14T22:04:39.052Z] 22:04:39 INFO - adb Native cp support: True
[task 2019-09-14T22:04:39.154Z] 22:04:39 INFO - adb Native chmod -R support: True
[task 2019-09-14T22:04:39.256Z] 22:04:39 INFO - adb Native chown -R support: True
[task 2019-09-14T22:04:39.358Z] 22:04:39 INFO - adb Setting SELinux Permissive
[task 2019-09-14T22:04:39.755Z] 22:04:39 INFO - adb Native pidof support: True
[task 2019-09-14T22:04:43.789Z] 22:04:43 INFO - Android sdk version '24'; will use this to filter manifests
[task 2019-09-14T22:04:43.789Z] 22:04:43 INFO - :::
[task 2019-09-14T22:04:43.789Z] 22:04:43 INFO - ::: Running test verification step "1. Run each test 10 times in one browser."...
[task 2019-09-14T22:04:43.789Z] 22:04:43 INFO - :::
....

Comment 1

[Intermittent Failures Robot]

3 failures in 4637 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-09-09&endday=2019-09-15&tree=all

Comment 2

[Geoff Brown [:gbrown]]

(In reply to Intermittent Failures Robot from comment #1)
Some of these belong in bug 1575266.

Comment 3

[Geoff Brown [:gbrown]]

...and others are bug 1578256.

Comment 4

[Intermittent Failures Robot]

25 failures in 869 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 24
• mozilla-inbound: 1

Platform breakdown:

• windows10-aarch64: 24
• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-09-20&endday=2019-09-20&tree=all

Comment 5

[Intermittent Failures Robot]

48 failures in 4350 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #26 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 10
• mozilla-central: 31
• mozilla-inbound: 7

Platform breakdown:

• android-em-7-0-x86_64: 22
• windows10-64-ccov: 3
• windows10-aarch64: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-09-16&endday=2019-09-22&tree=all

Comment 6

[Intermittent Failures Robot]

5 failures in 4530 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 3
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 7

[Intermittent Failures Robot]

6 failures in 4292 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 1
• autoland: 3
• mozilla-esr68: 1
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 5
• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-09-30&endday=2019-10-06&tree=all

Comment 8

[Intermittent Failures Robot]

5 failures in 3801 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• autoland: 2
• try: 1
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 3
• android-hw-p2-8-0-arm7-api-16-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 9

[Intermittent Failures Robot]

21 failures in 3964 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• try: 10
• mozilla-central: 2
• mozilla-inbound: 2

Platform breakdown:

• android-em-7-0-x86_64: 11
• lint: 1
• linux64: 1
• linux64-shippable-qr: 6
• linux64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 10

[Intermittent Failures Robot]

6 failures in 4182 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-inbound: 1
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-10-21&endday=2019-10-27&tree=all

Comment 11

[Intermittent Failures Robot]

1 failures in 4314 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-10-28&endday=2019-11-03&tree=all

Comment 12

[Intermittent Failures Robot]

7 failures in 4850 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-esr68: 1
• mozilla-central: 3
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 5
• android-em-7-0-x86_64-beta: 1
• android-hw-p2-8-0-arm7-api-16-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 13

[Geoff Brown [:gbrown]]

Recent failures are Android reftests, just like bug 1575266.

Comment 14

[Intermittent Failures Robot]

7 failures in 4207 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 2
• autoland: 4
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 3
• android-em-7-0-x86_64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-11-11&endday=2019-11-17&tree=all

Comment 15

[Intermittent Failures Robot]

8 failures in 5444 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 5

Platform breakdown:

• android-em-7-0-x86_64-qr: 2
• android-em-7-0-x86_64: 5
• android-hw-p2-8-0-arm7-api-16-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-11-18&endday=2019-11-24&tree=all

Comment 16

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #13)

Recent failures are Android reftests, just like bug 1575266.

That's still the biggest issue. There are also some regular crashes; I'm not as concerned about those.

Comment 17

[Intermittent Failures Robot]

6 failures in 4331 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1
• try: 1

Platform breakdown:

• android-em-7-0-x86_64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-11-25&endday=2019-12-01&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 4867 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-12-09&endday=2019-12-15&tree=all

Comment 19

[Bogdan Tara[:bogdan_tara | bogdant]]

Seems to have started to perma fail on Tier 2 on Central:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=windows%2C10%2Cx64%2Cccov%2Cdebug%2Cxpcshell%2Ctests%2Ctest-windows10-64-ccov%2Fdebug-xpcshell-e10s&fromchange=e94a014eb27e172be7f6b3860806803159d3d1f2&tochange=de230ce6bd5058c0608cddfb4b6c1fb3afb62759&selectedJob=282248296

Is it useful to backfill these? https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C10%2Cx64%2Cccov%2Cdebug%2Cxpcshell%2Ctests%2Ctest-windows10-64-ccov%2Fdebug-xpcshell-e10s&tochange=c2b76b2150345b2465770a37a0a713bd33f400fc&fromchange=0450c6ad5b8502e78973fe9462c0cb6a37f6b40c&selectedJob=282262848

Comment 20

[Geoff Brown [:gbrown]]

The regression is here:

https://treeherder.mozilla.org/#/jobs?repo=autoland&tochange=355cde565ce159fe6c4c53aedc1c412d8b570e04&searchStr=windows%2Cxpcshell&fromchange=0f937f06f71786f665bee862a33919e3b5bd1580

Would you backout https://hg.mozilla.org/integration/autoland/rev/355cde565ce159fe6c4c53aedc1c412d8b570e04, please?

Comment 21

[Cosmin Sabou [:CosminS]]

Backout: https://hg.mozilla.org/integration/autoland/rev/a8bcde65765354e3c4f5d4e8c5f7fc5ff288960c

Comment 22

[Intermittent Failures Robot]

102 failures in 195 pushes (0.523 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 91
• mozilla-central: 11

Platform breakdown:

• windows10-64-ccov: 102

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-12-21&endday=2019-12-21&tree=all

Comment 23

[Intermittent Failures Robot]

104 failures in 4582 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 11
• autoland: 93

Platform breakdown:

• windows10-64-ccov: 104

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-12-16&endday=2019-12-22&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures in 1365 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2019-12-23&endday=2019-12-29&tree=all

Comment 25

[Geoff Brown [:gbrown]]

Today there are several geckoview-junit failures which refer to TestHarnessException, added in

https://hg.mozilla.org/integration/autoland/rev/8b6baaf1e987714e9d18c11b718947c0f1a84899

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=283743870&repo=autoland&lineNumber=2206

Comment 26

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1581345 - Add TEST-UNEXPECTED-FAIL to TestHarnessException.

This is so that treeherder can pick up the failure line.

Comment 27

[[ex-Mozilla] Agi Sferro | :agi]

Ah, that's my bad. I uploaded a patch that should get rid of that.

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/337f50d46831
Add TEST-UNEXPECTED-FAIL to TestHarnessException. r=gbrown

Comment 29

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/337f50d46831

Comment 30

[Intermittent Failures Robot]

27 failures in 4472 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 25
• try: 1
• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 4
• android-hw-p2-8-0-arm7-api-16-qr: 2
• android-em-7-0-x86_64: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 31

[Intermittent Failures Robot]

11 failures in 3773 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 4

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 6
• android-hw-p2-8-0-android-aarch64: 3
• macosx1014-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-01-13&endday=2020-01-19&tree=all

Comment 32

[Geoff Brown [:gbrown]]

Some of these (osx) have been fixed by backout.

Some appear to be OOM - test app killed by Android. We should follow-up if they continue.

Comment 33

[Geoff Brown [:gbrown]]

These might also be device issues -- all Android failures are on Pixel2-10 and Pixel2-12.

Comment 34

[Intermittent Failures Robot]

11 failures in 4004 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 3
• mozilla-esr68: 1
• try: 1

Platform breakdown:

• android-hw-p2-8-0-arm7-api-16: 3
• android-em-7-0-x86_64: 5
• windows7-32: 1
• android-hw-p2-8-0-android-aarch64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-01-20&endday=2020-01-26&tree=all

Comment 35

[Geoff Brown [:gbrown]]

The android-em failures reported in comment 34 are dups of bug 1610676 - now resolved.

Comment 36

[Geoff Brown [:gbrown]]

The android-hw failures are all on Pixel2-10 and Pixel2-12, as noted previously.

Comment 37

[Intermittent Failures Robot]

6 failures in 2190 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 3
• android-em-7-0-x86_64-qr: 1
• windows10-64-ccov: 1
• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-01-27&endday=2020-02-02&tree=all

Comment 38

[Geoff Brown [:gbrown]]

Recent android-hw mochitest-webgl failures show something like "remoteautomation.py | exception reading log: [Errno 2] No such file or directory: '/tmp/tmpUoc2Z6'".

Comment 39

[Intermittent Failures Robot]

2 failures in 3364 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-esr68: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 1
• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-02-03&endday=2020-02-09&tree=all

Comment 40

[Intermittent Failures Robot]

6 failures in 4690 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 2
• mozilla-beta: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 2
• android-hw-p2-8-0-arm7-api-16: 1
• android-hw-p2-8-0-android-aarch64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-02-10&endday=2020-02-16&tree=all

Comment 41

[Intermittent Failures Robot]

1 failures in 5233 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-02-24&endday=2020-03-01&tree=all

Comment 42

[Intermittent Failures Robot]

4 failures in 5483 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 2

Platform breakdown:

• android-em-7-0-x86_64-qr: 2
• android-em-7-0-x86_64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-03-02&endday=2020-03-08&tree=all

Comment 43

[Intermittent Failures Robot]

4 failures in 5336 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-beta: 2
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 1
• android-hw-p2-8-0-android-aarch64: 2
• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-03-09&endday=2020-03-15&tree=all

Comment 44

[Intermittent Failures Robot]

3 failures in 4955 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform breakdown:

• windows10-64-ccov: 2
• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-03-16&endday=2020-03-22&tree=all

Comment 45

[Intermittent Failures Robot]

1 failures in 4773 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• android-em-7-0-x86_64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-03-23&endday=2020-03-29&tree=all

Comment 46

[Intermittent Failures Robot]

2 failures in 5015 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64: 1
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-03-30&endday=2020-04-05&tree=all

Comment 47

[Intermittent Failures Robot]

2 failures in 4823 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 48

[Intermittent Failures Robot]

18 failures in 4528 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5
• autoland: 13

Platform breakdown:

• windows10-64-ccov: 2
• android-hw-p2-8-0-android-aarch64-qr: 15
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 49

[Intermittent Failures Robot]

27 failures in 5582 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• autoland: 16
• try: 1
• comm-central: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 24
• macosx1014-64: 1
• linux64: 1
• linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 50

[Geoff Brown [:gbrown]]

Almost all recent failures are android-hw-p2-8-0-android-aarch64-qr/crashtest-e10s; might have a similar cause as bug 1630774.

Comment 51

[Intermittent Failures Robot]

13 failures in 5423 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 4
• comm-central: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 9
• linux32: 1
• windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 52

[Intermittent Failures Robot]

3 failures in 5276 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform breakdown:

• windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 53

[Intermittent Failures Robot]

1 failures in 5488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 54

[Intermittent Failures Robot]

2 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform breakdown:

• windows10-64-ccov: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 55

[Intermittent Failures Robot]

3 failures in 5458 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform breakdown:

• windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-05-25&endday=2020-05-31&tree=all

Comment 56

[Intermittent Failures Robot]

1 failures in 4949 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-06-01&endday=2020-06-07&tree=all

Comment 57

[Intermittent Failures Robot]

1 failures in 5211 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 58

[Intermittent Failures Robot]

2 failures in 4035 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• windows7-32: 1
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-06-15&endday=2020-06-21&tree=all

Comment 59

[Intermittent Failures Robot]

1 failures in 5252 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 60

[Intermittent Failures Robot]

2 failures in 4516 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-06-29&endday=2020-07-05&tree=all

Comment 61

[Intermittent Failures Robot]

1 failures in 5265 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-07-06&endday=2020-07-12&tree=all

Comment 62

[Intermittent Failures Robot]

2 failures in 4014 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-07-13&endday=2020-07-19&tree=all

Comment 63

[Intermittent Failures Robot]

8 failures in 4289 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 2
• android-em-7-0-x86_64: 5
• macosx1014-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 64

[Intermittent Failures Robot]

9 failures in 4618 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 7

Platform breakdown:

• windows10-64-ccov: 2
• android-em-7-0-x86_64: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 65

[Intermittent Failures Robot]

6 failures in 4100 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-shippable: 3
• windows10-64-ccov: 2
• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-08-17&endday=2020-08-23&tree=all

Comment 66

[Intermittent Failures Robot]

3 failures in 4046 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform breakdown:

• windows10-64-ccov: 2
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 67

[Intermittent Failures Robot]

3 failures in 2984 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• try: 1
• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 2
• android-hw-p2-8-0-android-aarch64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 68

[Intermittent Failures Robot]

3 failures in 3410 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• try: 1

Platform breakdown:

• windows10-64-ccov: 1
• android-hw-p2-8-0-android-aarch64: 1
• android-hw-p2-8-0-android-aarch64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-09-07&endday=2020-09-13&tree=all

Comment 69

[Intermittent Failures Robot]

16 failures in 3760 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• autoland: 13
• mozilla-central: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 2
• android-em-7-0-x86_64: 12
• linux1804-64-shippable: 1
• android-hw-p2-8-0-android-aarch64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-09-14&endday=2020-09-20&tree=all

Comment 70

[Intermittent Failures Robot]

8 failures in 4019 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• try: 2
• mozilla-beta: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-shippable: 3
• android-hw-p2-8-0-android-aarch64: 4
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-09-21&endday=2020-09-27&tree=all

Comment 71

[Intermittent Failures Robot]

19 failures in 744 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 2
• autoland: 17

Platform breakdown:

• android-em-7-0-x86_64: 17
• android-em-7-0-x86_64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-10-02&endday=2020-10-02&tree=all

Comment 72

[Intermittent Failures Robot]

33 failures in 4195 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #31 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 29
• mozilla-central: 1
• try: 3

Platform breakdown:

• android-em-7-0-x86_64: 29
• windows10-64-ccov: 1
• android-hw-p2-8-0-android-aarch64: 1
• android-em-7-0-x86_64-qr: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-09-28&endday=2020-10-04&tree=all

Comment 73

[Andreea Pavel [:apavel]]

Recent android failures here are bug 1668645, please disregard.

Comment 74

[Intermittent Failures Robot]

21 failures in 4130 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 18

Platform breakdown:

• windows10-64-ccov: 3
• android-em-7-0-x86_64: 17
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-10-05&endday=2020-10-11&tree=all

Comment 75

[Intermittent Failures Robot]

11 failures in 3694 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 7

Platform breakdown:

• windows10-64-ccov: 3
• android-em-7-0-x86_64: 7
• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-10-12&endday=2020-10-18&tree=all

Comment 76

[Intermittent Failures Robot]

3 failures in 4230 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64: 2
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-10-19&endday=2020-10-25&tree=all

Comment 77

[Intermittent Failures Robot]

6 failures in 4275 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-beta: 3
• autoland: 1

Platform breakdown:

• windows10-64-ccov-qr: 2
• windows7-32-shippable: 2
• windows10-64-shippable: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-10-26&endday=2020-11-01&tree=all

Comment 78

[Intermittent Failures Robot]

7 failures in 4128 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 4
• autoland: 1
• mozilla-release: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 2
• windows10-64-ccov-qr: 3
• macosx1014-64-qr: 1
• android-hw-p2-8-0-android-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1581345&startday=2020-11-02&endday=2020-11-08&tree=all

Comment 79

[Intermittent Failures Robot]

3 failures in 4040 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• try: 1

Platform breakdown:

• windows10-64-ccov-qr: 2
• linux1804-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2020-11-09&endday=2020-11-15&tree=all

Comment 80

[Intermittent Failures Robot]

5 failures in 4810 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• try: 3

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 3
• windows10-64-ccov-qr: 1
• macosx1014-64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2020-11-16&endday=2020-11-22&tree=all

Comment 81

[Intermittent Failures Robot]

6 failures in 1479 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• try: 1
• autoland: 2
• mozilla-beta: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 3
• android-hw-p2-8-0-android-aarch64-shippable-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2020-11-23&endday=2020-11-29&tree=all

Comment 82

[Intermittent Failures Robot]

14 failures in 3712 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 8
• try: 1

Platform breakdown:

• android-em-7-0-x86_64-qr: 2
• android-hw-p2-8-0-android-aarch64-qr: 6
• windows10-64-ccov-qr: 2
• android-hw-p2-8-0-android-aarch64-shippable-qr: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2020-11-30&endday=2020-12-06&tree=all

Comment 83

[Geoff Brown [:gbrown]]

Recent failures on Android show:
https://treeherder.mozilla.org/logviewer?job_id=323806791&repo=autoland&lineNumber=1771
[task 2020-12-07T19:18:58.270Z] 19:18:58 INFO - 0 ERROR The 'prefs' key must be set in the DEFAULT section of a manifest. Fix the following manifests: dom/html/test/mochitest.ini

I'll look into that...

Comment 84

[Geoff Brown [:gbrown]]

Seems related to bumpy landing in bug 1680596.

Comment 85

[Geoff Brown [:gbrown]]

Looks like it was a temporary hitch on autoland, perhaps addressed by additional fixes, or backouts.

Comment 86

[Intermittent Failures Robot]

16 failures in 4060 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 7
• try: 2

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 6
• android-hw-p2-8-0-android-aarch64-shippable-qr: 7
• android-em-7-0-x86_64-qr: 1
• android-em-7-0-x86_64: 1
• windows10-64-ccov-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 87

[Intermittent Failures Robot]

18 failures in 3421 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• mozilla-beta: 1
• try: 5
• autoland: 6

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 11
• android-hw-p2-8-0-android-aarch64-shippable-qr: 6
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2020-12-14&endday=2020-12-20&tree=all

Comment 88

[Intermittent Failures Robot]

3 failures in 1450 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 1
• macosx1014-64-qr: 1
• windows10-64-ccov-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2020-12-21&endday=2020-12-27&tree=all

Comment 89

[Intermittent Failures Robot]

1 failures in 2956 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-01-04&endday=2021-01-10&tree=all

Comment 90

[Intermittent Failures Robot]

3 failures in 3810 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-beta: 1

Platform breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
• windows10-64-ccov-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-01-11&endday=2021-01-17&tree=all

Comment 91

[Intermittent Failures Robot]

1 failures in 3488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows7-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-01-18&endday=2021-01-24&tree=all

Comment 92

[Intermittent Failures Robot]

1 failures in 3891 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-01-25&endday=2021-01-31&tree=all

Comment 93

[Intermittent Failures Robot]

3 failures in 3966 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform and build breakdown:

• windows7-32: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-02-01&endday=2021-02-07&tree=all

Comment 94

[Intermittent Failures Robot]

2 failures in 4322 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 95

[Intermittent Failures Robot]

3 failures in 3737 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-central: 1

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-02-15&endday=2021-02-21&tree=all

Comment 96

[Intermittent Failures Robot]

9 failures in 4327 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 3

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 2
  • debug: 2
• windows10-64-ccov-qr: 2
  • opt: 2
• linux1804-64-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64: 3
  • opt: 3
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 97

[Intermittent Failures Robot]

4 failures in 3683 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• try: 1

Platform and build breakdown:

• linux1804-64: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1
• android-hw-p2-8-0-android-aarch64: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 98

[Intermittent Failures Robot]

4 failures in 4622 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 2

Platform and build breakdown:

• android-hw-p2-8-0-arm7-api-16: 1
  • debug: 1
• android-hw-p2-8-0-android-aarch64-qr: 2
  • debug: 2
• windows10-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 99

[Intermittent Failures Robot]

10 failures in 4257 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-central: 2
• try: 1
• autoland: 6

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1
• windows10-64-ccov-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 3
  • debug: 2
  • opt: 1
• android-hw-p2-8-0-android-aarch64: 1
  • opt: 1
• android-em-7-0-x86_64: 2
  • opt: 1
  • debug: 1
• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 100

[Intermittent Failures Robot]

54 failures in 714 pushes (0.076 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 54

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 36
  • debug: 35
  • opt: 1
• android-em-7-0-x86_64: 18
  • debug: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-03-26&endday=2021-03-26&tree=all

Comment 101

[Andreea Pavel [:apavel]]

There are 67 total failures in the last 7 days on

• android-em-7-0-x86_64-qr debug and opt
• android-em-7-0-x86_64 debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=334579471&repo=autoland&lineNumber=1939

[task 2021-03-26T17:52:28.484Z] 17:52:28 INFO - SUITE-START | Running 0 tests
[task 2021-03-26T17:52:28.484Z] 17:52:28 INFO - 0 INFO TEST-START | Shutdown
[task 2021-03-26T17:52:28.485Z] 17:52:28 INFO - 1 INFO Passed: 0
[task 2021-03-26T17:52:28.485Z] 17:52:28 INFO - 2 INFO Failed: 0
[task 2021-03-26T17:52:28.485Z] 17:52:28 INFO - 3 INFO Todo: 0
[task 2021-03-26T17:52:28.485Z] 17:52:28 INFO - 4 INFO Mode: e10s
[task 2021-03-26T17:52:28.485Z] 17:52:28 INFO - 5 INFO SimpleTest FINISHED
[task 2021-03-26T17:52:28.509Z] 17:52:28 INFO - --------- beginning of main
[task 2021-03-26T17:52:28.509Z] 17:52:28 INFO - 03-26 17:52:26.667 I/log ( 3055): logcat cleared
[task 2021-03-26T17:52:28.509Z] 17:52:28 INFO - 03-26 17:52:27.184 D/AndroidRuntime( 3100): >>>>>> START com.android.internal.os.RuntimeInit uid 0 <<<<<<
[task 2021-03-26T17:52:28.510Z] 17:52:28 INFO - 03-26 17:52:27.185 D/AndroidRuntime( 3100): CheckJNI is ON
[task 2021-03-26T17:52:28.510Z] 17:52:28 INFO - 03-26 17:52:27.210 W/art ( 3100): Unexpected CPU variant for X86 using defaults: x86_64
[task 2021-03-26T17:52:28.510Z] 17:52:28 INFO - 03-26 17:52:27.214 D/ICU ( 3100): No timezone override file found: /data/misc/zoneinfo/current/icu/icu_tzdata.dat
[task 2021-03-26T17:52:28.511Z] 17:52:28 INFO - 03-26 17:52:27.234 E/memtrack( 3100): Couldn't load memtrack module (No such file or directory)
[task 2021-03-26T17:52:28.511Z] 17:52:28 INFO - 03-26 17:52:27.234 E/android.os.Debug( 3100): failed to load memtrack module: -2
[task 2021-03-26T17:52:28.511Z] 17:52:28 INFO - 03-26 17:52:27.235 I/Radio-JNI( 3100): register_android_hardware_Radio DONE
[task 2021-03-26T17:52:28.511Z] 17:52:28 INFO - 03-26 17:52:27.242 D/AndroidRuntime( 3100): Calling main entry com.android.commands.am.Am
[task 2021-03-26T17:52:28.511Z] 17:52:28 INFO - --------- beginning of system
[task 2021-03-26T17:52:28.512Z] 17:52:28 INFO - 03-26 17:52:27.243 I/ActivityManager( 1618): Force stopping org.mozilla.geckoview.test appid=10062 user=0: from pid 3100
[task 2021-03-26T17:52:28.512Z] 17:52:28 INFO - 03-26 17:52:27.244 D/AndroidRuntime( 3100): Shutting down VM
[task 2021-03-26T17:52:28.512Z] 17:52:28 INFO - 03-26 17:52:27.246 D/CarrierSvcBindHelper( 1847): No carrier app for: 0

[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - systime: Fri Mar 26 17:52:28 GMT 2021
[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - disk:
[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - Filesystem 1K-blocks Used Available Use% Mounted on
[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - /dev/block/vdc 3525632 350624 3175008 10% /data
[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - /dev/block/vda 2539312 912448 1626864 36% /system
[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - /dev/fuse 3525632 350624 3175008 10% /storage/emulated
[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - os: sdk_phone_x86_64-userdebug 7.0 NYC 4174735 test-keys
[task 2021-03-26T17:52:28.706Z] 17:52:28 INFO - id: emulator-5554
[task 2021-03-26T17:52:28.707Z] 17:52:28 INFO - Test root: /data/local/tmp/test_root
[task 2021-03-26T17:52:28.707Z] 17:52:28 INFO - Buffered messages finished
[task 2021-03-26T17:52:28.707Z] 17:52:28 INFO - SUITE-END | took 0s
[task 2021-03-26T17:52:28.719Z] 17:52:28 ERROR - Return code: 1
[task 2021-03-26T17:52:28.719Z] 17:52:28 ERROR - No tests run or test summary not found
[task 2021-03-26T17:52:28.719Z] 17:52:28 INFO - TinderboxPrint: mochitest-plain<br/><em class="testfail">T-FAIL</em>
[task 2021-03-26T17:52:28.719Z] 17:52:28 INFO - ##### mochitest-plain log ends
[task 2021-03-26T17:52:28.719Z] 17:52:28 WARNING - # TBPL WARNING #
[task 2021-03-26T17:52:28.719Z] 17:52:28 WARNING - setting return code to 1

There was a spike here between 25th-27th of March. Geoff can you take a look?

Comment 102

[Geoff Brown [:gbrown]]

Our test scheduling algorithm is imperfect: Sometimes it tries to run a test on a platform where the test doesn't run. That's what happened here: The scheduler instructed the harness to run browser-chrome tests on Android; the harness refused to run them, since browser-chrome is not supported on Android; since no tests were run, the task failed. afaict, that only happened once, but then someone re-ran the task multiple times, and also backfilled it, and then re-ran the backfills -- and they all failed too, of course.

I believe :ahal understands this scheduling limitation and that it is difficult to fix, so the current plan is to just tolerate occasional failures like this.

Comment 103

[Intermittent Failures Robot]

66 failures in 3967 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 64
• mozilla-central: 2

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 41
  • debug: 39
  • opt: 2
• windows7-32: 1
  • debug: 1
• windows10-64-ccov-qr: 2
  • opt: 2
• android-hw-p2-8-0-android-aarch64: 1
  • opt: 1
• macosx1014-64-qr: 1
  • debug: 1
• android-em-7-0-x86_64: 20
  • opt: 1
  • debug: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-03-22&endday=2021-03-28&tree=all

Comment 104

[Intermittent Failures Robot]

18 failures in 644 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 2

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 12
  • debug: 8
  • opt: 4
• windows10-64-ccov-qr: 2
  • opt: 2
• android-em-7-0-x86_64: 4
  • opt: 1
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-04-01&endday=2021-04-01&tree=all

Comment 105

[Intermittent Failures Robot]

32 failures in 3125 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-release: 1
• mozilla-central: 5
• autoland: 26

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1
• windows10-64-ccov-qr: 5
  • opt: 5
• android-em-7-0-x86_64-qr: 19
  • opt: 6
  • debug: 13
• android-em-7-0-x86_64: 6
  • debug: 4
  • opt: 2
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-03-29&endday=2021-04-04&tree=all

Comment 106

[Intermittent Failures Robot]

4 failures in 3854 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 3

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 3
  • debug: 3
• windows7-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 107

[Intermittent Failures Robot]

33 failures in 870 pushes (0.038 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 33

Platform and build breakdown:

• android-em-7-0-x86_64: 2
  • debug: 2
• android-em-7-0-x86_64-qr: 31
  • debug: 4
  • opt: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-04-15&endday=2021-04-15&tree=all

Comment 108

[Intermittent Failures Robot]

47 failures in 4255 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #39 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 45
• try: 1
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 36
  • opt: 29
  • debug: 7
• android-em-7-0-x86_64: 8
  • opt: 5
  • debug: 3
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 109

[Intermittent Failures Robot]

11 failures in 3224 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 6
  • debug: 5
  • opt: 1
• android-em-7-0-x86_64: 3
  • debug: 2
  • opt: 1
• windows7-32: 1
  • debug: 1
• windows7-32-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 110

[Intermittent Failures Robot]

13 failures in 3706 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 2
• mozilla-beta: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 8
  • debug: 7
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 2
  • opt: 2
• android-hw-p2-8-0-arm7-api-16-shippable: 1
  • opt: 1
• android-em-7-0-x86_64: 2
  • opt: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-04-26&endday=2021-05-02&tree=all

Comment 111

[Intermittent Failures Robot]

2 failures in 3279 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64: 1
  • debug: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-05-03&endday=2021-05-09&tree=all

Comment 112

[Intermittent Failures Robot]

7 failures in 3061 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 1
• mozilla-beta: 1

Platform and build breakdown:

• android-em-7-0-x86_64: 2
  • opt: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 3
  • debug: 3
• android-hw-p2-8-0-android-aarch64: 1
  • debug: 1
• windows7-32-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-05-17&endday=2021-05-23&tree=all

Comment 113

[Intermittent Failures Robot]

7 failures in 2908 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7

Platform and build breakdown:

• android-em-7-0-x86_64: 3
  • debug: 3
• android-em-7-0-x86_64-qr: 3
  • debug: 3
• android-hw-p2-8-0-android-aarch64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-05-24&endday=2021-05-30&tree=all

Comment 114

[Intermittent Failures Robot]

9 failures in 3065 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 7
• mozilla-central: 2

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 6
  • debug: 4
  • opt: 2
• android-em-7-0-x86_64: 1
  • debug-isolated-process: 1
• android-hw-p2-8-0-android-aarch64: 1
  • debug: 1
• android-hw-p2-8-0-android-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-05-31&endday=2021-06-06&tree=all

Comment 115

[Intermittent Failures Robot]

11 failures in 3502 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 9
• mozilla-central: 1

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable: 1
  • opt: 1
• linux1804-64: 1
  • debug: 1
• linux1804-64-ccov: 1
  • opt: 1
• android-em-7-0-x86_64: 3
  • debug: 2
  • opt: 1
• android-em-7-0-x86_64-qr: 5
  • debug: 4
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-06-07&endday=2021-06-13&tree=all

Comment 116

[Geoff Brown [:gbrown]]

Recent failures here are as in comment 102, where the test harness is asked to run only manifests that are not run on that platform. Most recent failures are on Android and involve mochitests under browser/, which are usually excluded from Android in moz.build files.

Comment 117

[Intermittent Failures Robot]

16 failures in 2888 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• mozilla-central: 1

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64: 3
  • opt: 2
  • debug: 1
• android-em-7-0-x86_64-qr: 10
  • opt: 2
  • debug: 8
• android-em-7-0-x86_64: 3
  • opt: 1
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-06-14&endday=2021-06-20&tree=all

Comment 118

[Andreea Pavel [:apavel]]

There are 361 total failures in the last 7 days on

• android-em-7-0-x86_64 opt and debug
• android-hw-p2-8-0-android-aarch64 opt and debug
  Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=343433814&repo=autoland&lineNumber=1677

[task 2021-06-21T23:02:13.676Z] 23:02:13 INFO - ##### mochitest-plain log begins
[task 2021-06-21T23:02:13.676Z] 23:02:13 INFO - Running command: ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtestsremote.py', '--app=org.mozilla.geckoview.test', '--remote-webserver=10.0.2.2', '--xre-path=/builds/worker/workspace/build/hostutils/host-utils-85.0a1.en-US.linux-x86_64', '--utility-path=/builds/worker/workspace/build/hostutils/host-utils-85.0a1.en-US.linux-x86_64', '--http-port=8854', '--ssl-port=4454', '--certificate-path=/builds/worker/workspace/build/tests/certs', '--symbols-path=/builds/worker/workspace/build/symbols', '--quiet', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_raw.log', '--log-raw-level=info', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_errorsummary.log', '--log-tbpl-level=info', '--extra-profile-file=fonts', '--extra-profile-file=hyphenation', '--screenshot-on-fail', '--deviceSerial=emulator-5554', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--setpref=layers.d3d11.enable-blacklist=false', 'browser/extensions/formautofill/test/mochitest/creditCard/mochitest.ini', '--enable-webrender'] in /builds/worker/workspace/build/tests/mochitest
[task 2021-06-21T23:02:13.676Z] 23:02:13 INFO - Copy/paste: /builds/worker/workspace/build/venv/bin/python -u /builds/worker/workspace/build/tests/mochitest/runtestsremote.py --app=org.mozilla.geckoview.test --remote-webserver=10.0.2.2 --xre-path=/builds/worker/workspace/build/hostutils/host-utils-85.0a1.en-US.linux-x86_64 --utility-path=/builds/worker/workspace/build/hostutils/host-utils-85.0a1.en-US.linux-x86_64 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/worker/workspace/build/tests/certs --symbols-path=/builds/worker/workspace/build/symbols --quiet --log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_raw.log --log-raw-level=info --log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-plain_errorsummary.log --log-tbpl-level=info --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --deviceSerial=emulator-5554 --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --setpref=layers.d3d11.enable-blacklist=false browser/extensions/formautofill/test/mochitest/creditCard/mochitest.ini --enable-webrender
[task 2021-06-21T23:02:13.677Z] 23:02:13 INFO - Using env: (same as previous command)
[task 2021-06-21T23:02:14.078Z] 23:02:14 INFO - adb Using adb 1.0.41
[task 2021-06-21T23:02:14.168Z] 23:02:14 INFO - adb /system/bin/ls -1A supported
[task 2021-06-21T23:02:15.458Z] 23:02:15 INFO - adb Native cp support: True
[task 2021-06-21T23:02:15.481Z] 23:02:15 INFO - adb Native chmod -R support: True
[task 2021-06-21T23:02:15.504Z] 23:02:15 INFO - adb Native chown -R support: True
[task 2021-06-21T23:02:15.600Z] 23:02:15 INFO - adb Native flaky pidof support: True
[task 2021-06-21T23:02:15.625Z] 23:02:15 INFO - adb adbd running as root
[task 2021-06-21T23:02:15.720Z] 23:02:15 INFO - adb Setting SELinux Permissive
[task 2021-06-21T23:02:16.150Z] 23:02:16 INFO - adb Setting test_root to /data/local/tmp/test_root
[task 2021-06-21T23:02:17.205Z] 23:02:17 INFO - Android sdk version '24'; will use this to filter manifests
[task 2021-06-21T23:02:17.205Z] 23:02:17 INFO - Checking for ssltunnel processes...
[task 2021-06-21T23:02:17.212Z] 23:02:17 INFO - Checking for xpcshell processes...
[task 2021-06-21T23:02:17.581Z] 23:02:17 INFO - No tests were found for flavor 'plain' and the following manifest filters:
[task 2021-06-21T23:02:17.581Z] 23:02:17 INFO - skip_if, run_if, fail_if, subsuite(name=None), pathprefix(['browser/extensions/formautofill/test/mochitest/creditCard/mochitest.ini'])
[task 2021-06-21T23:02:17.581Z] 23:02:17 INFO - Make sure the test paths (if any) are spelt correctly and the corresponding
[task 2021-06-21T23:02:17.581Z] 23:02:17 INFO - --flavor and --subsuite are being used. See mach mochitest --help for a
[task 2021-06-21T23:02:17.581Z] 23:02:17 INFO - list of valid flavors.
[task 2021-06-21T23:02:17.584Z] 23:02:17 INFO - SUITE-START | Running 0 tests
[task 2021-06-21T23:02:17.584Z] 23:02:17 INFO - 0 INFO TEST-START | Shutdown
[task 2021-06-21T23:02:17.584Z] 23:02:17 INFO - 1 INFO Passed: 0
[task 2021-06-21T23:02:17.584Z] 23:02:17 INFO - 2 INFO Failed: 0
[task 2021-06-21T23:02:17.584Z] 23:02:17 INFO - 3 INFO Todo: 0
[task 2021-06-21T23:02:17.584Z] 23:02:17 INFO - 4 INFO Mode: e10s
[task 2021-06-21T23:02:17.585Z] 23:02:17 INFO - 5 INFO SimpleTest FINISHED
[task 2021-06-21T23:02:17.585Z] 23:02:17 INFO - Buffered messages finished
[task 2021-06-21T23:02:17.585Z] 23:02:17 INFO - SUITE-END | took 0s
[task 2021-06-21T23:02:17.620Z] 23:02:17 ERROR - Return code: 1
[task 2021-06-21T23:02:17.620Z] 23:02:17 ERROR - No tests run or test summary not found
[task 2021-06-21T23:02:17.620Z] 23:02:17 INFO - TinderboxPrint: mochitest-plain<br/><em class="testfail">T-FAIL</em>
[task 2021-06-21T23:02:17.620Z] 23:02:17 INFO - ##### mochitest-plain log ends
[task 2021-06-21T23:02:17.620Z] 23:02:17 WARNING - # TBPL WARNING #
[task 2021-06-21T23:02:17.621Z] 23:02:17 WARNING - setting return code to 1
[task 2021-06-21T23:02:17.621Z] 23:02:17 WARNING - The mochitest-plain suite: mochitest-plain ran with return status: WARNING

Some of the failures are bad workers actually and they got quarantined:

• i-019abfdda4e50312c
• i-015f148acc3237c75
• i-03da0d43756e6ef90
• i-03503a6d7bcafa64a
• i-0279348ecb51aa1a5
• i-0eb7e043d1c7f66b4
• i-0ca80bb895e8c43ff
• i-0aa717f2921f921fe
• i-0a31cd98cb3c747f7
• i-086c7a883ef7ff21c
• i-06bcdafd152c18607

Comment 119

[Intermittent Failures Robot]

150 failures in 565 pushes (0.265 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 150

Platform and build breakdown:

• android-em-7-0-x86_64: 101
  • opt: 100
  • debug: 1
• linux1804-64-qr: 42
  • debug: 42
• windows7-32-qr: 2
  • opt: 2
• android-em-7-0-x86_64-qr: 5
  • opt: 1
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-06-21&endday=2021-06-21&tree=all

Comment 120

[Intermittent Failures Robot]

163 failures in 3115 pushes (0.052 failures/push) were associated with this bug in the last 7 days.

This is the #6 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 156
• mozilla-central: 6
• try: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 10
  • opt: 1
  • debug: 9
• android-em-7-0-x86_64: 102
  • opt: 100
  • debug: 2
• windows7-32-qr: 2
  • opt: 2
• linux1804-64-qr: 42
  • debug: 42
• windows10-32-shippable-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-06-21&endday=2021-06-27&tree=all

Comment 121

[Intermittent Failures Robot]

3 failures in 723 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 3

Platform and build breakdown:

• windows10-32-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-06-28&endday=2021-07-04&tree=all

Comment 122

[Intermittent Failures Robot]

8 failures in 2387 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 2
• mozilla-central: 4
• try: 2

Platform and build breakdown:

• android-em-7-0-x86_64: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • debug: 1
• windows10-32-shippable-qr: 5
  • opt: 5
• windows10-64-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-07-05&endday=2021-07-11&tree=all

Comment 123

[Intermittent Failures Robot]

16 failures in 2224 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 10
• mozilla-central: 2
• autoland: 4

Platform and build breakdown:

• windows10-64-devedition-qr: 5
  • opt: 5
• windows10-32-shippable-qr: 7
  • opt: 7
• android-em-7-0-x86_64-qr: 2
  • debug: 2
• android-em-7-0-x86_64: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-07-12&endday=2021-07-18&tree=all

Comment 124

[Intermittent Failures Robot]

15 failures in 2940 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 12
• mozilla-beta: 2
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 4
  • debug: 2
  • opt: 2
• android-em-7-0-x86_64: 2
  • opt: 2
• windows10-32-shippable-qr: 1
  • opt: 1
• windows10-64-devedition-qr: 1
  • opt: 1
• linux1804-64: 2
  • debug: 2
• android-em-7-0-x86_64-lite-qr: 4
  • debug: 3
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-07-19&endday=2021-07-25&tree=all

Comment 125

[Intermittent Failures Robot]

5 failures in 3131 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• autoland: 3
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-shippable: 1
  • opt: 1
• android-em-7-0-x86_64-lite-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 2
  • debug: 1
  • debug-isolated-process: 1
• android-hw-p2-8-0-arm7-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-07-26&endday=2021-08-01&tree=all

Comment 126

[Intermittent Failures Robot]

2 failures in 2829 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-08-02&endday=2021-08-08&tree=all

Comment 127

[Intermittent Failures Robot]

1 failures in 3025 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-08-09&endday=2021-08-15&tree=all

Comment 128

[Intermittent Failures Robot]

4 failures in 2899 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 4
  • debug-isolated-process: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-08-16&endday=2021-08-22&tree=all

Comment 129

[Intermittent Failures Robot]

3 failures in 3084 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 2

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-08-30&endday=2021-09-05&tree=all

Comment 130

[Marian-Vasile Laza]

this is currently perma-failing on android-em-7-0-x86_64-lite-qr debug platform when we re-trigger or backfill, is there something we can do about it?

Comment 131

[Marian-Vasile Laza]

Aryx: re https://bugzilla.mozilla.org/show_bug.cgi?id=1581345 - yes, it's permanent and only goes away if a new push schedules tests which actually run on that platform

Comment 132

[Intermittent Failures Robot]

40 failures in 769 pushes (0.052 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 40

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 36
  • debug: 35
  • opt: 1
• android-em-7-0-x86_64-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-09-09&endday=2021-09-09&tree=all

Comment 133

[Geoff Brown [:gbrown]]

(In reply to Intermittent Failures Robot from comment #132)

40 failures in 769 pushes (0.052 failures/push) were associated with this bug yesterday.

Most are android/debug mochitest-plain. I see

MOZHARNESS_TEST_PATHS': '{"mochitest-plain": ["browser/extensions/formautofill/test/mochitest/mochitest.ini"]}'

and

[task 2021-09-09T17:54:18.865Z] 17:54:18     INFO -  No tests were found for flavor 'plain' and the following manifest filters:
[task 2021-09-09T17:54:18.866Z] 17:54:18     INFO -  skip_if, run_if, fail_if, subsuite(name=None), pathprefix(['browser/extensions/formautofill/test/mochitest/mochitest.ini'])

In general, tests under browser/ are not run on android.

:ahal - Remind/inform me, is there a simple way of avoiding this?

Comment 134

[Andrew Halberstadt [:ahal]]

My hunch is that when we package tests we perform a DIRS traversal of the moz.build files. On Android, this means the moz.build files under /browser never get processed and it's like those tests never existed in the first place. But in the decision task we don't have a build config, so we do a file-system traversal of moz.build files instead. This means we process all moz.build files no matter what and those tests get picked up. Since they don't bother skipping on android, the test selection algorithm thinks they are fair game to assign to android tasks.

I'm not sure the best way to solve this, but here are a few ideas:

1. Add skip-if = os == "android" to the default section of every manifest under /browser. This isn't ideal as we can hit the same issue the other way around as well (plus there are other "apps" other than browser and mobile/android). Also we'd need to remember to add this for new manifests. But on the plus side, it is easy and can make the problem here much better in the short term.

2. Hard-code some logic to deal with this in the test selection algorithm. I think given a task we'd need to guess what the MOZ_BUILD_APP is, determine what their locations in-tree are (or hardcode them), and filter out any manifests that start with that prefix. The risk here is that a manifest might be included across file system boundaries. E.g mobile/android/moz.build could theoretically define a MOCHITEST_BROWSER_CHROME_MANIFESTS variable that points to a manifest in browser. If we have cases like that, we could lose test coverage and not notice.. so imo this is too risky.

3. Implement a sort of hybrid file-system + MOZ_BUILD_APP traversal of moz.build files. This would likely be the most accurate way of solving the issue but we'd need to resolve tests once per MOZ_BUILD_APP, so it would cause a significant perf regression to the decision task. Plus I'm not sure how possible this is or not.

4. Start doing a file-system traversal for test packaging as well. This means each test package would contain all tests (so package size will increase). This is basically the same as #1 except that we would enforce that all manifests add appropriate skip-if keys to their DEFAULT sections.

5. Implement some way for manifestparser to pick up default values from the file system (e.g, browser/browser.ini.defaults). This way we could define default skip-ifs for all manifests under a specific directory.. But given that A) manifestparser doesn't currently walk the file system and B) I'd like us to move off of manifestparser at some point, this isn't my preferred solution either.

Honestly after writing all this out, I'm kind of leaning towards #1.

Comment 135

[Andrew Halberstadt [:ahal]]

One way to mitigate the risk of #2 is to make sure we don't do this filtering on backstop pushes. That way we would guarantee no loss of coverage, though if cross-boundary tests existed, they would only run every 20th push on autoland.

Comment 136

[Geoff Brown [:gbrown]]

Now that's a full answer - thanks so much!

I'll have a look at #1.

Comment 137

[Intermittent Failures Robot]

61 failures in 3266 pushes (0.019 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 1
• autoland: 60

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 11
  • debug: 2
  • opt: 8
  • debug-isolated-process: 1
• android-em-7-0-x86_64-lite-qr: 48
  • debug: 47
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-09-06&endday=2021-09-12&tree=all

Comment 138

[Intermittent Failures Robot]

8 failures in 3375 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8

Platform and build breakdown:

• android-em-7-0-x86_64-lite-qr: 3
  • debug: 3
• android-em-7-0-x86_64-qr: 4
  • opt: 3
  • debug: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-09-13&endday=2021-09-19&tree=all

Comment 139

[Intermittent Failures Robot]

2 failures in 2873 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-shippable-qr: 1
  • opt: 1
• android-hw-p2-8-0-arm7-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-09-20&endday=2021-09-26&tree=all

Comment 140

[Intermittent Failures Robot]

1 failures in 3626 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-10-04&endday=2021-10-10&tree=all

Comment 141

[Intermittent Failures Robot]

4 failures in 3315 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 2

Platform and build breakdown:

• android-hw-p2-8-0-arm7-shippable-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2
• android-em-7-0-x86_64-lite-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-10-11&endday=2021-10-17&tree=all

Comment 142

[Intermittent Failures Robot]

7 failures in 3186 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 6
  • debug-isolated-process: 6
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-10-18&endday=2021-10-24&tree=all

Comment 143

[Intermittent Failures Robot]

1 failures in 2845 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-10-25&endday=2021-10-31&tree=all

Comment 144

[Intermittent Failures Robot]

13 failures in 3030 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• autoland: 7

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 4
  • opt: 4
• windows7-32-qr: 2
  • opt: 2
• android-hw-p2-8-0-android-aarch64-qr: 6
  • opt: 5
  • debug: 1
• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-11-01&endday=2021-11-07&tree=all

Comment 145

[Intermittent Failures Robot]

5 failures in 2723 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-beta: 2

Platform and build breakdown:

• windows7-32-qr: 2
  • opt: 2
• android-em-7-0-x86_64-qr: 1
  • opt: 1
• windows7-32-shippable-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-11-08&endday=2021-11-14&tree=all

Comment 146

[Intermittent Failures Robot]

8 failures in 3259 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 7
  • debug-isolated-process: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-11-15&endday=2021-11-21&tree=all

Comment 147

[Intermittent Failures Robot]

6 failures in 2819 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 2

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 2
  • debug: 1
  • opt: 1
• android-em-7-0-x86_64-lite-qr: 1
  • opt: 1
• android-hw-p2-8-0-arm7-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-11-22&endday=2021-11-28&tree=all

Comment 148

[Intermittent Failures Robot]

2 failures in 3170 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-11-29&endday=2021-12-05&tree=all

Comment 149

[Intermittent Failures Robot]

4 failures in 3319 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 3
  • opt: 2
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-12-06&endday=2021-12-12&tree=all

Comment 150

[Intermittent Failures Robot]

5 failures in 3263 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 4

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• android-hw-p2-8-0-android-aarch64-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-12-13&endday=2021-12-19&tree=all

Comment 151

[Intermittent Failures Robot]

1 failures in 2286 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2021-12-20&endday=2021-12-26&tree=all

Comment 152

[Intermittent Failures Robot]

6 failures in 2801 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6

Platform and build breakdown:

• linux1804-64-qr: 3
  • debug: 3
• android-hw-p2-8-0-android-aarch64-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-01-03&endday=2022-01-09&tree=all

Comment 153

[Intermittent Failures Robot]

1 failures in 3060 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-01-10&endday=2022-01-16&tree=all

Comment 154

[Intermittent Failures Robot]

3 failures in 3445 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• macosx1015-64-devedition-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-01-24&endday=2022-01-30&tree=all

Comment 155

[Intermittent Failures Robot]

2 failures in 3165 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-01-31&endday=2022-02-06&tree=all

Comment 156

[Intermittent Failures Robot]

5 failures in 3711 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 1

Platform and build breakdown:

• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1
• linux1804-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-02-07&endday=2022-02-13&tree=all

Comment 157

[Intermittent Failures Robot]

3 failures in 3281 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 2
  • opt: 1
  • debug: 1
• android-hw-p2-8-0-arm7-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-02-14&endday=2022-02-20&tree=all

Comment 158

[Intermittent Failures Robot]

3 failures in 2992 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform and build breakdown:

• linux1804-64-ccov-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-02-21&endday=2022-02-27&tree=all

Comment 159

[Intermittent Failures Robot]

1 failures in 3063 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-02-28&endday=2022-03-06&tree=all

Comment 160

[Intermittent Failures Robot]

1 failures in 2837 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-03-07&endday=2022-03-13&tree=all

Comment 161

[Intermittent Failures Robot]

1 failures in 3304 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• android-hw-p2-8-0-arm7-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-03-14&endday=2022-03-20&tree=all

Comment 162

[Intermittent Failures Robot]

2 failures in 3341 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-03-21&endday=2022-03-27&tree=all

Comment 163

[Intermittent Failures Robot]

6 failures in 3323 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 4

Platform and build breakdown:

• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 2
  • debug-isolated-process: 2
• windows10-64-2004-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-03-28&endday=2022-04-03&tree=all

Comment 164

[Cristian Tuns]

This is perma failing after this push(Bug 1751492). Can you please take a look?
Failure log: https://treeherder.mozilla.org/logviewer?job_id=373533786&repo=autoland

Comment 165

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The usual situation where all tests are skipped.

Comment 166

[Joel Maher ( :jmaher ) (UTC -8)]

the root cause here for the increase is that we selectively schedule tests in the CI by manifest. This failure that we are seeing is a variant where for a given manifest ALL the tests fail for a specific variant. As the variant is not near shipping, I thing a good solution would be to only run tests on default variant (have an allowlist).
We have had 38 failures since Jan 1

• 13 of those are socketprocess_networking variant for xpcshell (included in recent spike this week)
• 6 of those are test-verify (android debug-isolated-process)
• 10 are jittest on geckoview where there was an issue with adb+harness and we didn't report the results - i.e. a timeout despite tests running
• the other 9 are bad commits, crashes, or other device/harness errors

Another hacky solution would be to always have a dummy test that will pass (basically test_pass.js which is an ok(true)), then it doesn't matter what is skipped.

I would prefer the option for #1 and it would be 2 actions:

1. change the smart scheduling to not run variants, prefer to add a boolean field shippable to variants.yml and have smart scheduling consume that (even from tip of m-c)
2. do not run test-verify on debug-isolated-process geckoview builds.

:marco, do you have thoughts on adjusting the scheduling algorithm?

Comment 167

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1581345 - add socketprocess_networking to the taskcluster chunking code to reduce zero test chunks. r=ahal!

Comment 168

[Marco Castelluccio [:marco]]

We talked about this off Bugzilla, and I think no changes to the smart scheduling algorithm will be needed after the patch Joel attached. Feel free to re-add needinfo back if I was mistaken.

Comment 169

[Intermittent Failures Robot]

40 failures in 2922 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 38
• mozilla-central: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows10-64-2004-qr: 25
  • debug: 16
  • opt: 9
• windows10-32-2004-qr: 4
  • debug: 4
• android-hw-p2-8-0-arm7-qr: 1
  • debug: 1
• windows7-32-qr: 3
  • opt: 3
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1
• macosx1015-64-qr: 5
  • opt: 4
  • debug: 1
• windows10-64-2004-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-04-04&endday=2022-04-10&tree=all

Comment 170

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/859502846859
add socketprocess_networking to the taskcluster chunking code to reduce zero test chunks. r=ahal

Comment 171

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/859502846859

Comment 172

[Norisz Fay [:noriszfay]]

Hello Joel, this appeared as a TV on autoland here . Could you take a look at it please? Thank you!

Comment 173

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Let's track the remaining issues in bug 1764722.

Comment 174

[Intermittent Failures Robot]

24 failures in 2948 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #44 most frequent failure this week.

Repository breakdown:

• autoland: 23
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-qr: 10
  • opt: 6
  • debug: 4
• macosx1015-64-qr: 4
  • opt: 4
• windows10-32-2004-qr: 7
  • debug: 7
• windows7-32-qr: 1
  • opt: 1
• android-hw-p2-8-0-android-aarch64-qr: 1
  • debug: 1
• android-em-7-0-x86_64-qr: 1
  • debug-isolated-process: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1581345&startday=2022-04-11&endday=2022-04-17&tree=all