1686308 - Intermittent The mochitest suite: mochitest-browser-chrome ran with return status: FAILURE

Status: RESOLVED FIXED

(Core :: Widget, defect)

Description

[Treeherder Bug Filer]

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=326466746&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/IlrMtZ4FSEO_waD48OxajA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-01-12T17:07:24.974Z] 17:07:24     INFO - TEST-INFO | widget/tests/browser/browser_test_clipboardcache.js | This test created 1 hidden window(s)
[task 2021-01-12T17:07:24.975Z] 17:07:24     INFO - TEST-INFO | widget/tests/browser/browser_test_clipboardcache.js | This test created 1 hidden docshell(s)
[task 2021-01-12T17:07:24.975Z] 17:07:24     INFO - runtests.py | Application ran for: 0:00:32.698000
[task 2021-01-12T17:07:24.976Z] 17:07:24     INFO - zombiecheck | Reading PID log: c:\users\task_1610466857\appdata\local\temp\tmpmnziqvpidlog
[task 2021-01-12T17:07:24.977Z] 17:07:24     INFO - ==> process 2232 launched child process 8676 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.0.1420584867\509873736" -parentBuildID 20210112155511 -prefsHandle 2192 -prefMapHandle 2180 -prefsLen 1 -prefMapSize 243258 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 1876 gpu)
[task 2021-01-12T17:07:24.977Z] 17:07:24     INFO - ==> process 2232 launched child process 7704 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.6.1582453199\1881930683" -childID 1 -isForBrowser -prefsHandle 2888 -prefMapHandle 2884 -prefsLen 1684 -prefMapSize 243258 -parentBuildID 20210112155511 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 2912 tab)
[task 2021-01-12T17:07:24.977Z] 17:07:24     INFO - ==> process 2232 launched child process 10684 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.13.821065802\1596333860" -childID 2 -isForBrowser -prefsHandle 3252 -prefMapHandle 3248 -prefsLen 1790 -prefMapSize 243258 -parentBuildID 20210112155511 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 3224 tab)
[task 2021-01-12T17:07:24.977Z] 17:07:24     INFO - ==> process 2232 launched child process 7040 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.20.1786689089\1598782762" -childID 3 -isForBrowser -prefsHandle 3184 -prefMapHandle 3132 -prefsLen 1830 -prefMapSize 243258 -parentBuildID 20210112155511 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 3388 tab)
[task 2021-01-12T17:07:24.977Z] 17:07:24     INFO - ==> process 2232 launched child process 7224 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.27.1889178576\322481101" -parentBuildID 20210112155511 -prefsHandle 3760 -prefMapHandle 3752 -prefsLen 2820 -prefMapSize 243258 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 3884 rdd)
[task 2021-01-12T17:07:24.977Z] 17:07:24     INFO - ==> process 2232 launched child process 4204 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.31.578326464\1708485305" -childID 4 -isForBrowser -prefsHandle 4408 -prefMapHandle 4396 -prefsLen 10279 -prefMapSize 243258 -parentBuildID 20210112155511 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 4388 tab)
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - ==> process 2232 launched child process 10072 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.38.794264668\1924073260" -childID 5 -isForBrowser -prefsHandle 4772 -prefMapHandle 4768 -prefsLen 11369 -prefMapSize 243258 -parentBuildID 20210112155511 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 4692 tab)
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - ==> process 2232 launched child process 9652 ("Z:\task_1610466857\build\application\firefox\firefox.exe" -contentproc --channel="2232.45.284909113\851592949" -childID 6 -isForBrowser -prefsHandle 5140 -prefMapHandle 5136 -prefsLen 11638 -prefMapSize 243258 -parentBuildID 20210112155511 -appdir "Z:\task_1610466857\build\application\firefox\browser" - 2232 "\\.\pipe\gecko-crash-server-pipe.2232" 5112 tab)
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 7040
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 8676
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 4204
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 10072
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 7224
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 9652
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 7704
[task 2021-01-12T17:07:24.978Z] 17:07:24     INFO - zombiecheck | Checking for orphan process with PID: 10684
[task 2021-01-12T17:07:24.979Z] 17:07:24     INFO - Stopping web server
[task 2021-01-12T17:07:24.989Z] 17:07:24     INFO - Stopping web socket server
[task 2021-01-12T17:07:25.029Z] 17:07:25     INFO - Stopping ssltunnel
[task 2021-01-12T17:07:25.047Z] 17:07:25     INFO - leakcheck | Processing log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks.log
[task 2021-01-12T17:07:25.047Z] 17:07:25     INFO - TEST-INFO | leakcheck | default process: leak threshold set at 0 bytes
[task 2021-01-12T17:07:25.047Z] 17:07:25     INFO - TEST-INFO | leakcheck | forkserver process: leak threshold set at 0 bytes
[task 2021-01-12T17:07:25.047Z] 17:07:25     INFO - TEST-INFO | leakcheck | gmplugin process: leak threshold set at 20000 bytes
[task 2021-01-12T17:07:25.047Z] 17:07:25     INFO - TEST-INFO | leakcheck | gpu process: leak threshold set at 0 bytes
[task 2021-01-12T17:07:25.047Z] 17:07:25     INFO - TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - TEST-INFO | leakcheck | rdd process: leak threshold set at 400 bytes
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - TEST-INFO | leakcheck | socket process: leak threshold set at 0 bytes
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - TEST-INFO | leakcheck | tab process: leak threshold set at 1000 bytes
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - TEST-INFO | leakcheck | vr process: leak threshold set at 0 bytes
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks.log
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, default process 2232
[task 2021-01-12T17:07:25.048Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.049Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.049Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.049Z] 17:07:25     INFO -    0 |TOTAL                                 |       40        0| 2510421        0|
[task 2021-01-12T17:07:25.054Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.054Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 1927 entries
[task 2021-01-12T17:07:25.054Z] 17:07:25     INFO - TEST-PASS | leakcheck | default no leaks detected!
[task 2021-01-12T17:07:25.054Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_gpu_pid8676.log
[task 2021-01-12T17:07:25.054Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.054Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, gpu process 8676
[task 2021-01-12T17:07:25.054Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.055Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.055Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.056Z] 17:07:25     INFO -    0 |TOTAL                                 |       23        0|  115773        0|
[task 2021-01-12T17:07:25.056Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.057Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 294 entries
[task 2021-01-12T17:07:25.057Z] 17:07:25     INFO - TEST-PASS | leakcheck | gpu no leaks detected!
[task 2021-01-12T17:07:25.058Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_rdd_pid7224.log
[task 2021-01-12T17:07:25.058Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.059Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, rdd process 7224
[task 2021-01-12T17:07:25.059Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO -    0 |TOTAL                                 |       29        0|    4963        0|
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 176 entries
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO - TEST-PASS | leakcheck | rdd no leaks detected!
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_tab_pid10072.log
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.060Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 10072
[task 2021-01-12T17:07:25.061Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.061Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.062Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.062Z] 17:07:25     INFO -    0 |TOTAL                                 |       37        0|   60369        0|
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 884 entries
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_tab_pid10684.log
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 10684
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.063Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.064Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.064Z] 17:07:25     INFO -    0 |TOTAL                                 |       36        0|  159082        0|
[task 2021-01-12T17:07:25.066Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.066Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 1048 entries
[task 2021-01-12T17:07:25.067Z] 17:07:25     INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2021-01-12T17:07:25.068Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_tab_pid4204.log
[task 2021-01-12T17:07:25.068Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.068Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 4204
[task 2021-01-12T17:07:25.068Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.068Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.068Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.068Z] 17:07:25     INFO -    0 |TOTAL                                 |       31        0|  290396        0|
[task 2021-01-12T17:07:25.073Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.073Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 1002 entries
[task 2021-01-12T17:07:25.073Z] 17:07:25     INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2021-01-12T17:07:25.073Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_tab_pid7040.log
[task 2021-01-12T17:07:25.073Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.073Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 7040
[task 2021-01-12T17:07:25.073Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.074Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.074Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.074Z] 17:07:25     INFO -    0 |TOTAL                                 |       41        0|   92829        0|
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 853 entries
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_tab_pid7704.log
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 7704
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.076Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.077Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.077Z] 17:07:25     INFO -    0 |TOTAL                                 |       41        0|   79076        0|
[task 2021-01-12T17:07:25.081Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.081Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 1146 entries
[task 2021-01-12T17:07:25.081Z] 17:07:25     INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2021-01-12T17:07:25.081Z] 17:07:25     INFO - leakcheck | Processing leak log file c:\users\task_1610466857\appdata\local\temp\tmpfekmcf.mozrunner\runtests_leaks_tab_pid9652.log
[task 2021-01-12T17:07:25.081Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.081Z] 17:07:25     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 9652
[task 2021-01-12T17:07:25.082Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.082Z] 17:07:25     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-01-12T17:07:25.082Z] 17:07:25     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2021-01-12T17:07:25.083Z] 17:07:25     INFO -    0 |TOTAL                                 |       37        0|   14446        0|
[task 2021-01-12T17:07:25.083Z] 17:07:25     INFO - 
[task 2021-01-12T17:07:25.083Z] 17:07:25     INFO - nsTraceRefcnt::DumpStatistics: 417 entries
[task 2021-01-12T17:07:25.083Z] 17:07:25     INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2021-01-12T17:07:25.083Z] 17:07:25     INFO - runtests.py | Running tests: end.
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO - Buffered messages finished
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO -  TEST-INFO | checking window state
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO -  Browser Chrome Test Summary
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO -  	Passed: 31311
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO -  	Failed: 348
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO -  	Todo: 20
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO -  	Mode: e10s
[task 2021-01-12T17:07:25.133Z] 17:07:25     INFO -  *** End BrowserChrome Test Results ***
[task 2021-01-12T17:07:25.134Z] 17:07:25     INFO - Buffered messages finished
[task 2021-01-12T17:07:25.134Z] 17:07:25     INFO - SUITE-END | took 2729s
[task 2021-01-12T17:07:25.223Z] 17:07:25    ERROR - Return code: 1
[task 2021-01-12T17:07:25.224Z] 17:07:25     INFO - TinderboxPrint: mochitest-mochitest-browser-chrome<br/>503/0/0
[task 2021-01-12T17:07:25.224Z] 17:07:25    ERROR - # TBPL FAILURE #
[task 2021-01-12T17:07:25.224Z] 17:07:25  WARNING - setting return code to 2
[task 2021-01-12T17:07:25.224Z] 17:07:25    ERROR - The mochitest suite: mochitest-browser-chrome ran with return status: FAILURE```

Comment 1

[Intermittent Failures Robot]

3 failures in 3810 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform breakdown:

• windows10-64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-01-11&endday=2021-01-17&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 3488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1014-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-01-18&endday=2021-01-24&tree=all

Comment 3

[Intermittent Failures Robot]

2 failures in 3891 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• windows10-64: 1
  • debug: 1
• windows7-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-01-25&endday=2021-01-31&tree=all

Comment 4

[Intermittent Failures Robot]

4 failures in 4322 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform and build breakdown:

• windows7-32: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 3737 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows7-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-02-15&endday=2021-02-21&tree=all

Comment 6

[Intermittent Failures Robot]

4 failures in 4327 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• autoland: 1

Platform and build breakdown:

• linux1804-64: 3
  • debug: 3
• windows10-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-02-22&endday=2021-02-28&tree=all

Comment 7

[Intermittent Failures Robot]

2 failures in 3683 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• linux1804-64-asan: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 8

[Intermittent Failures Robot]

4 failures in 4622 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• try: 2

Platform and build breakdown:

• windows7-32: 1
  • debug: 1
• windows10-64-qr: 1
  • debug: 1
• linux1804-64: 2
  • opt: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-03-08&endday=2021-03-14&tree=all

Comment 9

[Intermittent Failures Robot]

3 failures in 4257 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• windows7-32: 2
  • debug: 2
• windows10-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 10

[Intermittent Failures Robot]

4 failures in 3967 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64: 1
  • opt: 1
• windows7-32: 2
  • debug: 2
• windows10-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-03-22&endday=2021-03-28&tree=all

Comment 11

[Intermittent Failures Robot]

8 failures in 3125 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• try: 1
• mozilla-central: 2

Platform and build breakdown:

• windows7-32: 1
  • debug: 1
• windows10-64: 1
  • debug: 1
• linux1804-64-asan: 3
  • opt: 3
• linux1804-64: 1
  • opt: 1
• linux1804-64-tsan: 1
  • opt: 1
• windows10-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-03-29&endday=2021-04-04&tree=all

Comment 12

[Intermittent Failures Robot]

16 failures in 3854 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64: 6
  • opt: 6
• linux1804-64-asan: 6
  • opt: 6
• windows10-64: 1
  • debug: 1
• windows7-32: 2
  • debug: 2
• linux1804-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-04-05&endday=2021-04-11&tree=all

Comment 13

[Intermittent Failures Robot]

17 failures in 4255 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 16
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64: 4
  • opt: 4
• windows10-64: 1
  • debug: 1
• linux1804-64-asan: 11
  • opt: 11
• linux1804-64-tsan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-04-12&endday=2021-04-18&tree=all

Comment 14

[Intermittent Failures Robot]

17 failures in 3224 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-beta: 2
• mozilla-central: 6

Platform and build breakdown:

• linux1804-64: 2
  • opt: 2
• linux1804-64-asan: 7
  • opt: 7
• linux1804-64-shippable: 3
  • opt: 3
• windows7-32: 1
  • debug: 1
• windows10-64-qr: 2
  • debug: 2
• windows10-64: 2
  • asan: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-04-19&endday=2021-04-25&tree=all

Comment 15

[Intermittent Failures Robot]

5 failures in 3706 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-beta: 1
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan: 2
  • opt: 2
• linux1804-64-shippable: 1
  • opt: 1
• macosx1014-64-qr: 1
  • debug: 1
• windows7-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-04-26&endday=2021-05-02&tree=all

Comment 16

[Intermittent Failures Robot]

7 failures in 3279 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 4
• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-shippable: 1
  • opt: 1
• windows10-64: 2
  • debug: 1
  • asan: 1
• linux1804-64-asan: 2
  • opt: 2
• windows10-64-qr: 1
  • debug: 1
• windows7-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-05-03&endday=2021-05-09&tree=all

Comment 17

[Intermittent Failures Robot]

4 failures in 3061 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-central: 1
• autoland: 2

Platform and build breakdown:

• windows10-64-qr: 1
  • debug: 1
• windows7-32: 2
  • debug: 2
• windows10-64: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-05-17&endday=2021-05-23&tree=all

Comment 18

[Intermittent Failures Robot]

10 failures in 2908 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 6
• try: 1
• mozilla-release: 1

Platform and build breakdown:

• windows10-64-qr: 2
  • debug: 2
• linux1804-64-asan: 2
  • opt: 2
• linux1804-64-asan-qr: 2
  • opt: 2
• linux1804-64: 1
  • debug: 1
• linux1804-64-tsan: 1
  • opt: 1
• windows10-32: 1
  • debug: 1
• linux1804-64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-05-24&endday=2021-05-30&tree=all

Comment 19

[Intermittent Failures Robot]

10 failures in 3065 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3
• autoland: 7

Platform and build breakdown:

• linux1804-64: 3
  • opt: 3
• windows10-32: 3
  • debug: 3
• linux1804-64-tsan: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-05-31&endday=2021-06-06&tree=all

Comment 20

[Intermittent Failures Robot]

19 failures in 3502 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 5
• autoland: 10
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64: 4
  • opt: 4
• windows10-64: 1
  • debug: 1
• linux1804-64-tsan: 2
  • opt: 2
• windows10-64-qr: 4
  • debug: 4
• macosx1015-64-qr: 1
  • debug: 1
• linux1804-64-asan: 6
  • opt: 6
• windows10-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-06-07&endday=2021-06-13&tree=all

Comment 21

[Intermittent Failures Robot]

11 failures in 2888 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 2

Platform and build breakdown:

• windows10-64-qr: 5
  • debug: 5
• linux1804-64-tsan: 3
  • opt: 3
• linux1804-64-asan: 2
  • opt: 2
• windows10-32: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-06-14&endday=2021-06-20&tree=all

Comment 22

[Intermittent Failures Robot]

18 failures in 3115 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• mozilla-central: 3

Platform and build breakdown:

• windows7-32-qr: 12
  • opt: 12
• windows10-64: 2
  • debug: 2
• windows10-64-qr: 2
  • debug: 2
• linux1804-64-asan: 1
  • opt: 1
• windows10-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-06-21&endday=2021-06-27&tree=all

Comment 23

[Intermittent Failures Robot]

11 failures in 2387 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11

Platform and build breakdown:

• windows10-32-qr: 2
  • debug: 2
• windows10-64-qr: 2
  • debug: 2
• linux1804-64-asan: 1
  • opt: 1
• windows7-32-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-07-05&endday=2021-07-11&tree=all

Comment 24

[Intermittent Failures Robot]

6 failures in 2224 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• autoland: 4

Platform and build breakdown:

• linux1804-64: 2
  • opt: 2
• linux1804-64-asan: 1
  • opt: 1
• windows10-64-qr: 2
  • debug: 2
• windows10-32-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-07-12&endday=2021-07-18&tree=all

Comment 25

[Intermittent Failures Robot]

8 failures in 2940 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• autoland: 6

Platform and build breakdown:

• windows10-64-qr: 4
  • debug: 4
• windows7-32-qr: 3
  • opt: 3
• linux1804-64-tsan: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-07-19&endday=2021-07-25&tree=all

Comment 26

[Intermittent Failures Robot]

18 failures in 591 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18

Platform and build breakdown:

• linux1804-64-asan-qr: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-07-28&endday=2021-07-28&tree=all

Comment 27

[Intermittent Failures Robot]

15 failures in 630 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15

Platform and build breakdown:

• linux1804-64-asan-qr: 13
  • opt: 13
• windows7-32-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-07-29&endday=2021-07-29&tree=all

Comment 28

[Intermittent Failures Robot]

22 failures in 622 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 20
  • opt: 20
• windows10-64-qr: 1
  • debug: 1
• windows7-32-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-07-30&endday=2021-07-30&tree=all

Comment 29

[Intermittent Failures Robot]

68 failures in 3131 pushes (0.022 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 67
• mozilla-central: 1

Platform and build breakdown:

• windows10-64-qr: 4
  • debug: 4
• windows7-32-qr: 8
  • opt: 8
• linux1804-64-asan-qr: 56
  • opt: 56

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-07-26&endday=2021-08-01&tree=all

Comment 30

[Intermittent Failures Robot]

19 failures in 460 pushes (0.041 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18
• mozilla-esr91: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 18
  • opt: 18
• windows10-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-02&endday=2021-08-02&tree=all

Comment 31

[Intermittent Failures Robot]

18 failures in 561 pushes (0.032 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18

Platform and build breakdown:

• linux1804-64-asan-qr: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-03&endday=2021-08-03&tree=all

Comment 32

[Intermittent Failures Robot]

23 failures in 580 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23

Platform and build breakdown:

• windows7-32-qr: 3
  • opt: 3
• linux1804-64-asan-qr: 19
  • opt: 19
• windows10-32-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-04&endday=2021-08-04&tree=all

Comment 33

[Intermittent Failures Robot]

21 failures in 625 pushes (0.034 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 20
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 19
  • opt: 19
• linux1804-64-qr: 1
  • opt: 1
• windows10-32-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-05&endday=2021-08-05&tree=all

Comment 34

[Alexandru Michis [:malexandru]]

This has increased in frequency since the 28th of July, and it now mostly appears on linux1804-64-asan-qr.
Joel, has there been any work done on optimized pushes or worker changes that might've caused the above?

Comment 35

[Intermittent Failures Robot]

19 failures in 466 pushes (0.041 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 17
• try: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 12
  • opt: 12
• windows7-32-qr: 2
  • opt: 2
• windows10-32-qr: 1
  • debug: 1
• linux1804-64-qr: 2
  • opt: 2
• windows10-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-06&endday=2021-08-06&tree=all

Comment 36

[Cosmin Sabou [:CosminS]]

The failures here are actually a byproduct of the manifest related changes. You'll find that on these classified fails there are actually no tests that were run, all of them were skipped. I suggest we star these moving forward as expected fail after we check the log. Aryx, any suggestions?

Comment 37

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Yes, this sounds like the best option under the current conditions.

Comment 38

[Intermittent Failures Robot]

107 failures in 2829 pushes (0.038 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 103
• mozilla-esr91: 1
• try: 3

Platform and build breakdown:

• linux1804-64-asan-qr: 91
  • opt: 91
• windows10-64-qr: 3
  • debug: 3
• windows7-32-qr: 7
  • opt: 7
• windows10-32-qr: 3
  • debug: 3
• linux1804-64-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-02&endday=2021-08-08&tree=all

Comment 39

[Intermittent Failures Robot]

29 failures in 633 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• try: 4

Platform and build breakdown:

• windows7-32-qr: 2
  • opt: 2
• linux1804-64-asan-qr: 25
  • opt: 25
• linux1804-64-tsan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-10&endday=2021-08-10&tree=all

Comment 40

[Joel Maher ( :jmaher ) (UTC -8)]

we keep running into this issue- and possibly we need to just accept a test run with no tests to run.

Comment 41

[Joel Maher ( :jmaher ) (UTC -8)]

:ahal, this area of code could be changed to allow zero results:
https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#2997

currently with zero tests/results we get a failure and an orange job- hence this bug! I have seen questions around this topic recently.

Do you think we should keep failing and solve this specific bug another way, should we remove the clause to return 1, or something else?

Comment 42

[Andrew Halberstadt [:ahal]]

I'm not entirely sure what the best course of action is here..

On one hand, we do keep hitting these intermittents and they're annoying.

On the other hand, usually this means there is a missing value in the mozinfo guess logic. So hiding these failures means we won't ever catch these bugs anymore. Furthermore, I'd be worried we make some change in the harness that causes tests to stop running and we'd never notice because the tasks in CI are all green.

I think I'd vote we leave it as is and focus on making the mozinfo guess better. Here's how I go about fixing these:

1. This almost always happens when a task runs only a single manifest, and that manifest has a skip-if in the DEFAULT section. In this case the manifest in question is browser/components/extensions/test/browser/browser.ini. And the skip-if in question is here (found by comparing the failing platform to the annotation).

2. Next I compare that skip-if to the mozinfo guess logic and try to figure out what is missing / inaccurate. In this case it is os_version.

3. Finally I add pprint(task, indent=2) (behind an if statement that catches roughly the tasks I'm interested in) to the guess_mozinfo_from_task function and try to figure out how I can guess the appropriate value. In this case it looks like we can derive the os_version from the test-platform attribute.

I'll put up a patch here shortly.

Comment 43

[Andrew Halberstadt [:ahal]]

I'll also add that as we fix more of these, I expect the frequency of these types of bugs to go down dramatically. Though we'll need to remember to keep guess_mozinfo_from_task up to date when adding new platforms and the like.

Comment 44

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1686308 - [taskgraph] Add 'os_version' to the guess_mozinfo_from_task function, r?jmaher!

Comment 45

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/275924e3f7fb
[taskgraph] Add 'os_version' to the guess_mozinfo_from_task function, r=jmaher

Comment 46

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/275924e3f7fb

Comment 47

[Alexandru Michis [:malexandru]]

This is still appearing, but with tests actually running, is it ok to reopen this bug?

Comment 48

[Intermittent Failures Robot]

51 failures in 3025 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 45
• try: 4
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• windows7-32-qr: 2
  • opt: 2
• linux1804-64-asan-qr: 43
  • opt: 43
• linux1804-64-tsan-qr: 2
  • opt: 2
• windows10-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-09&endday=2021-08-15&tree=all

Comment 49

[Andrew Halberstadt [:ahal]]

I think that's a separate issue, probably stemming from the Failed: 1623.

Looks like a whole lot of assertions may have failed during the test run? Though if that's the case I'm surprised there's no better error messages being printed.

Comment 50

[Alexandru Michis [:malexandru]]

This one has the same test-skip behavior as the previous failures.

Comment 51

[Cosmin Sabou [:CosminS]]

Hi, so it seems that there are two different issues here:

• first one where tests are ran but there's nothing relevant that I found in the log - filed Bug 1726151
• second one where all the tests are skipped in the manifest and nothing ran. log - filed Bug 1726603 (still hitting this pretty frequent on trees)

Ahal, having a second look at these kind of failure would be greatly appreciated. Thank you.

Comment 52

[Andrew Halberstadt [:ahal]]

Thanks for filing new bugs! Much appreciated over re-using old ones :)

1. I'll add a comment here, not entirely sure what's going on
2. Posted a patch

Comment 53

[Intermittent Failures Robot]

13 failures in 2899 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13

Platform and build breakdown:

• linux1804-64-asan-qr: 12
  • opt: 12
• windows10-32-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2021-08-16&endday=2021-08-22&tree=all

Comment 54

[Intermittent Failures Robot]

1 failures in 2694 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr91: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1686308&startday=2022-05-16&endday=2022-05-22&tree=all