1240672 - Intermittent Linux crashtest TEST-UNEXPECTED-FAIL | Shutdown | application terminated with exit code -11 or test_webvtt_disabled.html | application terminated with exit code -11 after Assertion failure: IsEmpty(), at xpcom/threads/nsEventQueue.cpp:35

Status: RESOLVED DUPLICATE of bug 1240178

(Core :: XPCOM, defect, P3)

Description

[Nigel Babu [:nigelb]]

https://treeherder.mozilla.org/logviewer.html#?job_id=20018826&repo=mozilla-inbound


 07:28:16     INFO -  --DOMWINDOW == 10 (0x7f05d0e72c00) [pid = 1106] [serial = 1636] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/483444.xml]

 07:28:16     INFO -  --DOMWINDOW == 9 (0x7f05d0f62000) [pid = 1106] [serial = 1638] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/485217.xml]

 07:28:16     INFO -  --DOMWINDOW == 8 (0x7f05d2962400) [pid = 1106] [serial = 1641] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/485286.xml]

 07:28:16     INFO -  --DOMWINDOW == 7 (0x7f05d0a77400) [pid = 1106] [serial = 1644] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/527558_1.xml]

 07:28:16     INFO -  --DOMWINDOW == 6 (0x7f05d0b27400) [pid = 1106] [serial = 1647] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/528300.xml]

 07:28:16     INFO -  --DOMWINDOW == 5 (0x7f05d0f42800) [pid = 1106] [serial = 1650] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/528488.xml]

 07:28:16     INFO -  --DOMWINDOW == 4 (0x7f05d2964800) [pid = 1106] [serial = 1653] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/528963.xml]

 07:28:16     INFO -  --DOMWINDOW == 3 (0x7f05d0a86000) [pid = 1106] [serial = 1665] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/667315.xml]

 07:28:16     INFO -  --DOMWINDOW == 2 (0x7f05d0b1c000) [pid = 1106] [serial = 1669] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/1205163.xml]

 07:28:16     INFO -  --DOMWINDOW == 1 (0x7f05d0b3e800) [pid = 1106] [serial = 1775] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/editor/composer/crashtests/428844-1-inner.xhtml]

 07:28:16     INFO -  --DOMWINDOW == 0 (0x7f05d0a76800) [pid = 1106] [serial = 1629] [outer = (nil)] [url = file:///home/worker/workspace/build/tests/reftest/tests/dom/xslt/crashtests/182460-select.xml]

 07:28:16     INFO -  Assertion failure: IsEmpty(), at /home/worker/workspace/build/src/xpcom/threads/nsEventQueue.cpp:35

 07:28:16     INFO -  #01: ??? (/home/worker/workspace/build/application/firefox/libxul.so)

 07:28:16     INFO -  #02: ??? (/home/worker/workspace/build/application/firefox/libxul.so)

 07:28:16     INFO -  #03: ??? (/home/worker/workspace/build/application/firefox/libxul.so)

 07:28:16     INFO -  #04: ??? (/home/worker/workspace/build/application/firefox/libxul.so)

 07:28:16     INFO -  #05: ??? (/home/worker/workspace/build/application/firefox/libnspr4.so)

 07:28:16     INFO -  #06: ??? (/home/worker/workspace/build/application/firefox/libnspr4.so)

 07:28:16     INFO -  #07: ??? (/home/worker/workspace/build/application/firefox/libnspr4.so)

 07:28:16     INFO -  #08: __nptl_deallocate_tsd (/lib/x86_64-linux-gnu/libpthread.so.0)

 07:28:16     INFO -  #09: start_thread (/lib/x86_64-linux-gnu/libpthread.so.0)

 07:28:16     INFO -  #10: clone (/lib/x86_64-linux-gnu/libc.so.6)

 07:28:16  WARNING -  TEST-UNEXPECTED-FAIL | Shutdown | application terminated with exit code -139

 07:28:16     INFO -  TEST-INFO | leakcheck | default process: leak threshold set at 0 bytes

 07:28:16     INFO -  TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes

 07:28:17     INFO -  TEST-INFO | leakcheck | tab process: leak threshold set at 5000 bytes

 07:28:17     INFO -  TEST-INFO | leakcheck | geckomediaplugin process: leak threshold set at 0 bytes

 07:28:17  WARNING -  TEST-UNEXPECTED-FAIL | leakcheck | default process: missing output line for total leaks!

 07:28:17     INFO -  TEST-INFO | leakcheck | missing output line from log file /tmp/tmpDOIz3M.mozrunner/runreftest_leaks.log

 07:28:17     INFO -  == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, plugin process 1439

 07:28:17     INFO -       |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|

 07:28:17     INFO -       |                                      | Per-Inst   Leaked|   Total      Rem|

 07:28:17     INFO -     0 |TOTAL                                 |       52        0|     563        0|

 07:28:17     INFO -  nsTraceRefcnt::DumpStatistics: 28 entries

 07:28:17     INFO -  TEST-PASS | leakcheck | plugin process: no leaks detected!

 07:28:17     INFO -  ==> process 1350 will purposefully crash

 07:28:17     INFO -  TEST-INFO | leakcheck | plugin process: deliberate crash and thus no leak log

 07:28:17     INFO -  == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, plugin process 1356

 07:28:17     INFO -       |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|

 07:28:17     INFO -       |                                      | Per-Inst   Leaked|   Total      Rem|

 07:28:17     INFO -     0 |TOTAL                                 |       47        0|     911        0|

 07:28:17     INFO -  nsTraceRefcnt::DumpStatistics: 30 entries

 07:28:17     INFO -  TEST-PASS | leakcheck | plugin process: no leaks detected!

 07:28:17     INFO -  REFTEST INFO | runreftest.py | Running tests: end.

 07:28:17    ERROR - Return code: 117

 07:28:17     INFO - TinderboxPrint: reftest-crashtest<br/>2966/0/52

 07:28:17  WARNING - # TBPL FAILURE #

 07:28:17  WARNING - setting return code to 2

 07:28:17  WARNING - The reftest suite: crashtest ran with return status: FAILURE

 07:28:17     INFO - Running post-action listener: _package_coverage_data

 07:28:17     INFO - Running post-action listener: _resource_record_post_action

 07:28:17     INFO - Running post-run listener: _resource_record_post_run

 07:28:18     INFO - Total resource usage - Wall time: 1541s; CPU: 96.0%; Read bytes: 171171840; Write bytes: 5605457920; Read time: 59544; Write time: 2430652

 07:28:18     INFO - install - Wall time: 27s; CPU: 100.0%; Read bytes: 0; Write bytes: 70139904; Read time: 0; Write time: 48032

 07:28:18     INFO - run-tests - Wall time: 1514s; CPU: 96.0%; Read bytes: 170598400; Write bytes: 5530320896; Read time: 59328; Write time: 2381032

 07:28:18     INFO - Running post-run listener: _upload_blobber_files

 07:28:18  WARNING - Blob upload gear skipped. Missing cmdline options.

 07:28:18     INFO - Running post-run listener: copy_logs_to_upload_dir

 07:28:18     INFO - Copying logs to upload dir...

 07:28:18     INFO - mkdir: /home/worker/workspace/build/upload/logs

 07:28:18     INFO - Copying logs to upload dir...

 07:28:18  WARNING - returning nonzero exit status 2

Comment 1

[Intermittent Failures Robot]

15 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 5

Platform breakdown:
* linux64: 14
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-01-19&endday=2016-01-19&tree=all

Comment 2

[Benjamin Smedberg]

Some event queue is being destroyed without being properly emptied. From the log it's very difficult to tell which process or thread this is. There are no symbols in the stack trace. Fortunately since this is Linux, an rr recording would help a lot. Does this happen enough that we could isolate it in rr?

Comment 3

[Phil Ringnalda (:philor)]

Yes, you can - the relatively few number of instances starred as this bug just reflect the fact that it's tough to get a summary that will match what treeherder's looking for, since we stick a branch-specific path in assertion failure message, so nobody but Nigel knew he had filed it. If crashtest is too uncooperative about failing, mochitest-4 might work, or the fact that it's also failing in a media/ mochitest might be a clue about which crashtests to run-until-failure.

Comment 4

[Intermittent Failures Robot]

36 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 24
* fx-team: 6
* b2g-inbound: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 28
* linux32: 5
* b2g-emu-ics: 2
* osx-10-6: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-01-21&endday=2016-01-21&tree=all

Comment 5

[Intermittent Failures Robot]

32 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 22
* fx-team: 6
* mozilla-central: 3
* b2g-inbound: 1

Platform breakdown:
* linux64: 23
* osx-10-6: 3
* linux32: 3
* b2g-emu-ics: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-01-22&endday=2016-01-22&tree=all

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

Note: better assertion stack:
 10:14:29     INFO -  --DOMWINDOW == 0 (0x7f3c37c0f000) [pid = 1935] [serial = 1626] [outer = (nil)] [url = file:///builds/slave/test/build/tests/reftest/tests/dom/xslt/crashtests/182460-select.xml]
 10:14:29     INFO -  Assertion failure: IsEmpty(), at /builds/slave/m-in-l64-d-0000000000000000000/build/src/xpcom/threads/nsEventQueue.cpp:35
 10:14:29     INFO -  #01: nsThread::nsChainedEventQueue::~nsChainedEventQueue() [mfbt/RefPtr.h:55]
 10:14:29     INFO -  #02: nsThread::~nsThread() [xpcom/glue/nsTArray.h:2095]
 10:14:29     INFO -  #03: nsThread::~nsThread() [xpcom/threads/nsThread.cpp:508]
 10:14:29     INFO -  #04: nsThread::Release() [xpcom/threads/nsThread.cpp:179]
 10:14:29     INFO -  #05: _PR_DestroyThreadPrivate [nsprpub/pr/src/threads/prtpd.c:237]
 10:14:29     INFO -  #06: _pt_thread_death_internal [nsprpub/pr/src/pthreads/ptthread.c:888]
 10:14:29     INFO -  #07: _pt_thread_death [nsprpub/pr/src/pthreads/ptthread.c:865]
 10:14:29     INFO -  #08: libpthread.so.0 + 0x7c83
 10:14:29     INFO -  #09: libpthread.so.0 + 0x7ea8
 10:14:29     INFO -  #10: libc.so.6 + 0xf42ed

Comment 7

[Intermittent Failures Robot]

21 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* fx-team: 4
* try: 3
* mozilla-central: 3

Platform breakdown:
* linux64: 10
* linux32: 9
* b2g-emu-ics: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-01-23&endday=2016-01-23&tree=all

Comment 8

[Intermittent Failures Robot]

149 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 93
* fx-team: 28
* try: 11
* b2g-inbound: 10
* mozilla-central: 7

Platform breakdown:
* linux64: 105
* linux32: 30
* b2g-emu-ics: 8
* osx-10-6: 5
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-01-18&endday=2016-01-24&tree=all

Comment 9

[Nathan Froyd [:froydnj]]

khuey flipped this back to an NS_ASSERTION.  I'm not sure if that answers the ni?...

Comment 10

[Intermittent Failures Robot]

6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* fx-team: 3
* try: 1
* mozilla-aurora: 1
* b2g-inbound: 1

Platform breakdown:
* linux64: 4
* linux32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-01-25&endday=2016-01-31&tree=all

Comment 11

[Intermittent Failures Robot]

9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 9

Platform breakdown:
* windows7-32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-06-06&endday=2016-06-12&tree=all

Comment 12

[Intermittent Failures Robot]

8 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 8

Platform breakdown:
* windows7-32: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-06-20&endday=2016-06-26&tree=all

Comment 13

[Intermittent Failures Robot]

10 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 10

Platform breakdown:
* windows7-32: 8
* windows8-64: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-06-27&endday=2016-07-03&tree=all

Comment 14

[Intermittent Failures Robot]

6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 6

Platform breakdown:
* windows7-32: 5
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-07-04&endday=2016-07-10&tree=all

Comment 15

[Intermittent Failures Robot]

5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 5

Platform breakdown:
* windows7-32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1240672&startday=2016-07-11&endday=2016-07-17&tree=all

Comment 16

[BMO Automation]

Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.