Intermittent test_speech_queue.html | This test left crash dumps behind, but we weren't expecting it to! | Main app process exited normally | application crashed [@ MessageLoop::DeletePendingTasks()] | Assertion failure: work_queue_.empty(), at /builds/

RESOLVED WORKSFORME

Status

()

P3
critical
RESOLVED WORKSFORME
4 years ago
7 months ago

People

(Reporter: cbook, Assigned: billm)

Tracking

({assertion, crash, intermittent-failure})

Trunk
x86
Mac OS X
assertion, crash, intermittent-failure
Points:
---

Firefox Tracking Flags

(e10s-, firefox36 wontfix, firefox37 affected, firefox38 affected, firefox39 affected, firefox-esr31 unaffected, b2g-v2.1 unaffected, b2g-v2.2 affected, b2g-v2.5 affected, b2g-master affected)

Details

(Whiteboard: [e10s], URL)

Attachments

(4 attachments)

(Reporter)

Description

4 years ago
Rev5 MacOSX Mountain Lion 10.8 mozilla-central debug test mochitest-3

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=550346&repo=mozilla-central

14:14:59 INFO - Assertion failure: work_queue_.empty(), at /builds/slave/m-cen-osx64-d-0000000000000000/build/ipc/chromium/src/base/message_loop.cc:410 

14:15:18 INFO - 255 INFO TEST-UNEXPECTED-ERROR | /tests/dom/media/webspeech/synth/test/test_speech_queue.html | This test left crash dumps behind, but we weren't expecting it to! 

14:49:11 WARNING - PROCESS-CRASH | Main app process exited normally | application crashed [@ MessageLoop::DeletePendingTasks()]
14:49:11 INFO - Crash dump filename: /var/folders/qg/zn406_js1y7777hyb4dqlbth00000w/T/tmpelH1T_.mozrunner/minidumps/B78298A1-C95A-42D0-B2AD-C661A843FF04.dmp
14:49:11 INFO - Operating system: Mac OS X
14:49:11 INFO - 10.8.0 12A269
14:49:11 INFO - CPU: amd64
14:49:11 INFO - family 6 model 42 stepping 7
14:49:11 INFO - 8 CPUs
14:49:11 INFO - Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
14:49:11 INFO - Crash address: 0x0
14:49:11 INFO - Thread 0 (crashed)
14:49:11 INFO - 0 XUL!MessageLoop::DeletePendingTasks() [message_loop.cc:53d84829b2b8 : 410 + 0x0]
14:49:11 INFO - rbx = 0x00007fff7b240c68 r12 = 0xffffffffffffffff
14:49:11 INFO - r13 = 0x0000000000000002 r14 = 0x00007fff5fbfd9a8
14:49:11 INFO - r15 = 0x0000000000000000 rip = 0x000000010087dc3f
14:49:11 INFO - rsp = 0x00007fff5fbfd870 rbp = 0x00007fff5fbfd8d0
14:49:11 INFO - Found by: given as instruction pointer in context
14:49:11 INFO - 1 XUL!MessageLoop::~MessageLoop() [message_loop.cc:53d84829b2b8 : 178 + 0x7]
14:49:11 INFO - rbx = 0x0000000000000001 r12 = 0xffffffffffffffff
14:49:11 INFO - r13 = 0x0000000000000002 r14 = 0x00007fff5fbfd9a8
14:49:11 INFO - r15 = 0x0000000000000000 rip = 0x000000010087d878
14:49:11 INFO - rsp = 0x00007fff5fbfd8e0 rbp = 0x00007fff5fbfd970
14:49:11 INFO - Found by: call frame info
14:49:11 INFO - 2 XUL!XRE_InitChildProcess [nsEmbedFunctions.cpp:53d84829b2b8 : 557 + 0x4]
14:49:11 INFO - rbx = 0x00007fff5fbfd900 r12 = 0x000000010692f000
14:49:11 INFO - r13 = 0x0000000000000002 r14 = 0x0000000000000000
14:49:11 INFO - r15 = 0x00007fff5fbfeed0 rip = 0x0000000102dcc546
14:49:11 INFO - rsp = 0x00007fff5fbfd980 rbp = 0x00007fff5fbfee90
14:49:11 INFO - Found by: call frame info
14:49:11 INFO - 3 plugin-container!main [plugin-container.cpp:53d84829b2b8 : 158 + 0x9]
14:49:11 INFO - rbx = 0x000000000000000a r12 = 0x0000000000000000
14:49:11 INFO - r13 = 0x0000000000000000 r14 = 0x00007fff5fbfeed0
14:49:11 INFO - r15 = 0x0000000000000000 rip = 0x0000000100000e0b
14:49:11 INFO - rsp = 0x00007fff5fbfeea0 rbp = 0x00007fff5fbfeeb0
14:49:11 INFO - Found by: call frame info
14:49:11 INFO - 4 plugin-container!start + 0x33
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
This is also affecting the Marionette unit tests, however these crashes are currently not being reported. When we landed bug 1038868 we started to report these crashes as failures, but it got backed out due to the spike of crashes reported.
(Reporter)

Comment 18

4 years ago
Kairo: do you know if we see this crashes also in the real world websites ?
Severity: normal → critical
Flags: needinfo?(kairo)
Comment hidden (Treeherder Robot)

Comment 20

4 years ago
(In reply to Carsten Book [:Tomcat] from comment #18)
> Kairo: do you know if we see this crashes also in the real world websites ?

https://crash-stats.mozilla.com/search/?signature=~MessageLoop%3A%3ADeletePendingTasks has one crash in the last week with this signature.
Flags: needinfo?(kairo)
Comment hidden (Treeherder Robot)
It's quite unclear if that crash signature on 34 has anything at all to do with this; the stack backtraces are rather different (and this was an assertion, there's no reason to be sure it would crash in an opt build).
Bug 1038868 apparently makes this into a really bad fail (by exposing that it's failing).  That one happens in an entirely different Mn test, not related to speech or media.  These seem to be shutdown-timing problems with IPC shutdown specific to Mac.  CC-ing IPC, shutdown and Mac folk, since this doesn't seem to be media-related at all (or at most indirectly).
Component: WebRTC → IPC
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
I've also hit this in Nightly by opening a non-e10s window, then closing the e10s window.  Same stack
Created attachment 8525106 [details]
gdb dump of data from failure
Note: while the test failures are Mac, this failure was on Linux Inbound
Comment hidden (Treeherder Robot)
I can also reproduce this by opening Nightly (now e10s by default), visiting a site, and then closing Nightly.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
This happens almost every time shutting down the browser in my debug builds...
Whiteboard: [e10s]
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)

Comment 66

4 years ago
Requesting e10s tracking on this: per comment 31 and various comments in #e10s, this is a bad shutdown crash that gets in the way of running and testing debug builds.
tracking-e10s: --- → ?
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
tracking-e10s: ? → m5+
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Assignee: nobody → wmccloskey
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Is there any progress on this? It's blocking us from landing bug 1038868 for over two months now, which will improve our ability to check and report crashes in Firefox OS on devices.
Flags: needinfo?(wmccloskey)
Dave, if you have an STR that I can reproduce, it would help a lot here. I don't see the crashes that other people have seen.

That said, I probably won't get to this for a few more weeks.
Flags: needinfo?(wmccloskey)
It happened reliably on shutdown with builds from a month ago (see e.g. comment 31 and comment 41), so that's a place to start looking at least.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
It (still) happens all the time for me when shutting down my inbound debug builds after doing webrtc testing. (likely the webrtc tests have nothing to do with it, and I think I crash similarly without it.   I'm sure there's a timing component to this (which is why it's intermittent.)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
For me, this happens all the time, while trying to run marionette-unit tests on Firefox debug builds on MacOSX10.9.5:
MINIDUMP_STACKWALK=/Users/mwargers/mercurial/tools/breakpad/osx/minidump_stackwalk python runtests.py --binary=/Users/mwargers/mozilla-central/obj-x86_64-apple-darwin13.4.0/dist/NightlyDebug.app/Contents/MacOS/firefox-bin --type=browser --symbols-path=/Users/mwargers/mozilla-central/obj-x86_64-apple-darwin13.4.0/dist/firefox-38.0a1.en-US.mac64.crashreporter-symbols.zip  tests/unit-tests.ini

With a release build, I don't get this crash.
Comment hidden (Treeherder Robot)
Created attachment 8567637 [details]
stack.txt

Ok, I crash every time, while trying to run layout/base/tests/marionette/test_selectioncarets.py
I attached the stack that I get while running that. Unfortunately, it doesn't seem like a useful stack to me.
I'm surprised these tests seem to work just fine on treeherder, while they are crashing for me, locally.
(In reply to Martijn Wargers [:mwargers] (QA) from comment #125)
> I'm surprised these tests seem to work just fine on treeherder, while they
> are crashing for me, locally.

This is *very* timing-sensitive, so it's not that surprising.  I get failures just trying to exit local debug builds.

Any ETA?  I presume it's still blocking an important landing
Flags: needinfo?(wmccloskey)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
I'm hoping to look into some IPC bugs in a week or two. I don't know if that will fix this crash or not though.
Flags: needinfo?(wmccloskey)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Created attachment 8577004 [details] [diff] [review]
add debugging output

I'm pretty sure this is caused by MessageChannel enqueuing its OnMaybeDequeueOne task too late, but I want to make sure of that before I go on a crusade.
Attachment #8577004 - Flags: review?(dvander)
Attachment #8577004 - Flags: review?(dvander) → review+
Keywords: leave-open
status-b2g-v2.1: --- → unaffected
status-b2g-v2.2: --- → affected
status-b2g-master: --- → affected
status-firefox36: --- → wontfix
status-firefox37: --- → affected
status-firefox38: --- → affected
status-firefox39: --- → affected
status-firefox-esr31: --- → unaffected
Comment hidden (Treeherder Robot)
Unexpected task! Unknown:Unknown:-1

That is disappointing. Maybe things have been shut down to the point where the data is no longer available.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
tracking-e10s: m5+ → -
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(In reply to Martijn Wargers [:mwargers] (QA) from comment #124)
> Ok, I crash every time, while trying to run
> layout/base/tests/marionette/test_selectioncarets.py
> I attached the stack that I get while running that. Unfortunately, it
> doesn't seem like a useful stack to me.

I don't seem to get this one anymore, running ./mach marionette-test layout/base/tests/marionette/test_selectioncarets.py with a recently updated debug build.
QA Whiteboard: QAExclude
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
I can see the same crash when closing firefox after opening ./toolkit/devtools/server/tests/browser/animation.html
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Created attachment 8602570 [details] [diff] [review]
task_tracking.diff

In case of the crash in comment #180, attaching patch is needed to output BirthPlace.

The place was http://hg.mozilla.org/mozilla-central/file/617dbce26726/ipc/glue/MessageChannel.cpp#l737
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
10 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7
* fx-team: 2
* b2g-inbound: 1

Platform breakdown:
* osx-10-6: 8
* linux64: 1
* linux32: 1

For more details, see:
http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2015-09-28&endday=2015-10-04&tree=all
status-b2g-v2.5: --- → affected
I can get this reproduced on OS X debug builds fairly easily, by having content in one tab, opening a new tab, closing the old tab before the new one finished loading.

Updated

3 years ago
See Also: → bug 1236350
5 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 3
* mozilla-central: 1
* b2g-inbound: 1

Platform breakdown:
* windows7-32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-01-11&endday=2016-01-17&tree=all
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* fx-team: 3

Platform breakdown:
* windows7-32: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-01-18&endday=2016-01-24&tree=all
26 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 18
* fx-team: 6
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 25
* osx-10-6: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-01-25&endday=2016-01-31&tree=all
15 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 12
* fx-team: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-02-08&endday=2016-02-14&tree=all
16 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-02-17&endday=2016-02-17&tree=all
15 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 11
* try: 2
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* windows7-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-02-18&endday=2016-02-18&tree=all
47 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 30
* fx-team: 13
* try: 2
* mozilla-central: 2

Platform breakdown:
* windows7-32: 47

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-02-22&endday=2016-02-28&tree=all
19 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 13
* fx-team: 5
* mozilla-central: 1

Platform breakdown:
* windows7-32: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-03-03&endday=2016-03-03&tree=all
70 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 43
* fx-team: 22
* mozilla-central: 4
* try: 1

Platform breakdown:
* windows7-32: 70

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1090921&startday=2016-02-29&endday=2016-03-06&tree=all

Comment 253

2 years ago
Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.
Priority: -- → P3
This seems to have stopped. We did a lot of work on this area of the code, so I'm not surprised.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.