Open Bug 1555897 Opened 5 years ago Updated 1 year ago

Intermittent mac OSX Mn WPT TV mda <test-name> | application crashed [@ CrashReporter::TerminateHandler]

Categories

(Toolkit :: Crash Reporting, defect, P3)

Product:
Toolkit
Component:
Crash Reporting
Type:
defect

Tracking

()

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: crash, intermittent-failure, regression)

Crash Data

Filed by: rmaries [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=249320286&repo=mozilla-inbound
Full log: https://queue.taskcluster.net/v1/task/XKeae5VqT4a8OEv6iicICw/runs/0/artifacts/public/logs/live_backing.log

05:44:25 INFO - TEST-START | testing/marionette/harness/marionette_harness/tests/unit/test_window_close_chrome.py TestCloseWindow.test_close_window_for_browser_window_with_single_tab
05:44:32 INFO - PROCESS-CRASH | testing/marionette/harness/marionette_harness/tests/unit/test_window_close_chrome.py TestCloseWindow.test_close_window_for_browser_window_with_single_tab | application crashed [@ CrashReporter::TerminateHandler()]
05:44:32 INFO - Crash dump filename: /var/folders/yx/vwxxzp055pg5dfqns8qj48_8000017/T/tmp2DnwC6.mozrunner/minidumps/9D37449E-B68A-44B3-9309-499AF26D7C75.dmp
05:44:32 INFO - Operating system: Mac OS X
05:44:32 INFO - 10.14.0 18A391
05:44:32 INFO - CPU: amd64
05:44:32 INFO - family 6 model 69 stepping 1
05:44:32 INFO - 4 CPUs
05:44:32 INFO -
05:44:32 INFO - GPU: UNKNOWN
05:44:32 INFO -
05:44:32 INFO - Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
05:44:32 INFO - Crash address: 0x0
05:44:32 INFO - Process uptime: 25 seconds
05:44:32 INFO -
05:44:32 INFO - Thread 1 (crashed)
05:44:32 INFO - 0 XUL!CrashReporter::TerminateHandler() [nsExceptionHandler.cpp:53af18412fd188c4db1af090874f68d1f6a804cf : 1360 + 0x15]
05:44:32 INFO - rax = 0x000000010ed1294a rdx = 0x00434c4e47432b2b
05:44:32 INFO - rcx = 0x0000000108134788 rbx = 0x0000000000002000
05:44:32 INFO - rsi = 0x00007fff59918c52 rdi = 0x000000010daaf9f0
05:44:32 INFO - rbp = 0x000070000edbeeb0 rsp = 0x000070000edbeeb0
05:44:32 INFO - r8 = 0x00000001082000b8 r9 = 0x00000000000000c4
05:44:32 INFO - r10 = 0x000070000edbece8 r11 = 0x0000000000000000
05:44:32 INFO - r12 = 0x0000000000000114 r13 = 0x0000000113c06200
05:44:32 INFO - r14 = 0x0000000000000000 r15 = 0x0000000000000000
05:44:32 INFO - rip = 0x000000010daafa05
05:44:32 INFO - Found by: given as instruction pointer in context
05:44:32 INFO - 1 libc++abi.dylib!std::__terminate(void ()()) + 0x8
05:44:32 INFO - rbp = 0x000070000edbeed0 rsp = 0x000070000edbeec0
05:44:32 INFO - rip = 0x00007fff59925dfe
05:44:32 INFO - Found by: previous frame's frame pointer
05:44:32 INFO - 2 libc++abi.dylib!std::terminate() + 0x33
05:44:32 INFO - rbp = 0x000070000edbeef0 rsp = 0x000070000edbeee0
05:44:32 INFO - rip = 0x00007fff59925e73
05:44:32 INFO - Found by: previous frame's frame pointer
05:44:32 INFO - 3 libobjc.A.dylib!lockAndFinishInitializing(objc_class, objc_class*) + 0x92
05:44:32 INFO - rbp = 0x000070000edbeef0 rsp = 0x000070000edbeef0
05:44:32 INFO - rip = 0x00007fff5b123a82
05:44:32 INFO - Found by: call frame info
05:44:32 INFO - 4 libdispatch.dylib!_dispatch_client_callout + 0x1c
05:44:32 INFO - rbp = 0x000070000edbef10 rsp = 0x000070000edbef00
05:44:32 INFO - rip = 0x00007fff5c1a0ddf
05:44:32 INFO - Found by: previous frame's frame pointer
05:44:32 INFO - 5 libdispatch.dylib!_dispatch_root_queue_drain + 0x29f
05:44:32 INFO - rbp = 0x000070000edbef80 rsp = 0x000070000edbef20
05:44:32 INFO - rip = 0x00007fff5c1aee74
05:44:32 INFO - Found by: previous frame's frame pointer
05:44:32 INFO - 6 libdispatch.dylib!_dispatch_worker_thread2 + 0x5a
05:44:32 INFO - rbp = 0x000070000edbef90 rsp = 0x000070000edbef90
05:44:32 INFO - rip = 0x00007fff5c1af4b1
05:44:32 INFO - Found by: previous frame's frame pointer
05:44:32 INFO - 7 libsystem_pthread.dylib!_pthread_wqthread + 0x26b
05:44:32 INFO - rbp = 0x000070000edbefd0 rsp = 0x000070000edbefa0
05:44:32 INFO - rip = 0x00007fff5c3e06ee
05:44:32 INFO - Found by: previous frame's frame pointer
05:44:32 INFO - 8 libsystem_pthread.dylib!start_wqthread + 0xd
05:44:32 INFO - rbp = 0x000070000edbeff8 rsp = 0x000070000edbefe0
05:44:32 INFO - rip = 0x00007fff5c3e0415
05:44:32 INFO - Found by: previous frame's frame pointer
05:44:32 INFO - 9 libdispatch.dylib!_dispatch_root_queues_init_once + 0xcd
05:44:32 INFO - rsp = 0x000070000edbf090 rip = 0x00007fff5c1af457
05:44:32 INFO - Found by: stack scanning

Flags: needinfo?(onosi.telea)
Component: Marionette → Crash Reporting
Product: Testing → Toolkit
Version: Version 3 → unspecified
Summary: Intermittent testing/marionette/harness/marionette_harness/tests/unit/test_window_close_chrome.py TestCloseWindow.test_close_window_for_browser_window_with_single_tab | application crashed [@ CrashReporter::TerminateHandler()] → Intermittent macosx Mn testing/marionette/harness/marionette_harness/tests/unit/<test_name> TestCloseWindow.test_close_window_for_browser_window_with_single_tab | application crashed [@ CrashReporter::TerminateHandler()]
Summary: Intermittent macosx Mn testing/marionette/harness/marionette_harness/tests/unit/<test_name> TestCloseWindow.test_close_window_for_browser_window_with_single_tab | application crashed [@ CrashReporter::TerminateHandler()] → Intermittent mac OSX Mn WPT TV <test-name> | application crashed [@ CrashReporter::TerminateHandler()]

Hey Gabriele, do you have an idea what's wrong here? Looks like this spiked a lot in the last two weeks.

Flags: needinfo?(onosi.telea) → needinfo?(gsvelto)

The crash is caused by an uncaught exception which has been thrown from Objective-C code. CrashReporter::TerminateHandler() gets called when an exception bubbles all the way up the stack without being caught. This line in the log suggests we're accessing freed memory from some Objective-C code:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249320286&repo=mozilla-inbound&lineNumber=24588

Unfortunately there's not much in the crashing thread's stack. It looks like a worker thread started with Apple's GCD. I can only speculate that this is a race of some sort where we created an object on the main thread, passed it to a worker thread then de-allocated it before the worker thread got a chance to work on it. That's assuming that the message above belongs to the crashing thread which I'm not 100% sure of.

Flags: needinfo?(gsvelto)

The line of code Gabriele is referring to is:

05:44:26 INFO - 2019-05-31 05:44:26.045 firefox[1581:10687] -[NSTaggedPointerString enumerateKeysAndObjectsUsingBlock:]: unrecognized selector sent to instance 0xe5e5e5e5e5e5e5e5

Markus, maybe you have an idea?

Flags: needinfo?(mstange)
Summary: Intermittent mac OSX Mn WPT TV <test-name> | application crashed [@ CrashReporter::TerminateHandler()] → Intermittent mac OSX Mn WPT TV mda <test-name> | application crashed [@ CrashReporter::TerminateHandler()]

Markus, would you mind having a look at it? Thanks

Is this issue gone (like Orangefactor claims)? No new bugs with the signature in Bugzilla and no classifications for more than 2 days. Searching the whole failure database of Treeherder times out.

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #26)

Is this issue gone (like Orangefactor claims)? No new bugs with the signature in Bugzilla and no classifications for more than 2 days. Searching the whole failure database of Treeherder times out.

I can't confirm that it is gone, I have refrained from running baseline try pushes due to the high load of macosx1014 machines. I suspect it may crop up at least once if I do run a baseline push, so I'd prefer to keep this bug open.

I took a look and the logs and don't have anything to add beyond what Gabriele said: Some Objective C code accesses a deleted object on a GCD thread. It's possible that we're responsible for causing this, but it's not clear from anything in the logs how that might have happened.

Flags: needinfo?(mstange)
Priority: -- → P3
Severity: critical → S2

Since the crash volume is low (less than 5 per week), the severity is downgraded to S3. Feel free to change it back if you think the bug is still critical.

For more information, please visit auto_nag documentation.

Severity: S2 → S3
Crash Signature: [@ CrashReporter::TerminateHandler()] → [@ CrashReporter::TerminateHandler]
Summary: Intermittent mac OSX Mn WPT TV mda <test-name> | application crashed [@ CrashReporter::TerminateHandler()] → Intermittent mac OSX Mn WPT TV mda <test-name> | application crashed [@ CrashReporter::TerminateHandler]
You need to log in before you can comment on or make changes to this bug.