1139388 - Hang and process kill in various performance tests

Status: RESOLVED FIXED

(DevTools :: Performance Tools (Profiler/Timeline), defect)

Comment 1

[Victor Porof [:vporof][:vp]]

Seen again for https://treeherder.mozilla.org/logviewer.html#?job_id=50900&repo=gum

Comment 2

[Victor Porof [:vporof][:vp]]

And https://treeherder.mozilla.org/logviewer.html#?job_id=51007&repo=gum

Comment 3

[Shu-yu Guo [:shu]]

These are timeouts, not crashes.

Comment 4

[Victor Porof [:vporof][:vp]]

https://treeherder.mozilla.org/logviewer.html#?job_id=51043&repo=gum

Comment 6

[Victor Porof [:vporof][:vp]]

Fixed by bug 1134029

Comment 7

[Victor Porof [:vporof][:vp]]

Apparently not.

Comment 8

[Victor Porof [:vporof][:vp]]

e.g 

https://treeherder.mozilla.org/logviewer.html#?job_id=59399&repo=gum
https://treeherder.mozilla.org/logviewer.html#?job_id=59279&repo=gum

Comment 10

[Shu-yu Guo [:shu]]

I've managed to trap one beachball so far. Deadlock?

(lldb) bt
* thread #1: tid = 0xa2583, 0x00007fff95e3448a libsystem_kernel.dylib`__semwait_signal + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff95e3448a libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff925beb98 libsystem_pthread.dylib`pthread_join + 445
  * frame #2: 0x0000000105a106b3 XUL`Thread::Join(this=0x000000012084cb80) + 35 at platform-macos.cc:132
    frame #3: 0x0000000105a1691c XUL`SamplerThread::RemoveActiveSampler(sampler=0x0000000134b14380) + 76 at platform-macos.cc:195
    frame #4: 0x0000000105a109a0 XUL`Sampler::Stop(this=0x0000000134b14380) + 128 at platform-macos.cc:333
    frame #5: 0x0000000105a137fb XUL`mozilla_sampler_stop() + 235 at platform.cpp:816
    frame #6: 0x0000000105a0f569 XUL`profiler_stop() + 9 at GeckoProfilerImpl.h:88
    frame #7: 0x0000000105a0f551 XUL`nsProfiler::StopProfiler(this=0x0000000127f0f220) + 17 at nsProfiler.cpp:87
    frame #8: 0x0000000101735c22 XUL`NS_InvokeByIndex(that=0x0000000127f0f220, methodIndex=4, paramCount=0, params=0x00007fff5fbeab10) + 578 at xptcinvoke_x86_64_unix.cpp:174
    frame #9: 0x0000000102592918 XUL`CallMethodHelper::Invoke(this=0x00007fff5fbeaac8) + 88 at XPCWrappedNative.cpp:2113
    frame #10: 0x00000001025834c7 XUL`CallMethodHelper::Call(this=0x00007fff5fbeaac8) + 263 at XPCWrappedNative.cpp:1450
    frame #11: 0x0000000102562634 XUL`XPCWrappedNative::CallMethod(ccx=0x00007fff5fbeaca0, mode=CALL_METHOD) + 260 at XPCWrappedNative.cpp:1417
    frame #12: 0x0000000102564893 XUL`XPC_WN_CallMethod(cx=0x0000000100429c40, argc=0, vp=0x00000001170e3488) + 931 at XPCWrappedNativeJSOps.cpp:1148
    frame #13: 0x0000000106eebebb XUL`js::CallJSNative(cx=0x0000000100429c40, native=0x00000001025644f0, args=0x00007fff5fbeb6d0)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) + 187 at jscntxtinlines.h:235
    frame #14: 0x0000000106e7fa45 XUL`js::Invoke(cx=0x0000000100429c40, args=CallArgs at 0x00007fff5fbeb6d0, construct=NO_CONSTRUCT) + 1269 at Interpreter.cpp:498
    frame #15: 0x0000000106e99fb5 XUL`Interpret(cx=0x0000000100429c40, state=0x00007fff5fbee5b8) + 51749 at Interpreter.cpp:2596
    frame #16: 0x0000000106e8d4a9 XUL`js::RunScript(cx=0x0000000100429c40, state=0x00007fff5fbee5b8) + 585 at Interpreter.cpp:448
    frame #17: 0x0000000106e7fb86 XUL`js::Invoke(cx=0x0000000100429c40, args=CallArgs at 0x00007fff5fbeedb0, construct=NO_CONSTRUCT) + 1590 at Interpreter.cpp:517
    frame #18: 0x0000000106e69464 XUL`js::Invoke(cx=0x0000000100429c40, thisv=0x00007fff5fbef1b0, fval=0x00007fff5fbef1e0, argc=3, argv=0x00007fff5fbef350, rval=JS::MutableHandleValue at 0x00007fff5fbeeeb0) + 900 at Interpreter.cpp:554
    frame #19: 0x00000001071ce6a7 XUL`js::jit::DoCallFallback(cx=0x0000000100429c40, frame=0x00007fff5fbef3e8, stub_=0x00000001191696c0, argc=3, vp=0x00007fff5fbef340, res=JS::MutableHandleValue at 0x00007fff5fbef298) + 1847 at BaselineIC.cpp:9648
    frame #20: 0x000000010078a03b

Comment 11

[Shu-yu Guo [:shu]]

Yeah, looks like a deadlock on Sampler::sRegisteredThreadsMutex when trying to stop SamplerThread.

RemoveActiveSampler locks sRegisteredThreadsMutex and tries to join the SamplerThread. SamplerThread is waiting on sRegisteredThreadsMutex in Sampler::Run, and can't join.

Comment 12

[Shu-yu Guo [:shu]]

Attachment: profiler-deadlock.patch

Not sure why those methods took the lock in the first place.

Comment 13

[Victor Porof [:vporof][:vp]]

https://treeherder.mozilla.org/logviewer.html#?job_id=60431&repo=gum

Comment 14

[Markus Stange [:mstange]]

Comment on attachment 8576534 [details] [diff] [review]
profiler-deadlock.patch

Review of attachment 8576534 [details] [diff] [review]:
-----------------------------------------------------------------

I agree. These locks were added in https://hg.mozilla.org/mozilla-central/rev/f2e44e02f874 , titled "Port multi-thread support to win/mac", but linux doesn't have that lock in that place.

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/91abcf24159b