Closed Bug 949347 Opened 11 years ago Closed 11 years ago

Intermittent PROCESS-CRASH | /tests/dom/camera/test/test_camera.html | application crashed [@ mozilla::layers::TextureClient::Finalize()] or timed out after 330 seconds and crashed

Categories

(Core :: Graphics: Layers, defect)

Product:
Core
Component:
Graphics: Layers
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla29
Project Flags:
blocking-b2g 1.4+
Tracking Flags:
Tracking Status
firefox27 --- unaffected
firefox28 --- unaffected
firefox29 --- fixed
firefox-esr24 --- unaffected
b2g-v1.2 --- unaffected
b2g-v1.3 --- unaffected
b2g-v1.3T --- unaffected
b2g-v1.4 --- fixed

People

(Reporter: cbook, Assigned: bjacob)

References

(
URL
)

Details

(Keywords: crash, intermittent-failure, smoketest)

Crash Data

Attachments

(3 files)

extracted crash log
87.50 KB, text/plain
Details
Refcount TextureChild
4.63 KB, patch
nical
: review+
sotaro
: review+
Details | Diff | Splinter Review
Temporary strong reference to the actor in case mActor might be nulled concurrently
1.12 KB, patch
nical
: review+
Details | Diff | Splinter Review
b2g_emulator_vm mozilla-inbound opt test mochitest-3 on 2013-12-11 20:27:18 PST for push c3610b50947e slave: tst-linux64-ec2-072 https://tbpl.mozilla.org/php/getParsedLog.php?id=31851641&tree=Mozilla-Inbound PROCESS-CRASH | /tests/dom/camera/test/test_camera.html | application crashed [@ mozilla::layers::TextureClient::Finalize()] 21:00:23 INFO - Operating system: Android 21:00:23 INFO - 0.0.0 Linux 2.6.29-00297-ge2ba18d #4 Tue Sep 24 09:35:47 UTC 2013 armv7l Android/full/generic:4.0.4.0.4.0.4/OPENMASTER/eng.cltbld.20131211.215655:eng/test-keys 21:00:23 INFO - CPU: arm 21:00:23 INFO - 0 CPUs 21:00:23 INFO - Crash reason: SIGSEGV 21:00:23 INFO - Crash address: 0x24 21:00:23 INFO - Thread 3 (crashed) 21:00:23 INFO - 0 libxul.so!mozilla::layers::TextureClient::Finalize() [TextureClient.cpp:c3610b50947e : 270 + 0x4] 21:00:23 INFO - r4 = 0x44713920 r5 = 0x448f4aa0 r6 = 0x44713920 r7 = 0x86698ae3 21:00:23 INFO - r8 = 0x448f4f80 r9 = 0x43b3a9a0 r10 = 0x80b17e09 fp = 0x0000015e 21:00:23 INFO - sp = 0x42f7cbd0 lr = 0x40060174 pc = 0x408d0aae 21:00:23 INFO - Found by: given as instruction pointer in context 21:00:23 INFO - 1 libxul.so!mozilla::RefPtr<mozilla::layers::GrallocTextureClientOGL>::~RefPtr [AtomicRefCountedWithFinalize.h : 36 + 0x5] 21:00:23 INFO - r4 = 0x44713920 r5 = 0x448f4aa0 r6 = 0x44713920 r7 = 0x86698ae3 21:00:23 INFO - r8 = 0x448f4f80 r9 = 0x43b3a9a0 r10 = 0x80b17e09 fp = 0x0000015e 21:00:23 INFO - sp = 0x42f7cbd8 pc = 0x408c00af 21:00:23 INFO - Found by: call frame info 21:00:23 INFO - 2 libxul.so!mozilla::layers::GrallocImage::~GrallocImage [GrallocImages.cpp:c3610b50947e : 69 + 0x7] 21:00:23 INFO - r4 = 0x448f4a00 r5 = 0x00000000 r6 = 0x44713920 r7 = 0x86698ae3 21:00:23 INFO - r8 = 0x448f4f80 r9 = 0x43b3a9a0 r10 = 0x80b17e09 fp = 0x0000015e 21:00:23 INFO - sp = 0x42f7cbe8 pc = 0x408c04b9 21:00:23 INFO - Found by: call frame info 21:00:23 INFO - 3 libxul.so!mozilla::layers::GrallocImage::~GrallocImage [GrallocImages.cpp:c3610b50947e : 69 + 0x3] 21:00:23 INFO - r4 = 0x448f4a00 r5 = 0x00000000 r6 = 0x0000015e r7 = 0x86698ae3 21:00:23 INFO - r8 = 0x448f4f80 r9 = 0x43b3a9a0 r10 = 0x80b17e09 fp = 0x0000015e 21:00:23 INFO - sp = 0x42f7cc38 pc = 0x408c04dd 21:00:23 INFO - Found by: call frame info 21:00:23 INFO - 4 libxul.so!CSF::CC_Call::Release() [CC_Call.h:c3610b50947e : 27 + 0x7] 21:00:23 INFO - r4 = 0x448f4a00 r5 = 0x00000000 r6 = 0x0000015e r7 = 0x86698ae3 21:00:23 INFO - r8 = 0x448f4f80 r9 = 0x43b3a9a0 r10 = 0x80b17e09 fp = 0x0000015e 21:00:23 INFO - sp = 0x42f7cc40 pc = 0x40594ed1 21:00:23 INFO - Found by: call frame info 21:00:23 INFO - 5 libxul.so!mozilla::VideoFrameContainer::SetCurrentFrame(nsIntSize const&, mozilla::layers::Image*, mozilla::TimeStamp) [nsAutoPtr.h : 887 + 0x5] 21:00:23 INFO - r4 = 0x43b3a9a0 r5 = 0x448f4a00 r6 = 0x0000015e r7 = 0x86698ae3 21:00:23 INFO - r8 = 0x448f4f80 r9 = 0x43b3a9a0 r10 = 0x80b17e09 fp = 0x0000015e 21:00:23 INFO - sp = 0x42f7cc50 pc = 0x40dbe481 21:00:23 INFO - Found by: call frame info 21:00:23 INFO - 6 libxul.so!mozilla::CameraPreviewMediaStream::SetCurrentFrame(nsIntSize const&, mozilla::layers::Image*) [CameraPreviewMediaStream.cpp:c3610b50947e : 113 + 0xf]
Nick, this seems likely to be from one of your recent pushes?
Flags: needinfo?(ncameron)
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #4) > Nick, this seems likely to be from one of your recent pushes? I don't think so. I don't think I've touched finalisation/destruction of texture clients. And the only thing I've pushed recently which is related (new textures content) is disabled on b2g. So, passing the buck to nical and bjacob because bug ptexture did finalisation stuff and I think it landed about when this started.
Flags: needinfo?(nical.bugzilla)
Flags: needinfo?(ncameron)
Flags: needinfo?(bjacob)
This is likely going to be my fault, bug 897452. Looking ASAP.
Blocks: PTexture
Flags: needinfo?(bjacob)
Alright, alright, making an emulator build, so I can reproduce locally...
Flags: needinfo?(nical.bugzilla)
Assignee: nobody → bjacob
Summary: Intermittent PROCESS-CRASH | /tests/dom/camera/test/test_camera.html | application crashed [@ mozilla::layers::TextureClient::Finalize()] → Intermittent PROCESS-CRASH | /tests/dom/camera/test/test_camera.html | application crashed [@ mozilla::layers::TextureClient::Finalize()] or timed out after 330 seconds and crashed
Implementation of AtomicRefCountedWithFinalize::Release() seems bad. It uses mRefCount directly to check refcount becomes 0. It should not use mRefCount directly. I could cause multiple calling Finalize() depends on the timing. ------------------------------ void Release() { MOZ_ASSERT(mRefCount > 0); if (0 == --mRefCount) { #ifdef DEBUG mRefCount = detail::DEAD; #endif T* derived = static_cast<T*>(this); derived->Finalize(); delete derived; } } ------------------------ android's MediaBuffer's implementation is like the following. It uses previous reference count got from __atomic_dec() like the following. void MediaBuffer::release() { if (mObserver == NULL) { CHECK_EQ(mRefCount, 0); delete this; return; } int prevCount = __atomic_dec(&mRefCount); if (prevCount == 1) { if (mObserver == NULL) { delete this; return; } mObserver->signalBufferReturned(this); } CHECK(prevCount > 0); } http://androidxref.com/4.4_r1/xref/frameworks/av/media/libstagefright/MediaBuffer.cpp#95
Many thanks Sotaro, that is absolutely true, and will save me a lot of time!
In fact, in if (0 == --mRefCount) { The value returned by --mRefCount comes from Atomic<int>::operator--, which returns the value returned on the stack by __sync_fetch_and_sub, so we should be safe there. This isn't in fact a shared variable. This is also the same as what MFBT's AtomicRefCounted does.
Hit this using the Buri device today on nightly after launching camera, using: Gaia 545aacf3feff6430140cc9ade757002df4895b77 SourceStamp b1e5ade62913 BuildID 20131217040201 Version 29.0a1
Keywords: smoketest
using same build as comment 52 - hit this crash when sending MMS with camera-taken picture, 100% repro
blocking-b2g: --- → 1.4?
Attached file extracted crash log —
Comment 0 does not include all crash log. This is extracted the crash log part from the following. https://tbpl.mozilla.org/php/getParsedLog.php?id=31851641&tree=Mozilla-Inbound
I found a problem in the crash log, GrallocImage's destructor is called on android binder ipc thread, and TextureClient::Finalize() is called in the thread. But in the Finalize(), "TextureChild* mActor" is used in the thread, Access to mActor have to be on ImageBridge thread in this case. Current TextureClient's implementation seems not care about the thread safety of "TextureChild* mActor".
TextureClient is used, referenced and going to be destructed as multi-threaded way. TextureChild need to protect "TextureChild* mActor" from such usage.
I manually ran into this crash : https://crash-stats.mozilla.com/report/index/bbf6233d-59f4-4cd7-a49f-2d2e22131220 I cannot reproduce it again. It occurred when I removed all my data from the sdcard using adb ( including pictures ) then launch gallery and pressed the button to go to the camera app. Frame Module Signature Source 0 libxul.so mozilla::layers::TextureClient::Finalize() gfx/layers/client/TextureClient.cpp 1 libxul.so mozilla::RefPtr<mozilla::layers::GrallocTextureClientOGL>::~RefPtr /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/gfx/layers/../../dist/include/mozilla/layers/AtomicRefCountedWithFinalize.h 2 libxul.so mozilla::layers::GrallocImage::~GrallocImage gfx/layers/GrallocImages.cpp 3 libxul.so mozilla::layers::GrallocImage::~GrallocImage gfx/layers/GrallocImages.cpp 4 libxul.so CSF::CC_Call::Release() media/webrtc/signaling/include/CC_Call.h 5 libxul.so mozilla::VideoFrameContainer::ClearCurrentFrame(bool) /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/content/media/../../dist/include/nsAutoPtr.h 6 libxul.so mozilla::dom::HTMLMediaElement::EndSrcMediaStreamPlayback() content/html/content/src/HTMLMediaElement.cpp 7 libxul.so mozilla::dom::HTMLMediaElement::AbortExistingLoads() content/html/content/src/HTMLMediaElement.cpp 8 libxul.so mozilla::dom::HTMLMediaElement::Load() content/html/content/src/HTMLMediaElement.cpp 9 libxul.so mozilla::dom::HTMLVideoElement::Load() /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/content/html/content/src/../../../../dist/include/mozilla/dom/HTMLVideoElement.h 10 libxul.so mozilla::dom::HTMLMediaElement::SetMozSrcObject(mozilla::DOMMediaStream&) content/html/content/src/HTMLMediaElement.cpp 11 libxul.so mozilla::dom::HTMLMediaElement::SetMozSrcObject(nsIDOMMediaStream*) content/html/content/src/HTMLMediaElement.cpp 12 libxul.so mozilla::dom::HTMLVideoElement::SetMozSrcObject(nsIDOMMediaStream*) /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/content/html/content/src/../../../../dist/include/mozilla/dom/HTMLVideoElement.h 13 libxul.so mozilla::dom::HTMLMediaElementBinding::set_mozSrcObject /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/dom/bindings/HTMLMediaElementBinding.cpp 14 libxul.so mozilla::dom::HTMLMediaElementBinding::genericSetter /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/dom/bindings/HTMLMediaElementBinding.cpp 15 libxul.so js::Invoke js/src/jscntxtinlines.h 16 libxul.so js::InvokeGetterOrSetter(JSContext*, JSObject*, JS::Value, unsigned int, JS::Value*, JS::MutableHandle<JS::Value>) js/src/vm/Interpreter.cpp 17 libxul.so bool js::baseops::SetPropertyHelper<(js::ExecutionMode)0>(js::ExecutionModeTraits<(js::ExecutionMode)0>::ContextType, JS::Handle<JSObject*>, JS::Handle<JSObject*>, JS::Handle<jsid>, unsigned int, JS::MutableHandle<JS::Value>, bool) js/src/vm/Shape-inl.h 18 libxul.so Interpret js/src/vm/Interpreter.cpp 19 libxul.so js::Invoke js/src/vm/Interpreter.cpp 20 libxul.so JS_CallFunctionValue(JSContext*, JSObject*, JS::Value, unsigned int, JS::Value*, JS::Value*) js/src/jsapi.cpp 21 libxul.so mozilla::dom::EventListener::HandleEvent(JSContext*, JS::Handle<JSObject*>, nsDOMEvent&, mozilla::ErrorResult&) /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/dom/bindings/EventListenerBinding.cpp 22 libxul.so nsEventListenerManager::HandleEventSubType(nsListenerStruct*, nsIDOMEvent*, mozilla::dom::EventTarget*) /builds/slave/b2g_m-cen_ham_ntly-00000000000/build/objdir-gecko/content/events/src/../../../dist/include/mozilla/dom/EventListenerBinding.h 23 libxul.so nsEventListenerManager::HandleEventInternal(nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent**, mozilla::dom::EventTarget*, nsEventStatus*) content/events/src/nsEventListenerManager.cpp 24 libxul.so nsEventTargetChainItem::HandleEvent(nsEventChainPostVisitor&, ELMCreationDetector&) content/events/src/nsEventListenerManager.h 25 libxul.so nsEventTargetChainItem::HandleEventTargetChain(nsTArray<nsEventTargetChainItem>&, nsEventChainPostVisitor&, nsDispatchingCallback*, ELMCreationDetector&) content/events/src/nsEventDispatcher.cpp 26 libxul.so nsEventDispatcher::Dispatch(nsISupports*, nsPresContext*, mozilla::WidgetEvent*, nsIDOMEvent*, nsEventStatus*, nsDispatchingCallback*, nsCOMArray<mozilla::dom::EventTarget>*) content/events/src/nsEventDispatcher.cpp 27 libxul.so nsEventDispatcher::DispatchDOMEvent(nsISupports*, mozilla::WidgetEvent*, nsIDOMEvent*, nsPresContext*, nsEventStatus*) content/events/src/nsEventDispatcher.cpp 28 libxul.so nsINode::DispatchEvent(nsIDOMEvent*, bool*) content/base/src/nsINode.cpp 29 libxul.so nsContentUtils::DispatchEvent(nsIDocument*, nsISupports*, nsAString_internal const&, bool, bool, bool, bool*) content/base/src/nsContentUtils.cpp 30 libxul.so nsContentUtils::DispatchTrustedEvent(nsIDocument*, nsISupports*, nsAString_internal const&, bool, bool, bool*) content/base/src/nsContentUtils.cpp 31 libxul.so nsDocument::UpdateVisibilityState() content/base/src/nsDocument.cpp 32 libxul.so nsRunnableMethodImpl<nsresult (mozilla::net::<unnamed>::CacheFilesDeletor::*)(), void, true>::Run 33 libxul.so nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp 34 libxul.so NS_ProcessNextEvent(nsIThread*, bool) xpcom/glue/nsThreadUtils.cpp 35 libxul.so mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp 36 libxul.so mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp 37 libxul.so MessageLoop::RunInternal() ipc/chromium/src/base/message_loop.cc 38 libxul.so MessageLoop::Run() ipc/chromium/src/base/message_loop.cc 39 libxul.so nsBaseAppShell::Run() widget/xpwidgets/nsBaseAppShell.cpp 40 libxul.so XRE_RunAppShell toolkit/xre/nsEmbedFunctions.cpp 41 libxul.so mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp 42 libxul.so MessageLoop::RunInternal() ipc/chromium/src/base/message_loop.cc 43 libxul.so MessageLoop::Run() ipc/chromium/src/base/message_loop.cc 44 libxul.so XRE_InitChildProcess toolkit/xre/nsEmbedFunctions.cpp 45 plugin-container main ipc/app/MozillaRuntimeMain.cpp 46 libc.so __libc_init bionic/libc/bionic/libc_init_dynamic.c 47 @0xb00045a9
Attachment #8350752 - Flags: review?(sotaro.ikeda.g)
Attachment #8350752 - Flags: review?(nical.bugzilla)
Attachment #8350752 - Flags: review?(nical.bugzilla) → review+
Comment on attachment 8350752 [details] [diff] [review] Refcount TextureChild Review of attachment 8350752 [details] [diff] [review]: ----------------------------------------------------------------- Look good. You are an expert of changing IPC object to reference counted!
Attachment #8350752 - Flags: review?(sotaro.ikeda.g) → review+
:) https://hg.mozilla.org/integration/mozilla-inbound/rev/0cc9c500afd9 Leaving open for now until it's confirmed to be fixed.
Whiteboard: [leave open]
Target Milestone: --- → mozilla29
I've replicated this on the following build: Device: Peak Build ID: 20131221064155 Gaia commit: c1ed307f Gecko: 29.0a1 The STR are: 1. Set a passcode/display lock 2. lock the device 3. Turn the screen on 4. Tap 'Camera' icon while phone is still locked. 5. Fiddle with the camera a bit and it will crash. Regular camera operation seems OK. I've also replicated this while just browsing in bookmarked pages but the above STR is much easier and would seem to correlate better with the TBPL failures.
See comment in the patch. The previous patch prevented us from dereferencing a dead actor, but it doesn't prevent us from dereferencing a null actor if mActor gets nulled concurrently.
Attachment #8355021 - Flags: review?(sotaro.ikeda.g)
Attachment #8355021 - Flags: review?(nical.bugzilla)
Attachment #8355021 - Flags: review?(nical.bugzilla) → review+
This bug is still included in 01/03 Master M-C smoketest report as the crash still occurs when sending MMS using camera-taken image. However, upon further investigation, it turned out that we hit bug 952170. So, I remove smoketest keyword here since this exact crash no longer reproduces and it no longer blocks smoketest. Device: Buri Master M-C (1.4) Mozilla RIL BuildID: 20140103040201 Gaia: 83cc63f728489a24256731adf558354bb2012a59 Gecko: 49d2fce9a86c Version: 29.0a1 Firmware Version: v1.2_20131115
Keywords: smoketest
Actually, that implies the second patch that landed here is confirmed to fix the bug, so we can close this out.
Status: NEW → RESOLVED
blocking-b2g: 1.4? → 1.4+
Closed: 11 years ago
Keywords: smoketest
Resolution: --- → FIXED
Whiteboard: [leave open]
Attachment #8355021 - Flags: review?(sotaro.ikeda.g)
Blocks: 1205559
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: