1247429 - I10t 481921.html,880404.html,882956.html,884459.html,907986-2.html,907986-4.html,933151.html,933156.html,944851.html,952756.html,986901.html,1028458.html,1041466.html | assertion count is more... "ASSERTION: Wrong size for this Shmem!"

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://treeherder.mozilla.org/logviewer.html#?job_id=21476804&repo=mozilla-inbound

###!!! ASSERTION: math on NS_UNCONSTRAINEDSIZE: 'NS_UNCONSTRAINEDSIZE != aState.mReflowState.ComputedLogicalBorderPadding().IStart(wm) && NS_UNCONSTRAINEDSIZE != aState.mReflowState.ComputedISize()', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/layout/generic/nsBlockFrame.cpp, line 1869
###!!! ASSERTION: Wrong size for this Shmem!: 'Error', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/ipc/glue/Shmem.cpp, line 459
###!!! ASSERTION: Wrong size for this Shmem!: 'Error', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/ipc/glue/Shmem.cpp, line 459
###!!! ASSERTION: Wrong size for this Shmem!: 'Error', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/ipc/glue/Shmem.cpp, line 459
###!!! ASSERTION: Wrong size for this Shmem!: 'Error', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/ipc/glue/Shmem.cpp, line 459
###!!! ASSERTION: Wrong size for this Shmem!: 'Error', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/ipc/glue/Shmem.cpp, line 459

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

As far as I can tell, this started happening somewhere in this range: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=f81f4e92f023&filter-searchStr=91c1aae070e037bd1d3378050613a86051cbb66d&tochange=432ef38dab95

Comment 2

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

I retrigger the e10s crash test of Linux x64 debug in my try [1] at least twenty times. I got all green.

[1] Try in bug 1246185 comment 3: https://treeherder.mozilla.org/#/jobs?repo=try&revision=411628cbd595

Comment 3

[Jan de Mooij [:jandem]]

My patches are two minor security fixes; pretty unlikely to have caused this. FWIW, the test uses setTimeout (800 and 200 ms) in a few places; that's asking for intermittent failures IMO.

Forwarding the NI to billm, as this is inside IPC code.


 12:52:43     INFO -  [Child 1938] WARNING: data callback fires before cubeb_stream_start() is called: file /builds/slave/m-in-l64-d-0000000000000000000/build/src/dom/media/AudioStream.cpp, line 644
 12:52:43     INFO -  [Parent 1882] ###!!! ASSERTION: Wrong size for this Shmem!: 'Error', file /builds/slave/m-in-l64-d-0000000000000000000/build/src/ipc/glue/Shmem.cpp, line 459
 12:52:43     INFO -  #01: mozilla::layers::PImageBridgeParent::OnMessageReceived(IPC::Message const&) [mfbt/AlreadyAddRefed.h:116]
 12:52:43     INFO -  #02: mozilla::ipc::MessageChannel::DispatchAsyncMessage(IPC::Message const&) [ipc/glue/MessageChannel.cpp:1444]

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

My changes only affected TC

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

The problem happens on the ImageBridge channel. We're sending a new shmem over and the size comes out wrong. That's a pretty serious problem. It sounds like memory corruption.

It's possible that the xrender patch could have caused this.

Comment 6

[Lee Salzman [:lsalzman]]

(In reply to Bill McCloskey (:billm) from comment #5)
> The problem happens on the ImageBridge channel. We're sending a new shmem
> over and the size comes out wrong. That's a pretty serious problem. It
> sounds like memory corruption.
> 
> It's possible that the xrender patch could have caused this.

It's definitely the recent changes to use-image-offscreen-surfaces that triggered this, since we are now sending Shmems around instead of X drawables. The Shmem code itself shouldn't really be causing the problem, in principle, and is probably just allowing us to see some underlying heap corruption that is going on. The alternative is that there is some use-after-dealloc going on, but that seems unlikely?

It might help if the errors were printing out the shmem magic/size so we could verify more what is going on.

Comment 7

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I have this in an rr recording.  The Shmem is created on the child and sent to the parent, and then deallocated before the parent actually gets around to receiving it.  Deallocating sets the size contained within the Shmem segment itself to zero, which trips the assertion when the parent finally does receive the message and attempt to sanity check things.

Comment 8

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Allocation

#0  mozilla::ipc::MessageChannel::Send (this=0x7f163a8ef868, aMsg=0x7f16231e2310)
    at /home/khuey/dev/gecko-dev/ipc/glue/MessageChannel.cpp:543
#1  0x00007f1653015867 in mozilla::layers::PImageBridgeChild::CreateSharedMemory (this=0x7f163a8ef800, aSize=115200, 
    aType=mozilla::ipc::SharedMemory::TYPE_BASIC, aUnsafe=true, aId=0x7f16342f5624)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/ipc/ipdl/PImageBridgeChild.cpp:618
#2  0x00007f1653016cb3 in mozilla::layers::PImageBridgeChild::AllocUnsafeShmem (this=0x7f163a8ef800, aSize=115200, 
    aType=mozilla::ipc::SharedMemory::TYPE_BASIC, aOutMem=0x7f163c043e80)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/ipc/ipdl/PImageBridgeChild.cpp:998
#3  0x00007f1653c9ea30 in mozilla::layers::ImageBridgeChild::AllocUnsafeShmem (this=0x7f163a8ef800, aSize=115200, 
    aType=mozilla::ipc::SharedMemory::TYPE_BASIC, aShmem=0x7f163c043e80)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp:931
#4  0x00007f1653c9ec34 in mozilla::layers::ProxyAllocShmemNow (aParams=0x7f163c043dc0, aBarrier=0x7f163c043d90, 
    aDone=0x7f163c043d4f) at /home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp:972
#5  0x00007f1653cc1363 in details::CallFunction<0ul, 1ul, 2ul, void (*)(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*), mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*> (
    function=0x7f1653c9eb15 <mozilla::layers::ProxyAllocShmemNow(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*)>, arg=...) at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:36
#6  0x00007f1653cc0b9c in DispatchTupleToFunction<void (*)(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*), mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*> (
    function=0x7f1653c9eb15 <mozilla::layers::ProxyAllocShmemNow(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*)>, arg=...) at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:54
#7  0x00007f1653cbf56e in RunnableFunction<void (*)(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*), mozilla::Tuple<mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*> >::Run (this=0x7f1624973c40)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:350
#8  0x00007f1652e02965 in MessageLoop::RunTask (this=0x7f16342f5980, task=0x7f1624973c40)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:364
#9  0x00007f1652e029dd in MessageLoop::DeferOrRunPendingTask (this=0x7f16342f5980, pending_task=...)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:372
#10 0x00007f1652e02e44 in MessageLoop::DoWork (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:459
#11 0x00007f1652e03327 in base::MessagePumpDefault::Run (this=0x7f163a6bb0d0, delegate=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_pump_default.cc:34
#12 0x00007f1652e0243d in MessageLoop::RunInternal (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:234
#13 0x00007f1652e023d2 in MessageLoop::RunHandler (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:227
#14 0x00007f1652e02363 in MessageLoop::Run (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:201
#15 0x00007f1652e1be4a in base::Thread::ThreadMain (this=0x7f1644f6c3c0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/thread.cc:172
#16 0x00007f1652e172aa in ThreadFunc (closure=0x7f1644f6c3c0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/platform_thread_posix.cc:36
#17 0x00007f165c5886aa in start_thread (arg=0x7f16342f6700) at pthread_create.c:333
#18 0x00007f1650bffeed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109


Deallocation:

#0  mozilla::ipc::Shmem::Dealloc (aSegment=0x7f16253e1b50) at /home/khuey/dev/gecko-dev/ipc/glue/Shmem.cpp:489
#1  0x00007f1653015b39 in mozilla::layers::PImageBridgeChild::DestroySharedMemory (this=0x7f163a8ef800, shmem=...)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/ipc/ipdl/PImageBridgeChild.cpp:667
#2  0x00007f1653016e3b in mozilla::layers::PImageBridgeChild::DeallocShmem (this=0x7f163a8ef800, aMem=...)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/ipc/ipdl/PImageBridgeChild.cpp:1028
#3  0x00007f1653c9efd6 in mozilla::layers::ImageBridgeChild::DeallocShmem (this=0x7f163a8ef800, aShmem=...)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp:1029
#4  0x00007f1653b99ac8 in mozilla::layers::ShmemTextureData::Deallocate (this=0x7f1624907820, aAllocator=0x7f163a8efba0)
    at /home/khuey/dev/gecko-dev/gfx/layers/BufferTexture.cpp:497
#5  0x00007f1653c2a631 in mozilla::layers::DestroyTextureData (aTextureData=0x7f1624907820, aAllocator=0x7f163a8efba0, 
    aDeallocate=true, aMainThreadOnly=false) at /home/khuey/dev/gecko-dev/gfx/layers/client/TextureClient.cpp:199
#6  0x00007f1653c2aada in mozilla::layers::DeallocateTextureClient (params=...)
    at /home/khuey/dev/gecko-dev/gfx/layers/client/TextureClient.cpp:294
#7  0x00007f1653c2ae78 in mozilla::layers::TextureClient::Destroy (this=0x7f1624ad12e0, aForceSync=false)
    at /home/khuey/dev/gecko-dev/gfx/layers/client/TextureClient.cpp:350
#8  0x00007f1653c2b5d1 in mozilla::layers::TextureClient::~TextureClient (this=0x7f1624ad12e0, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/gfx/layers/client/TextureClient.cpp:452
#9  0x00007f1653c2b6bc in mozilla::layers::TextureClient::~TextureClient (this=0x7f1624ad12e0, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/gfx/layers/client/TextureClient.cpp:453
#10 0x00007f1653b57d29 in mozilla::AtomicRefCountedWithFinalize<mozilla::layers::TextureClient>::Release (
    this=0x7f1624ad12e8)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/layers/AtomicRefCountedWithFinalize.h:148
#11 0x00007f1653cab32c in mozilla::AtomicRefCountedWithFinalize<mozilla::layers::TextureClient>::ReleaseManually (
    this=0x7f1624ad12e8, 
    funcName=0x7f1658cccb40 <mozilla::layers::ReleaseTextureClientNow(mozilla::layers::TextureClient*)::__FUNCTION__> "ReleaseTextureClientNow", fileName=0x7f1658cc8670 "/home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp", lineNum=507)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/layers/AtomicRefCountedWithFinalize.h:109
#12 0x00007f1653c9ce6a in mozilla::layers::ReleaseTextureClientNow (aClient=0x7f1624ad12e0)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp:507
#13 0x00007f1653b5b414 in details::CallFunction<0ul, void (*)(mozilla::layers::TextureClient*), mozilla::layers::TextureClient*> (function=0x7f1653c9cdf2 <mozilla::layers::ReleaseTextureClientNow(mozilla::layers::TextureClient*)>, arg=...)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:36
#14 0x00007f1653b5acde in DispatchTupleToFunction<void (*)(mozilla::layers::TextureClient*), mozilla::layers::TextureClient*> (function=0x7f1653c9cdf2 <mozilla::layers::ReleaseTextureClientNow(mozilla::layers::TextureClient*)>, arg=...)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:54
#15 0x00007f1653b5a7e6 in RunnableFunction<void (*)(mozilla::layers::TextureClient*), mozilla::Tuple<mozilla::layers::TextureClient*> >::Run (this=0x7f1624eff1f0) at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:350
#16 0x00007f1652e02965 in MessageLoop::RunTask (this=0x7f16342f5980, task=0x7f1624eff1f0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:364
#17 0x00007f1652e029dd in MessageLoop::DeferOrRunPendingTask (this=0x7f16342f5980, pending_task=...)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:372
#18 0x00007f1652e02e44 in MessageLoop::DoWork (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:459
#19 0x00007f1652e03327 in base::MessagePumpDefault::Run (this=0x7f163a6bb0d0, delegate=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_pump_default.cc:34
#20 0x00007f1652e0243d in MessageLoop::RunInternal (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:234
#21 0x00007f1652e023d2 in MessageLoop::RunHandler (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:227
#22 0x00007f1652e02363 in MessageLoop::Run (this=0x7f16342f5980)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:201
#23 0x00007f1652e1be4a in base::Thread::ThreadMain (this=0x7f1644f6c3c0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/thread.cc:172
#24 0x00007f1652e172aa in ThreadFunc (closure=0x7f1644f6c3c0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/platform_thread_posix.cc:36
#25 0x00007f165c5886aa in start_thread (arg=0x7f16342f6700) at pthread_create.c:333
#26 0x00007f1650bffeed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Comment 9

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Both of those stacks are async, so the actual allocation stack is:

#0  0x00007f1653cb1387 in RunnableFunction<void (*)(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*), mozilla::Tuple<mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*> >::RunnableFunction(void (*)(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*), mozilla::Tuple<mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*>&&) (this=0x7f1624973c40, 
    function=0x7f1653c9eb15 <mozilla::layers::ProxyAllocShmemNow(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*)>, 
    params=<unknown type in /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/bin/libxul.so, CU 0x53bab8e, DIE 0x54d2f51>) at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:342
#1  0x00007f1653cabcb4 in NewRunnableFunction<void (*)(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*), mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*>(void (*)(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*), mozilla::layers::AllocShmemParams*&&, mozilla::ReentrantMonitor*&&, bool*&&) (
    function=0x7f1653c9eb15 <mozilla::layers::ProxyAllocShmemNow(mozilla::layers::AllocShmemParams*, mozilla::ReentrantMonitor*, bool*)>) at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:365
#2  0x00007f1653c9edc2 in mozilla::layers::ImageBridgeChild::DispatchAllocShmemInternal (this=0x7f163a8ef800, 
    aSize=115200, aType=mozilla::ipc::SharedMemory::TYPE_BASIC, aShmem=0x7f163c043e80, aUnsafe=true)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp:1002
#3  0x00007f1653c9ea4f in mozilla::layers::ImageBridgeChild::AllocUnsafeShmem (this=0x7f163a8ef800, aSize=115200, 
    aType=mozilla::ipc::SharedMemory::TYPE_BASIC, aShmem=0x7f163c043e80)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp:933
#4  0x00007f1653b97ed2 in mozilla::layers::BufferTextureData::CreateInternal (aAllocator=0x7f163a8efba0, aDesc=..., 
    aMoz2DBackend=mozilla::gfx::BackendType::NONE, aBufferSize=115200, 
    aTextureFlags=mozilla::layers::TextureFlags::NO_FLAGS) at /home/khuey/dev/gecko-dev/gfx/layers/BufferTexture.cpp:117
#5  0x00007f1653b98193 in mozilla::layers::BufferTextureData::CreateForYCbCr (aAllocator=0x7f163a8efba0, aYSize=..., 
    aCbCrSize=..., aStereoMode=mozilla::StereoMode::MONO, aTextureFlags=mozilla::layers::TextureFlags::NO_FLAGS)
    at /home/khuey/dev/gecko-dev/gfx/layers/BufferTexture.cpp:167
#6  0x00007f1653c2cef7 in mozilla::layers::TextureClient::CreateForYCbCr (aAllocator=0x7f163a8efba0, aYSize=..., 
    aCbCrSize=..., aStereoMode=mozilla::StereoMode::MONO, aTextureFlags=mozilla::layers::TextureFlags::NO_FLAGS)
    at /home/khuey/dev/gecko-dev/gfx/layers/client/TextureClient.cpp:865
#7  0x00007f1653ccd406 in mozilla::layers::SharedPlanarYCbCrImage::Allocate (this=0x7f16240d65c0, aData=...)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/SharedPlanarYCbCrImage.cpp:179
#8  0x00007f1653cccda1 in mozilla::layers::SharedPlanarYCbCrImage::SetData (this=0x7f16240d65c0, aData=...)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/SharedPlanarYCbCrImage.cpp:91
#9  0x00007f16553a9b05 in mozilla::VideoData::SetVideoDataToImage (aVideoImage=0x7f16240d65c0, aInfo=..., aBuffer=..., 
    aPicture=..., aCopyData=true) at /home/khuey/dev/gecko-dev/dom/media/MediaData.cpp:250
#10 0x00007f16553aa165 in mozilla::VideoData::Create (aInfo=..., aContainer=0x7f16243f8740, aImage=0x0, aOffset=28672, 
    aTime=166833, aDuration=33367, aBuffer=..., aKeyframe=false, aTimecode=69, aPicture=...)
    at /home/khuey/dev/gecko-dev/dom/media/MediaData.cpp:354
#11 0x00007f16553aa246 in mozilla::VideoData::Create (aInfo=..., aContainer=0x7f16243f8740, aOffset=28672, aTime=166833, 
    aDuration=33367, aBuffer=..., aKeyframe=false, aTimecode=69, aPicture=...)
    at /home/khuey/dev/gecko-dev/dom/media/MediaData.cpp:388
#12 0x00007f165555f9f9 in mozilla::OggReader::DecodeTheora (this=0x7f16230c6800, aPacket=0x7f16258fa0d0, aTimeThreshold=0)
    at /home/khuey/dev/gecko-dev/dom/media/ogg/OggReader.cpp:880
#13 0x00007f165555fcde in mozilla::OggReader::DecodeVideoFrame (this=0x7f16230c6800, aKeyframeSkip=@0x7f163c0445df: false, 
    aTimeThreshold=0) at /home/khuey/dev/gecko-dev/dom/media/ogg/OggReader.cpp:922
#14 0x00007f16553b47e9 in mozilla::MediaDecoderReader::RequestVideoData (this=0x7f16230c6800, aSkipToNextKeyframe=false, 
    aTimeThreshold=0) at /home/khuey/dev/gecko-dev/dom/media/MediaDecoderReader.cpp:286
#15 0x00007f1655472f14 in mozilla::detail::MethodCallInvokeHelper<RefPtr<mozilla::MozPromise<RefPtr<mozilla::MediaData>, mozilla::MediaDecoderReader::NotDecodedReason, true> >, mozilla::MediaDecoderReader<bool, long>, 0ul, 1ul> (
    aMethod=&virtual mozilla::MediaDecoderReader::RequestVideoData(bool, long), aThisVal=0x7f16230c6800, aArgs=...)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/MozPromise.h:873
#16 0x00007f165546d344 in mozilla::detail::MethodCall<mozilla::MozPromise<RefPtr<mozilla::MediaData>, mozilla::MediaDecoderReader::NotDecodedReason, true>, mozilla::MediaDecoderReader<bool, long> >::Invoke (this=0x7f16231e2550)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/MozPromise.h:898
#17 0x00007f16554626db in mozilla::detail::ProxyRunnable<mozilla::MozPromise<RefPtr<mozilla::MediaData>, mozilla::MediaDecoderReader::NotDecodedReason, true>, mozilla::MediaDecoderReader<bool, long> >::Run (this=0x7f16231e27c0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/MozPromise.h:916
#18 0x00007f16528f2d8c in mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run (this=0x7f162497b1a0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/TaskDispatcher.h:192
#19 0x00007f16528e6946 in mozilla::TaskQueue::Runner::Run (this=0x7f162497b820)
    at /home/khuey/dev/gecko-dev/xpcom/threads/TaskQueue.cpp:170
#20 0x00007f16528f0b27 in nsThreadPool::Run (this=0x7f16318f74f0)
    at /home/khuey/dev/gecko-dev/xpcom/threads/nsThreadPool.cpp:227
#21 0x00007f16528edc22 in nsThread::ProcessNextEvent (this=0x7f16318ac400, aMayWait=true, aResult=0x7f163c04499f)
    at /home/khuey/dev/gecko-dev/xpcom/threads/nsThread.cpp:1018
#22 0x00007f1652953d05 in NS_ProcessNextEvent (aThread=0x7f16318ac400, aMayWait=true)
    at /home/khuey/dev/gecko-dev/xpcom/glue/nsThreadUtils.cpp:297
#23 0x00007f1652eaca34 in mozilla::ipc::MessagePumpForNonMainThreads::Run (this=0x7f1630b36a40, aDelegate=0x7f1630b12360)
    at /home/khuey/dev/gecko-dev/ipc/glue/MessagePump.cpp:355
#24 0x00007f1652e0243d in MessageLoop::RunInternal (this=0x7f1630b12360)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:234
#25 0x00007f1652e023d2 in MessageLoop::RunHandler (this=0x7f1630b12360)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:227
#26 0x00007f1652e02363 in MessageLoop::Run (this=0x7f1630b12360)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:201
#27 0x00007f16528ec186 in nsThread::ThreadFunc (aArg=0x7f16318ac400)
    at /home/khuey/dev/gecko-dev/xpcom/threads/nsThread.cpp:411
#28 0x00007f165cbbf320 in _pt_root (arg=0x7f1632bf90c0) at /home/khuey/dev/gecko-dev/nsprpub/pr/src/pthreads/ptthread.c:216
#29 0x00007f165c5886aa in start_thread (arg=0x7f163c045700) at pthread_create.c:333
#30 0x00007f1650bffeed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109


And deallocation:

#0  0x00007f1653b59751 in RunnableFunction<void (*)(mozilla::layers::TextureClient*), mozilla::Tuple<mozilla::layers::TextureClient*> >::RunnableFunction(void (*)(mozilla::layers::TextureClient*), mozilla::Tuple<mozilla::layers::TextureClient*>&&)
    (this=0x7f1624eff1f0, 
    function=0x7f1653c9cdf2 <mozilla::layers::ReleaseTextureClientNow(mozilla::layers::TextureClient*)>, 
    params=<unknown type in /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/bin/libxul.so, CU 0x4a586fe, DIE 0x4af6b2a>) at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:342
#1  0x00007f1653b58f30 in NewRunnableFunction<void (*)(mozilla::layers::TextureClient*), mozilla::layers::TextureClient*&>
    (function=0x7f1653c9cdf2 <mozilla::layers::ReleaseTextureClientNow(mozilla::layers::TextureClient*)>)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/task.h:365
#2  0x00007f1653c9cf1e in mozilla::layers::ImageBridgeChild::DispatchReleaseTextureClient (aClient=0x7f1624ad12e0)
    at /home/khuey/dev/gecko-dev/gfx/layers/ipc/ImageBridgeChild.cpp:530
#3  0x00007f1653cccb0c in mozilla::layers::SharedPlanarYCbCrImage::~SharedPlanarYCbCrImage (this=0x7f16240d65c0, 
    __in_chrg=<optimized out>) at /home/khuey/dev/gecko-dev/gfx/layers/ipc/SharedPlanarYCbCrImage.cpp:42
#4  0x00007f1653cccbca in mozilla::layers::SharedPlanarYCbCrImage::~SharedPlanarYCbCrImage (this=0x7f16240d65c0, 
    __in_chrg=<optimized out>) at /home/khuey/dev/gecko-dev/gfx/layers/ipc/SharedPlanarYCbCrImage.cpp:48
#5  0x00007f165383b799 in mozilla::layers::Image::Release (this=0x7f16240d65c0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/ImageContainer.h:190
#6  0x00007f165383e097 in RefPtr<mozilla::layers::Image>::AddRefTraitsReleaseHelper (aPtr=0x7f16240d65c0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:362
#7  0x00007f165383d35b in RefPtr<mozilla::layers::Image>::AddRefTraits<mozilla::layers::Image>::Release (
    aPtr=0x7f16240d65c0) at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:372
#8  0x00007f165383c35d in RefPtr<mozilla::layers::Image>::~RefPtr (this=0x7f1622f54d88, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:56
#9  0x00007f16553a9492 in mozilla::VideoData::~VideoData (this=0x7f1622f54d40, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/dom/media/MediaData.cpp:144
#10 0x00007f16553a94d6 in mozilla::VideoData::~VideoData (this=0x7f1622f54d40, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/dom/media/MediaData.cpp:146
#11 0x00007f16527be527 in mozilla::MediaData::Release (this=0x7f1622f54d40)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/MediaData.h:33
#12 0x00007f16554323e8 in RefPtr<mozilla::MediaData>::AddRefTraitsReleaseHelper (aPtr=0x7f1622f54d40)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:362
#13 0x00007f1655419606 in RefPtr<mozilla::MediaData>::AddRefTraits<mozilla::MediaData>::Release (aPtr=0x7f1622f54d40)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:372
#14 0x00007f1655400879 in RefPtr<mozilla::MediaData>::~RefPtr (this=0x7f16249aa0f8, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:56
#15 0x00007f16554754fa in mozilla::detail::TupleImpl<0ul, RefPtr<mozilla::MediaData> >::~TupleImpl (this=0x7f16249aa0f8, 
    __in_chrg=<optimized out>) at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/Tuple.h:106
#16 0x00007f1655475514 in mozilla::Tuple<RefPtr<mozilla::MediaData> >::~Tuple (this=0x7f16249aa0f8, 
    __in_chrg=<optimized out>) at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/Tuple.h:200
#17 0x00007f1655479364 in mozilla::detail::ListenerHelper<mozilla::AbstractThread, mozilla::EnableIf<mozilla::detail::TakeArgs<void (mozilla::MediaDecoderStateMachine::*)(RefPtr<mozilla::MediaData> const&)>::value, mozilla::MediaEventListener>::Type mozilla::MediaEventSourceImpl<(mozilla::ListenerMode)1, RefPtr<mozilla::MediaData> >::ConnectInternal<mozilla::AbstractThread, mozilla::MediaDecoderStateMachine, void (mozilla::MediaDecoderStateMachine::*)(RefPtr<mozilla::MediaData> const&)>(mozilla::AbstractThread*, mozilla::MediaDecoderStateMachine*, void (mozilla::MediaDecoderStateMachine::*)(RefPtr<mozilla::MediaData> const&))::{lambda(RefPtr<mozilla::MediaData>&&)#1}>::R<RefPtr<mozilla::MediaData> const&>::~R() (this=0x7f16249aa0c0, 
    __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/MediaEventSource.h:136
#18 0x00007f16554793b8 in mozilla::detail::ListenerHelper<mozilla::AbstractThread, mozilla::EnableIf<mozilla::detail::TakeArgs<void (mozilla::MediaDecoderStateMachine::*)(RefPtr<mozilla::MediaData> const&)>::value, mozilla::MediaEventListener>::Type mozilla::MediaEventSourceImpl<(mozilla::ListenerMode)1, RefPtr<mozilla::MediaData> >::ConnectInternal<mozilla::AbstractThread, mozilla::MediaDecoderStateMachine, void (mozilla::MediaDecoderStateMachine::*)(RefPtr<mozilla::MediaData> const&)>(mozilla::AbstractThread*, mozilla::MediaDecoderStateMachine*, void (mozilla::MediaDecoderStateMachine::*)(RefPtr<mozilla::MediaData> const&))::{lambda(RefPtr<mozilla::MediaData>&&)#1}>::R<RefPtr<mozilla::MediaData> const&>::~R() (this=0x7f16249aa0c0, 
    __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/MediaEventSource.h:136
#19 0x00007f165295344a in nsRunnable::Release (this=0x7f16249aa0c0)
    at /home/khuey/dev/gecko-dev/xpcom/glue/nsThreadUtils.cpp:35
#20 0x00007f165283c041 in nsCOMPtr<nsIRunnable>::~nsCOMPtr (this=0x7f16249778a8, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/nsCOMPtr.h:405
#21 0x00007f165284b736 in nsTArrayElementTraits<nsCOMPtr<nsIRunnable> >::Destruct (aE=0x7f16249778a8)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/nsTArray.h:523
#22 0x00007f16528493b2 in nsTArray_Impl<nsCOMPtr<nsIRunnable>, nsTArrayInfallibleAllocator>::DestructRange (
    this=0x7f1624fd1870, aStart=0, aCount=8)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/nsTArray.h:2014
#23 0x00007f1652844f57 in nsTArray_Impl<nsCOMPtr<nsIRunnable>, nsTArrayInfallibleAllocator>::RemoveElementsAt (
    this=0x7f1624fd1870, aStart=0, aCount=8)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/nsTArray.h:1656
#24 0x00007f165283eaa0 in nsTArray_Impl<nsCOMPtr<nsIRunnable>, nsTArrayInfallibleAllocator>::Clear (this=0x7f1624fd1870)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/nsTArray.h:1666
#25 0x00007f165283d620 in nsTArray_Impl<nsCOMPtr<nsIRunnable>, nsTArrayInfallibleAllocator>::~nsTArray_Impl (
    this=0x7f1624fd1870, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/nsTArray.h:828
#26 0x00007f1652836258 in nsTArray<nsCOMPtr<nsIRunnable> >::~nsTArray (this=0x7f1624fd1870, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/nsTArray.h:2088
#27 0x00007f16528f2c56 in mozilla::AutoTaskDispatcher::PerThreadTaskGroup::~PerThreadTaskGroup (this=0x7f1624fd1860, 
    __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/TaskDispatcher.h:164
#28 0x00007f1652902ec6 in mozilla::DefaultDelete<mozilla::AutoTaskDispatcher::PerThreadTaskGroup>::operator() (
    this=0x7f1624fd19f8, aPtr=0x7f1624fd1860)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/UniquePtr.h:482
#29 0x00007f16528fd9ea in mozilla::UniquePtr<mozilla::AutoTaskDispatcher::PerThreadTaskGroup, mozilla::DefaultDelete<mozilla::AutoTaskDispatcher::PerThreadTaskGroup> >::reset (this=0x7f1624fd19f8, aPtr=0x0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/UniquePtr.h:309
#30 0x00007f16528f64e1 in mozilla::UniquePtr<mozilla::AutoTaskDispatcher::PerThreadTaskGroup, mozilla::DefaultDelete<mozilla::AutoTaskDispatcher::PerThreadTaskGroup> >::~UniquePtr (this=0x7f1624fd19f8, __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/UniquePtr.h:254
#31 0x00007f165290a186 in mozilla::AutoTaskDispatcher::TaskGroupRunnable::~TaskGroupRunnable (this=0x7f1624fd19e0, 
    __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/TaskDispatcher.h:172
#32 0x00007f165290a1ca in mozilla::AutoTaskDispatcher::TaskGroupRunnable::~TaskGroupRunnable (this=0x7f1624fd19e0, 
    __in_chrg=<optimized out>)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/TaskDispatcher.h:172
#33 0x00007f165295344a in nsRunnable::Release (this=0x7f1624fd19e0)
    at /home/khuey/dev/gecko-dev/xpcom/glue/nsThreadUtils.cpp:35
#34 0x00007f16529051a0 in RefPtr<nsIRunnable>::AddRefTraitsReleaseHelper (aPtr=0x7f1624fd19e0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:362
#35 0x00007f1652901467 in RefPtr<nsIRunnable>::AddRefTraits<nsIRunnable>::Release (aPtr=0x7f1624fd19e0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:372
#36 0x00007f165290164d in RefPtr<nsIRunnable>::assign_assuming_AddRef (this=0x7f1621cb9760, aNewPtr=0x0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:43
#37 0x00007f1652901698 in RefPtr<nsIRunnable>::assign_with_AddRef (this=0x7f1621cb9760, aRawPtr=0x0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:34
#38 0x00007f1#39 0x00007f16528e6969 in mozilla::TaskQueue::Runner::Run (this=0x7f1624ebfc20)
    at /home/khuey/dev/gecko-dev/xpcom/threads/TaskQueue.cpp:178
#39 0x00007f16528e6969 in mozilla::TaskQueue::Runner::Run (this=0x7f1624ebfc20)
    at /home/khuey/dev/gecko-dev/xpcom/threads/TaskQueue.cpp:178
#40 0x00007f16528f0b27 in nsThreadPool::Run (this=0x7f16318f74f0)
    at /home/khuey/dev/gecko-dev/xpcom/threads/nsThreadPool.cpp:227
#41 0x00007f16528edc22 in nsThread::ProcessNextEvent (this=0x7f1625333400, aMayWait=true, aResult=0x7f1621cb999f)
    at /home/khuey/dev/gecko-dev/xpcom/threads/nsThread.cpp:1018
#42 0x00007f1652953d05 in NS_ProcessNextEvent (aThread=0x7f1625333400, aMayWait=true)
    at /home/khuey/dev/gecko-dev/xpcom/glue/nsThreadUtils.cpp:297
#43 0x00007f1652eaca34 in mozilla::ipc::MessagePumpForNonMainThreads::Run (this=0x7f1629190140, aDelegate=0x7f1626acbcc0)
    at /home/khuey/dev/gecko-dev/ipc/glue/MessagePump.cpp:355
#44 0x00007f1652e0243d in MessageLoop::RunInternal (this=0x7f1626acbcc0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:234
#45 0x00007f1652e023d2 in MessageLoop::RunHandler (this=0x7f1626acbcc0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:227
#46 0x00007f1652e02363 in MessageLoop::Run (this=0x7f1626acbcc0)
    at /home/khuey/dev/gecko-dev/ipc/chromium/src/base/message_loop.cc:201
#47 0x00007f16528ec186 in nsThread::ThreadFunc (aArg=0x7f1625333400)
    at /home/khuey/dev/gecko-dev/xpcom/threads/nsThread.cpp:411
#48 0x00007f165cbbf320 in _pt_root (arg=0x7f1632bf99c0) at /home/khuey/dev/gecko-dev/nsprpub/pr/src/pthreads/ptthread.c:216
#49 0x00007f165c5886aa in start_thread (arg=0x7f1621cba700) at pthread_create.c:333
#50 0x00007f1650bffeed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
6528fb59b in RefPtr<nsIRunnable>::operator= (this=0x7f1621cb9760, aRhs=0x0)
    at /home/khuey/dev/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/include/mozilla/RefPtr.h:153

Comment 10

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

So I guess the fundamental question here is "what's supposed to keep the Shmem alive until the parent reads it, and why did it fail?"  If someone can answer the first part of that I can poke through the recording and answer the second.

ni cpearce.

Comment 11

[Intermittent Failures Robot]

47 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 40
* fx-team: 5
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 36
* linux32: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1247429&startday=2016-02-08&endday=2016-02-14&tree=all

Comment 12

[Intermittent Failures Robot]

25 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 16
* fx-team: 9

Platform breakdown:
* linux64: 20
* linux32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1247429&startday=2016-02-15&endday=2016-02-15&tree=all

Comment 13

[Nicolas Silva [:nical]]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #7)
> I have this in an rr recording.  The Shmem is created on the child and sent
> to the parent, and then deallocated before the parent actually gets around
> to receiving it.  Deallocating sets the size contained within the Shmem
> segment itself to zero, which trips the assertion when the parent finally
> does receive the message and attempt to sanity check things.

Yes, every instance of this assertion I have come across is about the shmem being created and destroyed on the content side before the compositor receives the shmem. So far it's never been an issue because this happens to shmems that we never actually use on the compositor process (by definition), and the shmem code is robust enough to not crash when this race happens.

This highlights two potential problems:
- Why would we allocate shmems and never use them on the compositor ? sounds like a waste. We looked at this :jesup and :pehrsons, and that comes from some contrived situations in the media pipeline which are uncommon in real world use cases, so I think that for now we can ignore that. We need to store video frames into shared memory as early as possible to avoid expensive copies and it is very hard for the media code to predict whether it will have to drop the frame for X or Y reason. I consider this to not really be a problem.
- The way IPDL handles shmems internally doesn't give us any way to prevent this race (I can go into more detail if anyone is interested). I plan to make gfx textures use manually managed shmems instead of the automatically managed stuff that ipdl provides (but which inherently has this race), since we re-do all that lifetime management at a higher level because we have to do it for the non-shmem resources that sit under the same abstraction. That would fix the most common occurrences of this assertion among other things but I (and I think we all) have more important and lower-hanging fruits to pick at this point.

The recent spike comes from the fact that we switched some code paths from using xlib surfaces into using shmems instead so we use the latter more than we use to.

So: in the shorter term I suggest we just turn this assertion into a printf_stderr and move on to more important stuff since this hasn't affected stability nor correctness as far as I know.

Comment 14

[Lee Salzman [:lsalzman]]

Attachment: Warn instead of error if shmem deallocated before IPDL sends it

Comment 15

[Lee Salzman [:lsalzman]]

Did a bunch of retriggers on try just to make sure it was properly showing up as a warning now and not causing oranges: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a191d589b26b

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/221698ba7d33

Comment 17

[Intermittent Failures Robot]

22 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 10
* fx-team: 7
* try: 5

Platform breakdown:
* linux64: 18
* linux32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1247429&startday=2016-02-16&endday=2016-02-16&tree=all

Comment 18

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/221698ba7d33

Comment 19

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Nicolas Silva [:nical] (PTO until March 7th) from comment #13)
> So: in the shorter term I suggest we just turn this assertion into a
> printf_stderr and move on to more important stuff since this hasn't affected
> stability nor correctness as far as I know.

I'll clear the ni? then.

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/3a34e8b89130