Closed Bug 1240622 Opened 5 years ago Closed 5 years ago

Intermittent runner.py,automation.py | application crashed [@ mozilla::layers::CheckerboardEvent::UpdateRendertraceProperty(mozilla::layers::CheckerboardEvent::RendertraceProperty,mozilla::gfx::RectTyped<mozilla::CSSPixel,float>

Categories

(Core :: Panning and Zooming, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla46
Tracking Status
firefox46 --- fixed

People

(Reporter: aryx, Assigned: kats)

References

Details

(Keywords: crash, intermittent-failure)

Crash Data

Attachments

(1 file, 1 obsolete file)

https://treeherder.mozilla.org/logviewer.html#?job_id=20002643&repo=mozilla-inbound

13:36:11     INFO -  TEST-START | test_desktop_all.py TestDesktopUnits.test_units
13:36:16     INFO -  mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/E5WLtuLpREyQdgyobAJbkA/artifacts/public/build/firefox-46.0a1.en-US.win64.crashreporter-symbols.zip
13:36:20     INFO -  mozcrash Copy/paste: C:\slave\test\build\win32-minidump_stackwalk.exe c:\users\cltbld~1.t-w\appdata\local\temp\tmpbrzh4a.mozrunner\minidumps\3bbbd1eb-41da-484d-afb3-4504acc693a5.dmp c:\users\cltbld~1.t-w\appdata\local\temp\tmp1euft2
13:36:25     INFO -  mozcrash Saved minidump as C:\slave\test\build\blobber_upload_dir\3bbbd1eb-41da-484d-afb3-4504acc693a5.dmp
13:36:25     INFO -  mozcrash Saved app info as C:\slave\test\build\blobber_upload_dir\3bbbd1eb-41da-484d-afb3-4504acc693a5.extra
13:36:25    ERROR -  PROCESS-CRASH | runner.py | application crashed [@ mozilla::layers::CheckerboardEvent::UpdateRendertraceProperty(mozilla::layers::CheckerboardEvent::RendertraceProperty,mozilla::gfx::RectTyped<mozilla::CSSPixel,float> const &,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)]
13:36:25     INFO -  Crash dump filename: c:\users\cltbld~1.t-w\appdata\local\temp\tmpbrzh4a.mozrunner\minidumps\3bbbd1eb-41da-484d-afb3-4504acc693a5.dmp
13:36:25     INFO -  Operating system: Windows NT
13:36:25     INFO -                    6.2.9200
13:36:25     INFO -  CPU: amd64
13:36:25     INFO -       family 6 model 30 stepping 5
13:36:25     INFO -       8 CPUs
13:36:25     INFO -  Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
13:36:25     INFO -  Crash address: 0x80
13:36:25     INFO -  Process uptime: 4 seconds
13:36:25     INFO -  Thread 0 (crashed)
13:36:25     INFO -   0  xul.dll!mozilla::layers::CheckerboardEvent::UpdateRendertraceProperty(mozilla::layers::CheckerboardEvent::RendertraceProperty,mozilla::gfx::RectTyped<mozilla::CSSPixel,float> const &,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &) [CheckerboardEvent.cpp:d7c04b4c6cff : 64 + 0x0]
13:36:25     INFO -      rax = 0x000000251f7fec08   rdx = 0x0000000000000003
13:36:25     INFO -      rcx = 0x0000000000000000   rbx = 0x0000000000000000
13:36:25     INFO -      rsi = 0x0000002538474c40   rdi = 0x000000252ed8e058
13:36:25     INFO -      rbp = 0x000000251f7fece0   rsp = 0x000000251f7feb80
13:36:25     INFO -       r8 = 0x000000251f7fec08    r9 = 0x000000251f7fed20
13:36:25     INFO -      r10 = 0x000000251f7fed20   r11 = 0x000000251f7fed20
13:36:25     INFO -      r12 = 0x0000000000000001   r13 = 0x0000000000000001
13:36:25     INFO -      r14 = 0x000000251f7fede8   r15 = 0x0000000000000000
13:36:25     INFO -      rip = 0x000007fec77e02af
13:36:25     INFO -      Found by: given as instruction pointer in context
13:36:25     INFO -   1  xul.dll!mozilla::layers::AsyncPanZoomController::RequestContentRepaint(mozilla::layers::FrameMetrics const &,mozilla::gfx::PointTyped<mozilla::ParentLayerPixel,float> const &) [AsyncPanZoomController.cpp:d7c04b4c6cff : 2880 + 0x25]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7febe0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec77dc283
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   2  xul.dll!mozilla::layers::AsyncPanZoomController::RequestContentRepaint() [AsyncPanZoomController.cpp:d7c04b4c6cff : 2827 + 0x5]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7fed90   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec77dc3bb
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   3  xul.dll!nsRunnableMethodImpl<void ( mozilla::net::ChannelEventQueue::*)(void),1>::Run() [nsThreadUtils.h:d7c04b4c6cff : 870 + 0x3]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7fede0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec72d7d97
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   4  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:d7c04b4c6cff : 997 + 0x9]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7fee10   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec710a980
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   5  xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:d7c04b4c6cff : 297 + 0xd]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7fefe0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec71276a3
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   6  xul.dll!nsThread::Shutdown() [nsThread.cpp:d7c04b4c6cff : 807 + 0xb]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff010   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec710dd6f
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   7  xul.dll!nsRunnableMethodImpl<void ( mozilla::net::ChannelEventQueue::*)(void),1>::Run() [nsThreadUtils.h:d7c04b4c6cff : 870 + 0x3]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff040   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec72d7d97
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   8  xul.dll!nsThread::ProcessNextEvent(bool,bool *) [nsThread.cpp:d7c04b4c6cff : 997 + 0x9]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff070   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec710a980
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -   9  xul.dll!NS_ProcessNextEvent(nsIThread *,bool) [nsThreadUtils.cpp:d7c04b4c6cff : 297 + 0xd]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff240   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec71276a3
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  10  xul.dll!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) [MessagePump.cpp:d7c04b4c6cff : 95 + 0xb]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff270   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec735d9c0
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  11  xul.dll!MessageLoop::RunHandler() [message_loop.cc:d7c04b4c6cff : 227 + 0xf]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff2c0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec734876b
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  12  xul.dll!MessageLoop::Run() [message_loop.cc:d7c04b4c6cff : 201 + 0x8]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff2f0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec7348536
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  13  xul.dll!nsBaseAppShell::Run() [nsBaseAppShell.cpp:d7c04b4c6cff : 156 + 0xd]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff340   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec846f7bb
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  14  xul.dll!nsAppShell::Run() [nsAppShell.cpp:d7c04b4c6cff : 257 + 0x8]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff370   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec84b91e5
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  15  xul.dll!nsAppStartup::Run() [nsAppStartup.cpp:d7c04b4c6cff : 281 + 0xa]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff3a0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec8aefc4f
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  16  xul.dll!XREMain::XRE_mainRun() [nsAppRunner.cpp:d7c04b4c6cff : 4326 + 0xa]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff3d0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec8b2ac06
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  17  xul.dll!XREMain::XRE_main(int,char * * const,nsXREAppData const *) [nsAppRunner.cpp:d7c04b4c6cff : 4423 + 0x8]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff660   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec8b29281
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  18  xul.dll!XRE_main [nsAppRunner.cpp:d7c04b4c6cff : 4525 + 0x12]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff6e0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fec8b2b675
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  19  firefox.exe!do_main [nsBrowserApp.cpp:d7c04b4c6cff : 212 + 0x18]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ff880   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007f619511a12
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  20  firefox.exe!NS_internal_main(int,char * *) [nsBrowserApp.cpp:d7c04b4c6cff : 352 + 0xe]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ffaa0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007f619511666
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  21  firefox.exe!wmain [nsWindowsWMain.cpp:d7c04b4c6cff : 131 + 0xb]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ffb70   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007f619511e2e
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  22  firefox.exe!__tmainCRTStartup [crt0.c : 255 + 0x12]
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ffbe0   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007f619512c7c
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  23  kernel32.dll!BaseThreadInitThunk + 0x1a
13:36:25     INFO -      rbx = 0x0000000000000000   rbp = 0x000000251f7fece0
13:36:25     INFO -      rsp = 0x000000251f7ffc20   r12 = 0x0000000000000001
13:36:25     INFO -      r13 = 0x0000000000000001   r14 = 0x000000251f7fede8
13:36:25     INFO -      r15 = 0x0000000000000000   rip = 0x000007fee51f167e
13:36:25     INFO -      Found by: call frame info
13:36:25     INFO -  24  ntdll.dll!RtlUserThreadStart + 0x21
13:36:25     INFO -      rsp = 0x000000251f7ffc50   rip = 0x000007fee77cc3f1
13:36:25     INFO -      Found by: stack scanning
13:36:25     INFO -  25  KERNELBASE.dll!GetLegacyComposition + 0x1180
13:36:25     INFO -      rsp = 0x000000251f7ffc80   rip = 0x000007fee48809d0
13:36:25     INFO -      Found by: stack scanning
Assignee: nobody → bugmail.mozilla
Component: Graphics: Layers → Panning and Zooming
I guess mCheckerboardEvent is accessed from multiple threads unsynchronized. Really small window of opportunity for a crash like this, but I guess it does happen.
https://treeherder.mozilla.org/logviewer.html#?job_id=20034207&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=20034210&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=20064685&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=20075614&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=20068211&repo=mozilla-inbound
Summary: Intermittent runner.py | application crashed [@ mozilla::layers::CheckerboardEvent::UpdateRendertraceProperty(mozilla::layers::CheckerboardEvent::RendertraceProperty,mozilla::gfx::RectTyped<mozilla::CSSPixel,float> const &,std::basic_string<char,std::char → Intermittent runner.py,automation.py | application crashed [@ mozilla::layers::CheckerboardEvent::UpdateRendertraceProperty(mozilla::layers::CheckerboardEvent::RendertraceProperty,mozilla::gfx::RectTyped<mozilla::CSSPixel,float>
Oh, fun, we blacklist automation.py as a filename to search for bugs with, so these will only be starred manually, by people who already know the bug exists.
Attached patch PatchSplinter Review
Now without the deadlock possibility. Since windows jobs are severely backlogged it's probably better to just land this and see if it fixes the problem than wait for try retriggers to confirm.
Attachment #8710523 - Attachment is obsolete: true
Attachment #8710735 - Flags: review?(botond)
Comment on attachment 8710735 [details] [diff] [review]
Patch

Review of attachment 8710735 [details] [diff] [review]:
-----------------------------------------------------------------

::: gfx/layers/apz/src/AsyncPanZoomController.cpp
@@ +863,5 @@
>       mInputQueue(aInputQueue),
>       mAPZCId(sAsyncPanZoomControllerCount++),
>       mSharedLock(nullptr),
> +     mAsyncTransformAppliedToContent(false),
> +     mCheckerboardEventLock("CBELock")

Please prefix the name with "APZ", so that if it shows up somewhere, it's easier to identify where it comes from.

::: gfx/layers/apz/src/AsyncPanZoomController.h
@@ +1102,5 @@
>     * recording.
>     */
>  private:
> +  // Mutex protecting mCheckerboardEvent
> +  Mutex mCheckerboardEventLock;

Aside: should we be using Mutex instead of Monitor for APZCTreeManager::mTreeLock, too?
Attachment #8710735 - Flags: review?(botond) → review+
(In reply to Botond Ballo [:botond] from comment #7)
> 
> Please prefix the name with "APZ", so that if it shows up somewhere, it's
> easier to identify where it comes from.
> 

Will do

> > +  Mutex mCheckerboardEventLock;
> 
> Aside: should we be using Mutex instead of Monitor for
> APZCTreeManager::mTreeLock, too?

Yeah, probably. I can file a follow-up for it.
(Filed bug 1241991 as the follow-up)
https://hg.mozilla.org/mozilla-central/rev/f3ad9269e053
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
You need to log in before you can comment on or make changes to this bug.