Closed Bug 1166432 Opened 5 years ago Closed 4 years ago

Intermittent test_capture.html,test_peerConnection_captureStream_canvas_2d.html | application crashed [@ nvwgf2umx.dll + 0xef889][@ nvwgf2umx.dll + 0x17ec5]

Categories

(Core :: Graphics: Layers, defect)

x86_64
Windows 8
defect
Not set

Tracking

()

RESOLVED FIXED
mozilla42
Tracking Status
firefox40 --- unaffected
firefox41 --- affected
firefox42 --- fixed
firefox-esr38 --- unaffected

People

(Reporter: RyanVM, Assigned: BenWa)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: [gfx-noted])

Attachments

(4 files, 2 obsolete files)

08:04:36 WARNING - PROCESS-CRASH | dom/media/tests/mochitest/test_peerConnection_captureStream_canvas_2d.html | application crashed [@ nvwgf2umx.dll + 0xef889]
08:04:36 INFO - Crash dump filename: c:\users\cltbld~1.t-w\appdata\local\temp\tmpexfpwt.mozrunner\minidumps\d216daf1-61f0-4c8d-9219-7b980c44dc14.dmp
08:04:36 INFO - Operating system: Windows NT
08:04:36 INFO - 6.2.9200
08:04:36 INFO - CPU: amd64
08:04:36 INFO - family 6 model 30 stepping 5
08:04:36 INFO - 8 CPUs
08:04:36 INFO - Crash reason: EXCEPTION_ACCESS_VIOLATION_READ
08:04:36 INFO - Crash address: 0x4
08:04:36 INFO - Thread 28 (crashed)
08:04:36 INFO - 0 nvwgf2umx.dll + 0xef889
08:04:36 INFO - rbx = 0x0000000000000000 r12 = 0x0000000000000000
08:04:36 INFO - r13 = 0x0000000000000000 r14 = 0x0000000000000000
08:04:36 INFO - r15 = 0x000000845c5a7a58 rip = 0x000007ff1456f889
08:04:36 INFO - rsp = 0x000000845c14f9b0 rbp = 0x0000000000000000
08:04:36 INFO - Found by: given as instruction pointer in context
08:04:36 INFO - 1 nvwgf2umx.dll + 0x2d0736
08:04:36 INFO - rip = 0x000007ff14750737 rsp = 0x000000845c14f9c0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 2 nvwgf2umx.dll + 0x2c07af
08:04:36 INFO - rip = 0x000007ff147407b0 rsp = 0x000000845c14f9f0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 3 nvwgf2umx.dll + 0xaeafaf
08:04:36 INFO - rip = 0x000007ff14f6afb0 rsp = 0x000000845c14fa60
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 4 nvwgf2umx.dll + 0x33f4de
08:04:36 INFO - rip = 0x000007ff147bf4df rsp = 0x000000845c14faf0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 5 nvwgf2umx.dll + 0x193ff
08:04:36 INFO - rip = 0x000007ff14499400 rsp = 0x000000845c14fb50
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 6 nvwgf2umx.dll + 0xaeafaf
08:04:36 INFO - rip = 0x000007ff14f6afb0 rsp = 0x000000845c14fb68
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 7 nvwgf2umx.dll + 0x3fc8df
08:04:36 INFO - rip = 0x000007ff1487c8e0 rsp = 0x000000845c14fb90
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 8 nvwgf2umx.dll + 0xaeafaf
08:04:36 INFO - rip = 0x000007ff14f6afb0 rsp = 0x000000845c14fbc8
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 9 nvwgf2umx.dll + 0xb023f
08:04:36 INFO - rip = 0x000007ff14530240 rsp = 0x000000845c14fc00
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 10 nvwgf2umx.dll + 0x2cf1ae
08:04:36 INFO - rip = 0x000007ff1474f1af rsp = 0x000000845c14fc10
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 11 nvwgf2umx.dll + 0xadbe3
08:04:36 INFO - rip = 0x000007ff1452dbe4 rsp = 0x000000845c14fc70
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 12 nvwgf2umx.dll + 0xa9c4c
08:04:36 INFO - rip = 0x000007ff14529c4d rsp = 0x000000845c14fca0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 13 nvwgf2umx.dll + 0x1d5dc9
08:04:36 INFO - rip = 0x000007ff14655dca rsp = 0x000000845c14fcd0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 14 nvwgf2umx.dll + 0x9e02c6
08:04:36 INFO - rip = 0x000007ff14e602c7 rsp = 0x000000845c14fd00
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 15 nvwgf2umx.dll + 0x9e046d
08:04:36 INFO - rip = 0x000007ff14e6046e rsp = 0x000000845c14fd30
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 16 kernel32.dll + 0x167d
08:04:36 INFO - rip = 0x000007ff1b8d167e rsp = 0x000000845c14fd60
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 17 ntdll.dll + 0x1c3f0
08:04:36 INFO - rip = 0x000007ff1bf3c3f1 rsp = 0x000000845c14fd90
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 18 KERNELBASE.dll + 0x809cf
08:04:36 INFO - rip = 0x000007ff192b09d0 rsp = 0x000000845c14fdc0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - Thread 0
08:04:36 INFO - 0 ntdll.dll + 0x2bea
08:04:36 INFO - rbx = 0x000000845c5944d0 r12 = 0x0000000000000000
08:04:36 INFO - r13 = 0x0000000000000000 r14 = 0x000007f72ad7e000
08:04:36 INFO - r15 = 0x000000007ffe0382 rip = 0x000007ff1bf22bea
08:04:36 INFO - rsp = 0x000000844ebfdf28 rbp = 0x0000000000000000
08:04:36 INFO - Found by: given as instruction pointer in context
08:04:36 INFO - 1 ntdll.dll + 0x19d29
08:04:36 INFO - rip = 0x000007ff1bf39d2a rsp = 0x000000844ebfdf30
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 2 KERNELBASE.dll + 0x1a21
08:04:36 INFO - rip = 0x000007ff19231a22 rsp = 0x000000844ebfdf60
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 3 nvwgf2umx.dll + 0xaef84
08:04:36 INFO - rip = 0x000007ff1452ef85 rsp = 0x000000844ebfdf90
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 4 nvwgf2umx.dll + 0xb7c54
08:04:36 INFO - rip = 0x000007ff14537c55 rsp = 0x000000844ebfdfc0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 5 ntdll.dll + 0x103b3
08:04:36 INFO - rip = 0x000007ff1bf303b4 rsp = 0x000000844ebfe000
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 6 d3d11.dll + 0x4884c
08:04:36 INFO - rip = 0x000007ff15a0884d rsp = 0x000000844ebfe040
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 7 d3d11.dll + 0x5bb5b
08:04:36 INFO - rip = 0x000007ff15a1bb5c rsp = 0x000000844ebfe070
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 8 d3d11.dll + 0x3dc6c
08:04:36 INFO - rip = 0x000007ff159fdc6d rsp = 0x000000844ebfe090
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 9 d3d11.dll + 0x16219f
08:04:36 INFO - rip = 0x000007ff15b221a0 rsp = 0x000000844ebfe0d0
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 10 d3d11.dll + 0x5ba90
08:04:36 INFO - rip = 0x000007ff15a1ba91 rsp = 0x000000844ebfe120
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 11 xul.dll!mozilla::layers::SyncObjectD3D11::FinalizeFrame() [TextureD3D11.cpp:7d119f9b1e10 : 1116 + 0xc]
08:04:36 INFO - rip = 0x000007fef9ea6024 rsp = 0x000000844ebfe150
08:04:36 INFO - Found by: stack scanning
08:04:36 INFO - 12 xul.dll!mozilla::layers::ClientLayerManager::ForwardTransaction(bool) [ClientLayerManager.cpp:7d119f9b1e10 : 550 + 0x5]
08:04:36 INFO - rip = 0x000007fef9e777a1 rsp = 0x000000844ebfe6a0
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 13 xul.dll!mozilla::layers::ClientLayerManager::EndEmptyTransaction(mozilla::layers::LayerManager::EndTransactionFlags) [ClientLayerManager.cpp:7d119f9b1e10 : 353 + 0x14]
08:04:36 INFO - rip = 0x000007fef9e763d9 rsp = 0x000000844ebfe960
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 14 xul.dll!PresShell::Paint(nsView *,nsRegion const &,unsigned int) [nsPresShell.cpp:7d119f9b1e10 : 6329 + 0x10]
08:04:36 INFO - rip = 0x000007fefba56f5e rsp = 0x000000844ebfe990
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 15 xul.dll!nsViewManager::ProcessPendingUpdatesPaint(nsIWidget *) [nsViewManager.cpp:7d119f9b1e10 : 456 + 0x24]
08:04:36 INFO - rip = 0x000007fefb6b0260 rsp = 0x000000844ebfeb10
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 16 xul.dll!nsViewManager::ProcessPendingUpdatesForView(nsView *,bool) [nsViewManager.cpp:7d119f9b1e10 : 396 + 0xb]
08:04:36 INFO - rip = 0x000007fefb6b009b rsp = 0x000000844ebfeb70
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 17 xul.dll!nsViewManager::ProcessPendingUpdates() [nsViewManager.cpp:7d119f9b1e10 : 1086 + 0xe]
08:04:36 INFO - rip = 0x000007fefb6afef2 rsp = 0x000000844ebfebe0
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 18 xul.dll!nsRefreshDriver::Tick(__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:7d119f9b1e10 : 1776 + 0x2e]
08:04:36 INFO - rip = 0x000007fefb97656e rsp = 0x000000844ebfec10
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 19 xul.dll!mozilla::RefreshDriverTimer::TickDriver(nsRefreshDriver *,__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:7d119f9b1e10 : 194 + 0x22]
08:04:36 INFO - rip = 0x000007fefb9767d3 rsp = 0x000000844ebfef80
08:04:36 INFO - Found by: call frame info
08:04:36 INFO - 20 xul.dll!mozilla::RefreshDriverTimer::Tick(__int64,mozilla::TimeStamp) [nsRefreshDriver.cpp:7d119f9b1e10 : 185 + 0x2d]
08:04:36 INFO - rip = 0x000007fefb975a02 rsp = 0x000000844ebfefd0
08:04:36 INFO - Found by: call frame info
This and bug 1165928 seem very-likely related, and both are spiking at the moment. Can we get a Windows gfx person to look into this please?
Flags: needinfo?(milan)
See Also: → 1165928
Summary: Intermittent test_peerConnection_captureStream_canvas_2d.html | application crashed [@ nvwgf2umx.dll + 0xef889] → Intermittent test_capture.html,test_peerConnection_captureStream_canvas_2d.html | application crashed [@ nvwgf2umx.dll + 0xef889][@ nvwgf2umx.dll + 0x17ec5]
Andrew, Kyle, do you think this could be from your recent changes?
Flags: needinfo?(milan)
Flags: needinfo?(kfung)
Flags: needinfo?(acomminos)
(In reply to Milan Sreckovic [:milan] from comment #5)
> Andrew, Kyle, do you think this could be from your recent changes?

Probably not. The changes we made are only applicable to machines running Windows 8.1, which I don't believe any of the try servers use.
Flags: needinfo?(acomminos)
Flags: needinfo?(kfung)
I thought D2D1.1 is available on Windows 7 machines with an update?  Can you take a quick look anyway?  Maybe send to a try with a backed out patch for bug 1161642 and see if it makes a difference?  It did start on the same day as that bug landed, which may very well be a coincidence, but it's worth checking.
The primitive blends feature is still only available on Windows 8.1. We do a version check here:
https://dxr.mozilla.org/mozilla-central/source/gfx/2d/HelpersD2D.h#284

Here's a try push with the change reverted:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=20552b9708e9
Looking closer, Jeff, is this in the window manager?  Could it be related to the whale work?
Flags: needinfo?(jmuizelaar)
And adding Bas as well, as this is in FinalizeFrame, and maybe can help with bug 1160157?
Flags: needinfo?(bas)
(In reply to Milan Sreckovic [:milan] from comment #9)
> Looking closer, Jeff, is this in the window manager?  Could it be related to
> the whale work?

What did you see that prompted this question?
Flags: needinfo?(jmuizelaar) → needinfo?(milan)
(In reply to Jeff Muizelaar [:jrmuizel] from comment #11)
> (In reply to Milan Sreckovic [:milan] from comment #9)
> > Looking closer, Jeff, is this in the window manager?  Could it be related to
> > the whale work?
> 
> What did you see that prompted this question?

Random thoughts and shots in the dark and reports somebody had dwm crashing in this driver after a windows update.  So, nothing solid.
Flags: needinfo?(milan)
Whiteboard: [gfx-noted]
This was a test added by pehrsons (taking canvas data and pushing to a MediaStream).  Though the crash certainly seems to be a GFX crash...

Any traction here?  Still failing ~4 times/day
Flags: needinfo?(milan)
My tests landed around the time this started crashing. Might just be that they're triggering something old.
This patch seems to have upped the repro rate to permorange: https://treeherder.mozilla.org/#/jobs?repo=try&revision=470227546678
(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #75)
> This patch seems to have upped the repro rate to permorange:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=470227546678

Milan/Bas/Jeff - this patch should make it a lot easier to debug!
Flags: needinfo?(jmuizelaar)
Jeff's patch from comment 86 https://treeherder.mozilla.org/#/jobs?repo=try&revision=14f1b0b5402c

Benoit, can you follow up?
Flags: needinfo?(milan)
Flags: needinfo?(jmuizelaar)
Flags: needinfo?(bgirard)
Flags: needinfo?(bas)
Assignee: nobody → bgirard
Comment on attachment 8617340 [details] [diff] [review]
Here's a patch that might help debug this

Review of attachment 8617340 [details] [diff] [review]:
-----------------------------------------------------------------

https://treeherder.mozilla.org/#/jobs?repo=try&revision=14f1b0b5402c
Attachment #8617340 - Flags: review+
This looks green, we should land it and see if it helps with the intermittents.
Comment on attachment 8617340 [details] [diff] [review]
Here's a patch that might help debug this

Review of attachment 8617340 [details] [diff] [review]:
-----------------------------------------------------------------

::: gfx/2d/2D.h
@@ +388,5 @@
>    struct MappedSurface {
>      uint8_t *mData;
>      int32_t mStride;
> +#ifdef DEBUG
> +    RefPtr<DataSourceSurface> mSrc;

Am I missing some magic or is mSrc never set?
I don't see it set, but there could be magic.
Attached patch IndeedSplinter Review
Attachment #8617340 - Attachment is obsolete: true
Comment on attachment 8621251 [details] [diff] [review]
Indeed

Review of attachment 8621251 [details] [diff] [review]:
-----------------------------------------------------------------

::: gfx/2d/2D.h
@@ +469,5 @@
>    virtual bool Map(MapType, MappedSurface *aMappedSurface)
>    {
>      aMappedSurface->mData = GetData();
>      aMappedSurface->mStride = Stride();
> +    aMappedSurface->mSrc = this;

I imagine inside of #ifdef DEBUG
(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #107)
> Well, patch worked :-)
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=81fc7e34c961

Or not, well, don't know, but the assert is because of Map() of a surface which previously saw GetData().
Let's see this instead: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8bbea8a47a9e

It's based on a recent m-c with plenty of D2D GetData() replaced by Map().
Here, https://treeherder.mozilla.org/#/jobs?repo=try&revision=18cc762afc9c

So there's one assertion failure on WinXP, probably unrelated.
> 21:05:18     INFO -  Assertion failure: !mSrc->mIsMapped, at c:\builds\moz2_slave\try-w32-d-00000000000000000000\build\src\obj-firefox\dist\include\mozilla/gfx/2D.h:394
> 21:05:31     INFO -  #01: mozilla::gfx::GfxPatternToCairoPattern [gfx/2d/DrawTargetCairo.cpp:487]

I also applied the patch to the various D2D surfaces, but it didn't yield anything.

Seems to be a GetData/Map mix on the Win7 machine though, perhaps that's something.
Looks like the debug patch didn't catch anything. I'm not too familiar with the code here but I have some time. I'll run a manual audit of the Map/GetData and see if I can spot something. Keeping ni?
Attached file Relevant crash stack (obsolete) —
Looks like we're calling DrawTargetWillChange for a target that's mapped.
Flags: needinfo?(bgirard)
Comment on attachment 8623239 [details]
Relevant crash stack

I'm not sure this is related to this bug actually. Let's make sure we don't mix things up.
Attachment #8623239 - Attachment is obsolete: true
Attached file Assert failure
Going to focus on this one failure:
https://hg.mozilla.org/try/annotate/18cc762afc9c0679acccc932aca6c13ae99c2cc3/gfx/2d/SourceSurfaceD2DTarget.cpp#l296

I'm not sure it's the same bug but fixing that may help here.
There is bug 1133119, but the timing isn't right, and that one would have just increased the number of times we fail the Map call, but thought I'd mention it anyway.
mMapped vs. mIsMapped is annoying.  Just sayin'.
Attached patch more assertSplinter Review
This is mostly a shot in the dark but could be worth running through try as well.

I don't know how we're getting here:
http://mxr.mozilla.org/mozilla-central/source/gfx/2d/DrawTargetD2D.cpp#1232

without having the surface mapped since we map it above and check of errors properly.
In this failure at least - https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=10921382 - one of the non-crashing threads (32) is in the middle of doing this:

00:40:33     INFO -   2  xul.dll!mozilla::layers::TextureHost::DestroyIPDLActor(mozilla::layers::PTextureParent *) [CompositableHost.cpp:a8e0bde30bd4 : 95 + 0xf]
00:40:33     INFO -   3  xul.dll!mozilla::layers::PTextureParent::Send__delete__(mozilla::layers::PTextureParent *) [PTextureParent.cpp:a8e0bde30bd4 : 64 + 0xc]
I still haven't reproduced this yet but I'll give it another try. I did find this handy command:
./mach mochitest --run-until-failure --repeat 300 ./dom/canvas/test/test_capture.html
Benoit, please see bug 1176363 comment 13 for a race between main thread and the compositor that could be related to this bug as well. Ideas for fixes welcome.
Flags: needinfo?(bgirard)
See Also: → 1176363
Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(bgirard)
Resolution: --- → FIXED
Target Milestone: --- → mozilla42
You need to log in before you can comment on or make changes to this bug.