1218395 - Dev Edition startup crashes in nsCOMPtr_base::~nsCOMPtr_base | nvd3d9wrap.dll@0x...

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect)

Description

[Robert Kaiser]

This bug was filed from the Socorro interface and is 
report bp-0eda266a-6c2f-4721-9322-82cac2151025.
=============================================================

I'm only filing this in GFX because of nvd3d9wrap.dll being involved in those crashes.

Those crashes started to appear and spike heavily on 2015-10-24 on Dev Edition, in both e10s-activated and e10s-deactivated builds (even more visible in crash reports right now with the experiment running, see metadata tabs).

The stacks very a lot, the only thing common seems to be that frame 0 is nsCOMPtr_base::~nsCOMPtr_base, frame 1 is in nvd3d9wrap.dll and frame 2 is in PLDHashTable.

About 80% of those crashes are withing the first 60s of uptime, so we consider them startup.

This is by far the largest crash issue on Dev Edition 43 at this point.

Comment 1

[Robert Kaiser]

[Tracking Requested - why for this release]:

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 43 since it is a new (top) crash.

Comment 3

[David Anderson [:dvander] - inactive, e-mail if emergency]

This crash looks exclusive to Intel adapters. There may be a more specific correlation. Driver versions:

 10.18.10.4276   57%    (0.25% of Telemetry sessions)
 9.17.10.4229    23%    (1.5% of Telemetry sessions)
 10.18.10.4252   11%    (1.0% of Telemetry sessions)

Devices:
 0x0166          69%       (4.9% of Telemetry sessions)
 0x0116          19%       (3.8% of Telemetry sessions)

Most of those Telemetry numbers are on the high end, meaning they're more prevalent given the large distribution of devices and drivers. But,

0x0166 on driver version 10.18.10.4276 apears in 0.09% of Telemetry sessions.
0x0166 on driver version 10.18.10.4252 appears in 0.58% of Telemetry sessions.
0x0116 on driver version 9.17.10.4229 appears in 0.6% of Telemetry sessions.

So it might be that a very small population of one specific adapter and driver is causing the majority of these crashes. It doesn't look like crash-stats can do multi-key aggregations though.

Comment 4

[Milan Sreckovic [:milan] (needinfo for best results)]

Could probably use some layout opinions.

Comment 5

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Robert Kaiser (:[e-mail]) from comment #0)
> This bug was filed from the Socorro interface and is 
> report bp-0eda266a-6c2f-4721-9322-82cac2151025.
> =============================================================
> 
> 
> Those crashes started to appear and spike heavily on 2015-10-24 on Dev
> Edition, ...

What are the patches that landed for this?

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

Need more big brains on this; anything we can glean from these crashes? (No offense to big brains I didn't needinfo :)

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Robert Kaiser (:[e-mail]) from comment #0)
> ...
> Those crashes started to appear and spike heavily on 2015-10-24 on Dev
> Edition...

Hopefully, not a result of disabling async plugin init.

When was the previous Dev Edition built?  I'd like to take a look at all the patches that were new on the 24th.

Comment 8

[Milan Sreckovic [:milan] (needinfo for best results)]

Shooting in the dark - bug 1210444? Maybe that SharedSurfaceTextureClient extra AddFlags should just have been mFlags |= mSurf->GetTextureFlags() instead?

Comment 9

[Jamie Nicol [:jnicol]]

I don't know the IPC code well enough to know if this could cause this. But yeah if it is responsible Milan's suggestion or just adding it to the parent constructor call would be fine (not sure why I didn't do the latter to be honest). I didn't intend for the AddFlags to have any side effect other than adding a flag.

Comment 10

[Milan Sreckovic [:milan] (needinfo for best results)]

Jamie, let's put together a patch to make that change, and uplift.  Even if it has nothing to do with this issue, it's probably safer, and it seems adequate.  Should check if fixes the original Android problem, of course.

Comment 11

[Jamie Nicol [:jnicol]]

Attachment: Avoid calling AddFlags from SharedSurfaceTextureClient constructor

Confirmed that this still fixes bug 1210444.

Comment 12

[Jamie Nicol [:jnicol]]

didn't mean for "hg bzexport" to assign me

Comment 13

[Milan Sreckovic [:milan] (needinfo for best results)]

The patch is speculative, so we don't want to close the bug when we land it, but it won't hurt to remove one unknown.

Comment 14

[Matt Woodrow (:mattwoodrow)]

Do we have a regression range for this?

Comment 15

[Milan Sreckovic [:milan] (needinfo for best results)]

So far the "started to appear and spike heavily on 2015-10-24 on Dev Edition", but I don't know when the previous Dev Edition was.

Comment 16

[Bas Schouten (:bas.schouten)]

I can't figure out exactly how the above mentioned patch would cause this problem. I strongly suspect this bug has absolutely nothing to do with graphics. The D3D DLL probably just gets in there by accident. Having said I think this patch improves the quality of the code, so let's still do this. Let's just not assume it fixes anything :-).

Comment 17

[Robert Kaiser]

Dev Edition has daily builds, so the regression range would be a day before that. The first build ID affected there is 20151024004043, looking into archive.m.o I find https://archive.mozilla.org/pub/firefox/nightly/2015/10/2015-10-24-00-40-43-mozilla-aurora/firefox-43.0a2.en-US.win32.json with moz_source_stamp": "68f165595f3a1081999a4b11edd5e5e7127f901d" and for the build before I find https://archive.mozilla.org/pub/firefox/nightly/2015/10/2015-10-23-00-40-26-mozilla-aurora/firefox-43.0a2.en-US.win32.json with "moz_source_stamp": "6c1457027394d198932267c646135a9d8ab3876c" so the regression range would be https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=6c1457027394d198932267c646135a9d8ab3876c&tochange=68f165595f3a1081999a4b11edd5e5e7127f901d

Comment 18

[Robert Kaiser]

All that said, it looks like those signatures are very flaky, I actually see them appearing with the build from the 24th and the 27th but not for builds in between, which makes me think that something else might be wrong there.

Comment 19

[Robert Kaiser]

FWIW, this could even be connected to the startup crashes on beta in bug 1218473

Comment 20

[Jamie Nicol [:jnicol]]

Try run for above patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8c7f03aa91d7

Requesting checkin please, and leave open since we don't think this will fix the problem.

Comment 21

[Jamie Nicol [:jnicol]]

Comment on attachment 8679726 [details] [diff] [review]
Avoid calling AddFlags from SharedSurfaceTextureClient constructor

And uplift this to aurora since it's low risk and might fix it.

Approval Request Comment
[Feature/regressing bug #]: possibly bug 1210444
[User impact if declined]: possibly this startup crash
[Describe test coverage new/current, TreeHerder]: Try run on aurora: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c8cbecc0150f I am assuming the Android 4.3 API11+ debug failures are unrelated.
[Risks and why]: Very low. very small change which does the same thing but removes potentially risky code.
[String/UUID change made/needed]: None

Comment 22

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Robert Kaiser (:[e-mail]) from comment #19)
> FWIW, this could even be connected to the startup crashes on beta in bug
> 1218473

My guess in IRC, mostly based on the flakiness of signatures and the same hardware Intel/Nvidia combinations.

Comment 23

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8679726 [details] [diff] [review]
Avoid calling AddFlags from SharedSurfaceTextureClient constructor

OK, let's uplift this speculative fix. Since Bas thinks it improves the code anyway, sounds like a reasonable idea.

Comment 24

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7e2286a4141c

Comment 25

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/9470fd1a1c89

Comment 26

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7e2286a4141c

Comment 27

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/7e2286a4141c

Comment 28

[Carsten Book [:Tomcat]]

removing the b2g 2.5 flag since this commit has been reverted due to an incorrect merge, sorry for the confusion

Comment 29

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Jamie, this signature has jumped up to #4 spot in DevEd44 crash stats (last 3 days of data). I know we uplifted a fix a few weeks back. Just wanted to bring this to your attention in case we have better diagnostics in place and want to review the recent crash dumps to root cause the problem.

Comment 30

[Jamie Nicol [:jnicol]]

I doubt I'm the best person for this to be assigned to. I don't have the hardware or expertise. Milan, could you reassign to somebody better suited? (Or if you I'll take a look.)

Comment 31

[Milan Sreckovic [:milan] (needinfo for best results)]

Right - are these all Optimus bugs, dual GPU configurations?

Comment 32

[u279076]

(In reply to Milan Sreckovic [:milan] from comment #31)
> Right - are these all Optimus bugs, dual GPU configurations?

100% of these are on dual-GPU systems with the following NVIDIA chipsets:
* 70% with Kepler
** 45% with Kepler GK107GLM
** 15% with Kepler GK106GLM
** 10% with Kepler GK107
* 30% with Fermi
** 15% with Fermi GF106GLM
** 10% with Fermi GF108M
**  5% with Fermi GF108M

Here are the Intel chipsets which show up as GPU#1 on these systems:
69% Ivybridge
25% Haswell
 6% Sandybridge

Comment 33

[Milan Sreckovic [:milan] (needinfo for best results)]

How about the existence of _etoured.dll?

Comment 34

[u279076]

(In reply to Milan Sreckovic [:milan] from comment #33)
> How about the existence of _etoured.dll?

Module correlations don't work for these crashes (not sure why) so the only way to know for sure would be to check every crash report manually. I manually checked 50 reports at random and they all have _etoured.dll as the first module. It's probably safe to assume most if not all are in the same boat.

Comment 35

[Milan Sreckovic [:milan] (needinfo for best results)]

That's making me think this could be related, if not a duplicate - see bug 1218473 comment 44.

Comment 36

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

So one thing that's somewhat interesting here is that the regression window on nightly appears to be *newer* than the regression window on aurora, at least assuming I'm using Socorro correctly, and it stores old data reliably.

https://crash-stats.mozilla.com/search/?product=Firefox&release_channel=aurora&signature=%24nsCOMPtr_base%3A%3A~nsCOMPtr_base+|+nvd3d9wrap.dll%40&date=%3E2015-07-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
is a search for crashes on the aurora channel.  Those, as stated in comment 0, appear to have started on 2015-10-24 (you have to look at the Aggregations tab of each separate crash).

The same search for the nightly channel is:
https://crash-stats.mozilla.com/search/?product=Firefox&release_channel=nightly&signature=%24nsCOMPtr_base%3A%3A~nsCOMPtr_base+|+nvd3d9wrap.dll%40&date=%3E2015-07-01&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
On nightly, the crashes appear to have started on 2015-11-07.

It might be worth trying to figure out if the crashes are from a small number of users -- and whether it's possible that the trigger for the crashes starting was some software update not-from-us, or whether it's possible instead that we just happened to hit the right users on nightly later than we hit them on aurora.

Comment 37

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Hi,

Based on Socorro reports, all provided crash signatures shows that in the last 28 days, the latest Aurora (46.0a2) is not affected. The latest Aurora version that is present in the crash signatures on the latest 28 days was version 45.0a2. Most of the signatures does not have crash reports. Can someone confirm that this issue was fixed on the latest Aurora 46.0a2? 

Removing "regressionwindow-wanted" keyboard since the regression was provided in comment 17.

Thanks,
Cosmin.

Comment 38

[u279076]

I'm closing this bug report as there's only been three reports of this crash recently, all on versions of Firefox we no longer support.

Comment 39

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.