Crash in [@ agcore@0x696135 ] [@glgProcessPixelsWithProcessor] while watching Netflix on demand movies

RESOLVED INCOMPLETE

Status

Plugins Graveyard
Silverlight (Microsoft)
--
critical
RESOLVED INCOMPLETE
9 years ago
2 years ago

People

(Reporter: marcia, Assigned: Andy Rivas)

Tracking

({crash})

x86
Mac OS X
crash

Details

(Whiteboard: [crashkill], crash signature)

Attachments

(1 attachment)

1.20 MB, application/octet-stream
Details
Although I have not been yet able to reproduce this crash using Firefox 3.5.5, stephend confirms he has hit it, and there are currently about 751 Mac users who have crashed as well in a 5 week period - see http://tinyurl.com/ydh9zw8 for the full list.

All of the comments consistently mention that they were watching netflix at the time of the crash. Netflix uses silverlight in their on demand implementation.
My version of Silverlight is 3.0.40818.0, in case that matters, and I'm on 10.5.

Comment 2

9 years ago
checking --- 20091209-crashdata.csv agcore@0x696135
release total-crashes
              agcore@0x696135 crashes
                         pct.
all     227791  58      0.000254619
3.0.15  43273   3       6.93273e-05
3.5.5   124561  42      0.000337184
3.6b4   20364   5       0.000245531
3.6b3   884             0
3.6b2   1015            0
3.6b1   2623            0

all releases
   1 3.0.1
   3 3.0.15
   1 3.0.3
   1 3.5.1
   1 3.5.2
   2 3.5.3
   1 3.5.4
  42 3.5.5
   1 3.5.6
   5 3.6b4

os breakdown
21      0.362069        Mac OS X10.6.2 10C540
20      0.344828        Mac OS X10.5.8 9L31a
10      0.172414        Mac OS X10.5.8 9L30
2       0.0344828       Mac OS X10.6.1 10B504
2       0.0344828       Mac OS X10.5.6 9G55
1       0.0172414       Mac OS X10.6.0 10A432
1       0.0172414       Mac OS X10.5.7 9J61
1       0.0172414       Mac OS X10.4.11 8S2167


netflix is definitely the site to try and reproduce this.
 
domains of sites
  44 http://www.netflix.com
   3 http://www.facebook.com
   1 https://mail.google.com
   1 http://www.tnaflix.com
   1 http://www.sky.com
   1 http://www.sesamestreet.org
   1 http://www.bestbuy.com
   1 http://gyao.yahoo.co.jp
   1 http://ezxcess.antlabs.com
   1 http://community.babycenter.com
   1 http://apps.facebook.com

some example urls look like

   1 http://www.sky.com/skycom/skyplayer?openChannel=501
   1 http://www.sesamestreet.org/classicclips

a variety of movie ids and tracking id's on netfix following the form...

   1 http://www.netflix.com/WiPlayer?movieid=863311&trkid=SANITIZED
Silverlight 3 (3.0.40818.0) is the version I was using to try to repro on both 10.5 and 10.6, but have not had any luck yet.

Comment 4

9 years ago
strong correlation to noscript

  agcore@0x52a713|EXC_BAD_ACCESS / KERN_PROTECTION_FAILURE (16 crashes)
    100% (16/16) vs.   1% (286/19185) {73a6fe31-595d-460b-a920-fcc0f8843232} (NoScript, https://addons.mozilla.org/addon/722)

         19% (3/16) vs.   0% (30/19185) 1.9.9.14
         13% (2/16) vs.   0% (71/19185) 1.9.9.15
         69% (11/16) vs.   1% (163/19185) 1.9.9.18

Comment 5

9 years ago
er, that's actually a different signature, and different bug that we should file.

Comment 6

9 years ago
we don't get version info out of the silverlight .dlls but we do get checksums and they show high correlation to a specific version.

 agcore@0x696135|EXC_BAD_INSTRUCTION / EXC_I386_INVOP (16 crashes)
    100% (16/16) vs.   3% (636/19185) coreclr
     100% (16/16) vs.   2% (474/19185) checksum: C1FE40E014A5379B6359E9DB2A863C870
    100% (16/16) vs.   4% (739/19185) agcore
     100% (16/16) vs.   3% (529/19185) checksum: D2B3FB1BA93C15B56385408FDD62AB380
Adding Andy to the bug in case he can help or knows someone who can :)
(Assignee)

Comment 8

9 years ago
Can we get a copy of the raw dumps for these crashes?  It will help us with our investigation.

Thanks
Unfortunately we are not able to share minidumps with third parties per our privacy policy. And even if we were, there are currently not much useful past what's in the public crash reports you can get out of them on a Mac.
I haven't hit this in a long time (if even).
(Assignee)

Comment 11

9 years ago
Did you upgrade your Mac?  Upgrade your FireFox?
(In reply to comment #11)
> Did you upgrade your Mac?  Upgrade your FireFox?

Both:

1) I'm now on 10.6
2) Running 3.6

Silverlight plugin version is still 3.0.40818.0, though.
In reviewing crash stats there are still users hitting this crash and not all crashes are related to Netflix - http://tinyurl.com/yacoa7p. For 3.6 it ranks #56 in the crash stats and #45 for 3.5.7.

Updated

8 years ago
Component: Plug-ins → Silverlight (Microsoft)
Product: Core → Plugins
QA Contact: plugins → microsoft-silverlight
Version: 1.9.1 Branch → 3.x
This is the #19 top crash on Mac when I search for 3.6.3/3.6.4 data but shows up with this stack: [@glgProcessPixelsWithProcessor] so adding it to the summary. http://tinyurl.com/28xapg3 is the link to those crashes. Perhaps we are crashing more due to the new version of Silverlight?
Summary: Crash in [@ agcore@0x696135 ] while watching Netflix on demand movies → Crash in [@ agcore@0x696135 ] [@glgProcessPixelsWithProcessor] while watching Netflix on demand movies

Comment 15

8 years ago
I took a sample of 100 crashes with the signature agcore@0x696135  and only two debug id's showed up.

  62 agcore D2B3FB1BA93C15B56385408FDD62AB380
   1 agcore D570188656AE296DC19BB8EA95E1F0150

and for glgProcessPixelsWithProcessor they were all a single debug id

 100 agcore 86B0B18BAC3460286443EE3430D9FBCE0

we should see if we can translate these debug id's into sliverlight versions

there are about 139 different agcore stack signatures so maybe this is the only signature we might see for this particular version with other versions showing up with other signatures. here would be the list to review for different versions showing up as similar crashes on netflix or other sites that tickle the possible silverlight bug.

agcore signature breakdown
signature distribution
     139
signature list
  89 agcore@0x6960a5
  63 agcore@0x696135
  17 agcore.dll@0x1be957
  13 agcore.dll@0x14a6b3
  10 agcore@0x510914
   9 agcore@0x695af5
   7 agcore@0x4a8a75
   5 agcore@0x68d035
   5 agcore@0x5ba872
   5 agcore@0x4f27bf
   5 agcore@0x4a7b25
   4 agcore@0x52a653
   4 agcore@0x5109b4
   4 @0x0 | agcore@0x5154b0
   3 agcore@0x68cfa5
   3 agcore@0x647a11
   3 agcore@0x516bab
   3 agcore@0x514103
   3 agcore@0x4f285f
   3 agcore@0x3d3807
   3 agcore@0x279f0f
   3 agcore@0x2617ff
   3 agcore.dll@0x2421b9
   3 agcore.dll@0x1bddc6
(Assignee)

Comment 16

8 years ago
Silverlight team is looking at this issue again with the additional data.

Updated

8 years ago
Assignee: nobody → andy.rivas

Comment 17

8 years ago
Hi, 

I am Andy's counterpart on the Silverlight team. I am trying to see how we can better resolve these issues. It would be really helpful if we can narrow down the version of Silverlight that caused the problems. 

Chris, you mention checksums generated to distinguish the binaries. How are the checksums generated and is there a way to reverse lookup the corresponding version of the silverlight binary? That would immensly help us. 

Thanks
Sunil
(In reply to comment #17)
> Chris, you mention checksums generated to distinguish the binaries. How are the
> checksums generated and is there a way to reverse lookup the corresponding
> version of the silverlight binary? That would immensly help us.

The checksums (aka Module or Debug Identifiers) are generated by the Breakpad crash reporting software, based on a number of characteristics of the binary.  You can't "reverse-engineer" the version from the ID, but you can use the same process to generate the IDs from binaries for which you know the versions.

We maintain a manual listing/mapping for Flash on the Mozilla Wiki; the page includes instructions for generating the Module/Debug IDs, so you can follow those instructions to create a similar list with Silverlight releases: https://wiki.mozilla.org/CrashKill/Mac_Flash_Identifiers

If you don't have a dump_syms binary (as part of your Mozilla build: $objdir/toolkit/crashreporter/google-breakpad/src/tools/mac/dump_syms/dump_syms), I or someone else can upload one for you.

Comment 19

8 years ago
I do not have the dump_syms binary. It would be great if you could share that out. I will be able to make the bug data more actionable on our side.

Comment 20

8 years ago
Is there a reason why these identifiers are listed? Can the same be done for Silverlight plugin versions?

Comment 21

8 years ago
Created attachment 458802 [details]
dump_syms

the identifiers are what breakpad deals in. converting them into something more meaningful is mostly outside its scope. it's roughly equivalent to the magic hashes used for pdb files. there's no meaningful version for a dll which is guaranteed to be unique. but given a given dll you can consistently run dump_syms and generate the identifier (you or someone else has to manually identify the version...).

this dump_syms should be from about m-c rev a80066930eb5

timeless-mbp:MacOS timeless$ pwd; (ls agcore coregen| xargs -n1 /tmp/dump_syms -a i386 ) |grep ^MODULE
/Library/Internet Plug-Ins/Silverlight.plugin/Contents/MacOS
MODULE mac i386 536E0F08163446380DA8B6F97BF2861A0 agcore
MODULE mac i386 C4E86D512C540FD1370378C62E93F45E0 coregen

timeless-mbp:MacOS timeless$ grep -A1 PluginDesc ../Info.plist|xargs echo
<key>WebPluginDescription</key> <string>4.0.50524.0</string>
Crash Signature: [@ agcore@0x696135 ] [@glgProcessPixelsWithProcessor]

Comment 22

2 years ago
Closing old bugs in the Plugins component. We aren't going to track issues in 3rd-party plugins in the Mozilla bug tracker. In addition, support for NPAPI plugins will be removed at the end of this year; for more details see the post at https://blog.mozilla.org/futurereleases/2015/10/08/npapi-plugins-in-firefox/

If there is a serious bug in Firefox, it needs to be filed in the "Core" product, "Plug-Ins" component.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → INCOMPLETE
Product: Plugins → Plugins Graveyard
You need to log in before you can comment on or make changes to this bug.