560584 - Firefox 3.6.4 Crash Report [@ libpthread-2.5.so@0x7300 ]

Status: RESOLVED WONTFIX

(Core Graveyard :: Plug-ins, defect)

Description

[chris hofmann]

there is a possible new crash in 3.6.4 

checking --- libpthread-2.5.so 20100419-crashdata.csv
found in: 3.6.4
release total-crashes
              libpthread-2.5.so crashes
                         pct.
all     366784  12      3.27168e-05
3.6.4   11871   12      0.00101087

with a stack that looks like

http://crash-stats.mozilla.com/report/index/93fd1af9-ad8d-4511-81b5-4a7ee2100420

Frame  	Module  	Signature [Expand]  	Source
0 	libpthread-2.5.so 	libpthread-2.5.so@0x7300 	
1 	libflashplayer.so 	libflashplayer.so@0x1af067 	
2 	libflashplayer.so 	libflashplayer.so@0x1f9251 	
3 	libflashplayer.so 	libflashplayer.so@0xb972c 	
4 	libflashplayer.so 	libflashplayer.so@0x26137 	
5 	libflashplayer.so 	libflashplayer.so@0x2203f 	
6 	libflashplayer.so 	libflashplayer.so@0x5a7539 	
7 	ld-2.5.so 	ld-2.5.so@0xe7ed 	
8 	libc-2.5.so 	libc-2.5.so@0x2bd38 	
9 	libXt.so.6.0.0 	libXt.so.6.0.0@0x19169 	
10 	libXt.so.6.0.0 	libXt.so.6.0.0@0x18e5b 	
11 	libXt.so.6.0.0 	libXt.so.6.0.0@0x194f5 	
12 	libXt.so.6.0.0 	libXt.so.6.0.0@0x18b1e 	
13 	libXt.so.6.0.0 	libXt.so.6.0.0@0x17646 	
14 	libXt.so.6.0.0 	libXt.so.6.0.0@0x1773b 	
15 	libXt.so.6.0.0 	libXt.so.6.0.0@0x1783c 	
16 	libflashplayer.so 	libflashplayer.so@0x2d306 	
17 	libflashplayer.so 	libflashplayer.so@0x25ef2 	
18 	libflashplayer.so 	libflashplayer.so@0x2a531 	
19 	libxul.so 	mozilla::plugins::PluginInstanceChild::AnswerNPP_SetWindow 	dom/plugins/PluginInstanceChild.cpp:598
20 	libxul.so 	mozilla::plugins::PPluginInstanceChild::OnCallReceived 	PPluginInstanceChild.cpp:1072
21 	libxul.so 	mozilla::plugins::PPluginModuleChild::OnCallReceived 	PPluginModuleChild.cpp:380
22 	libxul.so 	mozilla::ipc::RPCChannel::DispatchIncall 	ipc/glue/RPCChannel.cpp:485
23 	libxul.so 	mozilla::ipc::RPCChannel::Incall 	ipc/glue/RPCChannel.cpp:471
24 	libxul.so 	mozilla::ipc::RPCChannel::OnMaybeDequeueOne 	ipc/glue/RPCChannel.cpp:413
25 	libxul.so 	RunnableMethod<mozilla::ipc::RPCChannel, void 	ipc/chromium/src/base/tuple.h:383
26 	libxul.so 	mozilla::ipc::RPCChannel::DequeueTask::Run 	RPCChannel.h:421
27 	libxul.so 	MessageLoop::RunTask 	ipc/chromium/src/base/message_loop.cc:336
28 	libxul.so 	MessageLoop::DeferOrRunPendingTask 	ipc/chromium/src/base/message_loop.cc:344
29 	libxul.so 	MessageLoop::DoWork 	ipc/chromium/src/base/message_loop.cc:444
30 	libxul.so 	base::MessagePumpForUI::HandleDispatch 	ipc/chromium/src/base/message_pump_glib.cc:264
31 	libxul.so 		ipc/chromium/src/base/message_pump_glib.cc:109
32 	libglib-2.0.so.0.1200.3 	libglib-2.0.so.0.1200.3@0x2b1a1 	
33 	libglib-2.0.so.0.1200.3 	libglib-2.0.so.0.1200.3@0x2e195 	
34 	libglib-2.0.so.0.1200.3 	libglib-2.0.so.0.1200.3@0x2e6ed 	
35 	libxul.so 	base::MessagePumpForUI::RunWithDispatcher 	ipc/chromium/src/base/message_pump_glib.cc:195
36 	libxul.so 	base::MessagePumpForUI::Run 	ipc/chromium/src/base/message_pump_glib.h:59
37 	libxul.so 	MessageLoop::RunInternal 	ipc/chromium/src/base/message_loop.cc:216
38 	libxul.so 	MessageLoop::RunHandler 	ipc/chromium/src/base/message_loop.cc:199
39 	libxul.so 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:173
40 	libxul.so 	base::Thread::ThreadMain 	ipc/chromium/src/base/thread.cc:165
41 	libxul.so 	ThreadFunc 	ipc/chromium/src/base/platform_thread_posix.cc:26
42 	libpthread-2.5.so 	libpthread-2.5.so@0x549a 	
43 	libc-2.5.so 	libc-2.5.so@0xd142d

more reports at

http://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&signature=libpthread-2.5.so%400x7300&version=Firefox%3A3.6.4

no urls and not much to go on at this point.

nominating to make sure we keep an eye on it for the next few days, and to see if anyone spots a problem in ipc or plugin code that might affect just Linux or be cross platform.

Comment 1

[juan becerra [:juanb]]

There are several crash reports with this signature in 3.6.4, but none in 3.6.3plugin1 nor in 3.6.5pre, at least none in the top 300, and all of these happened in Linux.

Comment 2

[chris hofmann]

(In reply to comment #1)
> There are several crash reports with this signature in 3.6.4, but none in
> 3.6.3plugin1 nor in 3.6.5pre, at least none in the top 300, and all of these
> happened in Linux.

likely to be a beta popluation deal where we need over 100k beta testers to start seeing it and even at that level only 12 crashes were generated on the 19th.

Comment 3

[Karl Tomlinson (:karlt)]

All crashes are on
"Linux 2.6.18-128.1.6.el5.centos.plus #1 SMP Thu Apr 2 12:53:13 EDT 2009 x86_64"
running an x86 firefox.

This looks like a fairly old OS.

Module identifier is:
libflashplayer.so  	 	9F60EA7237C2DB7834E97F0BDD08661D0

Other different crash reports have:
libflashplayer.so  	 	9BC43748D925AB0C97B43BB3C7019E210

I suspect this is an older version of Flash Player.

(In reply to comment #0)
> Frame      Module      Signature [Expand]      Source
> 0     libpthread-2.5.so     libpthread-2.5.so@0x7300     
> 1     libflashplayer.so     libflashplayer.so@0x1af067     
> 2     libflashplayer.so     libflashplayer.so@0x1f9251     
> 3     libflashplayer.so     libflashplayer.so@0xb972c     
> 4     libflashplayer.so     libflashplayer.so@0x26137     
> 5     libflashplayer.so     libflashplayer.so@0x2203f     
> 6     libflashplayer.so     libflashplayer.so@0x5a7539     
> 7     ld-2.5.so     ld-2.5.so@0xe7ed     
> 8     libc-2.5.so     libc-2.5.so@0x2bd38     

Similarities to bug 545819 make me suspect a crash in a Flash Player atexit handler.

> 9     libXt.so.6.0.0     libXt.so.6.0.0@0x19169     
> 10     libXt.so.6.0.0     libXt.so.6.0.0@0x18e5b     
> 11     libXt.so.6.0.0     libXt.so.6.0.0@0x194f5     
> 12     libXt.so.6.0.0     libXt.so.6.0.0@0x18b1e     
> 13     libXt.so.6.0.0     libXt.so.6.0.0@0x17646     
> 14     libXt.so.6.0.0     libXt.so.6.0.0@0x1773b     
> 15     libXt.so.6.0.0     libXt.so.6.0.0@0x1783c     
> 16     libflashplayer.so     libflashplayer.so@0x2d306     
> 17     libflashplayer.so     libflashplayer.so@0x25ef2     
> 18     libflashplayer.so     libflashplayer.so@0x2a531     
> 19     libxul.so mozilla::plugins::PluginInstanceChild::AnswerNPP_SetWindow

Old old versions of Flash Player are not XEmbed but Xt plugins.

Xt plugins will exit when run OOP - bug 544088.

A consolation is that we are probably doing this person a favor by not
running their old Flash plugin.

Comment 4

[chris hofmann]

should a fix for this or bug 544088 be blocking 1.9.2?   or 1.9.3?

the fact that this is showing up in crash data means at least some people are hitting it in the early beta data, although I suspect this will get burried by windows and mac crashes pretty quickly.

if this wasn't a security and stability release we could pre-announce drop in support for Xt Plugins.

Comment 5

[Karl Tomlinson (:karlt)]

The current plan for 1.9.2 is to only whitelist plugins for OOPP that are known XEmbed plugins.  We didn't consider Flash Player versions prior to Flash Player 9 Update 3, version identifier 9,0,115,0, which have the same filename and so are enabled by the same pref but are Xt plugins.

http://blogs.adobe.com/penguin.swf/2007/12/flash_player_9_update_3_final.html

I guess we could do something to enable these, but disabling them may be quite appropriate for a security release.

For 1.9.3, we are currently blacklisting known Xt plugins (bug 543802), though we don't know what other Xt plugins are out there.  We should consider whether bug 544088 blocks 1.9.3.

Comment 6

[timeless]

karlt: as these bugs have stack traces, i'm going to move them to plugins:, if you want a bug for blocking things, i think you can file one which isn't tagged as crash.

Comment 7

[chris hofmann]

blizzard, any thoughts on the impact of this?  

We are seeing crashes show up for Xt plugins in the crash data.

Karl outline some possibilities for how by maybe backing off OOPP whitelisting until a major release or maybe some advanced blocklisting for 3.6.4 to get people to move forward to non-Xt versions.

timeless, I'm going to move this back to core plugins as our blocking tracker bug sinces its unlikely we will get a fix for this on the plugin side.

Comment 8

[Mike Beltzner [:beltzner, not reading bugmail]]

Blocking, as this reads to me as "Linux users with older versions of Flash have a bad time with OOPP" - this is either a dupe of bug 544088 (if you want that to mean that we need to blockist Xt plugins) at which point that should block, or if you interpret bug 544088 to mean that we should implement OOPP support for Xt plugins, then this is the one that should block.

Confused yet?

Comment 9

[Benjamin Smedberg]

Yes, absolutely. I thought the answer to this was going to be "those versions of Flash are hopeless insecure, and we should refuse to load them altogether". Because they all have the same filename and don't export version numbers, it would be *very* difficult to blacklist this, and implementing Xt plugins is a multi-week project.

Comment 10

[Christopher Blizzard (:blizzard)]

They don't export version numbers, but we can tell from the description, can't we?

Comment 11

[Benjamin Smedberg]

I don't know, does anyone know where I can download these older versions?

In case it's not clear, I think this bug should be WONTFIX, or we should implement an even more aggresive blocking strategy.

Comment 12

[Christopher Blizzard (:blizzard)]

There are flash archives out there.  If you want I can try and dig up the URL.

There's a flag in the NSAPI for if a plugin uses the new XEmbed stuff.  (It might actually be a callback?)  If it doesn't, shouldn't we just not load it?

Comment 13

[Christopher Blizzard (:blizzard)]

http://kb2.adobe.com/cps/142/tn_14266.html

Comment 14

[chris hofmann]

The net effect of dropping support for Xt Plugins on short notice would
probably mean at least some users just find the update to 3.6.4 crashier since
there this crashes the browser, right? 

Some users might stick on  or go back to 3.6.3 or earlier.   Thats not a good
option for those users either.  

It would also be good to try and get some numbers that breakdown flash version
distribution on linux.  Any ideas there?  I see if I can find some.

Looks like the release date was Dec 2007 so we are dealing with plugins more
than 2 years out of date -->
http://blogs.adobe.com/penguin.swf/2007/12/flash_player_9_update_3_final.html

Comment 15

[Benjamin Smedberg]

No, this isn't crashing the browser, it's only crashing the plugin process. Due to the uniformity of the reports, I'd wager that this is a single user.

Comment 16

[chris hofmann]

ominature was no help to get a breakdown of linux + flash version as far as I could see.

I grabbed a sample of 200 crash reports on linux where libflashplayer running and here was the breakdown.  If the debug ID's can be used as a proxy for version numbers then this would be the distribution. Only 25% running what hopefully is one of most recent versions might mean slow flash update patterns, but this is just guessing.  Charles, do you know if there is an easy way to translate the debug id's, or have some numbers that you could share with us in the bug or privately?

50 libflashplayer.so 617F25534A9BB6F7F0A0820B6216D3530
32 libflashplayer.so 9BC43748D925AB0C97B43BB3C7019E210
22 libflashplayer.so 745026564F2B12E0E3B2715D145A0F690
19 libflashplayer.so 72973D3A409794225D71BF5796E80EBC0
10 libflashplayer.so E8E4615D8D0DFC9AE0BE3E3E69920AB80
10 libflashplayer.so 87934424747BF50DA30C8528037E75EE0
5 libflashplayer.so EECE3AF0BC567DB9F2692ACAAA974F7F0
5 libflashplayer.so E68208AF7DD76CE7A5016EB5EF110AC00
5 libflashplayer.so 75C4D5A84AA5F22F07AD159D020895EF0
5 libflashplayer.so 3F4FE6DB0E1FC865B3049718AC3521980
4 libflashplayer.so D708CF1409A7A3DE15F591C53914549B0
4 libflashplayer.so ADF15068159E275F6F66F9A3D006F2F90
4 libflashplayer.so 9BF4278859263B0053642AE35840B3610
4 libflashplayer.so 74E1B5911E3F3E5E80BB24EC2CDC5F660
3 libflashplayer.so FCDA6F558B67CF6B4063475C74EBBF4B0
3 libflashplayer.so 9F60EA7237C2DB7834E97F0BDD08661D0
2 libflashplayer.so B90802BF65F9CE3B9841550D7C0FDE400
2 libflashplayer.so 49ACE7A2810BC1EE5958F095C25F09E20
2 libflashplayer.so 390DEC869F4C2E7D7C03885CD7A231440
1 libflashplayer.so F1490E942E837412B93F7F6A82C744AD0
1 libflashplayer.so F0B1F424C57BBC81825967A5591BE82D0
1 libflashplayer.so CF1E82198F9EC8CB28BA0D972D7FDDB70
1 libflashplayer.so CD5D2174D75A0118490922BAE78C3B230
1 libflashplayer.so AB1A186B5B8C190D5E9A2C683D85461F0
1 libflashplayer.so 9EF10ADE595DA88B6297999314BB04D90
1 libflashplayer.so 3EE4F5D300F6AA9410F47A6BA24ECB900
1 libflashplayer.so 23AE5B6205B1D58FBE4150AB185049FC0
1 libflashplayer.so 01E9F2FB89FF8A927955CA54DDF8F4920
total 200

Comment 17

[Karl Tomlinson (:karlt)]

The debug ids are google breakpad's own brand of identifier.  The method
changed recently (for 3.6.4) from an md5sum of the text section to something
like an XOR of the first 4096 bytes of the text section.

http://hg.mozilla.org/releases/mozilla-1.9.2/diff/62605e3b7322/toolkit/crashreporter/google-breakpad/src/common/linux/file_id.cc

This means that the same file has different identifiers before and after this
change and so how to match the ids in comment 16 depends on which versions the
crash reports were from.

> 50 libflashplayer.so 617F25534A9BB6F7F0A0820B6216D3530

This is the old id for x86_32 LNX 10,0,45,2 (latest stable).

> 32 libflashplayer.so 9BC43748D925AB0C97B43BB3C7019E210

This is the new id for x86_32 LNX 10,0,45,2 (latest stable).

> 4 libflashplayer.so D708CF1409A7A3DE15F591C53914549B0

This is the new id for x86_64 LNX 10,0,45,2 (latest stable).
I guess we would only have new ids for x86_64.

Comment 18

[timeless]

chofmann: ok

Comment 19

[Karl Tomlinson (:karlt)]

Debug ids for the versions at http://kb2.adobe.com/cps/142/tn_14266.html
with numbers of crashes from comment 16:


x86_32 Xt versions:

9,0,31,0
old
> 4 libflashplayer.so ADF15068159E275F6F66F9A3D006F2F90
new
> 1 libflashplayer.so 23AE5B6205B1D58FBE4150AB185049FC0
bp-98f7b1dd-ad25-4a35-b413-e96c82100422
bp-b82150c7-a766-4727-b09f-6b67c2100419
bp-baada217-7170-4ceb-bff7-931862100418

9,0,48,0
old (74e1b5911e3f3e5e80bb24ec2cdc5f66)
new
> 3 libflashplayer.so FCDA6F558B67CF6B4063475C74EBBF4B0
bp-69b87f60-fc6c-47b1-a380-a8ca02100423
bp-a6a138e8-9a41-42b5-9623-8ee152100424
bp-f20a75c3-e595-4162-8802-11f812100420


x86_32 XEmbed versions:

9,0,115,0
old (7d7e1c4c87e80d6a377824c7622a119f)
new (1E22D0749A8E26FA2843391E4A4349030)

9,0,124,0
old
> 22 libflashplayer.so 745026564F2B12E0E3B2715D145A0F690
new (F62B236AB783ADE90966D7196294880A0)

9,0,151,0
old (b57a2d934a231511f92727454d476012)
new (07136AC2BC0942CB6E42E914950709FE0)

9,0,152,0
old (a3a7882f07a851da11537d43091f1715)
new (8771458B40782B2A54BC1F3543C823480)

9,0,159,0
old 
> 2 libflashplayer.so B90802BF65F9CE3B9841550D7C0FDE400
new (49894D868298C97B68494F8F2CA95BAB0)

9,0,246,0 (July or August 2009 - February 2010)
old
> 1 libflashplayer.so AB1A186B5B8C190D5E9A2C683D85461F0
new (9DAAE6B6BD74568B0C1888D6066082F00)

9,0,262,0 latest Flayer Player 9
old
> 10 libflashplayer.so 87934424747BF50DA30C8528037E75EE0
new (D8744156EECF9DF8DD4E635E865993C40)

10,0,12,36
old
> 5 libflashplayer.so E68208AF7DD76CE7A5016EB5EF110AC00
new (441EBC7C89F93C9C514821ADA595FE600)

10,0,15,3
old
> 1 libflashplayer.so 9EF10ADE595DA88B6297999314BB04D90
new (CA13026CF51DC8A87603CC3A1510E5040)

10,0,22,87
old
> 10 libflashplayer.so E8E4615D8D0DFC9AE0BE3E3E69920AB80
new
> 1 libflashplayer.so F1490E942E837412B93F7F6A82C744AD0

10,0,32,18 (July or August - December 2009)
old
> 19 libflashplayer.so 72973D3A409794225D71BF5796E80EBC0
new
> 5 libflashplayer.so 3F4FE6DB0E1FC865B3049718AC3521980

10,0,42,34 (Dec 2009 - February 2010)
old
> 5 libflashplayer.so EECE3AF0BC567DB9F2692ACAAA974F7F0
new
> 4 libflashplayer.so 9BF4278859263B0053642AE35840B3610

LNX 10,0,45,2 (latest stable).
old
> 50 libflashplayer.so 617F25534A9BB6F7F0A0820B6216D3530
new
> 32 libflashplayer.so 9BC43748D925AB0C97B43BB3C7019E210


So, of the 175 reports with a regular x86_32 distribution Flash Player plugin,

92 (53%) are with the latest stable version (of Flash Player 9 or 10).

126 (72%) are from people who updated sometime in the last 9 months.
49 (28%) are from people who have not updated in the last 9 months.

8 (5%) are from people using Xt versions.

We could ask whether this is representative of proportions of people using
older versions or whether older versions show up more because of more crashes.

I also don't know whether the sample in comment 16 came from and what
proportion are likely to upgrade Firefox because they upgrade Flash Player.

However, the question is really:
Should we be engineering to enable people to run plugins that are old enough
that they likely have known security holes?

Comment 20

[Karl Tomlinson (:karlt)]

A few of the other ids I recognize (11 of the reports in comment 16):

x86_32 10,1,53,7 (10_1_rc_linux_040510)
old
> 1 libflashplayer.so CD5D2174D75A0118490922BAE78C3B230
new
> 2 libflashplayer.so 390DEC869F4C2E7D7C03885CD7A231440

x86_64 10,0,45,2 (latest stable).
> 4 libflashplayer.so D708CF1409A7A3DE15F591C53914549B0

x86_64 10,0,32,18
> 1 libflashplayer.so CF1E82198F9EC8CB28BA0D972D7FDDB70

The id in this bug report for an Xt plugin (new id)
> 3 libflashplayer.so 9F60EA7237C2DB7834E97F0BDD08661D0

And 14 other reports have unrecognized ids:

> 5 libflashplayer.so 75C4D5A84AA5F22F07AD159D020895EF0
> 4 libflashplayer.so 74E1B5911E3F3E5E80BB24EC2CDC5F660
> 2 libflashplayer.so 49ACE7A2810BC1EE5958F095C25F09E20
> 1 libflashplayer.so F0B1F424C57BBC81825967A5591BE82D0
> 1 libflashplayer.so 3EE4F5D300F6AA9410F47A6BA24ECB900
> 1 libflashplayer.so 01E9F2FB89FF8A927955CA54DDF8F4920

Comment 21

[Karl Tomlinson (:karlt)]

(In reply to comment #19)
> I also don't know whether the sample in comment 16 came from and what
> proportion are likely to upgrade Firefox because they upgrade Flash Player.

Er, after proof reading:
I also don't know where the sample in comment 16 came from and what
proportion are likely to upgrade Firefox before they upgrade Flash Player.

Comment 22

[chris hofmann]

The sample is from linux crash reports received on 2010 04 22.

Comment 23

[Mike Beltzner [:beltzner, not reading bugmail]]

Should we just blocklist these ancient versions of flash and WONTFIX this bug?

Comment 24

[Benjamin Smedberg]

As I currently understand it, we don't actually have the capability to blocklist these: Flash on Linux doesn't export version numbers, so we can't use the AMO blocklist, and we don't have a DLL blocklist on Linux either.

Comment 25

[chris hofmann]

If its a case where "Flash doesn't seem to work any more after I upgraded to 3.6.4" thats a bit easier to guide users to a fix and evangelize, than "Firefox seems more crashy with 3.6.4.   we could "user-doc-needed" this bug, and maybe start some evalgelism outreach to linux magazine and press contacts to inform users of the need to update flash for a number of reasons.

Comment 26

[Mike Beltzner [:beltzner, not reading bugmail]]

Cheng; please see above. Firefox 3.6.4 will break Flash for some Linux users who had really old (and insecure) versions. They'll need to update.

Chris: how will they be able to update?

Comment 27

[chris hofmann]

I think this page does auto sniffing of platform to provide the right package 
http://get.adobe.com/flashplayer/ 

and there are some installation instructions for the various .deb, tar.gz, .rpm, APT and YUM installer formats here

http://www.adobe.com/products/flashplayer/productinfo/instructions/

Comment 28

[Chris Ilias [:cilias]]

Re: user-doc-needed
I've filed an article request at <https://support.mozilla.com/en-US/forum/5/659753>