It first appeared in 14.0a1/20120315:
It's currently #3 top crasher in 14.0a1.
More reports at:
ajuma, do you have any recommendations for this topcrash? It looks like we're crashing in the Atrix MB860's GL driver.
We need to get our hands on the GL driver from this device so we can grab symbols and find out where in the driver this is happening.
Knowing the URLs where this crash happens would be helpful too.
There's one crash on LG-P990 CyanogenMod: bp-3a0f9dab-423b-4345-9823-5c33f2120318.
Only two URLs came up in the search:
The crash seemed to have died down. Last crash from build: 20120331031108
Found only one other URL:
(In reply to Naoki Hirata :nhirata from comment #5)
> The crash seemed to have died down. Last crash from build: 20120331031108
There are still recent crashes, but it's no longer a top crasher.
There's a spike in crashes from two users that use those devices unsupported on native Fennec, but supported on XUL Fennec.
It's within the top 10 top crashers now... not placing in topcrash due to comment 7
It affects 11 users over the last week.
Should we consider MB860 (2.3% of Android devices - source ) and LG-P99* as unsupported?
Well, I have an MB860/Atrix... would be nice to figure out what's going on :)
It's #6 top crasher in the first days of 14.0b1.
It occurs also after the fix of bug 748531.
It would be interesting to have recent URLs now that 14.0b1 is released.
Scoobidiver, what's the 14.0b1 that you are looking at? We have some bad data for 14b1 graphs. Yesterday was the first official release of 14b1 Native; 14b1 XUL is not released.
FWIW, those LG phones and the Atrix are all Tegra devices, and CgDrv_Create makes me think NVIDIA too.
(In reply to Naoki Hirata :nhirata from comment #13)
> Scoobidiver, what's the 14.0b1 that you are looking at? We have some bad
> data for 14b1 graphs. Yesterday was the first official release of 14b1
> Native; 14b1 XUL is not released.
This 14.0b1, https://crash-stats.mozilla.com/query/query?product=FennecAndroid&version=FennecAndroid%3A14.0b1&range_value=1&range_unit=weeks&query_search=signature&query_type=contains&query=&do_query=1, updated continuously contrarily to https://crash-stats.mozilla.com/topcrasher/byversion/FennecAndroid/14.0b1 updated once a day.
Yes, /query (advanced search) is doing live searches on what's in the database, topcrasher is generated once every day for the recent UTC day.
(In reply to Scoobidiver from comment #15)
> (In reply to Naoki Hirata :nhirata from comment #13)
> > Scoobidiver, what's the 14.0b1 that you are looking at? We have some bad
> > data for 14b1 graphs. Yesterday was the first official release of 14b1
> > Native; 14b1 XUL is not released.
> This 14.0b1,
> &query=&do_query=1, updated continuously contrarily to
> updated once a day.
Thanks Scoobi for the link. Would this link qualify the public beta (release date of 5/15) better?
(In reply to Naoki Hirata :nhirata from comment #17)
> Thanks Scoobi for the link. Would this link qualify the public beta
> (release date of 5/15) better?
My link is shorter because, with the date field, new crashes are not taken into account, and the reason, build_id, process_type and hang_type fields have their default value.
Ah. I was referring to adding the date of 5/22/2012 in the search field so that the start date would be from the official release date of 5/15.
These seem to have disappeared since the 0516 nightly build; I'm guessing that this is another bug 752368 dup. Also, given that most of the crashes are from CM users, not sure this needs to block. Keeping this open until we can verify crash numbers when next beta goes out.
(In reply to Vladimir Vukicevic (:vlad) from comment #20)
> These seem to have disappeared since the 0516 nightly build
Nightly is not statistically representative. There are still recent crashes in Aurora after the fix of bug 752368: bp-6a5085ee-1414-474f-bb31-542132120523.
True, I guess 3 crashes are enough to say "It's still happening", though much lower on the list. It could still be bug 752368 but in a different situation, but like I said, let's keep this open until the beta ships and see what happens.
Looks like this is happening on beta3 still:
Bug 757944 might help here.
I'd be surprised if bug 757944 helps, but maybe!
Bug 757944 has stalled - do we need to unstick it?
(In reply to JP Rosevear [:jpr] from comment #26)
> Bug 757944 has stalled - do we need to unstick it?
Bug 757944 has now landed on inbound. I'm also going to try to get symbols for these crashes, so that we can get a better idea of what might be causing this.
FWIW, the entry point into the driver is unsurprisingly glCompileShader()
(In reply to Jeff Muizelaar [:jrmuizel] from comment #28)
> FWIW, the entry point into the driver is unsurprisingly glCompileShader()
And it crashes because the compiler does a division by 0. It would be nice to know which shader is causing the crash.
Interestingly, there haven't been any of these on FF15 for quite a while.
Does anyone have one of the devices mentioned?
We suspect that this might be flash related; it would be helpful if we could get a module correlation for all of these crashes, to see if libflashplayer.so appears in all of them.
So looking more at this, these all look to be Tegra 2 devices that are running some form of ICS rom (almost all non-vendor-provided, e.g. CM9). The flash version is likely to be the same one that shipped with gingerbread on these phones (which is the stock vendor-provided one), and so we're most likely getting some sort of form of bug 703056, but we're not being blocked because the devices are running ICS.
I think we need a finer-grained block here somehow.
All of the crashes in b4/b5 are with 1) homebrew ICS; 2) libflashplayer.so with a 'debug id' of 79B73C212164CC7699C039D8B6C646570 . Some of the online reviews about CM9 on some of these devices say that flash does not work with the built in browser.
That 'debug id' is generated by taking the first 4096 bytes of the text section of the .so and xor'ing guid-sized chunks together, as seen here http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/common/linux/file_id.cc#144 .
So maybe we block flash on Tegra 2 devices that have libflashplayer.so that has that signature?
The latest libflashplayer.so that's served by the market is:
SDK version 8..14 (downloaded on my Atrix with gingerbread):
79 b7 31 21 14 20 cc 76 15 a7 39 d8 b6 c6 46 57 0
SDK version 15 onward (downloaded on my Galaxy Nexus with ICS):
79 b7 3c 21 cc 76 21 64 d8 39 c0 99 57 46 c6 b6 0
(note if anyone repeats this exercise: you can xor the bytes together, but the version that's displayed on breakpad is the GUID form -- int, short, short, bytes -- so you have to rearrange bytes appropriately). The flash version that we're crashing with is the ICS one which makes sense, that's what they'd get from the market given that they'll be prsenting an ICS SDK version.
So we have:
1) Homebrew ICS
2) Tegra 2
3) ICS Flash 11
Renomming. This only seems to happen with aftermarket android firmware; most common with CM9 but with others as well. I suggest we either minus and ignore this crash for now, or add some filtering based on kernel version and block some common kernel strings (CM9, cyanogen, nova, a few others). I may just write up that version anyway, since should just be able to look at /proc/version.
Created attachment 629877 [details] [diff] [review]
maybe like this?
Sort of band-aid-y, but should work to get rid of this crash. It would be nice to have a way to override our blacklisting of Flash, but meh.
Comment on attachment 629877 [details] [diff] [review]
maybe like this?
Review of attachment 629877 [details] [diff] [review]:
Should probably put vreader.close() in a finally() clause since readLine() and other stuff can throw. Looks fine otherwise.
Created attachment 630239 [details] [diff] [review]
Better; adds some logging as well, just in case.
Comment on attachment 630239 [details] [diff] [review]
Review of attachment 630239 [details] [diff] [review]:
Might as well put the bug number in the log message too :)
Pushed to inbound with bug # in log message:
Based on affected devices, I add the kill | raise signature: https://crash-stats.mozilla.com/report/list?signature=kill+|+raise
It's definitely the same set of devices, but it might not be the same issue. I wouldn't necessarily expect it to go away with this patch.
removing QA wanted. Not going to root a specific phone w/ CM specifically for this. Will monitor in Socorro.
Vlad, please nominate this for aurora and beta
Comment on attachment 630239 [details] [diff] [review]
[Approval Request Comment]
User impact if declined: ugly crashes with flash plugin on some devices, though largely ones with CyanogenMod and other custom firmware
Testing completed (on m-c, etc.): on m-c
Risk to taking this patch (and alternatives if risky): none; just adds some more checks to our existing flash blacklisting code
String or UUID changes made by this patch: none
There are still crashes in the trunk:
Yup, those are all on platforms where we didn't block it -- "p990-ics" is one of them, because it wasn't clear whether that was a custom firmware or an early build of ics for the p990. The third one there could be a normal motorola kernel; they have a bad habit of building some random git revision without giving it any useful name. Regardless, none of those crashes would have been stopped by the patch here, but the overall number should have gone down.
Target Milestone is for m-c.
There are still crashes: https://crash-stats.mozilla.com/report/list?signature=CgDrv_Create