855221 - crash in _cairo_quartz_draw_image @ CGAccessSessionRewind

Status: VERIFIED FIXED

(Core :: Graphics, defect)

Description

[Scoobidiver (away)]

Despite the stack trace, it seems to be a regression in Firefox as it first showed up in 22.0a1/20130325105600. The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=3acbf951b3b1&tochange=4d3250f3afea (best case)
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=0a10eca0c521&tochange=4d3250f3afea (worst case)
It might be caused by bug 716140.

Signature 	CoreGraphics@0x34cd3 More Reports Search
UUID	9614f5c4-d040-4c47-b3e6-d61c52130327
Date Processed	2013-03-27 07:07:56
Uptime	365
Last Crash	6.5 minutes before submission
Install Age	6.1 minutes since version was first installed.
Install Time	2013-03-27 07:01:34
Product	Firefox
Version	22.0a1
Build ID	20130326030941
Release Channel	nightly
OS	Mac OS X
OS Version	10.8.3 12D78
Build Architecture	amd64
Build Architecture Info	family 6 model 23 stepping 10
Crash Reason	EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash Address	0x10
App Notes 	
AdapterVendorID: 0x10de, AdapterDeviceID: 0x 8a0GL Context? GL Context+ GL Layers? GL Layers+ 
Processor Notes 	sp-processor10.phx1.mozilla.com_16437:2008; exploitablity tool: ERROR: unable to analyze dump
EMCheckCompatibility	True
Adapter Vendor ID	0x10de
Adapter Device ID	0x 8a0

Frame 	Module 	Signature 	Source
0 	CoreGraphics 	CoreGraphics@0x34cd3 	
1 	CoreGraphics 	CoreGraphics@0x34eeb 	
2 	CoreGraphics 	CoreGraphics@0x5fda3 	
3 	libRIP.A.dylib 	libRIP.A.dylib@0xd796

More reports at:
https://crash-stats.mozilla.com/report/list?signature=CoreGraphics%400x34cd3

Comment 1

[Scoobidiver (away)]

It occurs also on OS X 10.7 but at a lower volume.

More reports also at:
https://crash-stats.mozilla.com/report/list?signature=CoreGraphics%400x76fef

Comment 2

[Scoobidiver (away)]

With combined signatures, it's #3 top browser crasher in 22.0a1 and #2 in 23.0a1 on Mac OS X.

Here is a better stack trace:
Frame 	Module 	Signature 	Source
0 	CoreGraphics 	CGAccessSessionRewind 	
1 	CoreGraphics 	CGAccessSessionRelease 	
2 	CoreGraphics 	CGSImageDataUnlock 	
3 	libRIP.A.dylib 	ripc_DrawImages 	
4 	libRIP.A.dylib 	ripc_TilePattern 	
5 	libRIP.A.dylib 	ripc_GetColor 	
6 	libRIP.A.dylib 	ripc_Render 	
7 	libRIP.A.dylib 	ripc_DrawRects 	
8 	CoreGraphics 	CGContextFillRects 	
9 	CoreGraphics 	CGContextFillRect 	
10 	CoreGraphics 	CGContextDrawTiledImage 	
11 	XUL 	_cairo_quartz_draw_image 	cairo-quartz-surface.c:1895
12 	XUL 	_cairo_quartz_surface_paint_cg 	cairo-quartz-surface.c:2342
13 	CoreFoundation 	___CFBasicHashFindBucket1 	
14 	XUL 	_cairo_quartz_surface_paint 	cairo-quartz-surface.c:2382 

More reports also at: https://crash-stats.mozilla.com/report/list?signature=CGAccessSessionRewind

Comment 4

[Alex Keybl [:akeybl]]

Milan, who might be in the best position to perform stack/change inspection for this bug? Thanks :)

Comment 5

[Milan Sreckovic [:milan] (needinfo for best results)]

Joe is not back until Friday, and if this is related to bug 716140, he or Seth (covering in the interim) are good people for it. Seth, do you have the bandwidth right now?

Comment 6

[Seth Fowler [:seth] [:s2h]]

Milan, I already took a look at this and didn't have much luck. I'll be happy to circle back around to it if it starts sitting for too long without getting fixed, but I lack the experience with this part of the code to solve this quickly. Joe is probably a good person to triage this. I've needinfo'd him.

Comment 7

[Scoobidiver (away)]

Here are some comments:
"zooming/panning google maps and kaboom. weird."
"I was uploading pictures on iTunesConnect. I had several time this crash since yesterday evening latest update"
"using bill.com, had just looked at an expense I needed to approve."

Comment 8

[Marcia Knous [:marcia]]

I hit this crash this morning when my machine was idle. I had google maps open, but I don't believe I was doing anything at the time of the crash in maps. 

Some URLs for the [@ CGAccessSessionRewind ] crash:


3 	https://maps.google.com/
1 	https://www.facebook.com/dialog/oauth?client_id=139475280761&response_type=token
1 	http://fr.yahoo.com/
1 	https://maps.google.com/maps?oe=utf-8&rls=org.mozilla:en-US:unofficial&client=fi
1 	http://www.megazip.ru/ru/bike/yamaha/view/1124/13485/160973
1 	http://fr.geneawiki.com/index.php/L%27Allemand_Gothique
1 	http://thestiffcollar.com/index.php/edistributor?___store=default
1 	https://docs.google.com/viewer?a=v&pid=gmail&attid=0.1&thid=12d050bf68faf6aa&mt=
1 	https://maps.google.fr/
1 	https://docs.google.com/viewer?a=v&pid=gmail&attid=0.1&thid=13de9510ba0a6fcf&mt=
1 	http://www.emulroom.com/games/dendy/action/legend-of-kage-legenda-o-keydzhe/
1 	https://www.facebook.com/ajax/pagelet/generic.php/PhotoViewerInitPagelet?ajaxpip
1 	http://www.aol.com/
1 	https://images.4chan.org/pol/src/1365446781919.png
1 	https://maps.google.com/maps?oe=utf-8&client=firefox-nightly&q=27+Eastwind+St.&i
1 	http://it.volkswagen.com/it/models/nuova-golf/CC5.html
1 	http://leprosorium.ru/users/Pokras_Lampas
1 	http://www.artscow.com/photo-gifts/-160
1 	https://www.google.com/calendar/render?tab=Xc&pli=1
1 	http://www.google.com/
1 	http://www.nfl.com/
1 	http://this--is-moi.tumblr.com/post/36916980727/photoset_iframe/dying-slowly-by-
1 	http://advate.com/
1 	http://maps.google.com/
1 	http://www.weather.gov/
1 	http://www.emulroom.com/
1 	https://accounts.google.com/AddressNoLongerAvailable?service=wise&continue=https
1 	http://cantinetta.antinori.twinbitlabs.com/it/mosca/wine/vini-bianchi
1 	http://images.4chan.org/g/src/1365655944262.png
1 	https://maps.google.ca/maps?oe=utf-8&client=firefox-aurora&ie=UTF-8&q=toronto+cu
1 	javascript:false;
1 	http://zeroing.tumblr.com/tagged/sculpture
1 	http://failblog.cheezburger.com/failbook
1 	http://weeklyad.michaels.com/coupons/?storeId=9982&promotionCode=Michaels-130407
1 	https://docs.google.com/document/d/1-jaLcYBCfV-UqPGEQR8-FnJ_ekcfMs3OXCOQBQKgGTo/
1 	https://www.4chan.org/frames
1 	http://doodlestagsgraphics.proboards.com/index.cgi?board=general&action=post&thr
1 	http://m.163.com/iphone/software/31qsrj.html

Comment 9

[Joe Drew (not getting mail)]

Wonder if this is any different now that bug 857876 has landed. I hope so, but only time will tell.

Comment 10

[Tracy Walker [:tracy]]

Combined signatures would put this at #6. Will keep an eye on it.

Comment 11

[Tracy Walker [:tracy]]

Joe, has the patch in bug 856876 been uplifted to aurora?  Combined signatures put this at #7 on 22.0a2.

Comment 12

[Joe Drew (not getting mail)]

Not yet, but thanks for the reminder!

Comment 13

[:Felipe Gomes (needinfo for replies!)]

FWIW I got this crash today with Nightly 23.0a1 (2013-05-04) on Mountain Lion, so bug 857876 might not have covered it all. My crash:
https://crash-stats.mozilla.com/report/index/bp-ada2e150-4b7f-4f6f-bde1-b01d02130506

I'm almost sure it occurred when switching to a google maps tab that had been open for a long time (~1 day), across sleep/wake sessions.

I'll keep an eye if it happens again

Comment 14

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to :Felipe Gomes from comment #13)
> FWIW I got this crash today with Nightly 23.0a1 (2013-05-04) on Mountain
> Lion, so bug 857876 might not have covered it all. My crash:
> https://crash-stats.mozilla.com/report/index/bp-ada2e150-4b7f-4f6f-bde1-
> b01d02130506
> 
> I'm almost sure it occurred when switching to a google maps tab that had
> been open for a long time (~1 day), across sleep/wake sessions.
> 

Any non-default gfx. preferences set?

Comment 15

[Scoobidiver (away)]

It's #1 top browser crasher and accounts for 7.5% of crashes in 22.0b1 on Mac OS X.

Here are new comments:
"lots of tabs open, crashed on tab close"
"Was going though image tabs I opened from Tumblr when it crashed."
"Tried to use the Tinkercad website it just crashed."
"Just opened a new website… for SoapUI"
"posting on ebay"
"Playing Farmville on Facebook and it just quit on me. no warning"

Comment 16

[Joe Drew (not getting mail)]

So, interestingly, a crash I just looked at has a decoder thread concurrently writing into a PNG. I wonder if we were drawing that image, and if so, whether OS X really dislikes that.

https://crash-stats.mozilla.com/report/index/5d6a2c94-3299-4df1-a301-c41ed2130520

Comment 17

[Steven Michaud [:smichaud] (Retired)]

Very interesting!

You presumably mean the thread stack that contains a all to mozilla::image::nsPNGDecoder::WriteInternal.  I just looked at several of these crashlogs from Socorro, and all contain such a thread stack.

By the way, *all* of these crashes (on all versions of OS X) are in CGAccessSessionRewind(), at exactly the same place in that method's code.

A few days ago I started doing some analysis of these crashes.  But I need to spend another day or two before I'll have much to say.  With luck I'll have time for that later this week.

For my own future reference, here's an example where the thread stack that contains the call to mozilla::image::nsPNGDecoder::WriteInternal contains an unusual amount of code above it:

bp-26c381e3-fe94-4364-8622-aaa252130519

Comment 18

[Alex Limi (:limi) — Firefox UX Team]

Still happening in 24.0 Nightly (UX build in this case). I got it while viewing screenshots on Evernote/Skitch.

Comment 19

[Steven Michaud [:smichaud] (Retired)]

Kudos and congratulations and enormous thanks if you can figure out how to reproduce the crash.

Anything at all unusual (or even just non-vanilla) about your settings?

Comment 20

[Alex Keybl [:akeybl]]

(In reply to Scoobidiver from comment #15)
> It's #1 top browser crasher and accounts for 7.5% of crashes in 22.0b1 on
> Mac OS X.

This is bad - who's actively investigating this by the way?

Comment 21

[Joe Drew (not getting mail)]

I'm waiting on Steven for further information. Hopefully he can shed some light!

Comment 22

[Steven Michaud [:smichaud] (Retired)]

I've been working on this off and on for the last week.

I'll keep working on it, but so far I haven't discovered anything actionable.

Comment 23

[Joe Drew (not getting mail)]

I'll talk with folks about it and see what can be done.

Scoobidiver, how much crash volume do we have on Nightly? I ask because I wonder whether we can test out speculative fixes on Nightly instead of on Beta.

Comment 24

[Marcia Knous [:marcia]]

I added up all the signatures on the trunk - looks as if the last week we have about 51 crashes.

I reproduced this crash on the latest beta by loading one of the URLs and just leaving the browser idle - https://crash-stats.mozilla.com/report/index/bp-7accfd3c-d577-4471-8fe6-3447a2130524. The URL is in the crash report.

Comment 25

[Scoobidiver (away)]

(In reply to Joe Drew (:JOEDREW! \o/) from comment #23)
> Scoobidiver, how much crash volume do we have on Nightly?
In relative value, it's #5 top crasher in 24.0a1 on Mac OS X but the trunk is polluted by new temporary top crashers.
In absolute value, there are 0.08 crashes per 100 ADU in 22.0b1 and 0.06 crashes per 100 ADU in 24.0a1 so its volume hasn't changed.

New comment say:
"http://www.horsexpo.com/html/schedule.html"
"Tried to use the Tinkercad website it just crashed." (https://tinkercad.com/)

Comment 26

[Joe Drew (not getting mail)]

Attachment: probably fix

This makes us not flush in imgFrame::ImageUpdated(), which probably caused the bug. We already flush when necessary, so it's not only dangerous, it's also unnecessary!

Comment 27

[Joe Drew (not getting mail)]

https://tbpl.mozilla.org/?tree=Try&rev=1785f34473df

Comment 28

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 753675 [details] [diff] [review]
probably fix

Review of attachment 753675 [details] [diff] [review]:
-----------------------------------------------------------------

::: image/src/imgFrame.cpp
@@ +492,1 @@
>  {

ImageUpdated should have a comment that it is called on multiple threads.

Comment 29

[Steven Michaud [:smichaud] (Retired)]

(In reply to comment #24)

> The URL is in the crash report.

I don't see it, even in the raw dump.

Comment 30

[Joe Drew (not getting mail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2451cdbc2b06

Comment 31

[Joe Drew (not getting mail)]

Comment on attachment 753675 [details] [diff] [review]
probably fix

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 716140
User impact if declined: crashes on OS X
Testing completed (on m-c, etc.): just pushed to m-i, passed try
Risk to taking this patch (and alternatives if risky): Very low risk. Should be strictly better. If there's a bug, could possibly cause some images to not always be drawn completely, but this is *very* unlikely.
String or IDL/UUID changes made by this patch: none

Comment 32

[Marcia Knous [:marcia]]

http://www.comic-rocket.com/go?mark&nav=next&uri=http%3A//nonadventures.com/2009/04/04/a-broken-pumice/

(In reply to Steven Michaud from comment #29)
> (In reply to comment #24)
> 
> > The URL is in the crash report.
> 
> I don't see it, even in the raw dump.

Comment 33

[Steven Michaud [:smichaud] (Retired)]

Thanks Marcia.

But how long did you wait (how long did you leave the browser idle)?

Comment 34

[Alex Keybl [:akeybl]]

Comment on attachment 753675 [details] [diff] [review]
probably fix

Will approve Mon/Tue once this has had a day to bake on m-c.

Comment 35

[Joe Drew (not getting mail)]

When you do, set it as checkin-needed; I'm off Monday.

Comment 36

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/2451cdbc2b06

Comment 37

[Scoobidiver (away)]

There have been no crashes since 24.0a1/20130526.

Comment 38

[Joe Drew (not getting mail)]

https://hg.mozilla.org/releases/mozilla-beta/rev/0576c6f52402
https://hg.mozilla.org/releases/mozilla-aurora/rev/08a29ed728b9

Comment 39

[Manuela Muntean [Away]]

For 3 of the signatures: [@ CoreGraphics@0x34cd3], [@ CoreGraphics@0x76fef], and [@ CGAccessSessionRewind], I found some crash reports in Socorro regarding last month, marked with 2013-05-28 date in the build IDs, on Firefox 23.0a2.

More details here:

https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=CoreGraphics%400x34cd3&reason_type=contains&date=05%2F30%2F2013%2008%3A34%3A35&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=CoreGraphics%400x34cd3


https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=CoreGraphics%400x76fef&reason_type=contains&date=05%2F30%2F2013%2008%3A34%3A33&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=CoreGraphics%400x76fef


https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=CGAccessSessionRewind&reason_type=contains&date=05%2F30%2F2013%2008%3A34%3A59&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=CGAccessSessionRewind


I think we should wait a few more days, to see if the crash still reproduces with Firefox 22 beta 3 build.

Comment 40

[Scoobidiver (away)]

(In reply to Manuela Muntean [:Manuela] [QA] from comment #39)
> For 3 of the signatures: [@ CoreGraphics@0x34cd3], [@ CoreGraphics@0x76fef],
> and [@ CGAccessSessionRewind], I found some crash reports in Socorro
> regarding last month, marked with 2013-05-28 date in the build IDs, on
> Firefox 23.0a2.
The contrary (no crashes in 23.0a2/20130528) would be surprising as the patch landed in 23.0a2/20130529: http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?startdate=2+day+ago&enddate=now

Comment 41

[Scoobidiver (away)]

Looks good because no crashes since 23.0a2/20130529 and 22.0b3.

Comment 42

[Manuela Muntean [Away]]

Indeed, no more crashes since 23.0a2/20130529 and 22.0b3 in Socorro, for none of the signatures.