Created attachment 657600 [details]
Reproducible with ASAN-build from https://people.mozilla.com/~choller/firefox/asan/20120831-mozilla-central-linux64-debug-fcc533f691e9+asan.html
==1095== ERROR: AddressSanitizer heap-buffer-overflow on address 0x7fae06810023 at pc 0x7fae417985cc bp 0x7fff9dfb8c80 sp 0x7fff9dfb8c78
READ of size 1 at 0x7fae06810023 thread T0
#0 0x7fae417985cc in Convolve3x3 /home/attekett/firefox/src/content/svg/content/src/nsSVGFilters.cpp:4926
0x7fae06810023 is located 3 bytes to the right of 4000-byte region [0x7fae0680f080,0x7fae06810020)
allocated by thread T0 here:
#0 0x422e7c in posix_memalign ??:0
#1 0x7fae428f6565 in TryAllocAlignedBytes /home/attekett/firefox/src/gfx/thebes/gfxImageSurface.cpp:89
#2 0x7fae41764886 in nsSVGFE::SetupScalingFilter(nsSVGFilterInstance*, nsSVGFE::Image const*, nsSVGFE::Image const*, nsIntRect const&, nsSVGNumberPair*) /home/attekett/firefox/src/content/svg/content/src/nsSVGFilters.cpp:153
#3 0x7fae41782077 in nsSVGFELightingElement::Filter(nsSVGFilterInstance*, nsTArray<nsSVGFE::Image const*, nsTArrayDefaultAllocator> const&, nsSVGFE::Image const*, nsIntRect const&) /home/attekett/firefox/src/content/svg/content/src/nsSVGFilters.cpp:5009
#4 0x7fae4169d8e5 in nsSVGFilterInstance::Render(gfxASurface**) /home/attekett/firefox/src/layout/svg/base/src/nsSVGFilterInstance.cpp:583
#5 0x7fae416944d4 in nsSVGFilterFrame::PaintFilteredFrame(nsRenderingContext*, nsIFrame*, nsSVGFilterPaintCallback*, nsRect const*) /home/attekett/firefox/src/layout/svg/base/src/nsSVGFilterFrame.cpp:444
I'm looking into this, but it's going to take some time to grok this part of the filters code.
I don't understand the relevant parts of the SVG filter code in enough detail to feel satisfied that it's doing the right thing. That said, the issue seems to be that in nsSVGFELightingElement::Filter we end up looping over sourceData and targetData for each pixel in dataRect, but dataRect is 24x11 px while the images that own sourceData and targetData are only 25x10 px. In other words we loop over 264 pixels when only 250 have been allocated for the images. The root cause of the problem seems to be that SetupScalingFilter (called right at the top of nsSVGFELightingElement::Filter) has a rounding issue when scaling aDataRect which causes it to return the 24x11 px result.mDataRect.
Created attachment 660408 [details] [diff] [review]
Do we think the ESR affected by this bug?
Assuming ESR-10 is affected because SVG Filters were around then.
Comment on attachment 660408 [details] [diff] [review]
[Approval Request Comment]
Bug caused by (feature/regressing bug #): issue has been in the code since forever
User impact if declined: may or may not be exploitable in different ways in different versions and on different platforms
Testing completed (on m-c, etc.): baked on m-c for a couple of days
Risk to taking this patch (and alternatives if risky): very low risk
String or UUID changes made by this patch: none
mass remove verifyme requests greater than 4 months old