nsSVGMaskFrame::ComputeMaskAlpha doesn't account for transforms that are already applied to the gfxContext that is being painting to. For elements that are under a CSS transform that's scales the element up, this means that it creates surfaces that are too small (or, more to the point, surfaces that don't have enough resolution) when creating the mask pattern it returns. As a result, the more the element is scaled up, the more pixelated the masking becomes.
This isn't generally a problem for SVG, since the transforms aren't accumulated on the gfxMatrix as we descend down the tree. However, when switching to display list painting of SVG, we do accumulate the transforms as we descend the tree, and this bug causes various SVG reftests to fail.
Created attachment 637318 [details] [diff] [review]
Created attachment 637440 [details] [diff] [review]
Actually, that's not right. While falling asleep last night I got to wondering why I could use SetDeviceOffset() at all, and this morning I came up with a test that proves that I shouldn't be. Here's a new patch to fix that, along with that test.