782368 - Why are mask surfaces not playing nicely with each other?

Status: RESOLVED FIXED

(Core :: Graphics: Layers, defect)

Description

[Nick Cameron [:nrc]]

Find out why, fix it, undo bug 780868. See that bug for more details.

Comment 1

[Nick Cameron [:nrc]]

Investigations so far: I've managed to reliably recreate this bug using a small but ugly hack, using a D3D10 layer manager just to create the mask surface. Of course this could cause different bugs, but hopefully not. Without this hack I could not recreate the intermittent fails that we see on Try, even with an old version of the tree. There is nothing wrong with the actual surface as far as I can see. Every where in the code I look at it in the debugger it is fine, and Cairo seems to be using it fine. This is the same as Matt Woodrow observed. However, I am noticing problems that seem to be transform related. This doesn't make much sense because nothing about a transform is cached, and the transform is associated with the mask layer, not the mask image, but we do transform the context before drawing into it, so maybe that is an issue.

Comment 2

[Nick Cameron [:nrc]]

Further investigation reveals that it is probably nothing to do with transforms after all.

Comment 3

[Nick Cameron [:nrc]]

Well, it seems to be kinda related to transforms. My best guess is that Cairo D2D surfaces do something differently with transforms which does not play nice when Cairo uses that D2D surface as a mask for a non-D2D-surface-backed context. Specifically, I'm guess it is something to do with what happens when we draw outside the surface - if I use surface which is larger than required, the bug goes away. Obviously, it is not as simple as that to fix, because there is no bug when the D2D surface is used as a mask using the D3D10 backend, which is why I think there is something funny going on in Cairo.

Comment 4

[Nick Cameron [:nrc]]

Bas: any idea what could be going on here?

Comment 5

[Nick Cameron [:nrc]]

This is puzzling me. There is something 'wrong' with the surface - painting stuff into it to try to help debugging doesn't work properly. I tried to set rgba = 0.5 everywhere, but this has the same effect as rgba = 1, any value > 0.0 has the same effect. I've played around scaling and translating various surfaces and don't get anything like the results I expect. I've copied the surface to an intermediate image surface, and that makes no difference. That does show that the pixel data looks plausible (it is not all ffffffff, as you would expect from rendering), I can't judge if it is exactly right but it seems on the right track. However, it looks like the rgb channels are varying independently, which I would not expect, but possibly some kind of AA artifact? I see, for e.g., 0080ffff.

I think there is some problem in the Cairo D2D code, I can't think of anything else that could be causing this, but can't get any more specific.

Comment 6

[Nick Cameron [:nrc]]

I've just come back to this after it slipped under my radar for a while, and it looks like the problem has magically resolved it self. I don't think this is work trying to bisect though.

Try run with 780868 backed out: https://tbpl.mozilla.org/?tree=Try&rev=979facc49d8d

With the hack described above I couldn't recreate the problems (and it could previously reproduce the problem), nor could I see the problems with repeated runs of local or Try ref tests.

So, I vote we backout the patch from 780868 and see what happens.

Comment 7

[Nick Cameron [:nrc]]

Attachment: undo 780868

Comment 8

[Nick Cameron [:nrc]]

Attachment: hack patch

Keeping the mixed layer manager hack around for posterity and in case we need it again. Don't even think about landing this thing.

Comment 9

[Nick Cameron [:nrc]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/390fb4d21787

Comment 10

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/390fb4d21787