252067 - Add support for translucent windows for Win32 (Windows 2000 and later)

Status: RESOLVED FIXED

(Core :: XUL, enhancement)

Description

[Dainis Jonitis]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040614 Firefox/0.9
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7) Gecko/20040614 Firefox/0.9

Patch available to implement translucent windows.

Reproducible: Always
Steps to Reproduce:
See bug 113232 for more details

Run Mozilla with -chrome from:
http://bugzilla.mozilla.org/attachment.cgi?id=111901&action=view
Actual Results:  
The window background is not transparent. Octopus is drawn on white background.

Expected Results:  
The window background should be transparent

This is  a followup for bug 113232 that adds support for translucent windows for
Windows 2000 and Windows XP. Older Windows versions will not support this
because this patch uses UpdateLayeredWindow API function.

Comment 1

[Dainis Jonitis]

Attachment: diff -d -u -8

Comment 2

[Dainis Jonitis]

Attachment: diff -d -u -8 -w (for review purposes only)

Comment 3

[Dainis Jonitis]

The changed files:
1. gfx/src/nsBlender.cpp
2. widget/public/nsIWidget.h  (spelling and whitespace only)
3. widget/src/windows/nsWindow.h
4. widget/src/windows/nsWindow.cpp

While learning widget code I noticed many spelling errors in comments which I
fixed. I also replaced tab symbols with spaces. Both nsWindow.h and nsWindow.cpp
were changed to make the indentation actually be 2 spaces as declared in header.
These changes were not required, but I guess some cleanup is required from time
to time.

The real changes are at the end of nsWindow.cpp in #ifdef MOZ_XUL section.
Because the new utility function GetTopLevelHWND was created I changed all
places in nsWindow.cpp what were looking for topmost HWND to use it.

I also replaced the GetNativeData(NS_NATIVE_WINDOW) with mWnd when it was called
for 'this' or nsWindow type instead of nsIWidget.

I also changed the conditions in WM_WINDOWPOSCHANGED part to check only for
window growth to avoid useless system calls in case when window was minimized.


This code was compiled with MS VS2003.NET compiler and is not tested with VS6.
It is tested on Win98 where translucency is not supported.
It works as expected on Windows XP
On Windows 2000 the translucency works, but clicks on the buttons don't. Seems
that something is wrong with coordinates because the button click is triggered
when you click on the octopus image. This incompatibility between Win2K and
WinXP is strange. Still have to investigate. At the moment I do not have access
to Win2k machine to debug this - help wanted.

Except Win2k mouse coordinate strangeness everything seems to work fine. Code is
ready for review. 

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Could it be that Win2K is incorrectly compensating for non-existent window
decorations?

Dainis, does translucency (fractional alpha values) work if you throw a
translucent PNG in there? I'd love to see a screenshot of that :-)
 
In the future it's probably better to keep cleanup separate from actual
functionality changes.

Doing OnPaint at every Invalidate might not be a good idea, but let's not worry
about that for now. We should be batching invalidates at a higher level via the
view manager BeginUpdateViewBatch mechanism, anyway.

In nsWindow::Update, I think you don't need to do anything in the translucent
window case. The window is always up to date.

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

+  PRPackedBool  mIsTranslucent;
+
+  HBITMAP       mMemoryBitmap;
+  HDC           mMemoryDC;
+  PRUint8*      mAlphaMask;

These should be #ifdef XUL, right?

Comment 6

[Dainis Jonitis]

Memeber variables take only 13 bytes. Is it worth to put everything in MOZ_XUL
if in most cases it will be defined? Then it will require too much #ifdef's...
Probably I can always have mIsTransparent (which will always be PR_FALSE) and
use conditional compilation only for mMemoryBitmap, mMemoryDC and mAlphaMask.

Comment 7

[Dainis Jonitis]

Attachment: PNG with alpha transparency

Comment 8

[Dainis Jonitis]

Attachment: Chrome XUL for PNG with 256 levels of transparency

Comment 9

[Dainis Jonitis]

Attachment: Screenshot of 256 level alpha transparency

Comment 10

[Robert O'Callahan (:roc) (email my personal email if necessary)]

> Probably I can always have mIsTransparent (which will always be PR_FALSE) and
> use conditional compilation only for mMemoryBitmap, mMemoryDC and mAlphaMask.

Sounds good. It's only 13 bytes, but it's 13 bytes per nsWindow widget.

Thanks for the screenshot. Looks fantastic!

Comment 11

[Dainis Jonitis]

Attachment: diff -d -u -8

Comment 12

[Dainis Jonitis]

Attachment: diff -d -u -8 -w (for review purposes only)

Comment 13

[Dainis Jonitis]

Addressed Robert`s review notes:
1. Now all new member variables (including mIsTranslucent) are in #ifdef MOZ_XUL.
   Without define there should be zero overhead. The code size should shrink 
   because of removal of duplicates of GetTopLevelHWND and unneeded GetNativeData.
2. nsWindow::Update for translucent windows does nothing. Currently I just
   commented out call to UpdateTranslucentWindow, because I do not know how to
   enforce the nsWindow::Update to be called and test what happens.

Comment 14

[Dainis Jonitis]

About mouse clicks not working right on Win2000.
The WM_MOUSEMOVE, WM_LBUTTONDOWN and WM_LBUTTONUP are not generated if mouse
cursor is in the non-client area. The vertical coordinate that starts generating
messages is 23 pixels.
When window is minimized and then restored everything starts to work as expected. 
Why this is different on Win2000 compared to WinXP is still unclear to me.
Win2000 bug?

Comment 15

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I suspect Win2K bug. The uses of translucent windows in Win2K did not respond to
events.

Are the horizontal coordinates also off by a little bit?

Comment 16

[Robert O'Callahan (:roc) (email my personal email if necessary)]

So when you mousedown over the octopus, the correct coordinates are generated in
Win2K?

You could try overriding WM_NCHITTEST to always return HTCLIENT if you think the
cursor is in the window.

Comment 17

[Dainis Jonitis]

Aha, now I see why after minimizing/restoring it started to work. The WS_CAPTION
style for topmost window has gone.

If I change SetWindowTranslucencyInner to remove WS_CAPTION:

  HWND hWnd = GetTopLevelHWND(mWnd);
  LONG_PTR style = ::GetWindowLongPtr(hWnd, GWL_STYLE) & ~WS_CAPTION;
  LONG_PTR exStyle = ::GetWindowLongPtr(hWnd, GWL_EXSTYLE) & ~WS_EX_LAYERED;
  if (aTranslucent)
    exStyle |= WS_EX_LAYERED;
  ::SetWindowLongPtr(hWnd, GWL_STYLE, style);
  ::SetWindowLongPtr(hWnd, GWL_EXSTYLE, exStyle);

it started to work on Win2k, too. Now I have to figure out why WS_CAPTION
appeared on Win2k and test if it still works on WinXP.

Comment 18

[Dainis Jonitis]

Attachment: diff -d -u -8

Comment 19

[Dainis Jonitis]

Attachment: diff -d -u -8 -w (for review purposes only)

Comment 20

[Dainis Jonitis]

The latest patch makes mouse clicks to work correctly on Windows 2000.
Seems that Win2k has some limitation/bug that prevented HideWindowChrome to set
window style if window previously was already set to WS_EX_LAYERED mode in
SetWindowTranslucencyInner. To workaround that the WS_CAPTION bits are removed
from style in the SetWindowTranslucencyInner, before WS_EX_LAYERED mode is set.

Now this works on both Win2K and WinXP. 

Special thanks to Robert O'Callahan for initial implementation of translucency
on GTK and being very helpful while working on this bug! 
Now we need MNG for really cool technology demos :)

Comment 21

[Robert O'Callahan (:roc) (email my personal email if necessary)]

+#ifdef MOZ_XUL
+  PRPackedBool  mIsTranslucent;
+  HBITMAP       mMemoryBitmap;
+  HDC           mMemoryDC;
+  PRUint8*      mAlphaMask;
+#endif
     char        mLeadByte;

put mLeadByte above, among the packedbools. That might save some space.

Where are mMemoryDC and mMemoryBitmap destroyed?

+  } else
+  {
+    ::DeleteDC(mMemoryDC);
+    ::DeleteObject(mMemoryBitmap);
+    delete [] mAlphaMask;

I think you should set these to NULL here.

+            pPixel [0] = (*pAlpha * pPixel [0]) >> 8;
+            pPixel [1] = (*pAlpha * pPixel [1]) >> 8;
+            pPixel [2] = (*pAlpha * pPixel [2]) >> 8;

Shouldn't this be division by 255? (I'm not sure.) If so, nsBlender.cpp has a
fast way to do this.

Other than that, I think it's ready.

Comment 22

[Dainis Jonitis]

Attachment: diff -d -u -8

Comment 23

[Dainis Jonitis]

> put mLeadByte above, among the packedbools. That might save some space.
Moved #ifdef MOZ_XUL above all PRPackedBool variables. Moved mIsTranslucent
to the bottom of this define. Thus the define block starts with 4 byte HBITMAP
which follows the other 4 byte HBRUSH. No memory alignment holes.

> Where are mMemoryDC and mMemoryBitmap destroyed?
DeleteDC and DeleteObject release the GDI objects. There are no GDI handle
leaks. Now after relese I reset both mMemoryDC and mMemory bitmap to NULL
and mAlphaMask to nsnull.

> Shouldn't this be division by 255? (I'm not sure.) If so, nsBlender.cpp
> has a fast way to do this.
Now nsColor.h is included and FAST_DIVIDE_BY_255 macro used.

Comment 24

[Robert O'Callahan (:roc) (email my personal email if necessary)]

> DeleteDC and DeleteObject release the GDI objects. There are no GDI handle
> leaks.

I don't see where they are released when the nsWindow is destroyed normally.

Comment 25

[Dainis Jonitis]

> I don't see where they are released when the nsWindow is destroyed normally.

Normally? Who would think about such rare condition :)

Something like that in nsWindow::Destroy

#ifdef MOZ_XUL
    if (mIsTranslucent)
    {
      ::DeleteDC(mMemoryDC);
      ::DeleteObject(mMemoryBitmap);
      delete [] mAlphaMask;

      mMemoryDC = NULL;
      mMemoryBitmap = NULL;
      mAlphaMask = nsnull;
    }
#endif


Now I will wait before updating patch every hour :)

Comment 26

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Right :-). sr=roc with that.

Comment 27

[Ere Maijala (slow)]

Comment on attachment 153805 [details] [diff] [review]
diff -d -u -8

This patch doesn't apply cleanly to nsWindow.cpp. I'd really really appreciate
separate patches for the style and spelling fixes and the real functionality.

Comment 28

[Dainis Jonitis]

Functionality first, cleanup later? Or vice versa?

Comment 29

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Functionality first please

Comment 30

[Dainis Jonitis]

Attachment: step 1 of 2. New functionality only

Comment 31

[Dainis Jonitis]

Per Ere`s request I split the patch into the two parts. The first step adds only
the new functionality, but skips all the whitespace cleanup, etc.

I will attach the step 2 that does all the cleanup once this one will be in CVS
and I will heave something to compare against. As written above the cleanup was
not only the whitespace changes and spelling errors but also the optimizations
in calling GetNativeData, GetTopLevelHWND, not calling API functions if window
was getting smaller etc.

Comment 32

[Ere Maijala (slow)]

Comment on attachment 154014 [details] [diff] [review]
step 1 of 2. New functionality only

r=ere

Comment 33

[Ere Maijala (slow)]

Patch checked in on behalf of Dainis.

Please open a new bug for the cleanup work.

Comment 34

[Ere Maijala (slow)]

I had to back out the checkin as it doesn't compile in Visual C++ 6 or needs a
newer Platform SDK than creature has. 

Comment 35

[Ere Maijala (slow)]

These errors were reported:

c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7430)
: error C2065: 'LONG_PTR' : undeclared identifier
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7430)
: error C2146: syntax error : missing ';' before identifier 'style'
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7430)
: error C2065: 'style' : undeclared identifier
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7431)
: error C2146: syntax error : missing ';' before identifier 'exStyle'
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7431)
: error C2065: 'exStyle' : undeclared identifier
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7431)
: error C2039: 'GetWindowLongPtr' : is not a member of '`global namespace''
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7431)
: error C2065: 'GetWindowLongPtr' : undeclared identifier
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7441)
: error C2039: 'SetWindowLongPtr' : is not a member of '`global namespace''
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7441)
: error C2065: 'SetWindowLongPtr' : undeclared identifier
c:/builds/tinderbox/MozillaTrunk/WINNT_5.0_Clobber/mozilla/widget/src/windows/nsWindow.cpp(7574)
: error C2065: 'AC_SRC_ALPHA' : undeclared identifier

Comment 36

[Dainis Jonitis]

Seems that old Platform SDK on build machines caused Tinderbox build errors:

1. nsWindow.cpp(7430) : error C2065: 'LONG_PTR' : undeclared identifier
 That could be fixed by using GetWindowLong and SetWindowLong instead of
 GetWindowLongPtr and SetWindowLongPtr. Older versions return LONG, but newer
 ones are 64-bit ready.

2. nsWindow.cpp(7574) : error C2065: 'AC_SRC_ALPHA' : undeclared identifier
  #ifndef AC_SRC_ALPHA
  #define AC_SRC_ALPHA   0x01
  #endif

Comment 37

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Regarding point 1, it would be nice to use #ifdefs or whatever to ensure that
the code will stay working on Win64...

Comment 38

[Dainis Jonitis]

Attachment: This should fix build bustage

Comment 39

[Dainis Jonitis]

1. I changed the GetWindowLongPtr to nsToolkit::mGetWindowLong (and
   same for setter).
2. Defined the missing constant.

Someone can try to build with this?

Comment 40

[Ere Maijala (slow)]

Comment on attachment 154395 [details] [diff] [review]
This should fix build bustage

It looks ok to me, but you should ask someone with VC6 to try it. I don't have
a VC6 build at hand right now.

Comment 41

[Chris Thomas (CTho) [formerly [e-mail] [e-mail]]]

It compiled for me.  Is there a testcase somewhere? "Chrome XUL for PNG with 256
levels of transparency" gives me a XUL error:

 XML Parsing Error: not well-formed
Location: http://bugzilla.mozilla.org/attachment.cgi?id=153661&action=view
Line Number 10, Column 74:  <image
src="http://bugzilla.mozilla.org/attachment.cgi?id=153660&action=view"/>
-------------------------------------------------------------------------^

The "^" points to the = in "action=view".

Comment 42

[Robert O'Callahan (:roc) (email my personal email if necessary)]

That XUL wasn't properly XML-escaped. Try "dist\bin\mozilla -chrome
http://bugzilla.mozilla.org/attachment.cgi?id=111901&action=view"

But really, the main thing is that it builds. You used VC6, Chris?

Comment 43

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Chris says he used VC6. Let's reland this ASAP

Comment 44

[Ere Maijala (slow)]

Comment on attachment 154395 [details] [diff] [review]
This should fix build bustage

I'll land it.

Comment 45

[Ere Maijala (slow)]

Patch checked in.

Comment 46

[David G King]

This breaks my Win32/MinGW/cygwin build with an error 'pUpdateLayeredWindow'
undeclared.

Earlier it has errors :-

e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7270: error: `WINUSERA
PI' does not name a type
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7273: error: extra `;'

e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7276: error: expected
init-declarator before '*' token
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7276: error: expected
`,' or `;' before '*' token
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp: In function `PRBool I
sTranslucencySupported()':
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7290: error: `pUpdateL
ayeredWindow' undeclared (first use this function)
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7290: error: (Each und
eclared identifier is reported only once for each function it appears in.)
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7290: error: `UpdateLa
yeredWindowProc' undeclared (first use this function)
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7290: error: expected
primary-expression before ')' token
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7290: error: expected
`;' before '::' token
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp: In member function `n
sresult nsWindow::UpdateTranslucentWindow()':
e:/mozilla/source/mozilla/widget/src/windows/nsWindow.cpp:7587: error: `pUpdateL
ayeredWindow' undeclared (first use this function)
make[5]: *** [nsWindow.o] Error 1

Comment 47

[Chris Thomas (CTho) [formerly [e-mail] [e-mail]]]

I tried making navigator.xul's <window> be style="background-color:transparent",
and there were 4 major issues:
1. No border/title bar.  I'm guessing that's expected (it's the behavior in the
testcase).
2. The content area is an empty gray box, and I don't see any page content.
3. Every time I open a menu (e.g. file menu), the rectangle it will occupy
becomes transparent (I can see my desktop), then a moment later the menu
appears.  This is slow.
4. Loading the browser took a while - it seemed like my overflowed personal
toolbar folder was moving bookmarks into the ">>" dropdown list one at a time.

Comment 48

[Dainis Jonitis]

After fix for bug 253108 "Translucent xul windows on gtk2 breaks gtk embedding"
I checked how translucent windows work in mfcembed, viewer and winembed and
mouse clicks on buttons did not worked. But do not panic - most likely I know
what is wrong. Similarly as previous Win2K problem the WS_EX_LAYERED can not be
used together with some other style flags. In the embeding clients the different
window styles were used than in mozilla.exe and looks that some additional flags
should be masked out in SetWindowTranslucencyInner().

BTW do we actually need system menu for windows with hidden chrome? Now
HideWindowChrome leaves these flags on WS_SYSMENU | WS_MAXIMIZEBOX | S_MINIMIZEBOX. 

About MinGW build failure. The Platform SDK defines WINUSERAPI as
  #if !defined(_USER32_)
  #define WINUSERAPI DECLSPEC_IMPORT
  #else
  #define WINUSERAPI
  #endif

I will try to fix both these problems in a couple of days. But I would like to
first know your thoughts about system menu for translucent windows.

Comment 49

[Robert O'Callahan (:roc) (email my personal email if necessary)]

No system menu for translucent windows.

Translucency should never be applied to windows that aren't top level; it should
never be turned on for embedders' windows.

Comment 50

[Dainis Jonitis]

If someone wants to fix MinGW bustage immediately then please remove the
WINUSERAPI from UpdateLayeredWindowProc declaration and check in the changes.
Other declarations in nsWindow.cpp that use WINAPI (like HeapWalkProc,
GetProcessHeapsProc) do not include WINUSERAPI in their declarations. WINUSERAPI
expands to nothing anyway.

typedef BOOL WINAPI UpdateLayeredWindowProc (...

Comment 51

[David G King]

Hmmm, I didn't expect it to work (I get paranoid about Win32 sometimes), but
removing the WINUSERAPI as mentioned above fixed my Win32/MinGW/cygwin build.

I'll leave it to Dainis to do the formal patch so he can fix all aspects in one
patch. My local builds now have the patch, so I'm not concerned anymore.

Comment 52

[Dainis Jonitis]

Attachment: Fix MinGW, ignore translucency for embedded apps, remove system menu for translucent windows

Comment 53

[Ere Maijala (slow)]

Comment on attachment 154656 [details] [diff] [review]
Fix MinGW, ignore translucency for embedded apps, remove system menu for translucent windows

r=ere

Comment 54

[Ere Maijala (slow)]

The supplementary patch is now checked in.

Comment 55

[Dainis Jonitis]

See bug 254191 for code cleanup.

Comment 56

[Jeff Leigh]

I'm not seeing the correct behavior on Win 2000 sp4 with build 2004081709
(1.8a3). I'm still seeing the testcase with the white background instead of the
transparent one.

Comment 57

[Dainis Jonitis]

(In reply to comment #56)

Are you running Mozilla with a -chrome param like this? 
mozilla -chrome "file:///c:/warpzilla/mozilla/dist/bin/alphapng.xul"

What video card are you using? Video driver version, DirectX version?

Comment 58

[Jeff Leigh]

When I run it from the link in Comment #0 it shows the octopus on a white
background. If I run it from the command line with a local copy of the file, it
will make a tab in the taskbar, but nothing shows on the screen. And I have to
kill Mozilla from the Task Manager. Right-clicking on the tab has no effect.

Changing the xul file to a different background color will allow the window to
appear, but obviously not transparent.

GeForce2 MX
Driver: 6.14.10.5216
DirectX 8.0

Comment 59

[Dainis Jonitis]

Attachment: Standalone Win32 test application

Comment 60

[Dainis Jonitis]

(In reply to comment #58)
Try standalone Win32 test application skintest.exe - should display the same
octopus with alternating transparency. If this will work then the problem is
somwhere Mozilla related, otherwise something is wrong with your config. 

> GeForce2 MX
> Driver: 6.14.10.5216
> DirectX 8.0

Try installing DirectX 9.0c, nVidia ForceWare 61.77.

Comment 61

[Jeff Leigh]

DirectX 9.0 and the new nVidia drivers fixed the problem. Thanks for the help.

Comment 62

[Dainis Jonitis]

DirectX 9.0c should not be a requirement (although strongly recommended),
because the original Win2K had only 8.0.
Looks that buggy nVidia drivers were causing the problem. That info could be
added to knowledgebase or known problems list.
I feel relief, knowing that is not a problem with my code :)

Comment 63

[Nigel McFarlane]

This excellent work has some nits needing shakeout. See dependancies.

- N.

Comment 64

[Dainis Jonitis]

Why this bug was reopened? All symptoms in bug 264707 are present even without
changes in this bug. This bug did not made any regressions - should be FIXED.
For window transparency issues in anything older than Windows 2000 see bug 264708.

Comment 65

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Please see bug 298889 which is attempting to address the behavior noted in comment  47.