So I think I've more or less figured out what's going on here.
In bug 1534389, I made it so that we fire dispatch mouse move events down into Gecko if we detect mouse moves within a draggable region - this was so that we could get the Picture-in-Picture controls to display on hover when mousing over it (since the whole thing is more or less a draggable region).
I think we're seeing a tug-of-war over who can set the mouse cursor - the tug-of-war is between EventStateManager / PresShell and widget/windows/nsWindow.cpp.
Let's suppose that the mouse is at the top of the window, over a draggable region, and just about to enter the point where we expect the resize cursor to appear. Let's say we're 1 pixel away, so if we move the mouse up 2 pixels, we expect to transition from the default cursor to the resize cursor.
The widget layer first detects the mouse movement, and does a hittest via the WM_NCHITTEST message. The hittest resolves to HTCAPTION, so Windows interprets the result of the WM_NCHITTEST message as, "you're in a draggable region". So at this point, we still show the default pointer cursor.
Next, the WM_NCMOUSEMOVE event comes in. Thanks to my code in bug 1534389, since we determine that we're within a draggable region, we re-dispatch that as a normal WM_MOUSEMOVE message. This causes the mouse movement to be turned into a Gecko event, and sent down to the EventStateManager.
This next part is kinda timing dependent, and also where my understanding gets a bit fuzzy...
PresShell seems to have this notion of a synthesized mousemove event... I believe this is in the event that the underlying structure of the page changes without the mouse actually moving. Anyhow, one of those synthetic mouse move events seems to get queued up via PresShell::DidDoReflow.
So at this point, we have our synthetic mousemove all ready to go, waiting for FlushType::Display to fire.
The mouse moves again. The widget layer first detects the mouse movement, and does a hittest via the WM_NCHITTEST message. The hittest resolves to HTTOP, so Windows interprets the result of the WM_NCHITTEST message as, "you're at the top of a resizable window". So at this point, we show the resize cursor. Hooray!
And then our synthetic mousemove event is handled. The EventStateManager handles that synthetic event, and tells the widget layer to update the cursor again - this time, to the default cursor (the pointer), because the frame that the mouse is over (presumably) doesn't have a
cursor CSS rule set.
So then the cursor is set back to the pointer.
Thanks to the hittest, clicking still causes the resize action to work, but the mouse cursor doesn't really communicate this.
I think this is what's happening, and I think the reason that this is a problem now is that synthetic mouse events weren't ever in conflict like this with the widget hittest stuff. However, now that we're sending down WM_MOUSEMOVE events to Gecko when mousing over draggable regions with WM_NCMOUSEMOVE... the bug has kinda been revealed.