Ok so here's where I'm at now.
While messing around with solutions to this bug (shadows getting over-clipped by Webrender) I wrote this reftest:
== https://gankro.github.io/blah/webtests/1529992.html https://gankro.github.io/blah/webtests/1529992-ref.html
Using it, I discovered some issues with mBounds on nsDisplayText, which until now we had been forwarding to webrender directly, but which isn't actually correct for our usecase:
(attached image: the mBounds rect on two pieces of text with highly translated and blurred shadows -- note that the bounds are tight on two sides, and the shadow blurs are badly clipped on exactly those sides!)
mBounds represents the total area covered by the text and its shadows including the shadow offsets. This makes perfect sense for how gecko/invalidation works, but isn't what webrender wants. It just wants the area covered by the text. Webrender has two modes: assume the bounds are inflated for the shadows already, or don't. Previously we ran in the latter mode, we are currently running in the former mode. However, in either case webrender then applies the shadow offsets to those bounds. This has mostly been accidentally working since if there are low offsets or low blurs, it all works out well enough.
However with large shadow offsets and large blurs, we develop artifacts on the sides the shadow isn't on, because those sides are still ~tight against the glyphs, but webrender is translating them to where the shadow is, meaning we are missing the blur expansion on that side. In this regard the "don't assume the bounds are inflated" mode was more correct, because indeed the bounds aren't inflated in the way webrender wants.
But then we also have massively over-inflated bounds on the other side, which leads to extra work for the blur shader, and additional overdraw, because we think actual content might be in there. So we either get really bad overdraw (slow), or incorrect extreme blurs (wrong).
What we really want is mBounds without any acknowledgement of shadows. Then webrender can apply the inflations and offsets on its own terms and get everything nice and tidy.
However even without shadows, mBounds is sometimes extremely large, even if the text is obscured by something (such as an overflow:hidden ancestor or the screen's edge). To early reject some things, mattwoodrow introduced an intersection with the PaintRect of the nsDisplayText. For the non-shadows case this is fine, but with shadows we run into a problem: shadows can offset otherwise hidden glyphs into view. So these intersected bounds need to be inflated to include shadow offsets so hidden glyphs will be drawn (or alternatively: completely abandon the intersection with PaintRect if any shadows are present).
Finally, what I thought was an mBounds bug but wasn't: I have discovered that DisplayListBuilder::PushText is using the MergeClipLeaf optimization on text with ::selection shadows, which isn't supposed to happen for any shadows (understandable: everyone forgets ::selection shadows :)).