Created attachment 775298 [details] testcase ###!!! ABORT: clamped(): max must be greater than or equal to min: 'max >= min', file nsAlgorithm.h, line 66
The right hand side of the range rect is < left hand side because the left hand side is so large that a 1 CSS pixel width overflows. The nscoord is large because it comes from CSSIntPoint via NSIntPixelsToAppUnits(). NSFloatPixelsToAppUnits() clamps to nscoord_MIN,MAX. There is no reason why NSIntPixelsToAppUnits() won't overflow, so the clamping in NSFloatPixelsToAppUnits() suggests that NSIntPixelsToAppUnits() should also clamp if that is the strategy that we want to propagate.
Created attachment 780643 [details] [diff] [review] clamp CSS pixel to nscoord conversion to nscoord_MIN,MAX Conversion from device pixels seems less likely to need clamping because Device pixels often have their own limits (surface sizes), so are probably more likely known not to overflow when converted to nscoord. Conversion from CSS pixel is more likely to be converting from a document-specified value and so there are less likely any guarantees on the range. I've used NSToCoordRoundWithClamp to make it explicit that clamping is happening. Clamping/saturating on input may be less efficient than letting things overflow and picking a random tolerable value where necessary, but will give results closer to the authors intentions. It allows an author to use a large value to get a maximum effect. I don't know whether xptc would clamp before truncation to int32, but overflow at 2^31 would be more predictable than overflow at nscoord limits which depend on zoom. I think the main advantage of this approach is that it picks a common place where overflow can occur and deals with that there, hopefully saving Gecko bug fixers from having to fix some other bugs like this one.