Bug 1565065 Comment 2 Edit History

Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.

Took a quick look in gdb while doing triage. The code in question is:
```
static void ConstrainToCoordValues(float& aStart, float& aSize) {
  MOZ_ASSERT(aSize >= 0);
```
https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/layout/base/nsLayoutUtils.cpp#2245-2246


...and we have `aStart = inf` and `aSize = -nan(0x400000)`.  And NaN fails all comparisons, so it fails the >=0 comparison here.


How do we get `inf` and `NaN`? Well, up one level, in RoundGfxRectToAppRect, we have a passed-in `aRect` with these components:
```
  {
    x = 3.40282347e+38, 
    y = 0, 
    width = 6, 
    height = 6
  } 
```

And we scale up that rect using a call to `ScaleRoundOut()`, which basically sets `x = x*60` and then sets `width = XMost()*60 - x`.  Our `x` value is large enough that the 60x multiplication bumps it up to be infinity, and so XMost() is also infinity, so we end up with `x=inf` and `width=inf-inf=NaN`, and then those are the values we pass into this nsLayoutUtils API.

More than likely, nothing terrible happens as a result. But we should probably be clamping and/or checking for NaN here somewhere.  Probably nsLayoutUtils::RoundGfxRectToAppRect() should be prepared for having huge & possibly NaN/inf values in its rect variables (either from being passed-in or from creating them locally via the 60x scaling), and it should simply clamp those to something sensible like `(0,0,0,0)` perhaps.
Took a quick look in gdb while doing triage. The code in question is:
```
static void ConstrainToCoordValues(float& aStart, float& aSize) {
  MOZ_ASSERT(aSize >= 0);
```
https://searchfox.org/mozilla-central/rev/7088fc958db5935eba24b413b1f16d6ab7bd13ea/layout/base/nsLayoutUtils.cpp#2245-2246


...and we have `aStart = inf` and `aSize = -nan(0x400000)`.  And NaN fails all comparisons, so it fails the >=0 comparison here.


How do we get `inf` and `NaN`? Well, up one level, in RoundGfxRectToAppRect, we have a passed-in `aRect` with these components:
```
  {
    x = 3.40282347e+38, 
    y = 0, 
    width = 6, 
    height = 6
  } 
```

And we scale up that rect using a call to `ScaleRoundOut()`, which basically sets `x = x*60` and then sets `width = XMost()*60 - x`.  Our `x` value is large enough that the 60x multiplication bumps it up to be infinity, and so XMost() is also infinity, so we end up with `x = inf` and `width = (inf - inf) = NaN`, and then those are the values we pass into this nsLayoutUtils API.

More than likely, nothing terrible happens as a result. But we should probably be clamping and/or checking for NaN here somewhere.  Probably nsLayoutUtils::RoundGfxRectToAppRect() should be prepared for having huge & possibly NaN/inf values in its rect variables (either from being passed-in or from creating them locally via the 60x scaling), and it should simply clamp those to something sensible like `(0,0,0,0)` perhaps.

Back to Bug 1565065 Comment 2