398181 - "ASSERTION: Doing nscoord addition with values > nscoord_MAX" with inline-block, percentage margin

Status: RESOLVED FIXED

(Core :: Layout, defect)

Description

[Jesse Ruderman]

Attachment: testcase

Loading the testcase triggers lots of assertions, starting with:

###!!! ASSERTION: Doing nscoord addition with values > nscoord_MAX: 'a < nscoord_MAX && b < nscoord_MAX', file ../../dist/include/gfx/nsCoord.h, line 151

###!!! ASSERTION: nscoord addition will reach or pass nscoord_MAX: '(PRInt64)a + (PRInt64)b < (PRInt64)nscoord_MAX', file ../../dist/include/gfx/nsCoord.h, line 153

Comment 1

[Daniel Holbert [:dholbert]]

Attachment: fix (trivial)

Adding NSCoordSaturatingAdd call in nsTextFrameThebes.cpp fixes first assertion.

Adding NSCoordSaturatingSubtract call in nsFrame.cpp fixes second assertion.

(The only other change -- the added NSCoordSaturatingAdd call in nsFrame.cpp -- doesn't actually affect this bug's testcase.  The change is preventative, because currentLine could be nscoord_MAX there, and we shouldn't add directly to it.)

Comment 2

[Daniel Holbert [:dholbert]]

Attachment: testcase2 (using float)

Per jesse's suggestion, I made this testcase that depends on the preemptive change in this bug's fix.  (which was mentioned parenthetically in my last comment, at nsFrame.cpp:2869:
 -    currentLine += floats_done;
 +    currentLine = NSCoordSaturatingAdd(currentLine, floats_done);

After applying my patch's other two changes, but skipping the preemptive change, I get these assertions when loading testcase2:

  ###!!! ASSERTION: Doing nscoord subtraction with values > nscoord_MAX: 'a < nscoord_MAX && b < nscoord_MAX', file ../../dist/include/gfx/nsCoord.h, line 208
  ###!!! ASSERTION: nscoord subtraction will reach or pass nscoord_MAX: '(PRInt64)a - (PRInt64)b < (PRInt64)nscoord_MAX', file ../../dist/include/gfx/nsCoord.h, line 210

(I think we're hitting these in the added call to NSCoordSaturatingSubtract at nsFrame.cpp:2874)

After adding the third change, those assertions go away.

Comment 3

[Daniel Holbert [:dholbert]]

Checking in nsTextFrameThebes.cpp;
/cvsroot/mozilla/layout/generic/nsTextFrameThebes.cpp,v  <--  nsTextFrameThebes.cpp
new revision: 3.128; previous revision: 3.127
done
Checking in nsFrame.cpp;
/cvsroot/mozilla/layout/generic/nsFrame.cpp,v  <--  nsFrame.cpp
new revision: 3.766; previous revision: 3.765
done

Comment 4

[Jesse Ruderman]

Crashtests checked in.