1675095 - ComputeAutoSize() may return an inline size which is not in the range of [0, nscoord_MAX]

Status: NEW

(Core :: Layout, defect, P3)

Description

[Boris Chiou [:boris]]

When I was trying to always clamp the inline size between [0, nscoord_MAX], I got some more assertions in try:

1. layout/generic/crashtests/first-letter-638937-2.html
   (ASSERTION: should no longer use unconstrained inline size: 'availSize.ISize(wm) != NS_UNCONSTRAINEDSIZE', file /builds/worker/checkouts/gecko/layout/generic/nsFirstLetterFrame.cpp:164)
2. layout/generic/crashtests/323386-1.html
   (ASSERTION: computed inline size should always be computed: 'computedSize.ISize(wm) != NS_UNCONSTRAINEDSIZE', file /builds/worker/checkouts/gecko/layout/xul/nsBoxFrame.cpp:480)
3. layout/base/crashtests/453894-1.xhtml
   (ASSERTION: FinishReflowChild with unconstrained container width!: 'aContainerSize.width != NS_UNCONSTRAINEDSIZE', file /builds/worker/checkouts/gecko/layout/generic/nsContainerFrame.cpp:1188)

One of the reasons is that ComputeAutoSize() returns an inline size which is more than nscoord_MAX (e.g. 2147483646 in first-letter-638937-2.html because the margin is -2147483646), so my clamp makes it in the correct range and then I hit the assertion.

Perhaps we should always use something like NSCoordSaturatingSubtract to calculate availBased:
nscoord availBased = aAvailableISize - aMargin.ISize(aWM) - aBorderPadding.ISize(aWM);

However, the margin or the borderpadding may be an INT_MAX value, and this may hit another assertion:
MOZ_ASSERT_UNREACHABLE("Attempted to subtract [n - nscoord_MAX]");

We have to figure out what is the correct inline size that ComputeAutoSize() returns for these edge cases. Or just remove these assertions.

Comment 1

[Boris Chiou [:boris]]

Attachment: Bug 1665327 - Factor out the computation of min/max inline size for better readability.

We will add the computation for transferred min and max inline sizes in the
patch series, which may makes this function too large. Therefore, I would
like to move the computation of specified min and max inline sizes into
a static function, so it's be much clear to know the order of applying
transfered min and max sizes and spectied min and max sizes.