863891 - Warnings in branchTruncateDouble code, comparing signed-unsigned values

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Jeff Walden [:Waldo]]

INT64_MIN == 0x8000000000000000 compares a signed value (left) to an unsigned value (right), because the right side's number doesn't fit in int64_t.  Smacking an LL at the end doesn't help, because 0x8...0 doesn't fit in int64_t.  Unary minus isn't a part of a numeric literal, so -0x8....0LL is the negation of an unsigned, which is unsigned.

As the Intel manuals don't talk about the negative-most integer but rather about an "indefinite integer value" that happens to also be the most negative one, it seems like the best workaround is to eschew INT*_MIN entirely here, and just to write out the hex constants manually.  This patch does that (both x86 and x86-64, for consistency).

Comment 1

[Jeff Walden [:Waldo]]

Attachment: Don't reuse the *_MIN constants to implement the indefinite integer value

Comment 2

[Jeff Walden [:Waldo]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0eedc853429b

Comment 3

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/0eedc853429b