811271 - Float casting overflow in js/src/ctypes/CTypes.cpp

Status: NEW

(Core :: js-ctypes, defect)

Description

[Christian Holler (:decoder)]

Running this line of code in xpcshell

ctypes.Int64(-0x8000000000001000);

will cause a float casting overflow to happen, resulting in undefined behavior:

<unknown>: fatal error: value -9.22337e+18 is outside the range of representable values of type 'long'

The backtrace points to the Convert method of the ConvertImpl template:

> template<class TargetType, class FromType>
> struct ConvertImpl {
>   static JS_ALWAYS_INLINE TargetType Convert(FromType d) {
>     return TargetType(d);
>   }
> };

We must check that the integral part of the float value can fit the target type to avoid this (see also FLP34-C, CERT Secure Coding Standard). Not sure what the exact fix looks like here as I'm not familiar with that template jungle ^_^

Comment 2

[Jesse Ruderman]

Compilers don't exploit the undefinedness of float-cast-overflow too badly, so this isn't likely to be a security hole.  But we would like to find other float-cast-overflow bugs, and fixing this instance is probably easier than telling UBsan to ignore it.

UBSan is well-known now so there's no point in keeping this hidden.