621987 - unary - applied to unsigned var in ConvertImage

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

13:38 < vlad> bjacob_: msvc is giving a warning that I think is a problem
13:38 < vlad> in ConvertImage(), dst_delta = mPixelStoreFlipY ? -dstStride :
              dstStride;
13:38 < vlad> dstStride is unsigned, so applying unary - leaves it as unsigned
13:38 < vlad> which means it'll become a bignum instead of negative
13:38 < bjacob_> oh oh
13:39 < vlad> I think that needs to be -ptrdiff_t(dstStride)
13:39 < bjacob_> rather, math dstStride signed
13:39 < bjacob_> s/math/make
13:39 < bjacob_> amazing that this wouldn't have resulted in massive test
                 failure and even crashes

Comment 1

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: fix unsigned bug

OK, there's a real bug here, doing as you suggested a ptrdiff_t cast because outside of this line of code it generally makes sense for the strides to be unsigned.

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

http://hg.mozilla.org/mozilla-central/rev/f5a045d43b2b