454856 - gcc seems to miscompile some fp math, leading to off-by-1 errors in colors

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Attachment: small C testcase

In gfxColor.h, the Packed() function did things like:

(double r,g,b,a; are part of the struct)

  (((PRUint8)(a * 255.0) << 24) |
   ((PRUint8)(b * 255.0) << 16) |
   ((PRUint8)(g * 255.0) << 8) |
   ((PRUint8)(r * 255.0)));

At least gcc 4.2.3 on Ubuntu 8.04 is doing something weird there, because the results differ from when optimization is turned on or off.  The attached simple C testcase demonstrates the problem.  If an intermediate double variable is used to hold the result of a * 255.0, and the cast to PRUint8 is done as a separate step, the right value seems to result.  If -Os or -O2 is used, the right value appears in both cases.  gcc on MacOS X, both 4.0.1 and 4.2.1, end up with the right result.

Comment 1

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Attachment: fix gfxColor::Packed

Fix gfxColor::Packed to use a pattern where gcc does what we want in both debug and opt builds.  There's likely to be something more robust that can be done here, but I'm not sure what.

Comment 2

[Bobby Holley (:bholley)]

Comment on attachment 338172 [details] [diff] [review]
fix gfxColor::Packed

looks good - how is this tree closure going to affect landing?

Comment 3

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Checked in.