2013940 - Font editor values are rounded unexpectedly

Status: NEW

(DevTools :: Inspector, defect, P3)

Description

[Leo McArdle [:leo]]

Attachment: Screencast of STR

Discovered while working on https://bugzilla.mozilla.org/show_bug.cgi?id=1465026

STR:

1. Open data:text/html,<style>* { font-size: 1.009em; }</style>hello world!
2. Open devtools, switch to font panel
3. Click on Size input
4. Click out of Size input

Expected result: input value doesn't change from 1.009
Actual result: input value changes to 1.008

This is a result of floating point errors in our toFixed util in the font inspector, which uses Math.floor (the first function here):

function toFixedFloor(number, decimals = 1) {
  return Math.floor(number * Math.pow(10, decimals)) / Math.pow(10, decimals);
}

function toFixedRound(number, decimals = 1) {
  return Math.round(number * Math.pow(10, decimals)) / Math.pow(10, decimals);
}

console.log(toFixedFloor(1.009, 3), toFixedRound(1.009, 3));

console.log(toFixedFloor(0.9999, 3), toFixedRound(0.9999, 3));

As you can see when running that code:

toFixedFloor(1.009, 3) -> 1.008
toFixedRound(1.009, 3) -> 1.009

However, I'm unsure whether the floor behaviour was deliberate when handling values like 0.9999:

toFixedFloor(0.9999, 3) -> 0.999
toFixedRound(0.9999, 3) -> 1

Clearly the second is correct, but the first perhaps behaves more nicely in UI: not jumping from a decimal to an integer.

I'm not sure I agree, but wanted to check if there was some intention to this bug.

Comment 1

[Leo McArdle [:leo]]

As discussed in triage, we'll round these values properly.

Comment 2

[Leo McArdle [:leo]]

Attachment: Bug 2013940 - [devtools] fix floating point errors in font inspector rounding r=#devtools