[calculator] .001 / 2 should not equal .001

RESOLVED FIXED in B2G C3 (12dec-1jan)

Status

P2
normal
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: nhirata, Assigned: kgrandon)

Tracking

unspecified
B2G C3 (12dec-1jan)
ARM
Gonk (Firefox OS)

Details

Attachments

(1 attachment)

## Environment :
Otoro phone, build 2012-10-03 us
Taken from default.xml in b2g-distro: 
* "platform_build" revision= 795261940c8b11fb7dddd7a8e9dd8561fdc4fb64
* "gaia" revision= a3339652136fde9e5207e1bc1263a410c84f55fc 
* "releases-mozilla-central" revision= aecce9686d0b0e6ec25bb31e837eadace251a951
* "gonk-misc" revision= 202d2c62443c098b125e5d9d7b42226d98230e44
  
## Repro :
1. launch calculator app
2. type in .001 /2 =

## Expected :
1. .0005

## Actual :
1. .001

## Note :
Oh.  it's truncating at 3 decimal places and rounding up.  :|

Updated

6 years ago
Component: Gaia → Gaia::Calculator
That doesn't sound good. Should we try to fix this before release?
This is a very basic change if we simply want to increase precision -- has a decision been made here yet?
This is quite a basic case for calculator.
Nominate this as blocking.
blocking-basecamp: --- → ?
blocking+

calculator is not very useful without more precision than this
blocking-basecamp: ? → +
(Assignee)

Comment 6

6 years ago
It looks like we might want to add the following test cases:

Operations which involve:
1 - Positive very large scientific numbers,   1e+21
2 - Negative very large scientific numbers,  -1e+21
3 - Positive large small scientific numbers,  1e-21
4 - Negative very small scientific numbers,  -1e-21

I'll take a look at the code, but not sure if I'm the best person to solve this.
(Assignee)

Comment 7

6 years ago
I have some ideas for this - taking unless someone really wants it?
Assignee: nobody → kgrandon
My thought on this is that instead of setting a fixed precision on all calculations, a maximum number character length should be set to something like 10.  The final value of a calculation will give us a major hint as to what the user expects.  For example, results could end up look like:

12.2937294
1.23948329
18239.1234
21932194.2

It's hard to mistake the "scale" that any of those results are measured on.  No one gets to a huge number by accident, and no one gets to a decimal-heavy result without knowing that that's the type of scale that will be returned.

I'm happy to take this if people agree.
(Assignee)

Comment 9

6 years ago
Created attachment 690678 [details]
Github pull request pointer

Dale - Seen that you've done some work on Calculator - maybe you could take a look at this?
Attachment #690678 - Flags: review?(dale)
Priority: -- → P2

Updated

6 years ago
Target Milestone: --- → B2G C3 (12dec-1jan)
flagging for a testcase for this.  John shih, can you add a testcase?
Flags: in-moztrap?(jshih)
Driver retriage: We cannot hold the whole release back for this fix. Get review and land with approval once all P1 bugs are fixed.
blocking-basecamp: + → -
I think this constitutes data loss. Let's make sure to fix this for v1, regardless of blocking-basecamp (but as a lower priority as Dietrich notes).
tracking-b2g18: --- → +

Updated

6 years ago
tracking-b2g18: + → 19+
Comment on attachment 690678 [details]
Github pull request pointer

Huge apologies for dropping the ball on this commit, I missed a whole bunch of emails and didnt notice it until checking my requests.

As mentioned in the other thread, I think the implementation of the number parser is cleaner than a straight regex + conversion over all the exponents, however that doesnt fix the precision issue nor is it tested. 

I have proposed Andrea merges your tests + formatNumber fix into that patch and they can both go in together.
Attachment #690678 - Flags: review?(dale) → review-
This commit was combined with #819166 as they both overlapped in what they were fixing.

Merged in: https://github.com/daleharvey/gaia/commit/77a7a987566a6834307b981ce78c15943c4f1615
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED

Updated

6 years ago
tracking-b2g18: 19+ → +

Updated

6 years ago
Flags: in-moztrap?(jshih)
status-b2g18: --- → fixed
Component: Gaia::Calculator → Gaia::Calculator
Product: Boot2Gecko → Boot2Gecko Graveyard
You need to log in before you can comment on or make changes to this bug.