1504181 - Signature verification fails because of scientific notation

Status: RESOLVED FIXED

(Firefox :: Remote Settings Client, enhancement)

Description

[Mathieu Leplatre [:leplatrem]]

If the data contains some float numbers (eg. 0.000002942), the canonical JSON differs between the server and the client, which leads to signature verification failure.

The server serializes it as `2.942e-06` and the client as `0.000002942`.

Comment 1

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1504181 - Force scientific notation in CanonicalJSON r?glasserc

Mimic python for small float numbers

Comment 2

[Mathieu Leplatre [:leplatrem]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce5cc731c05e25339f4e0b32fbea89043ad3c91c

Comment 3

[Michael Cooper [:mythmon]]

It may be interesting to see that we also ran into this issue in Normandy. We had thought to modify the server or the API to accommodate the discrepancy instead of modifying Firefox. We discussed it a bit here: https://github.com/mozilla/normandy/issues/1210

Comment 4

[Mathieu Leplatre [:leplatrem]]

> Are there other users of CanonicalJSON besides remote-settings and normandy?
> 

Nope.

> I don't feel good about trying to force the formatting to be specifically the Python one. Maybe we should standardize Canonical JSON somewhere.
> 

Well yes, writing a standard is a good idea...

For example, I was super sad to see `1.3e+04` instead of `1.3e+4`...

> We had thought to modify the server or the API to accommodate the discrepancy instead of modifying Firefox

That was my intention too. But I didn't know what to set for the number of significant/fractional numbers... 

>>> format(0.000068272)
'6.8272e-05'

>>> format(decimal.Decimal(0.000068272), ".8g")
'0.000068272000'

>>> format(decimal.Decimal(0.000068272), "f")
'0.00006827199999999999957024654495540971765876747667789459228515625'

Whereas in Firefox the change generic (but ugly I admit)

Comment 5

[Michael Cooper [:mythmon]]

The change here is simple enough that I'm ok with it. I wanted to provide additional context, in case it were useful.

Comment 6

[Ethan Glasser-Camp (:glasserc)]

From the linked bugs, and per communication on IRC, Normandy still has this incompatibility, and since both Kinto and Normandy are written in Python, I guess this fixes the problem. I feel like this is papering over a huge mess of underdefined behavior about what "Canonical JSON" actually means, but maybe that's OK. Having this fix in Firefox master might not cover us as well as changing both Normandy and Kinto, but it's definitely better than nothing.

Comment 7

[Mathieu Leplatre [:leplatrem]]

I am considering a solution for both the client and server.

The scientific notation should not have any leading zero according to these specs:

- https://tools.ietf.org/html/draft-rundgren-predictable-serialization-for-json-00#section-3.3
- https://www.ecma-international.org/ecma-262/6.0/#sec-tostring-applied-to-the-number-type

I will update this patch, and we can deploy a fix on the server (PR https://github.com/Kinto/kinto-signer/pull/275)

Comment 8

[Mathieu Leplatre [:leplatrem]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7820c157f85b968505f6a73a63ce00fc4feed0cf

Comment 9

[Mathieu Leplatre [:leplatrem]]

We will be able to deploy a fix on the server only. The patch for this bug does not contain any code change, only tests. Hence no need to uplift required :)

Comment 10

[Mathieu Leplatre [:leplatrem]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ac25764bbeafe7c79fba476442df5bdfe8351b79

Comment 11

[Mathieu Leplatre [:leplatrem]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6ada14470e76a7519a24c21706ebf2d082af205b

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/233f5ec75924
Force scientific notation in CanonicalJSON r=glasserc

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/233f5ec75924