1172487 - Find a way to introduce new buchets past 2^53

Status: VERIFIED FIXED

(Marketplace Graveyard :: Code Quality, enhancement, P1)

Description

[Mathieu Pillard [:mat]]

With the current system, the maximum number of features (aka "buchets") we can pass to the API from fireplace is 53. This is because fireplace encodes the features into an integer, and the maximum safe integer in JavaScript is 2^53, also available as Number.MAX_SAFE_INTEGER in modern browsers with ES6.

It might appear to work if the resulting number is small enough but fail as soon as too many features are detected as present.

See also: http://blog.vjeux.com/2010/javascript/javascript-max_int-number-limits.html

We need to do something, because with the feature detection we need to add for bug 1172486 we're hitting that limit, and we won't be able to add new feature requirements until this is solved.

Various ideas (keep in mind we need to support the old system no matter what, for older versions of our package - parsing the version number to determine how to decode the string will be important on the API side):
- Use 2 integers (never combine them arithmetically)
- Play with negative numbers (we can use from 0 to -2^53, Number.MIN_SAFE_INTEGER)
- Keep the feature profile as a string of 1s and 0s
- Keep the feature profile as a string, but compress it

Comment 1

[Allen Short [:ashort]]

- send feature profile as a base64 string, decode it into a typed array. Stick '=' on the end to distinguish it from oldstyle buchets.

Comment 2

[Christopher Van Wiemeersch [:cvan]]

(In reply to Allen Short [:ashort] from comment #1)
> - send feature profile as a base64 string, decode it into a typed array.
> Stick '=' on the end to distinguish it from oldstyle buchets.

seems like it'd that'd be quite long.

what about just

10101101001
1,0,1,0,1,1,0,1,00,1

?

Comment 3

[Allen Short [:ashort]]

i'm saying that could be encoded as "BWk=".

Comment 4

[Mathieu Pillard [:mat]]

Encoding the number in base 64 you mean ? But then, we'd end up having the issue are trying to avoid: manipulating a number that's greater than 2^53 in JavaScript.

10101101001... format is one of my proposals, yeah. It'd be long - since we currently have 53 flags to pass, and it's only going to get worse - but at least it's simple...

Comment 5

[Allen Short [:ashort]]

decode it into a typedarray and you can deal with it in chunks, no need to represent it as a single number.

Comment 6

[Mathieu Pillard [:mat]]

Ah, I think I see it now, but to be efficient you'd need to pack the bits in groups of 8 yourself, since there is no TypedArray, working on bits, right ? Otherwise you end up using a Uint8Array and wasting 7 bits of space for each byte, and when turning it into base64 ending up with something longer than the equivalent raw string of 0s and 1s (with btoa(String.fromCharCode.apply(null, new Int8Array(arr))) I transform an array of 53 features in a string of 72 chars)

Comment 7

[Mathieu Pillard [:mat]]

Answering myself: it turns out other people have solved this before :)

There are lots of cool BitField implementations using TypedArrays, grouping bits like I suggest in comment 6. Take this gist for instance: https://gist.github.com/panzi/56bbd884e00a5cc548dc

An array of alternating true/false values is reduced to "VVVVVVVVFQ==", not bad! My favorite proposal so far, thanks for suggesting it, :ashort.

Comment 8

[Mathieu Pillard [:mat]]

API part done in https://github.com/mozilla/zamboni/commit/8a929b707620a0ccc49a688badfc1549520d849b

Comment 9

[Mathieu Pillard [:mat]]

Fixed in https://github.com/mozilla/fireplace/commit/69eb6392d7efe352eac2fec829eb3c76fea2a430

STR:
- Make sure to update your mkt-dev package to version 20150825.134039
- Follow the same steps as https://bugzilla.mozilla.org/show_bug.cgi?id=1188357#c2
- If possible test with multiple FxOS versions - the ones before 2.0 should have less features shown in the "View Feature Profile Information" page

Comment 10

[Mathieu Pillard [:mat]]

Note: like bug 1188357, this is not part of the weekly push, we'll make a new prod package when we need one.

Comment 11

[Victor Carciu]

Verified as fixed following steps from comment #9 on different FxOS versions, including versions before 2.0 (less features were shown as green)