843013 - monolith: Change record uuid in mysql to store bytes as binary(16) instead of hex as binary(32)

Status: RESOLVED FIXED

(Marketplace Graveyard :: Statistics, enhancement)

Description

[Hanno Schlichting]

Currently we store a record uuid as a binary(32) - as a hex representation of a uuid.uuid1().hex. That's wasting space, and we should just store the uuid.uuid1().bytes variant.

Comment 1

[Hanno Schlichting]

In addition to using a 16-byte variant, we should also change to a different uuid format, which has stable prefixes and doesn't screw BTree inserts and data locality.

Plus one: Have a key that lets us specify a simple "PARTITION BY KEY()", preferably the key denoting the year or year/month combination (http://dev.mysql.com/doc/refman/5.5/en/partitioning-overview.html - NDB cluster needs the partition key to be based on the primary key).

Comment 2

[Hanno Schlichting]

Done: Changed uuid format to have a stable prefix to put data from the same batch into the same buckets and avoiding totally random writes (while preserving a node-specific prefix to avoid writers on multiple hosts from conflicting too much).

Also used a url-safe base64-encoded version of the uuid, instead of the hex version. This only cuts it down to 24 bytes, but means less trouble in the ES URL/JSON API's.

Todo: Look into partition keys.

Comment 3

[Hanno Schlichting]

I looked into partitions and measured their overhead. At this point it's too much overhead without any clear benefit. And our PK doesn't give anything close to an even distribution for the provided (linear) key/hash algorithms.

We can look at this again if we consider cluster (ndb) storage or when we actually get tons of data.