433918 - Split deltas into multiple files

Status: RESOLVED FIXED

(Cloud Services :: General, defect, P1)

Description

[Dan Mills [:thunder]]

Deltas are currently stored in a single file.  Splitting them into multiple files will reduce bandwidth waste and reduce eliminate the need for regenerating the snapshot to reduce the deltas.

The structure on the server would look something like this:

user-data/
  engine-name/
    status.json
    keys.json
    snapshot.json
    deltas/
      1.json
      2.json
      3.json
      ..
      NN.json

It would probably be best to have the filenames be the actual snapshot number, not an offset from the snapshot.  That way the snapshot could be regenerated if desired and the deltas would still be correct (but old).

For example, if the snapshot version is 3 and the max version is 5, there would be two delta files, '4.json' and '5.json', instead of 1 and 2.

Comment 1

[Maksym Kortunov]

As was mentioned on the forum, the deltas may be combined into larger files to not create a huge list of small files. The deltas may be combined by a time interval (by a day for example) or by a file size (something like 200k will not be a problem for single file to download several times). The file name may contain the first delta number included in it.

Comment 2

[Dan Mills [:thunder]]

Yeah, that's a possibility.  We'd have to change the status file to contain more information, probably, but it's something we can explore after we split things up.

Comment 3

[Dan Mills [:thunder]]

I'm working on this now.

Comment 4

[Dan Mills [:thunder]]

Committed to hg (over multiple commits; the last one being 4cd2f00f5b7a).