1250357 - [propose] Solid compression for datareporting/archived files

Status: RESOLVED WONTFIX

(Toolkit :: Telemetry, enhancement)

Description

[YF (Yang)]

Attachment: files detail

I see thousands of small files (10~100KB) in my profile folder, they are jsonlz4. Why not solid archive them, obtain a continuous and higher compression ratio archive files? They looks will not be changed.

Comment 1

[Georg Fritzsche [:gfritzsche]]

This would:
* complicate saving & loading those files
* make us more vulnerable to file I/O errors (losing us all pings instead of one in case of errors)

We need to be able to save and load them individually (for about:telemetry and SelfSupport/Shield) and use the information in the filename to cheaply build an index of those files.
To put them in one archive file, i think we would need to either
a) start building and managing an index etc. or
b) switch to a format that has one & supports cheap random access

We have a hard limit on the maximum size of the archive directory, so i'm not sure what this kind of change would buy us here given the cost?

Comment 2

[YF (Yang)]

Sorry, just a suggestion / proposal, I can not provide more help.

I understand the situation now, but it is a significant burden relative to the entire profile folder. Just hope to improve it.

Comment 3

[Georg Fritzsche [:gfritzsche]]

Sure, i'm just trying to understand the motivation here.

What is the burden here, are you just worried about space usage?
We have a hard-limit on disk-space use, so i think this should not be an issue.

Comment 4

[YF (Yang)]

Not a necessity, but not perfect.

Causing burden for archive backup the profile (I don't use the Sync). Perhaps it is more suitable place in AppData\Local\Mozilla\Firefox\Profiles?

Comment 5

[Georg Fritzsche [:gfritzsche]]

I see, thanks - so this can impact full profile copying performance with e.g. fragmented disks.
For now i would recommend skipping this folder for profile backups if it has issues for you.

For Telemetry & other features this is part of the profile, so i would not want to move this elsewhere.
We might rewrite the archiving in the future in a way that addresses this, but i think in the nearer-term we will not realistically fix this specific bug (given the concerns outlined above).

Comment 6

[YF (Yang)]

Sorry to bother you again, but I still don't understand if there are technical limitations that prevent us from placing some folders in the "Local" folder (all of cache) instead of "Roaming" folder, like the "datareporting", "gmp" & "gmp-*", as well as other large volumes and anti-volatile, these non-critical folders occupy a large amount of space in the roaming folder[1].

[1]: https://en.wikipedia.org/wiki/Roaming_user_profile

Comment 7

[Georg Fritzsche [:gfritzsche]]

It depends on whether we expect the data in those folders to have the same life-time as the rest of the folders.

If it is a burden for your backups, can't you exclude these sub-directories?

Comment 8

[YF (Yang)]

Excluding them from backups is mostly possible through software configuration, but it is a burden for many users to manually backup them.

Therefore, while their persistence is a good thing for telemetry, but maybe this is not necessary? (such as UUID instead). Some users suggest to disable telemetry because of their large and privacy concerns.

Comment 9

[Georg Fritzsche [:gfritzsche]]

(In reply to YF (Yang) from comment #8)
> Therefore, while their persistence is a good thing for telemetry, but maybe
> this is not necessary? (such as UUID instead). Some users suggest to disable
> telemetry because of their large and privacy concerns.

Moving this elsewhere wouldn't address these concerns.
We do use this data for QA & support handling and for on-ramping plans to offer automated self-support & diagnostics.

The size is restricted to reasonable amounts [0].
Anyone concerned about privacy can complete disable the upload of data in the Firefox settings. The data in the archive is solely used locally.

I'm happy to investigate other options if this affects more users.

0: https://dxr.mozilla.org/mozilla-central/rev/e5a10bc7dac4ee2453d8319165c1f6578203eac7/toolkit/components/telemetry/TelemetryStorage.jsm#56

Comment 10

[Georg Fritzsche [:gfritzsche]]

For what its worth, we recently lowered the amount of days we keep pings archived from 180 to 60 in bug 1295555.
This will improve the situation somewhat.