1209390 - Use standard lz4 file format instead of the non-standard jsonlz4/mozlz4

Status: ASSIGNED

(Toolkit :: General, task, P5)

Description

[quadim]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0
Build ID: 20150918100310

Steps to reproduce:

Use Firefox.


Actual results:

Bookmark backup files (in "bookmarkbackups/") and other files (such as
things in "crashes/") are lz4-compressed files, but they use a
non-standard format.

Result: Users cannot avail themselves of standard, commonly available
tools to inspect these files, which contain *their* data.  Instead they
have to resort to Firefox-specific (or Mozilla-specific, same point)
hacks to access their data. [1]

[1] Such as using the Library GUI in Firefox to export bookmarks; or
    using Mozilla's lz4 interfaces through XPCOM.


Expected results:

Mozilla should use standard file formats.

You promised to switch to a standard format, once one was defined [2].
One was defined a while ago [3].  Standard tools are available since
some time [4].  Why are you still delaying?

[2] https://dxr.mozilla.org/mozilla-central/source/toolkit/components/workerlz4/lz4.js#49

[3] https://github.com/Cyan4973/lz4/blob/master/lz4_Block_format.md
    https://github.com/Cyan4973/lz4/blob/master/lz4_Frame_format.md

[4] For example: https://packages.debian.org/jessie/liblz4-tool

Comment 1

[vlakoff]

lz4.js file has been moved to /toolkit/components/lz4/lz4.js [1]

[1] http://mxr.mozilla.org/mozilla-central/source/toolkit/components/lz4/lz4.js

Comment 2

[Marco Bonardo [:mak]]

Places will use a different format, when the platform (toolkit) will move to a different format. And I think it will be the same for all the consumers.
So this is a more general Toolkit bug.

Comment 3

[Marco Bonardo [:mak]]

(In reply to Marco Bonardo [::mak] from comment #2)
> Places will use a different format, when the platform (toolkit) will move to
> a different format.

To clarify, if the compressor starts creating files in the more widely supported format, and the decompressor can still support the old format (I don't see why not, since there's an header it can detect), it will be enough to fix the lz4 component in toolkit to automatically move all the consumers to the new format.

Comment 4

[Avi Halachmi (:avih)]

I've created an unofficial stand-alone decompressor for `.jsonlz4` files.

The project with source code is hosted here: https://github.com/avih/dejsonlz4 .

Initial v0.1 release can be found here https://github.com/avih/dejsonlz4/releases and includes a Windows executable `dejsonlz4.exe`. It should hopefully compile easily elsewhere too.

Please take any discussions regarding this project to the project page on github.

Comment 5

[Anthony Thyssen]

Thank you  Avi Halachmi

Could you include in your source code "README.md" file, links back to these bugs, as well as a link to the GIT HUB source.  This lets people who come across the code later to locate the code's primary source and the reasons for its existence.


ASIDE: This problem was one of the reasons (not the only one) I later moved from firefox to a different browser.

Comment 6

[Avi Halachmi (:avih)]

(In reply to Anthony Thyssen from comment #5)
> Could you include in your source code "README.md" file ...

(In reply to Avi Halachmi (:avih) from comment #4)
> Please take any discussions regarding this project to the project page on github.

Comment 7

[S]

(In reply to Avi Halachmi (:avih) from comment #6)
> (In reply to Anthony Thyssen from comment #5)
> > Could you include in your source code "README.md" file ...
> 
> (In reply to Avi Halachmi (:avih) from comment #4)
> > Please take any discussions regarding this project to the project page on github.

Thanks for the piece of code to decompress. But do you have any idea to go the other way and compress?

Here at my side it generates Chinese characters which it shouldn't be. The size is the right size though (same as the original compressed file).

This is the code I used:
		std::ifstream js_in_file(iname, std::ifstream::binary);
	std::string js_code((std::istreambuf_iterator<char>(js_in_file)), std::istreambuf_iterator<char>());

	int returnSize =0;
	char* compressed = new char[js_code.size()];
	returnSize = LZ4_compress(js_code.c_str(), compressed, js_code.size());


		std::ofstream out(oname, std::ofstream::binary);
	out.write(compressed, returnSize);
	out.close();

Comment 8

[Georg Fritzsche [:gfritzsche]]

(In reply to S from comment #7)
> (In reply to Avi Halachmi (:avih) from comment #6)
> > (In reply to Avi Halachmi (:avih) from comment #4)
> > > Please take any discussions regarding this project to the project page on github.
> 
> Thanks for the piece of code to decompress. But do you have any idea to go
> the other way and compress?

Please take this to GitHub, this can't be answered here.

Comment 9

[H.-Dirk Schmitt]

The current obfuscation by changing the magic field in the lz4 compressed files doesn't provide any security enhancements. 
For advanced users it is now more complicated to "patch" or "synchronise"  configuration files.

So +1 for migrate to standard lz4 compressed or uncompressed files.

Comment 10

[Marco Bonardo [:mak]]

(In reply to H.-Dirk Schmitt from comment #9)
> The current obfuscation by changing the magic field in the lz4 compressed
> files doesn't provide any security enhancements. 

There is no obfuscation will, nor security enhancement here. At the time when lz4 was added there wasn't a standard format, thus a very simple header had been created in front of the payload.
Now a standard exists, but nobody internally had the time to convert the encoder/decoder for it, nor anyone volunteered to do that yet.

Comment 11

[ZeroUnderscoreOu]

I would like to support this change in regards to search engines. Currently, I'm forced to use side tools and export/import is severely limited.
Also, I don't really see how compression/signing really prevents search hijacking when (from my experience) most of the time it's done by side software with all the needed tools for that.

Comment 12

[Stefan Endrullis]

Are there any news on this?

Is Mozilla planning to replace the Firefox-specific lz4 format by a standard format?

I definitely want to vote for a standard format, because it's then much easier to work with in other tools.
For instance, I've written a Scala/Java library that reads the state of the Firefox session file.  I've chosen the JVM to be platform independent.  With a standard lz4 format there would not be any problem, since there are multiple lz4 libraries for Java.  However, this Firefox-specific format makes the whole decoding now much more complicated.  I can either try to reimplement the Firefox-specific implementation in Java or I have to deploy the library with several platform dependent tools like dejsonlz4 which do the decompression task for the specific platform.  Both solutions are cumbersome and therefore I really hope for a format change.

Are there any workarounds for tool developers like me?  For instance, is it possible to decompress the Firefox-specific format somehow with some standard lz4 decompressors by slightly changing the format or something like that?

Comment 13

[Stefan Endrullis]

OK, forget my last question.  Fortunately, the Firefox-specific format is actually the same as lz4 except that there is a 12 bytes prefix which can be just skipped for decompressing the file.  So there is a workaround for now. :)

Comment 14

[Marco Bonardo [:mak]]

yes, the only non-standard thing is the header.

Comment 15

[vlakoff]

FWIW, I'm using a PHP code for several years: https://gist.github.com/vlakoff/3139e310664285c6c83b

Also, note it's not the latest LZ4, but v1.3, which is not upward compatible.

Comment 16

[Graham Perrin]

(In reply to Stefan Endrullis from comment #12)

> … workarounds …

mozlz4-edit – Add-ons for Firefox
<https://addons.mozilla.org/addon/mozlz4-edit/

> … open, edit and save …

Comment 17

[janek]

Another utility, for people trying to work around this: https://gist.github.com/Tblue/62ff47bef7f894e92ed5

What I'm wondering now - why compress at all? For me, and I guess for most other users, this file doesn't even reach a Megabyte. Yes, lz4 can compress it to around 10%, but who cares at that size?

Comment 18

[Marco Bonardo [:mak]]

It does less I/O when storing and retrieving on disk.

Comment 19

[Marcel Partap]

Ok from a quick benchmark, I'd zuggest switching to zstd? It seems to compress/decompress slightly faster than LZ4 with triple the compression ratio...
(I unpacked a ~4MiB upgrade.jsonlz4 I had in my profile and duplicated its contents random times, than ran multitime with mozlz4 (python3 implemented, that may skew the results), zstd and pigz commands packing and unpacking the test file.. on a tmpfs, with BOINC paused on my hexacore machine)

652M test.json
123M test.json.gz
144M test.json.lz4
 43M test.json.zst

Compressing:

++ multitime -v zstd -kf test.json
===> Executing zstd -kf test.json
===> multitime results
1: zstd -kf test.json
            Mean        Std.Dev.    Min         Median      Max
real        1.266       0.000       1.266       1.266       1.266       
user        1.363       0.000       1.363       1.363       1.363       
sys         0.178       0.000       0.178       0.178       0.178       

++ multitime -v sh -c 'mozlz4 -c < '\''test.json'\'' > '\''test.json'\''.lz4'
===> Executing sh -c "mozlz4 -c < 'test.json' > 'test.json'.lz4"
===> multitime results
1: sh -c "mozlz4 -c < 'test.json' > 'test.json'.lz4"
            Mean        Std.Dev.    Min         Median      Max
real        1.353       0.000       1.353       1.353       1.353       
user        0.852       0.000       0.852       0.852       0.852       
sys         0.497       0.000       0.497       0.497       0.497       

++ multitime -v pigz -kf test.json
===> Executing pigz -kf test.json
===> multitime results
1: pigz -kf test.json
            Mean        Std.Dev.    Min         Median      Max
real        2.010       0.000       2.010       2.010       2.010       
user        22.482      0.000       22.482      22.482      22.482      
sys         0.454       0.000       0.454       0.454       0.454       

Decompression:

++ multitime -v zstd -dc test.json.zst
===> Executing zstd -dc test.json.zst
===> multitime results
1: zstd -dc test.json.zst
            Mean        Std.Dev.    Min         Median      Max
real        0.368       0.000       0.368       0.368       0.368       
user        0.363       0.000       0.363       0.363       0.363       
sys         0.004       0.000       0.004       0.004       0.004       

++ multitime -v sh -c 'mozlz4 -d < test.json.lz4 > /dev/null'
===> Executing sh -c "mozlz4 -d < test.json.lz4 > /dev/null"
===> multitime results
1: sh -c "mozlz4 -d < test.json.lz4 > /dev/null"
            Mean        Std.Dev.    Min         Median      Max
real        0.863       0.000       0.863       0.863       0.863       
user        0.408       0.000       0.408       0.408       0.408       
sys         0.455       0.000       0.455       0.455       0.455       

++ multitime -v pigz -dc test.json.gz
===> Executing pigz -dc test.json.gz
===> multitime results
1: pigz -dc test.json.gz
            Mean        Std.Dev.    Min         Median      Max
real        1.467       0.000       1.467       1.467       1.467       
user        2.184       0.000       2.184       2.184       2.184       
sys         0.174       0.000       0.174       0.174       0.174       

Comment 20

[mozbugz]

For anyone coming across this needing to decompress these files:

/* 
NOTE: BEFORE RUNNING THIS SCRIPT, CHECK THIS SETTING:
Type or paste about:config into the address bar and press Enter
Click the button promising to be careful
In the search box type devt and pause while Firefox filters the list
If devtools.chrome.enabled is false, double-click it to toggle to true

Paste this entire script into the command line at the bottom of the Browser Console (Windows: Ctrl+Shift+j)
Then press Enter to run the script. A file picker should promptly open.
*/

async function convert() {
  // Set up file chooser
  var fp = Components.classes["@mozilla.org/filepicker;1"]
    .createInstance(Components.interfaces.nsIFilePicker);
  fp.init(window, "Open File", Components.interfaces.nsIFilePicker.modeOpen);
  fp.appendFilter("Bookmark Backup Files", "*.jsonlz4");
  var result = await new Promise(resolve => fp.open(resolve));
  // Call file choose, proceed if a file was chosen
  if (result == Components.interfaces.nsIFilePicker.returnOK) {   var file = fp.file;
    // Check that file can be used
    if (file.exists() && file.isFile() && file.isReadable()) {
      var oldfile = fp.file.path;
      // Construct output file name
      var newfile = oldfile.replace(".jsonlz4", "_converted.json");
      // See: http://forums.mozillazine.org/viewtopic.php?p=14111285#p14111285
      var {utils:Cu} = Components;
      Cu.import("resource://gre/modules/osfile.jsm");
      var jsonString = await OS.File.read(oldfile,{ compression: "lz4" });
      // console.log(jsonString);
      OS.File.writeAtomic(newfile, jsonString);
    }
  }
}
convert()

Comment 21

[BugBot [:suhaib / :marco]]

The severity field for this bug is relatively low, S3. However, the bug has 25 votes and 52 CCs.
:mossop, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 22

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.

Comment 23

[Luis Antonio Castro Henriques]

People, it is important to support open formats.

I know Mozilla does not intend to prevent interoperability using this format. But other companies regularly try this. Remember Microsoft in the 90s with the "Embrace, extend, and extinguish"?

Mozilla should lead by example and support open formats as much as possible. This is a trivial modification and could be done easily.

I understand this is a very little thing, but remember your manifesto:

Principle 6

The effectiveness of the internet as a public resource depends upon interoperability (protocols, data formats, content), innovation and decentralized participation worldwide.

This should be accomplished on the little things too.

Comment 24

[Dave Townsend [:mossop]]

I don't think there is any value in posting further comments in support of this, we're on board with doing it, we just don't currently have the resources available to do it. If someone wants to work on implementing it then we would review a patch.

Comment 25

[Anirban Majumder]

Hi! I'm interested in working on this task. I've identified that IOUtils and AddonManagerStartup.cpp are using the custom mozLz40 header. I plan to update them to use the standard LZ4 frame format while maintaining backward compatibility. Could I be assigned to this bug, or is there a mentor who can give me a quick 'go-ahead' on this approach?

Comment 26

[u Ltd.]

IMHO It would be sufficient to reinterpret 0x0000 match offset as escape sequence. After that, token 0x10..0xff act as always while token 0x00..0x0f are reserved (e. g. could be 0x04 = EOT for output of unknown size).

This would make encoding easier (no dynamic memory allocation needed) so that the original problem, not enough tooling, would solve by itself.

Comment 27

[Anirban Majumder]

Attachment: Bug 1209390 - Switch IOUtils to standard LZ4 frame format. r?nika,#xpcom-reviewers

IOUtils::MozLZ4::Compress now writes standard LZ4 frames (via
LZ4FrameCompressionContext) instead of the custom mozLz40 header.
IOUtils::MozLZ4::Decompress auto-detects the format: standard LZ4
frames are decompressed via LZ4FrameDecompressionContext, while legacy
mozLz40 files are handled by a new DecompressLegacy helper for backward
compatibility. AddonManagerStartup::ReadFileLZ4 is similarly updated to
read both formats. File extensions remain .jsonlz4.

Comment 28

[Anirban Majumder]

While a custom escape sequence could simplify the internal encoder, it would result in yet another non-standard format. The primary goal of this bug is to ensure that Mozilla-generated files can be handled by standard external tools (like the lz4 CLI, Python, or Rust libraries) without requiring Mozilla-specific patches.

By adopting the Standard LZ4 Frame Format (RFC 6214), we solve the interoperability issue once and for all, benefit from built-in content checksums, and reduce long-term maintenance of custom 'mozLz40' logic. I have submitted patches D283790 and D283830 which implement this standard format while maintaining full backward compatibility for reading existing legacy files.

Revision: https://phabricator.services.mozilla.com/D283790