1803995 - Indexed DB truncates viewable JSON data at 10000 characters

Status: NEW

(DevTools :: Storage Inspector, defect, P3)

Description

[Stephen Monro]

Attachment: Data incorrectly formatted.

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0

Steps to reproduce:

I've been developing my app for a while now and am using the Index DB data storage to save some JSON formatted settings in a long string.

While irrelevant to this bug, for reference, I'm using Ionic Storage.

I set an array of data and store it - as an example using this method:
await this.storage.set('settings', this.settingsItems);

The stored JSON here:
https://gist.github.com/vrdriver/bbc4067abb2d3f9dea8a9ea79b1181cf

I know there are no issues with the data that is going in, as it is working fine in other browsers using Webkit, or the Chromium engine, and as a compiled app.

I just can't view the data correctly, now that I've hit the 10k character limit.

Actual results:

The data is stored fine, however, when I go to view that data now that it's over 10000 bytes, it truncates the data and shows corrupted 'parsed' values

This is the data that is returned when I hit 'copy' of the 'data' field of my settings field.
https://gist.github.com/vrdriver/b497e63c14d3938e46452137d242d641

Expected results:

The data should not have been truncated in the Indexed DB Inspector viewer, and the JSON data should be formatted correctly with the full amount of data in the side viewer.

It works fine for smaller values, but as it's truncated, nothing above that size works.

Comment 1

[Stephen Monro]

Attachment: The same app/data working correctly in Chrome.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'DevTools::Inspector' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Stephen Monro]

Further to this, if I log the same JSON data to the console, it formats and works perfectly.

Comment 4

[Nicolas Chevobbe [:nchevobbe]]

I was able to reproduce on https://ffx-devtools-storage-indexeddb-malformed-parsed-json.glitch.me/

Comment 5

[Nicolas Chevobbe [:nchevobbe]]

I found the culprit. The serialized data goes through parseItemValue (https://searchfox.org/mozilla-central/rev/196b6aa0427f38058fd43ac90ee94fcf2f436fb6/devtools/shared/storage/utils.js#122-129)

function parseItemValue(originalValue) {
  // Find if value is URLEncoded ie
  let decodedValue = "";
  try {
    decodedValue = decodeURIComponent(originalValue);
  } catch (e) {
    // Unable to decode, nothing to do
  }

Here we're using decodeURIComponent on the string, which caps it at 9999 characters

Comment 6

[Nicolas Chevobbe [:nchevobbe]]

(In reply to Nicolas Chevobbe [:nchevobbe] from comment #5)

I found the culprit. The serialized data goes through parseItemValue (https://searchfox.org/mozilla-central/rev/196b6aa0427f38058fd43ac90ee94fcf2f436fb6/devtools/shared/storage/utils.js#122-129)

function parseItemValue(originalValue) {
  // Find if value is URLEncoded ie
  let decodedValue = "";
  try {
    decodedValue = decodeURIComponent(originalValue);
  } catch (e) {
    // Unable to decode, nothing to do
  }

Here we're using decodeURIComponent on the string, which caps it at 9999 characters

Scratch that, I jumped to conclusion here, originalValue is already truncated.
This is because of https://searchfox.org/mozilla-central/rev/196b6aa0427f38058fd43ac90ee94fcf2f436fb6/devtools/server/actors/storage.js#2700-2707

// FIXME: Bug 1318029 - Due to a bug that is thrown whenever a
// LongStringActor string reaches DevToolsServer.LONG_STRING_LENGTH we need
// to trim the value. When the bug is fixed we should stop trimming the
// string here.
const maxLength = DevToolsServer.LONG_STRING_LENGTH - 1;
if (value.length > maxLength) {
  value = value.substr(0, maxLength);
}

I'm not sure what wasn't working, let's see if we still need to do this