1357294 - It's possible to get into a state with a broken index

Status: NEW

(Core :: Storage: IndexedDB, defect, P3)

Description

[Markus Stange [:mstange]]

My IndexedDB database on https://perf-html.io has gotten into a broken state after I've used the Gecko Profiler add-on for a few months.

The profiler stores symbol tables in an IndexedDB database and never deletes them. My <profiledir>/storage/default/https+++perf-html.io directory is currently 1.75GB big.

Here's where we create the database and the index: https://github.com/devtools-html/perf.html/blob/d468429e52e0d1a7fd892ddd7473955b8d3b4033/src/content/symbol-store-db.js#L25

The symptom I'm seeing is that I can't look up entries by the ['debugName', 'breakpadId'] index. I can insert new entries just fine, and I can look them up by the key that I get from the insertion, but I can't look them up by the index.

I've loaded the relevant sqlite file into an app called "DB Browser for SQLite" and it shows me that my object_data table has 3361 rows but my unique_index_data table only has 205 rows.

Bevis, do you know what could be causing this? Can I do anything to help debugging? Would it help if I sent you the sqlite file, or even whole directory? It is rather large though... I do have another DB for my local instance of perf-html.io ("localhost:4242") which is showing the same problem and is only 758.7MB big, so I could send that instead.

Comment 1

[Markus Stange [:mstange]]

The specific symptom is: We ask the index to look up the key for the element with index ['firefox', 'FAE077F598FE3E1F9D5C606DA3B23A580'], and it returns a null result, which means that no entry with this index exists. Even though there is already an entry with debugName == 'firefox' and breakpadId == 'FAE077F598FE3E1F9D5C606DA3B23A580'.
Then we handle the failure, by retrieving the symbol table from a different source, and we insert it into the object store, with debugName == 'firefox' and breakpadId == 'FAE077F598FE3E1F9D5C606DA3B23A580'. At this point, we should really get an exception, because there already is an element with those exact values for debugName and breakpadId, and we have an index based on the combination of those two properties. But the insertion succeeds, and now we have another copy of the same symbol table in the database.

Comment 2

[Bevis Tseng[:bevis][:btseng](Exited)]

(In reply to Markus Stange [:mstange] from comment #0)
> Bevis, do you know what could be causing this? Can I do anything to help
> debugging? Would it help if I sent you the sqlite file, or even whole
> directory? It is rather large though... I do have another DB for my local
> instance of perf-html.io ("localhost:4242") which is showing the same
> problem and is only 758.7MB big, so I could send that instead.

I haven't seen this problem before... :-(

To reproduce with the profile you provided, it will be helpful if there is a test page in the same web site that reproduces this problem.

Before having this test environment ready, it is helpful to check your browser console to see if there is any indexedDB error printed out when the symptom is happened.
Then, with the correct revision info, we can check which line of which source file this error is reported from.

Comment 3

[Markus Stange [:mstange]]

Attachment: Code that can be executed in the Scratchpad in order to reproduce the problem in a profile with a broken index

In my broken profile, if you go to perf-html.io, open the Firefox Scratchpad (Tools -> Web Developer -> Scratchpad), paste the code from this attachment, run it using Cmd+R, and then evaluate the following in the Web Console:

runTest('firefox', 'FAE077F598FE3E1F9D5C606DA3B23A580').then(console.log, console.error);

Then you will see a few messages, including the message "indexeddb put success IDBRequest { result: 1425, ...} ".

If you run it again, you'll get the message "indexeddb put success IDBRequest { result: 1426, ...} ".

In other words, every single put succeeds, even though those records have overlapping keys.

In a non-broken profile, you will instead see the message "indexeddb error IDBRequest { error: DOMError, ...} " the second time, where that error is a ConstraintError with the message "A mutation operation in the transaction failed because a constraint was not satisfied. For example, an object such as an object store or index already exists and a new one was being attempted to be created."

Comment 4

[Markus Stange [:mstange]]

I expect that it's fairly easy to find out why we behave the way we do once the index is already in the broken state, and that it's much harder to find out how I got into the broken state in the first place.

Comment 5

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Is more information needed than comment 3?

Comment 6

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Hey Markus, I am wondering how easy you run into this issue? Are you blocked by this? Thanks!

Comment 7

[Markus Stange [:mstange]]

I have run into this issue both with my main profile and with my testing profile, so I wouldn't be surprised if it exists in the profiles of other perf-html.io users as well. But I'm not blocked by it; soon I'm going to land a patch in perf.html that just deletes any existing database and starts from scratch, and this time with a size limit on the database. I hope that the smaller size will make it less likely to hit this problem.

Comment 8

[Bevis Tseng[:bevis][:btseng](Exited)]

Hi Markus,

Unfortunately, I still can't hit this problem by adding thousands of records with the testscript you provided in comment 3.
In addition, I haven't found any suspicious point yet in current IDBObjectStore::Put() implementation.
 
Would you mind to share the folder of "https+++perf-html.io" in your testing profile for further analysis?
With the broken storage folder and the same test script on perf-html.io, it shall be helpful to reproduce this problem from my local profile.

Thanks!

Comment 9

[Markus Stange [:mstange]]

I don't have the broken https+++perf-html.io folder any more, but here's my http+++localhost+4242 folder, which was showing the same problem: https://drive.google.com/file/d/0B1oqqAegSq7uZnEyei1sNjB2RUk/view?usp=sharing 

If it's not possible to just rename the folder and have it work on https+++perf-html.io, then you can start a local instance of perf.html at localhost:4242 by following the instructions at https://github.com/devtools-html/perf.html#development .

Does this let you reproduce the problem?

Yesterday, I landed and deployed the change to the DB's version and schema on perf-html.io. That means that perf-html.io will delete the broken index if you use the Gecko Profiler add-on to open perf-html.io. And that would mean that you'd no longer be able to reproduce the bug. So please don't use the Gecko Profiler add-on in your testing profile :)