1434579 - Devtools IndexedDB object store view shows max 50 objects

Status: RESOLVED FIXED

(DevTools :: Storage Inspector, defect, P2)

Description

[Simmo Saan]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:58.0) Gecko/20100101 Firefox/58.0
Build ID: 20180122225428

Steps to reproduce:

https://jsfiddle.net/3LtzL0ej/


Actual results:

Looking at IndexedDB object store "raindrops" shows only 50 objects.


Expected results:

The "raindrops" object store contains 10,000 objects. This is in no way indicated in the devtools nor is it possible to see the objects after the first 50.

Comment 1

[Simmo Saan]

Initial investigation suggests that TableWidget is not triggering the "scroll-end" event properly, which should load more objects to be shown.

Comment 2

[Julian Descottes [:jdescottes]]

From what I have seen, the TableWidget does trigger the event (at least on FF59), but fetching new items fails in the actor in :
https://searchfox.org/mozilla-central/rev/c56f656febb1aa58c72b218699e24a76425bd284/devtools/server/actors/storage.js#385

The issue starts with the following line. 
  let sliced = sorted.slice(offset, offset + size);

Sorted is typically an array of 50 items. Offset is > 50. So the slice returns an empty array. 
Even after fixing that the UI still doesn't update properly though.

Comment 3

[Julian Descottes [:jdescottes]]

Part of the issue is also with https://searchfox.org/mozilla-central/rev/c56f656febb1aa58c72b218699e24a76425bd284/devtools/server/actors/storage.js#2290-2298

This code snippet wants to get "all" the values, but to do so it calls getObjectStoreData which is capped to 50 entities. So we only ever retrieve the first 50 entities.

Comment 4

[Julian Descottes [:jdescottes]]

Attachment: wip-patch

Here is a first patch that seems to fix the current issues with pagination. On top of my previous comments two other issues:
- the method that checks if we are at the end of the scroll in TableWidget returns true too often (using bounding rect height instead of the scroll height)
- the update should not flash the rows when fetching 50 new rows as this freezes the UI for several seconds

Comment 5

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

The Storage inspector pagination has been tested many times so I am very surprised by this... is it only an issue for indexedDB Object Stores?

Comment 6

[Julian Descottes [:jdescottes]]

Yes the first issue is only about indexedDB and the way its version of getObjectStoreData is written. 

The other issues I mentioned in comment 4 apply to other stores as well:
- use of flashing that is very bad for performance
- fetching new results too early, which means, that as soon as you start scrolling you fetch hundreds of items rather than just the next 50

Comment 7

[Julian Descottes [:jdescottes]]

Attachment: Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

Review commit: https://reviewboard.mozilla.org/r/257846/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/257846/

Comment 8

[Julian Descottes [:jdescottes]]

Comment on attachment 8994589 [details]
Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/257846/diff/1-2/

Comment 9

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

Comment on attachment 8994589 [details]
Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

https://reviewboard.mozilla.org/r/257846/#review266372

Comment 10

[Julian Descottes [:jdescottes]]

Thanks for the review Mike!

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f2f7dc725274
Fix infinite scrolling for indexedDB in storage inspector;r=miker

Comment 12

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=Windows%2010%20x64%20asan%20Mochitests%20with%20e10s%20test-windows10-64-asan%2Fopt-mochitest-devtools-chrome-e10s-7%20M-e10s(dt7)&tochange=010c910436fe42d4e734d0f91f810662063d61eb&fromchange=f2f7dc725274d3ebd07a0212b5507d96c1ce12a6&selectedJob=190098832

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=190098832&repo=autoland&lineNumber=1846

Backout link: https://hg.mozilla.org/integration/autoland/rev/010c910436fe42d4e734d0f91f810662063d61eb

19:05:44    ERROR -  283 INFO TEST-UNEXPECTED-FAIL | devtools/client/storage/test/browser_storage_overflow.js | Test timed out -
19:05:44     INFO -  GECKO(5544) | console.error: (new Error("Connection closed, pending request to server1.conn31.child1/indexedDB29, type getStoreObjects failed\n\nRequest stack:\nrequest@resource://devtools/shared/base-loader.js -> resource://devtools/shared/protocol.js:1346:14\ngenerateRequestMethods/</frontProto[name]@resource://devtools/shared/base-loader.js -> resource://devtools/shared/protocol.js:1500:14\nfetchStorageObjects@resource://devtools/shared/base-loader.js -> resource://devtools/client/storage/ui.js:620:28\nasync*onHostSelect@resource://devtools/shared/base-loader.js -> resource://devtools/client/storage/ui.js:945:11\nAsync*emit@resource://devtools/shared/base-loader.js -> resource://devtools/shared/event-emitter.js:178:37\nemit@resource://devtools/shared/base-loader.js -> resource://devtools/shared/event-emitter.js:255:29\nset selectedItem@resource://devtools/shared/base-loader.js -> resource://devtools/client/shared/widgets/TreeWidget.js:83:7\nselectTreeItem@chrome://mochitests/content/browser/devtools/client/storage/test/head.js:522:3\nasync*runIndexedDBTests@chrome://mochitests/content/browser/devtools/client/storage/test/browser_storage_overflow.js:58:9\nasync*runTests@chrome://mochitests/content/browser/devtools/client/storage/test/browser_storage_overflow.js:34:9\nasync*@chrome://mochitests/content/browser/devtools/client/storage/test/browser_storage_overflow.js:11:9\nAsync*Tester_execTest/<@chrome://mochikit/content/browser-test.js:1103:34\nasync*Tester_execTest@chrome://mochikit/content/browser-test.js:1094:16\nnextTest/<@chrome://mochikit/content/browser-test.js:996:9\nSimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:795:59\n", "resource://devtools/shared/base-loader.js -> resource://devtools/shared/protocol.js", 1289))
19:05:44     INFO -  GECKO(5544) | console.error: (new Error("Connection closed, pending request to server1.conn31.child1/indexedDB29, type getStoreObjects failed\n\nRequest stack:\nrequest@resource://devtools/shared/base-loader.js -> resource://devtools/shared/protocol.js:1346:14\ngenerateRequestMethods/</frontProto[name]@resource://devtools/shared/base-loader.js -> resource://devtools/shared/protocol.js:1500:14\nfetchStorageObjects@resource://devtools/shared/base-loader.js -> resource://devtools/client/storage/ui.js:620:28\nasync*handleScrollEnd@resource://devtools/shared/base-loader.js -> resource://devtools/client/storage/ui.js:1102:11\nAsync*emit@resource://devtools/shared/base-loader.js -> resource://devtools/shared/event-emitter.js:178:37\nemit@resource://devtools/shared/base-loader.js -> resource://devtools/shared/event-emitter.js:255:29\nafterScroll@resource://devtools/shared/base-loader.js -> resource://devtools/client/shared/widgets/TableWidget.js:1011:7\nsetNamedTimeout/<@resource://devtools/shared/base-loader.js -> resource://devtools/client/shared/widgets/view-helpers.js:32:38\nnotify@resource://gre/modules/Timer.jsm:44:7\n", "resource://devtools/shared/base-loader.js -> resource://devtools/shared/protocol.js", 1289))
19:05:45     INFO -  284 INFO Removing tab.
19:05:45     INFO -  285 INFO Waiting for event: 'TabClose' on [object XULElement].
19:05:45     INFO -  286 INFO Got event: 'TabClose' on [object XULElement].
19:05:45     INFO -  287 INFO Tab removed and finished closing

Comment 13

[Cosmin Sabou [:CosminS]]

Also please take a look at these kind of failures: https://treeherder.mozilla.org/logviewer.html#?job_id=190107795&repo=autoland&lineNumber=15245

Comment 14

[Julian Descottes [:jdescottes]]

Comment on attachment 8994589 [details]
Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/257846/diff/2-3/

Comment 15

[Julian Descottes [:jdescottes]]

Need to add an async setup method in test page:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=aff1215b314954f8cc2dea09327793f35da29182

Comment 16

[Julian Descottes [:jdescottes]]

Comment on attachment 8994589 [details]
Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/257846/diff/3-4/

Comment 17

[Julian Descottes [:jdescottes]]

Comment on attachment 8994589 [details]
Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/257846/diff/4-5/

Comment 18

[Julian Descottes [:jdescottes]]

Comment on attachment 8994589 [details]
Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

Mike can you take another look? I had to change the test in order to make it pass. Most importantly I split the indexeddb and localstorage tests.

Thanks!

Comment 19

[Julian Descottes [:jdescottes]]

try https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab8af72d383d05544d2df663d9cfbd40347affef

Comment 20

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

Comment on attachment 8994589 [details]
Bug 1434579 - Fix infinite scrolling for indexedDB in storage inspector;

Splitting the test makes a lot of sense.

r+ assuming a green try.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/327631824cff
Fix infinite scrolling for indexedDB in storage inspector;r=miker

Comment 22

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/327631824cff