1860486 - IDBObjectStore.clear() performance regression that can appear like a hang after upgrading to SQLite 3.41.0

Status: RESOLVED FIXED

(Core :: Storage: IndexedDB, defect, P3)

Description

[Brian Birtles (:birtles)]

I haven't been able to make a reduced test case for this yet but the following STR reproduces consistently for me on Windows.

STR:

1. Go to https://hikibiki.app
2. Press the two cog icons and wait for the dictionaries to download (i.e. both progress bars to reach 100%)
3. Go to the language selector at the bottom of the page and choose "Français"

Expected results:

The two pink sections switch to "Checking for updates..." for about 15s (depends on your machine I guess) and then begin downloading again (i.e. the progress bars re-appear).

Actual results:

The two pink sections switch to "Checking for updates..." and just hang like that forever.

I ran this through Mozregression and it pointed to this changeset:

https://hg.mozilla.org/integration/autoland/rev/78445faba66006a8a33a01f854e87b516e2a6a23

The source for the web app is here: https://github.com/birchill/hikibiki-app/

But it's really the library it uses for downloading the data that is interesting: https://github.com/birchill/jpdict-idb/

I added some basic logging to the library and it revealed that it is hanging on this line:

https://github.com/birchill/jpdict-idb/blob/46239835b0977200b09b0b7e9e220f57712b76c8/src/store.ts#L296

This same bug occurs on this add-on which uses the same library: https://addons.mozilla.org/en-US/firefox/addon/10ten-ja-reader/

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

:RyanVM, since you are the author of the regressor, bug 1776566, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 2

[Brian Birtles (:birtles)]

This reproduces for me on Mac too.

Comment 3

[Marco Bonardo [:mak]]

Let's give this bug more visibility to the appropriate team

Comment 4

[Jens Stutte [:jstutte]]

Hi Jari, would you mind to take a look?

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1776566

Comment 6

[Jari Jalkanen]

Attachment: Bug 1860486 - Fix IDB object store clear() performance regression. r=#dom-storage

Comment 7

[Jari Jalkanen]

Attachment: Bug 1860486 - Test IDB object store clear() performance regression. r=#dom-storage

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9fc6ae90cc51
Test IDB object store clear() performance regression. r=dom-storage-reviewers,janv
https://hg.mozilla.org/integration/autoland/rev/a7618fbeaa67
Fix IDB object store clear() performance regression. r=dom-storage-reviewers,janv

Comment 9

[Natalia Csoregi [:nataliaCs]]

Backed out for causing failures on test_clear_object_store_with_indexes.js

• backout: https://hg.mozilla.org/integration/autoland/rev/e22abf3976f2653087769023f21c9c1615e9437d
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=EbiPx7hQT5uHE6L3813s3A.0&revision=a7618fbeaa677c6f785662c921e85e07713b290d
• failure log: https://treeherder.mozilla.org/logviewer?job_id=441146472&repo=autoland&lineNumber=5521

[task 2023-12-22T14:14:11.901Z] 14:14:11     INFO -  TEST-START | xpcshell.toml:dom/indexedDB/test/unit/test_clear_object_store_with_indexes.js
[task 2023-12-22T14:14:11.947Z] 14:14:11     INFO -  adb launch_application: am startservice -W -n 'org.mozilla.geckoview.test_runner/org.mozilla.geckoview.test_runner.XpcshellTestRunnerService$i0' -a android.intent.action.MAIN --es env0 XPCOM_DEBUG_BREAK=stack-and-abort --es env1 MOZ_CRASHREPORTER=1 --es env2 MOZ_CRASHREPORTER_NO_REPORT=1 --es env3 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env4 MOZ_DEVELOPER_REPO_DIR=/builds/worker/checkouts/gecko --es env5 MOZ_DEVELOPER_OBJ_DIR=/builds/worker/workspace/obj-build --es env6 MOZ_DISABLE_CONTENT_SANDBOX=1 --es env7 MOZ_FETCHES_DIR=/builds/worker/fetches --es env8 MOZ_DISABLE_SOCKET_PROCESS=1 --es env9 LD_LIBRARY_PATH=/data/local/tmp/test_root/xpcb --es env10 MOZ_LINKER_CACHE=/data/local/tmp/test_root/xpcb --es env11 GRE_HOME=/data/local/tmp/test_root/xpcb --es env12 XPCSHELL_TEST_PROFILE_DIR=/data/local/tmp/test_root/xpc/p/7dd4e7d3-3e39-4fc8-8621-883aa0843c3b --es env13 HOME=/data/local/tmp/test_root/xpc/p --es env14 XPCSHELL_TEST_TEMP_DIR=/data/local/tmp/test_root/xpc/tmp/0fb2ca58-c89d-43e3-bc25-990dea186e9c --es env15 MOZ_ANDROID_DATA_DIR=/data/local/tmp/test_root/xpcb --es env16 MOZ_IN_AUTOMATION=1 --es env17 MOZ_ANDROID_CPU_ABI=x86_64 --es env18 MOZHTTP2_PORT=43763 --es env19 MOZNODE_EXEC_PORT=45781 --es env20 MOZHTTP3_ECH= --es env21 TMPDIR=/data/local/tmp/test_root/xpc/p/7dd4e7d3-3e39-4fc8-8621-883aa0843c3b --es env22 XPCSHELL_MINIDUMP_DIR=/data/local/tmp/test_root/xpc/minidumps/7dd4e7d3-3e39-4fc8-8621-883aa0843c3b --es arg0 -g --es arg1 /data/local/tmp/test_root/xpcb --es arg2 --greomni --es arg3 /data/local/tmp/test_root/xpcb/geckoview-test_runner.apk --es arg4 -m --es arg5 -e --es arg6 'const _HEAD_JS_PATH = "/data/local/tmp/test_root/xpc/head.js";' --es arg7 -e --es arg8 'const _MOZINFO_JS_PATH = "/data/local/tmp/test_root/xpc/p/7dd4e7d3-3e39-4fc8-8621-883aa0843c3b/mozinfo.json";' --es arg9 -e --es arg10 'const _PREFS_FILE = "/data/local/tmp/test_root/xpc/user.js";' --es arg11 -e --es arg12 'const _TESTING_MODULES_DIR = "/data/local/tmp/test_root/xpc/m";' --es arg13 -f --es arg14 /data/local/tmp/test_root/xpc/head.js --es arg15 -e --es arg16 'const _HEAD_FILES = ["/data/local/tmp/test_root/xpc/dom/indexedDB/test/unit/xpcshell-head-parent-process.js"];' --es arg17 -e --es arg18 'const _JSDEBUGGER_PORT = 0;' --es arg19 -e --es arg20 'const _TEST_CWD = "/data/local/tmp/test_root/xpc/dom/indexedDB/test/unit";' --es arg21 -e --es arg22 'const _TEST_FILE = ["test_clear_object_store_with_indexes.js"];' --es arg23 -e --es arg24 'const _TEST_NAME = "xpcshell.toml:dom/indexedDB/test/unit/test_clear_object_store_with_indexes.js";' --es arg25 -e --es arg26 '_execute_test(); quit(0);' --ez use_multiprocess True --es out_file /data/local/tmp/test_root/xpc/logs/xpcshell-059c6d95-3752-4049-b65d-21b3b273eec3.log
[task 2023-12-22T14:14:12.183Z] 14:14:12     INFO -  remotexpcshelltests.py | xpcshell.toml:dom/indexedDB/test/unit/test_clear_object_store_with_indexes.js | 15898 | Launched Test App
[task 2023-12-22T14:16:23.901Z] 14:16:23     INFO -  remotexpcshelltests.py | xpcshell.toml:dom/indexedDB/test/unit/test_clear_object_store_with_indexes.js | 15898 | Application ran for: 0:02:11.999242
[task 2023-12-22T14:16:23.965Z] 14:16:23  WARNING -  TEST-UNEXPECTED-FAIL | xpcshell.toml:dom/indexedDB/test/unit/test_clear_object_store_with_indexes.js | xpcshell return code: 0
[task 2023-12-22T14:16:23.966Z] 14:16:23     INFO -  TEST-INFO took 132064ms

Comment 10

[Jari Jalkanen]

Attachment: Bug 1860486 - Use larger timeout for IDB object store clear on Android. r=#dom-storage

Depends on D195204

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ffaac6566c3c
Test IDB object store clear() performance regression. r=dom-storage-reviewers,janv
https://hg.mozilla.org/integration/autoland/rev/5741e1445fd9
Fix IDB object store clear() performance regression. r=dom-storage-reviewers,janv
https://hg.mozilla.org/integration/autoland/rev/4c051bde5ad2
Use larger timeout for IDB object store clear on Android. r=dom-storage-reviewers,janv

Comment 12

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/ffaac6566c3c
https://hg.mozilla.org/mozilla-central/rev/5741e1445fd9
https://hg.mozilla.org/mozilla-central/rev/4c051bde5ad2

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit BugBot documentation.