1874334 - DecreaseCachedQuotaUsage /builds/worker/checkouts/gecko/dom/fs/parent/datamodel/FileSystemDatabaseManagerVersion001.cpp

Status: RESOLVED FIXED

(Core :: DOM: File, defect, P2)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.zip

Found while fuzzing m-c 20231121-7bc9f9c659dd (--enable-debug --enable-fuzzing)

To reproduce via Grizzly Replay:

$ pip install fuzzfetch grizzly-framework
$ python -m fuzzfetch -d --fuzzing -n firefox
$ python -m grizzly.replay.bugzilla ./firefox/firefox <bugid>

Assertion failure: uint64_t(aDest) >= uint64_t(aArg), at /builds/worker/workspace/obj-build/dist/include/mozilla/dom/quota/AssertionsImpl.h:47

#0 0x7feb7ae32138 in AssertNoUnderflow<unsigned long, long> /builds/worker/workspace/obj-build/dist/include/mozilla/dom/quota/AssertionsImpl.h:47:3
#1 0x7feb7ae32138 in mozilla::dom::quota::OriginInfo::LockedDecreaseUsage(mozilla::dom::quota::Client::Type, long) /builds/worker/checkouts/gecko/dom/quota/OriginInfo.cpp:112:3
#2 0x7feb7addbf47 in mozilla::dom::quota::QuotaManager::DecreaseUsageForClient(mozilla::dom::quota::ClientMetadata const&, long) /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2520:17
#3 0x7feb7a038f56 in DecreaseCachedQuotaUsage /builds/worker/checkouts/gecko/dom/fs/parent/datamodel/FileSystemDatabaseManagerVersion001.cpp:1232:17
#4 0x7feb7a038f56 in mozilla::dom::fs::data::FileSystemDatabaseManagerVersion002::MergeFileId(nsTString<char> const&, mozilla::dom::fs::FileId const&, bool)::$_1::operator()(mozilla::dom::fs::FileId const&) const /builds/worker/checkouts/gecko/dom/fs/parent/datamodel/FileSystemDatabaseManagerVersion002.cpp:667:9
#5 0x7feb7a03819f in mozilla::dom::fs::data::FileSystemDatabaseManagerVersion002::MergeFileId(nsTString<char> const&, mozilla::dom::fs::FileId const&, bool) /builds/worker/checkouts/gecko/dom/fs/parent/datamodel/FileSystemDatabaseManagerVersion002.cpp:694:5
#6 0x7feb7a01e6b1 in mozilla::dom::fs::data::FileSystemDataManager::UnlockShared(nsTString<char> const&, mozilla::dom::fs::FileId const&, bool) /builds/worker/checkouts/gecko/dom/fs/parent/datamodel/FileSystemDataManager.cpp:498:3
#7 0x7feb7a005dad in mozilla::dom::FileSystemWritableFileStreamParent::RecvClose(bool, std::function<void (mozilla::void_t const&)>&&) /builds/worker/checkouts/gecko/dom/fs/parent/FileSystemWritableFileStreamParent.cpp:41:3
#8 0x7feb7a06936f in mozilla::dom::PFileSystemWritableFileStreamParent::OnMessageReceived(IPC::Message const&) /builds/worker/workspace/obj-build/ipc/ipdl/PFileSystemWritableFileStreamParent.cpp:157:102
#9 0x7feb7a0645f7 in mozilla::dom::PFileSystemManagerParent::OnMessageReceived(IPC::Message const&) /builds/worker/workspace/obj-build/ipc/ipdl/PFileSystemManagerParent.cpp:304:32
#10 0x7feb772d85cf in mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1813:25
#11 0x7feb772d5322 in mozilla::ipc::MessageChannel::DispatchMessage(mozilla::ipc::ActorLifecycleProxy*, mozilla::UniquePtr<IPC::Message, mozilla::DefaultDelete<IPC::Message>>) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1732:9
#12 0x7feb772d5fa2 in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::ActorLifecycleProxy*, mozilla::ipc::MessageChannel::MessageTask&) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1525:3
#13 0x7feb772d70ef in mozilla::ipc::MessageChannel::MessageTask::Run() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1623:14
#14 0x7feb765fbe5b in mozilla::TaskQueue::Runner::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskQueue.cpp:257:20
#15 0x7feb766268d5 in nsThreadPool::Run() /builds/worker/checkouts/gecko/xpcom/threads/nsThreadPool.cpp:341:14
#16 0x7feb7661ceed in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1192:16
#17 0x7feb76623dcd in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:480:10
#18 0x7feb772df6ce in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:300:20
#19 0x7feb771f8461 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:363:3
#20 0x7feb771f8461 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:345:3
#21 0x7feb766181d3 in nsThread::ThreadFunc(void*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:370:10
#22 0x7feb8abecd0f in _pt_root /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5
#23 0x7feb8b48dac2 in start_thread nptl/pthread_create.c:442:8
#24 0x7feb8b51ebf3 in __clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Comment 1

[Bugmon [:jkratzer for issues]]

Verified bug as reproducible on mozilla-central 20240112045806-1d2ccbe0bb6d.
The bug appears to have been introduced in the following build range:

Start: be6843e3475a7f536156aba03a849fe701932b0c (20230628154748)
End: 0df511b69760b69959818e34e07aa74a0ad7061a (20230628173448)
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=be6843e3475a7f536156aba03a849fe701932b0c&tochange=0df511b69760b69959818e34e07aa74a0ad7061a

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1824305

:jjalkanen, since you are the author of the regressor, bug 1824305, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1824305

Comment 4

[Bugmon [:jkratzer for issues]]

Successfully recorded a pernosco session. A link to the pernosco session will be added here shortly.

Comment 5

[Bugmon [:jkratzer for issues]]

A pernosco session for this bug can be found here.

Comment 6

[Intermittent Failures Robot]

1 failures in 3952 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 7

[Dianna Smith [:diannaS]]

:jari do you happen to know if this assertion failure has any user facing impact?

Comment 8

[Jari Jalkanen]

Attachment: WIP: Bug 1874334 - Add test to IDB crashtest suite

Comment 9

[Intermittent Failures Robot]

1 failures in 4009 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 10

[Intermittent Failures Robot]

1 failures in 3382 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-02-19&endday=2024-02-25&tree=all

Comment 11

[Intermittent Failures Robot]

3 failures in 4052 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 2
  • debug: 2
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-02-26&endday=2024-03-03&tree=all

Comment 12

[Intermittent Failures Robot]

3 failures in 4198 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• windows11-32-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-03-04&endday=2024-03-10&tree=all

Comment 13

[Jari Jalkanen]

Attachment: Bug 1874334 - Release blockers when WritableFileStream without actor is closing. r=#dom-storage

Comment 14

[Jari Jalkanen]

Attachment: Bug 1874334 - Add fuzzing finding to OPFS crashtest suite. r=#dom-storage

Comment 15

[Jari Jalkanen]

Attachment: Bug 1874334 - Update cached quota usage when OPFS temporary file is copied. r=#dom-storage

Comment 16

[Intermittent Failures Robot]

5 failures in 3849 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5

Platform and build breakdown:

• windows11-64-2009-qr: 3
  • debug: 3
• windows11-32-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-03-11&endday=2024-03-17&tree=all

Comment 17

[Jari Jalkanen]

There is some user facing impact even though nothing will crash or break.

Comment 18

[Jari Jalkanen]

Attachment: Bug 1874334 - Introduce control actor for WritableFileStreams. r=#dom-storage

Comment 19

[Jan Varga [:janv]]

Attachment: Bug 1874334 - Add worker related fuzzing finding to OPFS crashtest suite; r=#dom-storage

Comment 20

[Jan Varga [:janv]]

Attachment: Bug 1874334 - Abort open writable file streams during xpcom-will-shutdown shutdown phase; r=#dom-storage

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5e1a6b776f69
Add fuzzing finding to OPFS crashtest suite. r=dom-storage-reviewers,janv
https://hg.mozilla.org/integration/autoland/rev/ab8f6bc22e59
Add worker related fuzzing finding to OPFS crashtest suite; r=dom-storage-reviewers,jari
https://hg.mozilla.org/integration/autoland/rev/7684de0452a5
Update cached quota usage when OPFS temporary file is copied. r=dom-storage-reviewers,janv
https://hg.mozilla.org/integration/autoland/rev/a872d74549a4
Abort open writable file streams during xpcom-will-shutdown shutdown phase; r=dom-storage-reviewers,jstutte

Comment 22

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/5e1a6b776f69
https://hg.mozilla.org/mozilla-central/rev/ab8f6bc22e59
https://hg.mozilla.org/mozilla-central/rev/7684de0452a5
https://hg.mozilla.org/mozilla-central/rev/a872d74549a4

Comment 23

[Cristian Tuns]

It looks like we still have the failures here: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=VtIPHQZsS6G94Rhyoxey4g.0&revision=5343a8186c58bd98347495d7a5da51600b68c552&searchStr=windows%2C11%2Cx86%2C22h2%2Cwebrender%2Cdebug%2Ctest-windows11-32-2009-qr%2Fdebug-marionette%2Cmn
Can you please take another look?

Comment 24

[Jari Jalkanen]

I'm not sure if this patch from autoland was yet based on the supposed fix. Let's keep an eye on the situation.

Comment 25

[Intermittent Failures Robot]

1 failures in 3590 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-03-25&endday=2024-03-31&tree=all

Comment 26

[Intermittent Failures Robot]

1 failures in 3805 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-04-01&endday=2024-04-07&tree=all

Comment 27

[Bugmon [:jkratzer for issues]]

Testcase crashes using the initial build (mozilla-central 20231121183118-7bc9f9c659dd) but not with tip (mozilla-central 20240412214434-be4463b26a49.)

The bug appears to have been fixed in the following build range:

Start: 9c458764557de25f93134811a808f6c5b68b5683 (20240327123927)
End: bc7ca80e302a788b0c3cc63cb148c628351b4946 (20240327133848)
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=9c458764557de25f93134811a808f6c5b68b5683&tochange=bc7ca80e302a788b0c3cc63cb148c628351b4946

jjalkanen, can you confirm that the above bisection range is responsible for fixing this issue?
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.

Comment 28

[Intermittent Failures Robot]

1 failures in 3235 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-04-08&endday=2024-04-14&tree=all

Comment 29

[Jari Jalkanen]

The latest intermittent failure is for cache from test dom/cache/test/marionette/test_caches_delete_cleanup_after_shutdown.py and unrelated to the fuzzing finding. They have the same line in the logs only because they use the same underflow check helper from dom/quota.

Therefore, I would conclude that the above bisection range (and the attached patch) are responsible for fixing the fuzzing finding.

Comment 30

[Intermittent Failures Robot]

4 failures in 4252 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 3

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• windows11-32-2009-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-04-15&endday=2024-04-21&tree=all

Comment 31

[Intermittent Failures Robot]

1 failures in 4336 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-04-22&endday=2024-04-28&tree=all

Comment 32

[Intermittent Failures Robot]

6 failures in 4059 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2
• mozilla-beta: 1

Platform and build breakdown:

• windows11-64-2009-qr: 1
  • debug: 1
• windows11-32-2009-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-04-29&endday=2024-05-05&tree=all

Comment 33

[Intermittent Failures Robot]

2 failures in 3649 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• try: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-05-06&endday=2024-05-12&tree=all

Comment 34

[Intermittent Failures Robot]

1 failures in 4547 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-05-13&endday=2024-05-19&tree=all

Comment 35

[Intermittent Failures Robot]

3 failures in 3995 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• windows11-64-2009-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-05-20&endday=2024-05-26&tree=all

Comment 36

[Intermittent Failures Robot]

2 failures in 4202 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• windows11-32-2009-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-05-27&endday=2024-06-02&tree=all

Comment 37

[Intermittent Failures Robot]

3 failures in 3649 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• linux1804-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-06-03&endday=2024-06-09&tree=all

Comment 38

[Intermittent Failures Robot]

1 failures in 4625 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-06-10&endday=2024-06-16&tree=all

Comment 39

[Intermittent Failures Robot]

1 failures in 4577 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-06-17&endday=2024-06-23&tree=all

Comment 40

[Intermittent Failures Robot]

1 failures in 4116 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1874334&startday=2024-06-24&endday=2024-06-30&tree=all

Comment 41

[Jari Jalkanen]

The line Assertion failure: uint64_t(aDest) >= uint64_t(aArg), at /builds/worker/workspace/obj-build/dist/include/mozilla/dom/quota/AssertionsImpl.h:47 is present in the logs whenever a quota client has an underflow. The crashes from dom/cache/test/marionette/test_caches_delete_cleanup_after_shutdown.py and dom/workers/test/marionette/test_service_workers_disabled.py should not be associated with this bug which has not been reproducible after the patch landed.