1800202 - (Re-)building folder cache leads to MSF deletion

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[b5]

Attachment: Local Folders.7z

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36

Steps to reproduce:

In the past, deleting panacea.dat led to a mass deletion of MSF files since for rebuilding the folder cache, too many MSF files were opened at once. This is well-known, see bug 1093217 comment #5. (Truncated to null bytes was a no-longer-existing side effect of a missing MSF being replaced with an empty file, see bug 1724849.)

This hasn't changed with the new folder cache. Drop the attached 910 (1 + 9 + 90 + 810) folders into "Local Folders" and start a debug version of TB. You can see mork asserts about not being able to open the databases. This happens while updating the folder cache with the new folders.

You can also see it by switching on the "Total" in the folder pane columns. The correct totals of 90 are only show for the first six or seven top level folders.

Note that the 910 folder example was prepared by [e-mail] for bug 1554188.

Comment 1

[b5]

https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/1800202-close-db-in-ReadDBFolderInfo.patch

Comment 2

[Thomas D. (:thomas8)]

Confirming based on reporter's comment 0.
Looks like a serious enough architectural problem which we should address - just having lots of folders is not a very extraordinary scenario, but may happen quite easily over time wrt archiving etc.

Magnus, who can fix this?
Can we use the patch linked in comment 1 to fix this?

Comment 3

[b5]

There are more issues in this area, see bug 1795178 comment #1.

Comment 4

[Magnus Melin [:mkmelin]]

(In reply to Thomas D. (:thomas8) from comment #2)

Can we use the patch linked in comment 1 to fix this?

Seems we could.
Apply the patch from bug 1787609 and the patch from comment 1 on top.
Ben, perhaps reconsider taking the patch in bug 1787609 unless bug 1795178 is likely to see a quick resolution

Comment 5

[Ben Campbell]

(In reply to Magnus Melin [:mkmelin] from comment #4)

(In reply to Thomas D. (:thomas8) from comment #2)

Can we use the patch linked in comment 1 to fix this?

Seems we could.
Apply the patch from bug 1787609 and the patch from comment 1 on top.

Yeah, seems like the prudent thing to do in the interim.
Sorry - I should have been more open to a short-term fix earlier.

Comment 6

[Magnus Melin [:mkmelin]]

Attachment: Bug 1800202 - Close db after use in nsMsgDBFolder::ReadDBFolderInfo(). r=benc

This is run from discoverFolders() at startup and can cause MSF loss if too many folders are left open.

Depends on D162520

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/fd0e7ca7df18
Close db after use in nsMsgDBFolder::ReadDBFolderInfo(). r=benc

Comment 8

[Magnus Melin [:mkmelin]]

Comment on attachment 9304374 [details]
Bug 1800202 - Close db after use in nsMsgDBFolder::ReadDBFolderInfo(). r=benc

[Approval Request Comment]
User impact if declined: can help some with .msf problems
Testing completed (on c-c, etc.): beta
Risk to taking this patch (and alternatives if risky): should be safe

Comment 9

[Wayne Mery (:wsmwk)]

Comment on attachment 9304374 [details]
Bug 1800202 - Close db after use in nsMsgDBFolder::ReadDBFolderInfo(). r=benc

[Triage Comment]
Approved for esr102

Comment 10

[Rob Lemley [:rjl]]

Thunderbird 102.7.0:
https://hg.mozilla.org/releases/comm-esr102/rev/f2e307ab4e8a