Open Bug 1554188 Opened 5 years ago Updated 16 days ago

saved search / virtual folders kills index files or re-indexes frequently

Categories

(Thunderbird :: General, defect)

Product:
Thunderbird
Component:
General
Platform:
x86_64
Windows 10
Type:
defect

Tracking

(Not tracked)

Status:
UNCONFIRMED

People

(Reporter: bugzilla, Unassigned)

References

Details

Attachments

(4 files)

Screenshot 2022-03-22 112610.png
21.47 KB, image/png
Details
Screenshot 2022-03-22 114146.png
8.26 KB, image/png
Details
Screenshot 2022-07-28.png
16.48 KB, image/png
Details
WIP: Bug 1554188 - More aggressively close databases during search. r=benc
48 bytes, text/x-phabricator-request
Details | Review

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:67.0) Gecko/20100101 Firefox/67.0

Steps to reproduce:

After years I have now generated a nice setup for interested developers with one folder and 10 sub-folders. Each sub-folder contains again 10 sub-folders and each sub-folder contains exactly one file. I generated a search for TEST and saved as virtual folder. If you add this setup to TB60 with the local folders ad-don you can watch yourself what happens.

Actual results:

I generated a search for TEST and saved as virtual folder. If you show the number of files in the main mail tree you will see that after some restarts TB does not see the files contained in each folder any more. In most cases it will show less files than are there. If you again generate a new search for "whatever" you will force manual re-indexing and everything is okay.

Expected results:

No frequent re-indexing should occur?

I will send the set-up on request as ZIP.

Alfred, want to take a crack at the testcase being offered?

Component: Untriaged → General
Flags: needinfo?(infofrommozilla)

(In reply to bugzilla@arge-metallguss.de from comment #0)

Steps to reproduce:

After years I have now generated a nice setup for interested developers with one folder and 10 sub-folders. Each sub-folder contains again 10 sub-folders and each sub-folder contains exactly one file. I generated a search for TEST and saved as virtual folder.

OK, I've done so, but I can't see an Issue. The number of found mails stays on 100.

I've tested it with TB60.7 and TB trunk.

If you add this setup to TB60 with the local folders ad-don you can watch yourself what happens.

Maybe the ADD-ON is the problem. So please test it in Safe-Mode without any ADD-ONs enabled.

I will send the set-up on request as ZIP.

If Save-Mode doesn't help and you think it makes a difference, feel free to do so.

Flags: needinfo?(infofrommozilla)

(In reply to Alfred Peters from comment #2)

(In reply to bugzilla@arge-metallguss.de from comment #0)
If Save-Mode doesn't help and you think it makes a difference, feel free to do so.

Hello Alfred,
I just sent my setup. Currently the filter finds 417 files. If I generate a new search for TEST it locates 471 files. In the 9 sub-folders there are 90 files each so there should 810 files. I also did test with all add-ons deactivated.

Regards,
Thomas

(In reply to bugzilla@arge-metallguss.de from comment #3)

(In reply to Alfred Peters from comment #2)

There was probably a misunderstanding. In comment # 0 you have written about 10*10 folders.

I just sent my setup. Currently the filter finds 417 files. If I generate a new search for TEST it locates 471 files. In the 9 sub-folders there are 90 files each so there should 810 files. I also did test with all add-ons deactivated.

Indeed your example has 9109 = 810 Folders.
OK, that changes everything.
Now I can confirm the bug.

I'll take a look.

(In reply to Alfred Peters from comment #4)

(In reply to bugzilla@arge-metallguss.de from comment #3)

(In reply to Alfred Peters from comment #2)

There was probably a misunderstanding. In comment # 0 you have written about 10*10 folders.

You are correct. If I recall it properly it happens above 500 folders.

I just sent my setup. Currently the filter finds 417 files. If I generate a new search for TEST it locates 471 files. In the 9 sub-folders there are 90 files each so there should 810 files. I also did test with all add-ons deactivated.

Indeed your example has 9109 = 810 Folders.
OK, that changes everything.
Now I can confirm the bug.

I'll take a look.

Thank you very much.

Not very much going on here?

Just installed TB68.4.1. No change so far. Unfortunately the SavedSearchThemAll add-on does not work any more in this version. So for all my filters I have to add each new folder manually. Very useful function, if it would work :-)

I just wanted to note that this issue is still pending. As I said 5 or even 10 years ago the function permanently forgets moved folders, it causes constant re-indexing and in the most recent version of TB91.6.2 it also causes frequent errors because indexing on start prevents TB to run the filters properly. Please fix if you can.

OS: Unspecified → Windows 10
Hardware: Unspecified → x86_64

Some more hints for the developers:

#1 when downloading e-mails immediately after starting TB, I now get frequent filter errors claiming that the target directories do not exist
#2 when I turn off immediately downloading e-mails after starting Thunderbird, then select all virtual folders one after another, then as described indexing takes place every time I select the next virtual folder. When ALL virtual folders are manually indexed (by selecting them individually) and I start download e-mail then, everything works fine

May be this also helps:
#1 My filters run on a POP3 account and target directories are placed in local folders.
#2 I have 500+ folders :-)
#3 total size of all e-mails in local folders is around 5GB
#4 total number of e-mail in local folders is 20.000 (my complete e-mail archive)

Version: 60 → Thunderbird 91

Thanks very much for the additional information. Do you have the precise wording for "filter errors claiming that the target directories do not exist"? How many pop accounts? Roughly how many virtual folders?

(Please maintain version # at the originally/oldest reported versoin - it helps with our quality tracking.)

Flags: needinfo?(bugzilla)
Version: Thunderbird 91 → 60

(In reply to bugzilla@arge-metallguss.de from comment #0)

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:67.0) Gecko/20100101 Firefox/67.0

Steps to reproduce:

After years I have now generated a nice setup for interested developers with one folder and 10 sub-folders. Each sub-folder contains again 10 sub-folders and each sub-folder contains exactly one file. I generated a search for TEST and saved as virtual folder.

Can you post a screen shot of what the folder hierarchy looks like? (obscure the folder names if you prefer)

Here comes a preview of the folder structure. All folders have at least subfolders.

rough translation of the message: The message could not be filtered to the "..." folder because adding a message failed. Make sure that the folder is displayed correctly or try to repair the folder.

Flags: needinfo?(bugzilla)

In the meantime I have installed TB 91.9.1 an I can tell you this "defect" has gotten even worse.

After EVERY start of TB the virtual folders are messed up, that means TB forgets basically all virtual folder settings.

#1 the search criteria are lost
#2 the folders to search are also lost

So I can newly set up the virtual folders every day. A very practical function I would say.

In TB 78 or was it 68 we still had the saved-search-them-all addon, that would restore the search in all local folders, but that was somehow optimized away. Under "normal" conditions one would call this a downgrade, but I do not want to be impolite, since I have not paid anything for TB for now.

In the meantime I have installed TB 91.11.0 and now it happens when opening Thunderbird after a restart ALL VIRTUAL FOLDERS are lost. That means the entries are still there, but the content is completely lost. NO filter CRITERIA and NO FOLDERS selected.

I wonder how many people are using virtual folders. I can't be that many, otherwise this issue would have been fixed years ago ;-)

Today I made an interesting observation. As ALWAYS when starting TB some of my filters were damaged. It tried to setup a damaged filter and had 10 results. When searching manually I received 50 results for the same folder range. Interestingly AFTER the manual search the virtual filter showed the same number as the manual search. To me it looks like the index is somehow f.....

Next hint for the potential solver of this problem. If you manually search for a text like '123123' TB WILL NOT INDEX all folders!

I had a search for "status is not read". Then it would index ALL folders and afterward the virtual folder will show all results.

I think I may have encountered the same bug. I am experiencing issues with Saved Searches, specifically:
#1 the search criteria are lost
#2 the folders to search are also lost

The issue occurs on 102 and 91.11.0.

As a workaround, I am able to save a backup of the virtualFolders.dat file, and overwrite it from backup whenever the Saved Searches are overwritten. The Saved Searches are broken within a few minutes of loading Thunderbird. I am searching 247 folders across 247 pop email accounts.

This issue only started for me last week. I have been using Thunderbird for 10 years.

(In reply to yflow-wolfy from comment #19)

I think I may have encountered the same bug. I am experiencing issues with Saved Searches, specifically:
#1 the search criteria are lost
#2 the folders to search are also lost

The issue occurs on 102 and 91.11.0.

As a workaround, I am able to save a backup of the virtualFolders.dat file, and overwrite it from backup whenever the Saved Searches are overwritten. The Saved Searches are broken within a few minutes of loading Thunderbird. I am searching 247 folders across 247 pop email accounts.

Thanks for your workaround.

What I found in addition is that the recursion of indexing into third level folders obviously does not work. Virtual folders therefore do NOT show all results. If you start a manual search on ALL local folders TB will start indexing the missing 3rd level sub-folders. After that MANUAL search, the virtual folder will also show ALL results, after a WHILE!

(In reply to yflow-wolfy from comment #19)

I think I may have encountered the same bug. I am experiencing issues with Saved Searches, specifically:
#1 the search criteria are lost
#2 the folders to search are also lost

The issue occurs on 102 and 91.11.0.

As a workaround, I am able to save a backup of the virtualFolders.dat file, and overwrite it from backup whenever the Saved Searches are overwritten. The Saved Searches are broken within a few minutes of loading Thunderbird. I am searching 247 folders across 247 pop email accounts.

Thanks for your hint with backup and restore. It saves me a lot of time. Anyway, after restoring I only get complete results when running a random MANUAL search BEFORE clicking on any virtual folder. With every click on a virtual folder I see reoccurring indexing and sometimes different search results.

I think I have resolved this issue on my Thunderbird profile, and I think it was caused by a data issue. I found a corrupt (mbox?) file in my profile. Once I fixed the mbox file the issue with saved folders went away.

Thunderbird could help us out more here by accurately reporting data issues and failing a bit more gracefully (rather than corrupting the virtualFolders.dat file) when it encounters an issue indexing.

You could try removing all folders from your saved searches, and adding them back one by one (or half at a time) to pinpoint the issue and see if we have similar/the same issues?

Here comes a nice workaround until someone takes care of fixing the issue:

#1 make a copy of virtualFolders.dat
#2 restore the virtualFolders.dat from the copy
#3 start TB and AVOID loading any mails from your mail server - offline mode
#4 open a search on local folders and search for Status > is not > read - WAIT
#5 you can now safely select one virtual folder after another
#6 you will have indexing every time your open the next virtual folder, be patient and WAIT

If you don't stick to the rules virtualFolders.dat will be damaged immediately. You need to close TB and restore virtualFolders.dat from your copy. Same game, same procedure, same waiting, every day, every time you open TB. Enjoy if you can ;-)

Let's pray together for a fix to come soon.

Just for the purpose of debugging.

If you have several virtual folders and you scroll through them top > down the probability of a damage is almost 100%. If you scroll through the virtual folders bottom > up high probability of NO damage.

Once one single virtual folder is damaged, you need to close TB and restore virtualfolders.dat.

Severity: normal → S3
See Also: → 1787609

[Bug 1787609] getMsgHdrForMsgId() may cause massive loss of MSF files since it potentially opens all folders in the profile hitting the limit of open files

That is exactly what is happening when you open TB. It immediately kills random MSF files. I assume 500 is the limit I reported years ago.

On Windows there is an open file limit of 512 files. There are occasions where TB opens more folder databases at the same time, see for example bug 1787609. MSF files that can't be opened due to the open file limit are deemed invalid and are deleted. The patch in bug 1787609 tries to mitigate this a bit. However, profiles with 1800 or even 2700 folder known to us survive without MSF loss under "normal" circumstances. It's not true that TB (quote) "immediately kills random MSF files" when opened. There must be some special circumstances causing this. According to the report, setting up a search folder which searches/opens more than 512 folders will cause folder loss.

This area is looked at in bug 1787609 and it's spin-off, bug 1795178.

Correct, I assumed that everybody was aware that we were talking about saved search / virtual folders.

One recent addition of symptoms I noticed is that TB in many cases cannot filter successfully any more. Once you start TB it fetches mails from several accounts but all filters claim that the target directory in local folders is not existent. If you start TB in offline mode and search e.g. for status is not read in local folders TB will start indexing numerous files. When finished you can go online, check all accounts and all filters will work without trouble.

There was an issue with disappearing MSF files that was addressed in bug 1726319 as well... several linked bugs are mentioned in the discussions.

Some context: We are understanding disappearing MSF files better now. "Traditionally" they got deleted for users with more than ~500 folders when through some operation too many folder databases were opened. The ones that couldn't be opened any more and were deleted. This "traditionally" happened under these circumstances:

  1. Old (pre 102) folder cache panacea.dat deleted, see bug 1093217 comment #5. (Truncated to null bytes was a no-longer-existing side effect of a missing MSF being replaced with an empty file, see bug 1724849.) We haven't tested yet what happens when the "modern" folder fache folderCache.json gets deteted.
  2. Search opening too many folders at once, this bug 1554188 here, also bug 1787609 and likely also bug 1551173 comment #25 (save search as cause).

At least issue 2. is still present.

In modern times due to some issues with the new folder cache, also too many folder databases were opened leading to MSF file loss. This issues have now been fixed: Bug 1726319 (mentioned in comment #30), bug 1773822, bug 1776280, bug, 1778888, bug 1788901.

Finally, bug 1795178 is aiming at resolving the issue of too many files being kept open.

Thanks for confirming the 500 folder limit. Would it be possible to raise that limit to 2500?

Have a look at https://bugzilla.mozilla.org/show_bug.cgi?id=1726319#c151 and the next 10-15 or so comments, it appears to be a windows level limit, not set by TB. Bug 1760925 is also a placeholder to look into filehandles at some point.

This fixes the wrong results in search and virtual folders and the disappearing MSF files:
https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/1554188-close-databases-during-search.patch
It needs a small rebase to trunk and depends on
https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/1787609-fix-findMsgIdInFolder.patch

(In reply to b5 from comment #34)

This fixes the wrong results in search and virtual folders and the disappearing MSF files:
https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/1554188-close-databases-during-search.patch
It needs a small rebase to trunk and depends on
https://github.com/Betterbird/thunderbird-patches/blob/main/102/bugs/1787609-fix-findMsgIdInFolder.patch

I have tried Betterbird and I can confirm, in the meantime, virtual folders work, indexing works. All is well. I will check TB every now and then see if it is fixed there.

Offtopic: I would also like to add that I have had massive problems with filters not working that would filter incoming mails into local folders. Using BB has also fixed this issue.

The patches referenced in comment #34 fix a search of more than 500 folders and related virtual folders. However, the solution is not complete at time of writing since a result set spanning more than 500 folders still causes problems when being selected, see comment here:
https://github.com/Betterbird/thunderbird-patches/blob/6459e4b125f89904c9088980231d63575a9a7cb2/102/bugs/1554188-close-databases-during-search.patch#L74

I have BB24preview4 up and running and I can confirm that all indexing problems that I can see are completely gone. Furthermore I had most of my filters that go in local folders NOT working on startup with TB when loading mails from my mail server, obviously since MSF-files were not existent on startup. When halting the loading of e-mails i.e. go to offline-mode on startup, doing a manual search in all local folders that would re-generate all MSF-files and the go online again, ALL filters will work properly.

(In reply to Magnus Melin [:mkmelin] from comment #38)

Currently the nsMsgDBFolder::GetMessageHeader change breaks comm/mailnews/compose/test/unit/test_bcc.js

We've noticed this too, there were also failures in comm/mailnews/compose/test/unit/test_sendMessageLater3.js. When run locally, both failures weren't reliably reproducible, so it looked more like flaky tests.

We recommend to install the test data supplied in attachment 9303024 [details] of bug 1800202 for testing. This shows that after applying the patch, search and virtual folder work, but this is only the first step. After selecting the 810 search results in the search window, things go downhill yet again with failures around the hunk in nsMsgSearchDBView.cpp which we haven't investigated further. So a more thorough fix to the "too many open MSF files" as intended in bug 1795178 and bug 1760925 is still needed.

(In reply to bugzilla@arge-metallguss.de from comment #39)

I have BB24preview4 up and running ...

This was shipped in the latest release, see release notes.

(In reply to b5 from comment #41)

(In reply to bugzilla@arge-metallguss.de from comment #39)

I have BB24preview4 up and running ...

This was shipped in the latest release, see release notes.

I can only state that ALL problems claimed in my bug description above are GONE. I am currently running BB24preview4 on my real world e-mails 22.5GB 3.330 files, 253 folders. For now my virtual folders issue and also my filters issue are SOLVED. Regards.

Just for reference:

https://bugzilla.mozilla.org/show_bug.cgi?id=754799

05/14/2012 ;-)

The comm/mailnews/compose/test/unit/test_bcc.js failure is 100% reproducible.
I traced it a bit, but not sure what to make of it.
https://searchfox.org/comm-central/rev/dce5f4ef2103e9cb7c7ea56c0e1689d14b2651a3/mailnews/compose/src/nsMsgSendLater.cpp#562 ends up getting empty account key, which will make things blow up. nsMsgHdr::GetStringProperty returns empty account key but success with the patch. Without closing the db it will find that account key. Maybe Mork doesn't like closing/opening this often.

Most of the added closing of DBs happens on code paths that are related to search (incl. virtual folders). So the only hunk that could affect "normal" operation would be the one in nsMsgDBFolder::GetMessageHeader(). Can you try adding nsMsgFolderFlags::Queue for the Outgoing folder here:
https://searchfox.org/comm-central/rev/5d6fd3036fd4dcf263792415e0025ea94d9a8322/mailnews/base/src/nsMsgDBFolder.cpp#4910

That fixes the issue for us.

(In reply to b5 from comment #44)

Most of the added closing of DBs happens on code paths that are related to search (incl. virtual folders). So the only hunk that could affect "normal" operation would be the one in nsMsgDBFolder::GetMessageHeader(). Can you try adding nsMsgFolderFlags::Queue for the Outgoing

That "fixes" it, but feels a bit like cheating. That folder happens to be exercised in the test, and probably we don't have a test that exercises the functionality in similar ways for other folders.

This is cheating. But you don't know whether the test actually represents a real-life use case and whether the test doesn't make invalid assumptions (see bug 1787609 comment #6, last sentences for a similar argument).

Ultimately, you need to define what the contract is when retrieving an nsIMsgDBHdr. Here's bad news: If you have a search with results from more than ~500 folders, you will need to display all the result messages (via their headers) in the result view. At the same time, you can't assume that all the databases from which the search results come are open. They will not be. So any code that assumes databases to be open just because a header was retrieved from them is wrong. Looks like some code is triggered by the test that makes a wrong assumption.

Based on user feedback we adjusted the solution. The full solution consists of two patches (here for trunk):
https://github.com/Betterbird/thunderbird-patches/blob/main/115/1554188-close-databases-during-search.patch
https://github.com/Betterbird/thunderbird-patches/blob/main/115/1554188-close-databases-during-search2.patch
Note that both nsMsgSearchDBView.cpp and nsMsgXFViewThread.cpp implemented code already in nsMsgDBFolder::GetMessageHeader(). Using the latter opens the door to more consistent database closure management.

We're still planning to improve this area further, for example getting all the selected headers just to count them is not optimal:
https://searchfox.org/comm-central/rev/2bf0ec9488bea95a776bef70a4a2be180855a785/mail/base/content/folderDisplay.js#2225,2271

(In reply to bugzilla@arge-metallguss.de from comment #42)

(In reply to b5 from comment #41)

(In reply to bugzilla@arge-metallguss.de from comment #39)

I have BB24preview4 up and running ...

This was shipped in the latest release, see release notes.

I can only state that ALL problems claimed in my bug description above are GONE. I am currently running BB24preview4 on my real world e-mails 22.5GB 3.330 files, 253 folders. For now my virtual folders issue and also my filters issue are SOLVED. Regards.

Just for reference:

https://bugzilla.mozilla.org/show_bug.cgi?id=754799

05/14/2012 ;-)

Everything still working as described above for 3 straight months now. As soon as I open TB most index files are messed up or killed. If I switch back to the above solution, all index files are completely restored and fully functional.

Status: UNCONFIRMED → RESOLVED
Closed: 1 year ago
Resolution: --- → WORKSFORME
Status: RESOLVED → UNCONFIRMED
Resolution: WORKSFORME → ---
See Also: → 1305207
See Also: → 1888791
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: