1905786 - Quick Filter search and clearing QF can be very slow since view is restored with primary and secondary sort order.

Status: RESOLVED INVALID

(MailNews Core :: Search, defect)

Description

[[e-mail]]

Attachment: init-search.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0

Steps to reproduce:

We're comparing a search on a folder with 190.000 messages and a size of 860 MB of data.

115.12.2:
time needed for search after entering the search term: 5-6 seconds
time needed to clear the search and restore the original unthreaded view: 3 seconds.

128.0b6: time needed for search after entering the search term: 20 seconds. The system becomes unresponsive during this time, see attached screenshot from the profiler.
time needed to clear the search and restore the original unthreaded view: 3 seconds: 4m 30sec.

Note that this folder is the folder where we conducted the compact experiments from bug 1904208, After we restored the folder to a state that was never touched by 128, the search times became like they used to be in 115.

So it appears that compaction causes permanent damage to the file or its index.

Comment 1

[[e-mail]]

Correction: time needed to clear the search and restore the original unthreaded view: 4m 30sec.

Comment 2

[Alessandro Castellani [:aleca]]

I can't reproduce with my Windows build.
I have a local folder with 200k messages, doing a quick search it's definitely not smooth at all, with a few seconds of UI jitter, but the list updates no later than 2 or 3 seconds.
Clearing the search restores the view immediately, both in threaded and unthreaded sorting.

Some caveats

• This folder is new and haven't done a folder compaction yet, I'm testing this right now.
• Indexing is still happening in this folder, with a 46% indexing completion at the time of the test (it's taking around 4 hours to index 200k messages).

Some random facts

• Same scenario on Linux takes a split second.
• Indexing is also insanely fast on Linux, completing in around 30 seconds.

Comment 3

[[e-mail]]

This folder is new and haven't done a folder compaction yet, I'm testing this right now.

It seems to be connected to compaction.

Indexing is still happening in this folder, with a 46% indexing completion at the time of the test (it's taking around 4 hours to index 200k messages).

We were using a local folder. Indexing/repairing takes under a minute. Looks like you're using an IMAP folder where compact will do something different.

Comment 4

[[e-mail]]

Apologies, you're referring to Gloda indexing. We have that switched off to avoid another complication.

Comment 5

[chrizilla]

(In reply to betterbird.project from comment #0)

So it appears that compaction causes permanent damage to the file or its index.

Do you suspect or see permanent damage to user data (keyword dataloss) ?

Comment 6

[Alessandro Castellani [:aleca]]

We're having completely different results from our tests so we're still investigating these findings.
Nonetheless, there's no dataloss at all because compaction or indexing doesn't affect any message, only local caching and local storage, which can be repaired and no message can be lost.

Comment 7

[[e-mail]]

(In reply to Alessandro Castellani [:aleca] from comment #6)

We're having completely different results from our tests so we're still investigating these findings.

This seems to only occur after compaction has locked up, see bug 1904208 comment #10. On a "virgin" folder there is no freeze in compacting, later we could trigger a 20 second freeze, nowhere near the original 5-6 minutes freeze we observed.

Nonetheless, there's no dataloss at all because compaction or indexing doesn't affect any message, only local caching and local storage, which can be repaired and no message can be lost.

For local folders, compaction rewrites the raw message files (removing all the "compacted away") messages, so dataloss is imaginable. However, we didn't see any. It's more likely that the message database (.msf) gets confused. As can be seen in comment #0, after starting a quick search, the system hangs for 20 seconds to sort a view that doesn't need sorting.

Comment 8

[[e-mail]]

The slowness of the QF is not related to compact. It is related to the primary and secondary sort order of the view.

See bug 1904208 comment #11 for details. Sorting is really slow by sender on large folders where all the messages are from one sender, like a folder with a lot of bugmail. This is already visible in 115.

Sorry about the wild goose chase. You may want to file a bug "View creation/sort is very slow on sort of column with identical values".