Open Bug 1849650 Opened 6 months ago Updated 10 days ago

Quick Filters is slow

Categories

(Thunderbird :: Search, defect, P2)

Thunderbird 115

Tracking

(thunderbird_esr115? fixed)

ASSIGNED
123 Branch
Tracking Status
thunderbird_esr115 ? fixed

People

(Reporter: doug.pinney, Assigned: welpy-cw)

References

(Blocks 1 open bug, Regression)

Details

(4 keywords, Whiteboard: [has performance profile])

Attachments

(2 files)

Steps to reproduce:

Tried to effectively and efficiently use the newly re-designed Quick Filter tool.

Actual results:

New version 115 completely borked the Quick Filters.

  • Seems to be much slower

  • Does not show results accumulate in window (which usually finds your target much quicker), but instead waits for full search to finish before showing any results.

  • New Toggle buttons and "blue lights" are unintelligible. What does light gray versus dark gray button mean? What does a "blue light" mean, since you can have something seem to show results when not blue.. OR is it a problem with

  • Toggle buttons lag - Click a button, it darkens, then sits... sometime later a blue light may turn on, but you don't get any indication search is running.. and if you click another filter button while it's running.. nothing happens right away. Should I click again? But then will that queue up clicks? And considering the fact that EACH BUTTON click can take a long time to produce any results.. you may be queuing up multiple clicks and multiple runs, w/o knowing if you ended up with filter ON or OFF?

  • Can't cancel midway through a search/filter. Very often you begin typing something, and off it goes to search, because one or more buttons were turned on.. and then you realize, NO, you didn't mean to search BODY.. that will take forever on 10K, 20K, 100K emails... BUT you can't cancel and adjust.. you must wait for it to FINISH first.

  • No spinner, or progress bar anymore. I thought the activity bar at the bottom was showing progress, but doesn't seem to anymore. ?? AND even if it DID work.. Please let us cancel a search, for heaven's sake!

Expected results:

  • Should begin showing results as soon as it finds any, because often your intended search target is found right away.. don't need to keep searching.

  • Need to be able to Cancel a Search Immediately (and have it cancel immediately, not lag along until it finally gives you control again).

  • If click a new filter button, during an ongoing search.. Needs to either cancel that search and start over with the new button.. OR at least register your click visibly (maybe in a different "pending" color?) Something. Some way for user to tell his/her new toggle choice was seen, accepted and either being worked on or pending.

See also bug 1831500 which covers part of the issues listed here.

Thank you for the bug report. However, please "Open a new bug report for each issue!" as specified in the reporting guidelines.

https://mzl.la/3QXHqKc lists the open quick filter bug reports for version 115 - some of your issues may be listed there.

(In reply to doug.pinney from comment #0)

Steps to reproduce:

Tried to effectively and efficiently use the newly re-designed Quick Filter tool.

Actual results:

New version 115 completely borked the Quick Filters.

  • Seems to be much slower

  • Does not show results accumulate in window (which usually finds your target much quicker), but instead waits for full search to finish before showing any results.

I think we don't currently have open performance bug reports for quick filter. Do you have body filter enabled?

Are you using a unified folder?
How large is the folder?

  • New Toggle buttons and "blue lights" are unintelligible. What does light gray versus dark gray button mean? What does a "blue light" mean, since you can have something seem to show results when not blue.. OR is it a problem with

Is it unclear that it's a toggle? Bug 1823978 - Quick Filter bar buttons have indicator that looks like empty textbox

  • Toggle buttons lag - Click a button, it darkens, then sits... sometime later a blue light may turn on, but you don't get any indication search is running.. and if you click another filter button while it's running.. nothing happens right away. Should I click again? But then will that queue up clicks? And considering the fact that EACH BUTTON click can take a long time to produce any results.. you may be queuing up multiple clicks and multiple runs, w/o knowing if you ended up with filter ON or OFF?

Sounds like part of your first issue.

  • Can't cancel midway through a search/filter. Very often you begin typing something, and off it goes to search, because one or more buttons were turned on.. and then you realize, NO, you didn't mean to search BODY.. that will take forever on 10K, 20K, 100K emails... BUT you can't cancel and adjust.. you must wait for it to FINISH first.

Perhaps bug 1831500 as previously suggested.

  • No spinner, or progress bar anymore. I thought the activity bar at the bottom was showing progress, but doesn't seem to anymore. ?? AND even if it DID work.. Please let us cancel a search, for heaven's sake!

related to above

Flags: needinfo?(doug.pinney)
Whiteboard: [closeme 2023-09-10]

These issues exist without body filter enabled, but worsen when it is enabled.

I am usually working within a subfolder that is associated with a Google Mail label. Not a unified folder.

The folder is quite large. I thought I mentioned that, but if not.. yes, many thousands of emails. Years of business emails. But, again, this did not become a problem until the changes in Thunderbird 115.

The toggle buttons might be decipherable, if there weren't so much lag on them, which causes confusion. Sometimes they are dark gray, sometimes light gray, and either one could have the blue light on or off.. Clicking them often doesn't register a change.

Possibly the core issue is that it does NOT start showing results as soon as it finds them.. but instead waits until all emails are found, then displays them. On a large folder, simple filter changes require the full search each time.. If you made a slight mistake, you can't simply adjust it and begin again, you must wait for it to finish. Developers/testers should be load testing with large numbers of emails (10K, 25K, 50K, simulating realistic business scenarios, if talking about years of email).

And yes, some kind of progress bar, or activity indicator is needed..

Flags: needinfo?(doug.pinney)

Also, if I didn't reiterate it: a way to quickly CANCEL a current search, seems to be a way to alleviate a lot of the trouble.
(that, and showing results as they're found, not waiting until all are found to display).

I'm curious.. will clicking the overall Quick Filter button itself cancel a current search job? (I'm doubting it)

The folder is quite large.

How many messages?

Flags: needinfo?(doug.pinney)
Keywords: perf
Summary: Quick Filters BAD on Thunderbird 115 (way slower, unintelligible toggles, can't cancel) → Quick Filters is slow
Whiteboard: [closeme 2023-09-10]

The filtered folder being discussed is about 24K msgs. The overall inbox has about 258K msgs.

However, before this recent change in Thunderbird v115... it seemed to handle it fine, due to the way it functioned.

Flags: needinfo?(doug.pinney)

Please post your results here next week with version 115.3.0

Will do.

Hoping 115.3.0 will address these issues. That'd be great!

(also, probably already included above but.. issue of when you happen to type just a few letters of your search keyword, but pause for a split second, for whatever reason, but cannot continue to type the rest of the keyword because the automatic search has kicked in.. so you have to wait till finished before typing remaining letters... and if you happen to type something wrong in the meantime, while you cannot see what you're typing.. you may have to correct that, wait again, and then finally finish typing your keyword. All of this is likely only intolerable when you have a large number of emails, and have to wait for entire search each time.)

(In reply to Wayne Mery (:wsmwk) from comment #7)

Please post your results here next week with version 115.3.0

Today I tried version 115.3.0 but the problem is still present.
I also do a search on the body and the list of emails found doesn't update immediately as it did in the version prior to 115.
I can force the display of found emails only by selecting "table view" from the menu

Yes, during my initial usage of 115.3.0, I also haven't seen much difference in functionality.

To Danilo's point, I do not seem to be able to "force the display of found emails" by selecting "table view" from the "message list display options" menu (next to Quick Filters button). In fact, the button is inoperable during a search.

It would help if TB developers could point to a specific feature adjusted, so we could check for any changes.

...
⭐ Also.. to reiterate, perhaps the two biggest tweaks needed here are:

  1. The ability to CANCEL a search at any point during the search, rather than having to wait until it's finished.
  2. Showing results as their found, accumulating over time, rather than having to wait for entire search to finish before displaying them.
Duplicate of this bug: 1855130

Please create a performance profile and post the URL here
https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Who is that request to create a performance profile directed at?

You want the end user to engage in work that should be done by development & testing team?

Are you (or team) not able to create a similar scenario, with the same volume of emails, and perform these tests and create same performance profiles yourselves? Only if you do so yourselves but cannot replicate what is being reported here (and elsewhere) should you resort to asking end user to make such efforts.

My 2 pennies.

(In reply to DeeP from comment #13)

Who is that request to create a performance profile directed at?

Anyone who is willing and able.

You want the end user to engage in work that should be done by development & testing team?

It's not reproducible here.

Are you (or team) not able to create a similar scenario, with the same volume of emails, and perform these tests and create same performance profiles yourselves? Only if you do so yourselves but cannot replicate what is being reported here (and elsewhere) should you resort to asking end user to make such efforts.

If I were able to reproduce then I would have done it myself. But even if I could reproduce, your issue might be different from mine.

Users who can reproduce are very often in the best position to collect accurate data which will lead to fixing their specific problem.

I see. Does "not reproducible here" refer to:

  1. the general scenario of having that many emails in the first place... or
  2. yes, you can reproduce that scenario, but still aren't seeing the reported behavior/perfomance problems?

And thanks so much for your time/effort, by the way. Much appreciated.

for a folder size of 164k messages I get results in 2.5 seconds, which seems reasonable to me.

Please create a performance profile and post the URL here
https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

I confirm this issue.

For a folder of 100k messages, it takes around 4 seconds to search. Version 115.3.1

Yesterday at home, it took around 10 seconds on the same folder. The version might be plain 115 on that computer, not entirely sure.

Wait no, it behaves actually quite weirdly.

When I have some quick filter already applied and then filter something different.

  • after 4 seconds after I stop typing, the results change to all e-mails listed (which is bad and confusing)
  • after 8 seconds it starts to show correct results finally

So two things bad imho: 1. It's very slow, and 2. The intermittent result is confusing.

(In reply to Wayne Mery (:wsmwk) from comment #16)

for a folder size of 164k messages I get results in 2.5 seconds, which seems reasonable to me.

Please create a performance profile and post the URL here
https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance

Wayne,

This should be a very basic performance profile, capturing one single search. Not complicating things by trying to cancel, or type new characters in search box while it's searching, or clicking different filter buttons mid-stage.

Let me know if it suffices.

https://share.firefox.dev/45uiixY

Thanks DeeP

nsMsgQuickSearchDBView::Open
nsMsgThreadedDBView::InitThreadedView 48%
nsMsgThreadedDBView::InitSort 24%
nsMsgDBView::ExpandByIndex 24%

Duplicate of this bug: 1857818

https://share.firefox.dev/45uiixY

Julienw and standard8 reviewed in ##joy-of-profiling:mozilla.org. standard8 summarizes:

don't think I was ever very educated about how threads work. There's a couple minor things (these won't help perf):

These params appear to be unused, could probably be dropped.
This line appears bogus. The only way mDone can be true there is if it is set to true on line 616, but there's a break on 617, so it'll never hit 620.

I think the main thing for me, would be to understand the nsMsgThreadEnumerator::nsMsgThreadEnumerator and MsgKeyFirstChildIndex algorithms and see if there's any optimisations there. The double loops... seem problematic.

The detailed conversation preceding:

julienw
mostly quick search DB work
it looks like we're enumerating messages twice
this calls ExpandByIndex first, then it sorts and call it again
but even if we wouldn't do that it would still be slow
is this a case of corrupted mork DB again?
actually I feel like I see the issue when I change the filters
I think this was faster earlier

Standard8
I don't think this is corrupted db. There's multiple levels of loops over threads here. I'm not even sure if they're looping over the same thread or not.

nsMsgThreadEnumerator is created here: https://searchfox.org/comm-central/rev/cce1113d98c79293806d9af893376050584f7192/mailnews/db/msgdb/src/nsMsgThread.cpp#766,769

second & third arguments are null pointers.
i.e. startKey is a null pointer.... so why iterate over the list looking for a null pointer? https://searchfox.org/comm-central/rev/cce1113d98c79293806d9af893376050584f7192/mailnews/db/msgdb/src/nsMsgThread.cpp#585-588,614

Why calling into MsgKeyFirstChildIndex with a null pointer?
Oh, maybe it is trying to find the last child in the thread, and have that as mResultHdr???
Oh, sorry. startKey is passed a value - parentKey
It is the unused filter and closure arguments that are null.
(they are never used).

darktrojan issued fixes in :

  • TM116 Bug 1820423 - Quick filter message list update voids subsequent keystrokes
  • TM114 Bug 1827003 - Slow message list quick filter

Similar to Bug 1820423 I can still reproduce the keystroke issue (no bug report for it), but I do not know whether it is related to this bug report:

  • a pop account with many messages to download (several hundred), account has a single filter on subject (note, my filters also can select on custom headers)
  • Inbox selected, while messages are downloading
  • type in quick search field with default QFB settings

Keystrokes can be delayed or disappear

Flags: needinfo?(geoff)
Severity: -- → S3
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: regression
Priority: -- → P2

If the current code isn't much changed in 115 then we could be seeing effects from Bug 1284753 - Quick filter UI is unresponsive after a quick search triggers via Bug 1055077 - JSMime header parsing makes quick filter slow.

I was able to reproduce on a local folder, not a virtual folder, on Windows VM with 14,000 messages. Sizes vary from 11KB to 1.2MB. Mean size around 21KB. 370 of the messages have html attachments.

I moved the 370 messages with html attachments to a seperate folder to test a theory. Unfortunately this triggered an automatic compact, and now I can longer reproduce signficant slowness - perhaps back to pre-115 2-3 seconds which would be hlped by patches from bug 1005413.

See Also: → 1055077, 1005413
Duplicate of this bug: 1859037
Blocks: 1863293
Duplicate of this bug: 1863293
Flags: needinfo?(geoff) → needinfo?(alessandro)
Keywords: triaged
See Also: → 1868152
Whiteboard: [has performance profile]

Whenever another character is typed into the search field of the quick filter bar, a new search
over the folder is started, which may result in many refreshes of the thread tree and can take a
long time, until the final search has completed.

This patch extends the delay of each individual search to 100 ms and cancels the previous one if
it has not started yet.

Assignee: nobody → h.w.forms
Status: NEW → ASSIGNED
Attachment #9372556 - Attachment description: Bug 1849650 - Improve quick search responsiveness. r=#thunderbird-reviewers → Bug 1849650 - Improve quick filter responsiveness. r=#thunderbird-reviewers
See Also: → 1873053
See Also: → 1874419

I don't know if it's relevant or not but in 115.6.1, when I get emails to my Inbox that I know I have a filter for, it usually take about 5 seconds for them to push to the folder I have them designated to go to. Whereas in previous ESR TB versions, it would be instantaneous. I can repro it fairly easy if you want me to test something.

While doing some testing in a debug build, I found that quick filtering threaded views in collapse-all state (which is the default) is much slower than any other view type. Searching for a term in a folder containing over 8000 messages took about 5 seconds in unthreaded and threaded expand-all modes, about 15 seconds in collapse-all with date descending, but over two minutes (!) with date ascending.

This regression caused by bug 1827042 is not as obvious in an optimized build, but should still be fixed by the following patch.

Regressed by: 1827042

Calling ensureThreadStateForQuickSearchView() before all messages have been loaded interferes
with the batch processing of search hits, which has a severe performance impact on collapsed
threaded single-folder views, especially when sorted by date ascending.

Blocks: 1874201
Duplicate of this bug: 1874201
Blocks: 1855125
Target Milestone: --- → 123 Branch

Pushed by john@thunderbird.net:
https://hg.mozilla.org/comm-central/rev/a541e64bad4d
Fix quick filter performance regression caused by bug 1827042. r=mkmelin

Pushed by john@thunderbird.net:
https://hg.mozilla.org/comm-central/rev/1cf175df13a9
Improve quick filter responsiveness. r=john.bieling

The work is still in progress here but we're gaining some wins.

Flags: needinfo?(alessandro)

It will be interesting to see what all this helps. For example I'm seeing terrible performance with a saved search that is using group by sort.

(In reply to Wayne Mery (:wsmwk) from comment #35)

It will be interesting to see what all this helps. For example I'm seeing terrible performance with a saved search that is using group by sort.

Do you see a difference between collapse-all and expand-all states?

Depends on: 1875577

(In reply to Hartmut Welpmann [:welpy-cw] from comment #36)

(In reply to Wayne Mery (:wsmwk) from comment #35)

It will be interesting to see what all this helps. For example I'm seeing terrible performance with a saved search that is using group by sort.

Do you see a difference between collapse-all and expand-all states?

With currently nightly, no difference

(In reply to Wayne Mery (:wsmwk) from comment #37)

(In reply to Hartmut Welpmann [:welpy-cw] from comment #36)

(In reply to Wayne Mery (:wsmwk) from comment #35)

It will be interesting to see what all this helps. For example I'm seeing terrible performance with a saved search that is using group by sort.

Do you see a difference between collapse-all and expand-all states?

With currently nightly, no difference

Good, in 115 expand-all is even slower, see bug 1875577 comment 4.

Comment on attachment 9372556 [details]
Bug 1849650 - Improve quick filter responsiveness. r=#thunderbird-reviewers

[Approval Request Comment]
Testing completed (on c-c, etc.): c-c and beta
Risk to taking this patch (and alternatives if risky): low

Attachment #9372556 - Flags: approval-comm-esr115?

Comment on attachment 9372741 [details]
Bug 1849650 - Fix quick filter performance regression caused by bug 1827042. r=#thunderbird-reviewers

[Approval Request Comment]
Testing completed (on c-c, etc.): c-c and beta
Risk to taking this patch (and alternatives if risky): low

Attachment #9372741 - Flags: approval-comm-esr115?

Comment on attachment 9372556 [details]
Bug 1849650 - Improve quick filter responsiveness. r=#thunderbird-reviewers

[Triage Comment]
Approved for esr115

Attachment #9372556 - Flags: approval-comm-esr115? → approval-comm-esr115+

Comment on attachment 9372741 [details]
Bug 1849650 - Fix quick filter performance regression caused by bug 1827042. r=#thunderbird-reviewers

[Triage Comment]
Approved for esr115

Attachment #9372741 - Flags: approval-comm-esr115? → approval-comm-esr115+
You need to log in before you can comment on or make changes to this bug.