Open Bug 564168 Opened 15 years ago Updated 2 months ago

Option to run quick filter on the server when using an IMAP server

Categories

(Thunderbird :: Search, enhancement)

Product:
Thunderbird
Component:
Search
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: kbsingh, Unassigned)

References

(Depends on 1 open bug)

Details

User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 Build Identifier: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.5pre) Gecko/20100505 Lanikai/3.1pre The new and improved quick search box on the top right of the message's pane is quite nice. But its also extremely slow to run on the client side, since in many cases the body of all messages is not available locally. Reproducible: Always Steps to Reproduce: 1.click into the search box in the 'Quick Filter bar' 2.Enter text 3.Results are displayed extremely slowly Actual Results: When the same search would run on the server side, it would be many times faster. For example, running a full search ( right click on a folder and select search, including subfolders ), takes between 12 to 13 seconds to complete on the client, but when run with the 'Run search on server' comples in under 2 seconds ( wallclock time ). The option of running quick-filter searches could be made available either as a preferences option or via a button on the quick filter bar.
Just as a FYI, there were specific attempts made in the code to force QuickSearch to use client search, even if the existing underlying folder was itself based on a server search. So the clear desire of the drivers is that quick search be done locally rather than on the server, if I understand it correctly. Perhaps what you would like though is that QuickSearch automatically revert to server search if the search includes a body term, and the underlying folders are not available locally? I don't think that loading QuickSearch with lots of choices is likely to fly, since you have advanced search for that. But you might comment on the new Quick Search in 3.1, and how you would like to integrate with that.
(In reply to comment #1) > > Perhaps what you would like though is that QuickSearch automatically revert to > server search if the search includes a body term, and the underlying folders > are not available locally? That's related to bug 543416 on the full-search behavior.
Component: General → Search
QA Contact: general → search
Hi, (In reply to comment #1) > Perhaps what you would like though is that QuickSearch automatically revert to > server search if the search includes a body term, and the underlying folders > are not available locally? That would solve the problem for me quite nicely.
I was about to open a bug for this when I found this one... I would like to make my own argument in favor of at least providing a config option (even if it must manually be applied via user.js)... There are at least 2 (3, if gmail does this, which I would be surprised if it didn't) IMAP servers that provide awesome server-side indexing of all message content (including body) - Cyrus, and Dovecot. For these servers, it is simply wrong to force me to have to download many dozens of Gigabytes of email just to be able to perform fast/quick filtering via the toolbar filter, when the server already has all of my messages fully indexed for me and can do this automatically with *no* local cache of emails. And yes, I know advanced search has the Edit > Find > Search Messages window that allows you to 'Run search on server', but I honestly can't remember the last time I used that window... and I have asked the 60+ people here in our office, and only 2 of them ever use it, and even then rarely. Hopefully you guys do understand that the Quickfilter is *much* more convenient, so why cripple it intentionally? Please, please consider a config option that can be applied on a per account basis, to *always* perform search on server', regardless of which type of search is performed (via Quickfilter toolbar or the Advanced Search window).
(In reply to Charles from comment #4) > I would like to make my own argument in favor of at least providing a config > option (even if it must manually be applied via user.js). that would work too. > There are at least 2 (3, if gmail does this, which I would be surprised if > it didn't) IMAP servers that provide awesome server-side indexing of all > message content (including body) - Cyrus, and Dovecot. gmail does support imap-sever-side-search, and it does it blindingly fast. Unless the user only has a few MB worth of email, or is on a dial up link to the internet, I seriously doubt that local search is going to be faster. > For these servers, it is simply wrong to force me to have to download many > dozens of Gigabytes of email just to be able to perform fast/quick filtering > via the toolbar filter, when the server already has all of my messages fully > indexed for me and can do this automatically with *no* local cache of emails. Even with a full local cache ( I have just over 20GB in my main mailbox ) - its a lot faster to search server side ( I use cyrus ) > And yes, I know advanced search has the Edit > Find > Search Messages window > that allows you to 'Run search on server', but I honestly can't remember the > last time I used that window... and I have asked the 60+ people here in our The other problem is that the option isnt sticky, you need to check it every time you do a search, which is again unhelpful. A good first step would be to atleast make that stick - KB ( I realise this is more commentary than helpful positive push for the bugreport )
There is an interesting thread on the question of server side searches here: http://groups.google.com/group/tb-enterprise/browse_thread/thread/4ee517c902c1075a/8380a26cef9619bf
In summary, the bottom line is, according to Timo (dovecot author), *all* IMAP servers are *required* to support SEARCH extension, so no reason to query that, but there is a new (a year or so old) RFC for SEARCH=FUZZY for a server to announce that it supports server side *indexes*, but apparently because it is s new, dovecot is the only one advertising it currently (but gmail and cyrus (when it is enabled) both support server side indexed searches)...
For the quick filter it might not be a good idea to run online - see bug 541400.
Depends on: 541400
Hmmm... rethinking making this bug depend on bug 541400... If the server has full text indexes, would bug 541400 really be a problem? Wouldn't the searches still be blindingly fast? (In reply to Karanbir Singh from comment #5) > gmail does support imap-sever-side-search, and it does it blindingly fast. Karanbir - can you confirm whether or not this is true for you when using Thunderbird? I just did a few preliminary tests, and it was fast, but I wouldn't call it 'blindingly fast', and there was another problem - the Search seemed to go on forever, and clicking the 'Stop' button didn't stop the search... > Even with a full local cache ( I have just over 20GB in my main mailbox ) - > its a lot faster to search server side ( I use cyrus ) And again to confirm - this is with Thunderbird? So, Magnus, do you still think we should make this bug depend on bug 541400?
(In reply to Charles from comment #9) > Karanbir - can you confirm whether or not this is true for you when using > Thunderbird? I just did a few preliminary tests, and it was fast, but I > wouldn't call it 'blindingly fast', and there was another problem - the > Search seemed to go on forever, and clicking the 'Stop' button didn't stop > the search... Using thunderbird/x86_64 Linux 10.0.1. ( User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.1) Gecko/20120225 Thunderbird/10.0.1 ) My gmail accounts local ImapMail folder directory is just over 3.8GB in size. Local search for 'tkmxpz5hbgPoc' takes 39 seconds on the laptop. From google - it returns in less than a 5 seconds ( I struggled to find a way to do an accurate benchmark, so these are wall-clock eyeball times ). The term tkmxpz5hbgPoc does not exist anywhere in the mails, so this should be doing something like a full-table-scan. My laptop is a quad i5/2.5Ghz and the disk is a crucial M4 ssd 128gb. I do not have gloda enabled. I also dont see the mail search running indefinitely. > > Even with a full local cache ( I have just over 20GB in my main mailbox ) - > > its a lot faster to search server side ( I use cyrus ) > And again to confirm - this is with Thunderbird? yes, same software as above. The server end is a dual core AMD 2218, 4gb of ram ( so ~ 6 years old machine ) with 2 500gb sata's in mdraid1 on linux/2.6.18/x86_64 running email exclusively for me and a couple of friends - the total /var/lib/imap/ dir for my account is just over 22 GB, the local ImapMail/ dir for this account under thunderbird is a hint over 20GB, I do not have gloda enabled. Searching for the same term ' tkmxpz5hbgPoc' takes 2 min and 32 seconds on the laptop, the server returns in 11 seconds. - KB
Charles: if the server is fast enough it probably isn't a problem. But not all servers are.
(In reply to Magnus Melin from comment #11) > Charles: if the server is fast enough it probably isn't a problem. But not > all servers are. thats ok, we dont need to solve every problem for everyone in thunderbird's codebase - but having the options to delegate the search to server, if the user desires, only increases the number of people who can get a better client side mail experience.
(In reply to Karanbir Singh from comment #10) > My gmail accounts local ImapMail folder directory is just over 3.8GB in > size. Local search for 'tkmxpz5hbgPoc' takes 39 seconds on the laptop. From > google - it returns in less than a 5 seconds ( I struggled to find a way to > do an accurate benchmark, so these are wall-clock eyeball times ). <snip> Well, this isn't really a fair comparison unless/until you enable GLODA, otherwise the local search is not an indexed search. (In reply to Karanbir Singh from comment #12) > (In reply to Magnus Melin from comment #11) >> Charles: if the server is fast enough it probably isn't a problem. But not >> all servers are. > thats ok, we dont need to solve every problem for everyone in thunderbird's > codebase - but having the options to delegate the search to server, if the > user desires, only increases the number of people who can get a better > client side mail experience. Agreed...
(In reply to Charles from comment #13) > Well, this isn't really a fair comparison unless/until you enable GLODA, > otherwise the local search is not an indexed search. gloda isnt an option for me, my mailbox size is over 20GB, and my laptop only has 128 GB SSD, I cant afford to lose another 20 GB for the mail index.
(In reply to Charles from comment #13) > Well, this isn't really a fair comparison unless/until you enable GLODA, > otherwise the local search is not an indexed search. GLODA search is used only by search box named "Global Search" in custimize panel, search box with "Search...<Ctrl+K>" where "Search the Web for:" is show in list down menue when search term is typed. As seen in term "Search the Web for:", "Global Search" of Tb is also a FUZZY search used by Google. GLODA search is not used by Quick Search(search box in Quick Filter Bar, Ctrl+Shift+K), Edit/Find/Search Messages(Search Messages of folder context menu), nor Saved Search Folder. And indexing is not used in these ordinal/non-GLODA search even when "Global Indexer and Search" is enabled, because of Non-GLODA seach. This bug is for non-GLODA search family. Please don't confuse.
In Edit/Find/Search Messages panel, "Body" is currently hidden, (a) if folder is IMAP Offline-use=Off folder(mail data is not held in local offline-store file by auto-sync) and (b) if "Run search on server" is unchecked. It's phenomenon of Bug 543416 pointed by rsx11m in Comment #2. This "hiding of Body" was intentionaly done in order to avoid user's confusion due to that "local Body Search with not-auto-sync'ed IMAP folder" can find nothing. Both Quick Search(search box in Quick Filter Bar, Ctrl+Shift+K) and Advanced Search of Edit/Find/Search Messages(Search Messages of folder context menu) are "per folder search". If "Quick Search" always executes "local search", "hiding of Body if IMAP Offline-use=off folder like Edit/Find/Search" can be a way to avoid user's confusion in Quick Search. Rename from "Quick Search" to "Quick Local Search" may be also a solution. "Per folder or per account Quick Search option for Local Search or Online search" proposed by Charles in Comment #4 can be a solution, but at least following issue in Online Search should be resolved before it's implemented, Bug 721167 Body search of Gmail IMAP includes headers (see dup'ed Bug 735665 for crispy explanation of phenomenon) in addition to Bug 541400 which is pointed by Magnus Melin in Comment #8.
(In reply to WADA from comment #15) > This bug is for non-GLODA search family. Please don't confuse. My point remains valid though... comparing a server side indexed search to a local UNindexed search is *not* a fair comparison.
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Linux → All
Hardware: x86_64 → All
Severity: normal → S3

Any movement on this?

In 2023 the server side is capable of vastly better indexing and search capability than the client side could ever hope for. Having server side as an option (even better, the default), would make people very happy.

Server-side search is definitely preferable to local search.
Adding a checkbox for optional server-side search in the Quick Filter Toolbar would be a valuable improvement.

There will be a project in the next year or two to overall search in general. Until that happens, I can pretty safely say the server search asked for here isn't going to come to quick filter in its present form - because:

  • it's not built for it
  • even if we made changes now to accommodate this, it would be obsoleted by the new work to come. We tend to avoid doing double work so that we user our manpower most efficiently for customer fixes and features.
  • Computers are much faster at local search than 5-10-15 years, so body search can be reasonably fast.
  • The main use case for server search IMO would be for users who have changed Thunderbird's default to not download message bodies. And for that you can use "Search messages" (ctrl+shift+F).

I'd search add-ons to see if there is something that can help you.

Summary: RFE: Option to run all searches on the server when using an IMAP server → Option to run quick filter on the server when using an IMAP server

(In reply to Wayne Mery (:wsmwk) from comment #20)

There will be a project in the next year or two to overall search in general. Until that happens, I can pretty safely say the server search asked for here isn't going to come to quick filter in its present form - because:

  • it's not built for it
  • even if we made changes now to accommodate this, it would be obsoleted by the new work to come. We tend to avoid doing double work so that we user our manpower most efficiently for customer fixes and features.
  • Computers are much faster at local search than 5-10-15 years, so body search can be reasonably fast.
  • The main use case for server search IMO would be for users who have changed Thunderbird's default to not download message bodies. And for that you can use "Search messages" (ctrl+shift+F).

I'd search add-ons to see if there is something that can help you.

Tech has moved on.

I have 100G+ of email, all attachments indexed, all images OCR’d, but this is useless because Thunderbird thinks it can do a better job at search and blocks the functionality.

No one cares how fast the local machine is. No one has time to waste 100G of local space to do a job already done remotely, far better than the local client can.

You need to log in before you can comment on or make changes to this bug.