Closed Bug 544100 Opened 16 years ago Closed 16 years ago

Make GLODA intelligent enough to take advantage of IMAP servers with fast server side search capability

Categories

(Thunderbird :: Search, enhancement)

Product:
Thunderbird
Component:
Search
Platform:
x86
All
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: tanstaafl, Unassigned)

Details

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 Build Identifier: 3.0.1 final I know this might sound strange, but hear me out... The Dovecot IMAP server has its own server side indexing scheme, and with recent versions, searching many many GBs/tens of thousands of messages (even across many dozens of folders) is almost instantaneous, so there is simply no need for all of the overhead of both downloading all of these GBs of messages and then indexing them locally. So, what I thought would be really cool is if TB could auto-detect when an IMAP account is being hosted by a Dovecot server (or any server that supports fast server side search capability), and disable offline/indexing for that account only and simply let the server do the heavy lifting for that account. Reproducible: Always
The traditional search function does support online searches. I think that the general feeling though is that online search is rarely the best choice, so it is sometimes ignored even when possible. For example, if you have a virtual search enabled as online, then try to add another view to it, that results in two searches in cascade, and both will be done online even if it could be done on the server.
(In reply to comment #0) > if TB could auto-detect when an IMAP account is being hosted by a Dovecot server Even if server returns text string as greeting and it contains "Dovecot", it never garantees that the sever is dovecot server, because any string can be set at server side. > (or any server that supports fast server side search capability), There is no standard way to know server side functinality except capability response and a part in response to some commands. Is there any standard for the "fast server side search capability"? > and disable offline/indexing for that account only "enable/disable (Gloda) indexing for that accout only" is already requested but is not implemented yet. So, it's impossible now. (For current online search of IMAP folder) Quick search: If "offline use=off", Online search. If "offline use=on", Local search. Edit/Find/Search Messages: There is option of Online search/Local search already. Saved Search Folder(Virtual Folder): If search target folders are IMAP folder only, option of Online search/Local search is already available. If local mail folder is mixed, the option is disabled, then Local search is forced. Smart Folder: Because "Smart Folder" is Virtual Folder internally, and because Inbox, Sent, ... of "Local Folders" is alwayes mixed in search target folders, Local search is always forced for all standard folders currently. How fast is the "fast server side search" of Dovecot when Online search by Tb? (search body ... comand etc. of IMAP is used by Tb) Can you provide data to us? Get IMAP log with timestamp and check response time etc. for "search ..." using Virtaul Folder or Edit/Find/Search Messages. SET NSPR_LOOG_MODULES=timestamp,imap:5
This would have an extremely high complexity factor and so is not suitable for core. The good news is that there is nothing stopping someone from trying their hand at implementing this as an extension.
Status: UNCONFIRMED → RESOLVED
Closed: 16 years ago
Component: Preferences → Search
QA Contact: preferences → search
Resolution: --- → WONTFIX
Hi Andrew, Are you mainly talking about the 'auto-detect' aspect? If so, I'd agree, and probably should have left that part out... What about a checkbox option for an account labeled something like: '[ ] Dovecot FTS Server Side Search Support' This would require the user to know that this particular server was a dovecto server with FTS search plugin enabled. This would obviously 'depend' on the enable/disable GLODA bug feature implemented, as enabling this option would have to also disable GLODA for this account at the same time. As for how much faster it is... my test server is down right now, and I never tested it for precise numbers with IMAP logging enabled, but I can tell you that on a mailbox with 10,000+ messages, a difference betwee a few SECONDS and a few MINUTES. The difference is staggering, and also dramatically reduces the load on the server, and so I feel getting this option enabled would really be an impressive feature, especially for corporate environments using TB... If this sounds more doable/reasonable, I can open a new Feature Request? Thanks...
(In reply to comment #4) > As for how much faster it is... my test server is down right now, and I never > tested it for precise numbers with IMAP logging enabled, but I can tell you > that on a mailbox with 10,000+ messages, a difference betwee a few SECONDS and > a few MINUTES. The difference is staggering, and also dramatically reduces the > load on the server, and so I feel getting this option enabled would really be > an impressive feature, especially for corporate environments using TB... If gloda is taking a few minutes to return search results, that is a gloda bug and not a fundamental limitation. It may also be a known gloda bug; right now we do not do so well on searches with a very high number of results. Searching for the word "the" for example turns out pretty poorly. > If this sounds more doable/reasonable, I can open a new Feature Request? Detecting a dovecot server with FTS support is not the hard part. I think you are assuming some things about the way gloda operates and how easy it would be for gloda to farm out its search to the IMAP server and still look like gloda to the rest of the system. It would be a lot of work and complexity and we simply do not have the manpower nor the priority to special-case for dovecot anytime in the foreseeable future. We're trying to be more realistic about feature requests in bugzilla which is why I wontfixed this. I would suggest if you would like to rally others who might share this need and interest and hopefully find someone with the time and skills to try their hand at such an extension that you create a getsatisfaction idea (in the Mozilla Messaging/Thunderbird section) on the subject; that is our preferred avenue for feature requests (blue sky or not) going forward.
(In reply to comment #5) > (In reply to comment #4) >> As for how much faster it is... my test server is down right now, and I never >> tested it for precise numbers with IMAP logging enabled, but I can tell you >> that on a mailbox with 10,000+ messages, a difference betwee a few SECONDS >> and a few MINUTES. The difference is staggering, and also dramatically >> reduces the load on the server, and so I feel getting this option enabled >> would really be an impressive feature, especially for corporate environments >> using TB... > If gloda is taking a few minutes to return search results, that is a gloda bug > and not a fundamental limitation. You don't understand. I am not using GLODA. GLODA requires local copies of the emails. We are talking about 16+ IMAP accounts eacg with multiple GB - some with10+GB - of email. Keeping local copies of all of these messages for use with GLODA is simply not an option. > It may also be a known gloda bug; right now we do not do so well on searches > with a very high number of results. Searching for the word "the" for example > turns out pretty poorly. Well, yeah... ;) in fact, there probably should be a list of simple words like that that GLODA would totally ignore - but that is beside the point... >> If this sounds more doable/reasonable, I can open a new Feature Request? > Detecting a dovecot server with FTS support is not the hard part. I think you > are assuming some things about the way gloda operates and how easy it would be > for gloda to farm out its search to the IMAP server and still look like gloda > to the rest of the system. Maybe the bug summary was poorly chosen. I was thinking more in terms of GLODA being smart enough to just *disable itself* when a server that had server-side indexes available. Dovecots FTS means 'Full Text Search', meaning, the bodies of all messages are fully indexed on the server. All I'm talking about is an intuitive way to take advantage of this - or, at a minimum, an option like I described above that would have to be intentionally enabled. > It would be a lot of work and complexity and we simply do not have the > manpower nor the priority to special-case for dovecot anytime in the > foreseeable future. ? I don't see this, maybe because you were misunderstanding the feature request? > We're trying to be more realistic about feature requests in bugzilla which is > why I wontfixed this. I would suggest if you would like to rally others who > might share this need and interest and hopefully find someone with the time > and skills to try their hand at such an extension that you create a > getsatisfaction idea (in the Mozilla Messaging/Thunderbird section) on the > subject; that is our preferred avenue for feature requests (blue sky or not) > going forward. Maybe *I'm* not understanding something... I honestly don't see how adding an option to simply tell GLODA to get out of the way could be all that complicated.
Charles, I'm not quite sure what is missing in the current program for you. If you don't want to use gloda, then why don't you just turn it off? If you want to do IMAP server-side searches, can't you just use the "Search on server" option of the advanced search dialog?
(In reply to comment #6) > Maybe *I'm* not understanding something... I honestly don't see how adding an > option to simply tell GLODA to get out of the way could be all that > complicated. I think I am indeed confused and rkent's sentiments echo my own. Either Thunderbird already does what you want (turn off gloda), or you want the faceted search UI which is built on top of gloda to not use gloda. My previous comments assume the latter... if you want the faceted search UI, the back-end needs to present itself as gloda.
(In reply to comment #2) > Is there any standard for the "fast server side search capability"? Not that I know of, nor does there need to be? If an account is flagged as supporting fast (maybe better word is 'fully indexed', including body text) server side search, then simply pass the search term to the server and wait for the response(s). > "enable/disable (Gloda) indexing for that accout only" is already requested > but is not implemented yet. So, it's impossible now. Thanks, do you know the bug #? I just searched and cannot find it... > (For current online search of IMAP folder) > Quick search: > If "offline use=off", Online search. > If "offline use=on", Local search. Clarification... what if I am using the undocumented opportunistic or 'on-demand' feature, where I have certain folders checked for offline use, but have the account level option 'Tools > Account Settings > Synch & Storage > Keep messages for this account on this computer' *disabled*. So, this would result in having *some* messages stored locally, and others not - what I want is for searches using the Quicksearch box to still use Online (server-side) searches only. (In reply to comment #7) > If you don't want to use gloda, then why don't you just turn it off? I did resort to that, but I would like to use it for my one and only POP3 account (which also has a ton of messages in it). > If you want to do IMAP server-side searches, can't you just use the "Search > on server" option of the advanced search dialog? Because I almost always only use the Quicksearch box. I only use the Advanced search box when I have an advanced search to run, which is not all that often - although, personally, I'd much prefer to be able to use simple boolean operators (AND(+), OR(/), NOT(-)) in the Quicksearch box (uh-oh - another Feature Request? Does it already exist? Will the Bug reporters buggy quicksearch box allow me to luck into finding it?).
(In reply to comment #8) > (In reply to comment #6) >> Maybe *I'm* not understanding something... I honestly don't see how adding an >> option to simply tell GLODA to get out of the way could be all that >> complicated. > I think I am indeed confused and rkent's sentiments echo my own. Either > Thunderbird already does what you want (turn off gloda), As I said earlier, I did resort to that for many reasons (consumed too much CPU, reindexing all the time, etc)... But, it has promise. The 2 features I really like and would like to be able to take advantage of are: 1. Fast searches of local mail (I still have one old POP account with tons (10+G) of mail that I'd like to be able to search quickly), and 2. It can search across all of your accounts. > or you want the faceted search UI which is built on top of gloda to not > use gloda. Ok, maybe here is where my confusion lies - what do you mean by 'multi-faceted search UI'? As far as a 'search UI' goes, I am familiar with the Quicksearch box, and the Advanced Search window. To the best of my knowledge, disabling GLODA, as far as the UI goes, only removes the 'Search All Messages' - which searches across all accounts/folders - choice from the Quicksearch box. Am I missing some other 'GLODA' search window/UI? > My previous comments assume the latter... if you want the faceted search UI, > the back-end needs to present itself as gloda. What I'd like is: 1. the ability to search across all accounts, and 2. not have to keep full local copies of IMAP messages to accomplish #1. The way I imagined this being accomplished was to have GLODA simply be smart enough to know which accounts didn't need to use it and act accordingly - hence the subject of this bug. Alternatively, if it would be too difficult for it to do this automatically, I'd be fine with a way to just tell it myself. But I just had another idea... Instead of making GLODA enable/disableable on a per account basis, why not make it simply make it auto-enabled for any folder that is set to full offline mode? You wouldn't even need a UI for that at all, and since it only works for accounts+folders that are set to full offline mode, it would just 'work'. Then, for any folders that weren't set to full offline mode, pass the query to the server, and let it deal with it.
You need to log in before you can comment on or make changes to this bug.