Open Bug 1205248 Opened 10 years ago Updated 1 year ago

global-messages-db.sqlite corrupted when mail/message is moved from IMAP folder to local folder, search doesn't find messages

Categories

(Thunderbird :: Search, defect, P3)

Product:
Thunderbird
Component:
Search
Version:
38 Branch
Type:
defect

Tracking

(thunderbird_esr68- affected)

Tracking Flags:
Tracking Status
thunderbird_esr68 - affected

People

(Reporter: spam, Unassigned)

References

(
URL
)

Details

(Keywords: dataloss)

User Story

https://support.mozilla.org/en-US/questions/1252397
https://support.mozilla.org/en-US/questions/1253814
https://support.mozilla.org/en-US/questions/1258666
https://support.mozilla.org/en-US/questions/1261826

Attachments

(1 file)

message moved on local folder from IMAP Exchange INBOX
4.37 KB, image/png
Details
User Agent: Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0 Build ID: 20150826023504 Steps to reproduce: - search for a text present in a known message contained on a IMAP folder - click on the related item listed in search results - the right mail is displayed - now move the message to a local folder - repeat the above search - click on the item listed in new search results - TB now displays a blank - deleting global-messages-db.sqlite forcing TB to rebuild it clear the problem, until next message move... Actual results: TB does not refer the right mail anymore. Expected results: The correct message should be shown.
errata corrige: - TB now displays a blank *message*
I am unable to reproduce 1. local account/folders is not disabled from search indexing, correct? 2. does this happen for a message moved from *any* imap folder, i.e. not just inbox? 3. please install glodaquilla addon and use the additional message pane columns columns to ensure that glodaID for the message that has been moved is non-zero,
Flags: needinfo?(spam)
Hello, - global indexing is active, I even don't know how to deactivate selectively for local folders - it happens for messages moved from every folder - it happens under Windows XP 32 bit with and IMAP account hosted by Exchange 2010, trying the same thing on Windows 8.1 64 bit with an IMAP account on a Cyrus server often works fine
Flags: needinfo?(spam)
please install glodaquilla addon. Enable the the glodaID column to see if the message that has been moved has a zero glodaID
Flags: needinfo?(spam)
Hello, here an example, no gloda id.
I believe I might be experiencing this problem also. I am running Thunderbird 52.5.0 (64-bit) on Mac OS X "El Capitan" 10.11.6. I tried to install the GlodaQuilla plugin. Version 0.3.3, Released June 7, 2011, seems to be their latest version, and it "works with Thunderbird 3.0pre - 24.*". Thunderbird disables the "Add to Thunderbird" button for this add-on. However, I can install it from the "Extensions" tab of the Add-on Manager, using the "Install Add-on from file" menu item. I just looked at one message which does not appear in the search results, but does appear when I search for it in its folder, via the Quick Filter bar. That message has a GlodaID of "b571".
(In reply to Wayne Mery (:wsmwk) from comment #4) > please install glodaquilla addon. Enable the the glodaID column to see if > the message that has been moved has a zero glodaID Hello, no glodaID for the message with bug
Flags: needinfo?(spam)

Similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1254982 "Open message in list" is empty for imap messages archived to local folders [faceted search results] (still indexed on Inbox, not on archive folder, fixed by rebuilding gloda database)

I am affected by this bug. It is the most severe flaw in Thunderbird that I am currently experiencing.
In my experience, The problem is corrected by rebuilding the Gloda database. Some, but not all, messages which are in the Local Folders and which Gloda cannot find have an empty cell in the Glodaquilla plugin's Gloda ID column. I am using current Thunderbird (presently 60.7.0) on Mac OS X 10.11.6.

Workaround: select the local folder into which the disappeared message was archived. Enable Quick Filter. Type search text into Quick Filter search box. Desired message is found.

This is a serious search problem, and easily reproduced. Can we assign someone to it to fix?

related to bug 1217958?

Severity: normal → critical
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(mkmelin+mozilla)
Keywords: dataloss
Summary: global-messages-db.sqlite gets corrupted when mails are being moved from IMAP folder to local folder → global-messages-db.sqlite corrupted when mail/message is moved from IMAP folder to local folder, search doesn't find messages
status-thunderbird_esr68: --- → affected
tracking-thunderbird_esr68: --- → ?
User Story: (updated)

Sounds to me global-messages-db.sqlite isn't corrupted, it's just that it doesn't know about the message in question. Like perhaps we just don't run indexing on it.

Starting points for debugging could be in https://searchfox.org/comm-central/rev/cb71ec93ae822bce37fb00e321ef201cce24ff1e/mailnews/db/gloda/modules/index_msg.js#579 or indexer.js - but there's a lot of abstractions in gloda so it's often fairly hard to understand what goes on :/

Given the age of this bug I don't think we need to track it for 68 though we might uplift if we find the problem.
Khushil, putting this in your backlog.

Assignee: nobody → khushil324
tracking-thunderbird_esr68: ? → -
Flags: needinfo?(mkmelin+mozilla)
Priority: -- → P3

Hey All,
Something interesting is going on here. This is not a bug actually. By default, all the local folders have indexingPriority set to -1. Because of this, TB does not index those folders in Global Search: https://searchfox.org/comm-central/rev/cb71ec93ae822bce37fb00e321ef201cce24ff1e/mailnews/db/gloda/modules/index_msg.js#1761
One can change this: Right click on the folder and go to properties, there will be a checkbox "Include messages in this folder in Global Search results". Tick that and you will see now messages coming up in the Global Search. The original author has given it as functionality which is very interesting.
This checkbox will be shown if there are some messages in the local folder.
Now, we can do one thing, set indexingPriority for all the local folders. Magnus and Wayne, what do you suggest?

Flags: needinfo?(vseerror)
Flags: needinfo?(mkmelin+mozilla)

I think the question is then, why is it set to -1, and where?
Unless there are reasons not to, we should just avoid that.

Flags: needinfo?(vseerror)
Flags: needinfo?(mkmelin+mozilla)

By default, all the local folders have indexingPriority set to -1

If it the default on your system then this is a different issue and definetly should be filed as a new bug.

But I don't see this on my system, so it's not the problem described in this bug

For SMTP like account folders:
Priority is set here: https://searchfox.org/comm-central/source/mailnews/db/gloda/modules/datastore.js#1929
For normal folders, indexingPriority will be a null string. In this case, new GlodaFolder() at time of creation will take default value which is 20. For Trash and SPAM like folders, stmt.getInt32(4) will give the value of -1 so indexingPriority will be set up as -1.

For Local Folders:
Priority is set here: https://searchfox.org/comm-central/source/mailnews/db/gloda/modules/datastore.js#2015. "stringPrio" is null for the new local folder which will be set up through JSM module and we are not assigning anything there. So, it will go to else statement: https://searchfox.org/comm-central/source/mailnews/db/gloda/modules/datastore.js#1961. But we are not passing the second argument in getDefaultIndexingPriority so it is setting up -1 for Trash and Junk and there is a comment not to add these folders into Global Search(Trash and Junk). And for other new local folders, it is setting up the default priority 20 which is the correct behavior. If folders have some special flags then the priority will be according to that.

My local folders, which demonstrate the problem that global search can't find results in them, have the checkbox "Include messages in this folder in Global Search results" ticked. So the absence of this checkbox tick can't be the full explanation of this bug.

Same problem is happening to me when moving messages from IMAP account folders to local folders. I've deleted global-messages-db.sqlite too but it is still happening and that file is not being created again.
I am with Thunderbid 60.9.0.
Messages in Local Folders are found in Global Search but some of them are empty when they are clicked. I've installed GlodaQuilla addon and these messages have a gloda id in local folder.
Actually, the only workaround that it is working for me is moving all messages from Local Folder to Inbox folder and then moving them back to local folder.
Nobody can fix that ??? This bug is present from 4 years ago :-(

I have the same problem since I switched from Pop3 to IMAP. I also have the checkbox "Include messages in this folder in Global Search results" ticked, so this is not the problem.

Please fix this, this is really a serious issue.

Thanks,
Klaus

I just had what I think is the problem described here. I moved some e-mails related to shopping to a local shopping folder. It was only several days ago. I think I was using 78.8.0 on Windows. Today (using 78.8.1) I searched for one of those e-mails using keyword target but only older e-mails from target appeared. The local folder has Include messages in global search results. When I searched via right-click > Search Messages it found the message.

I exited thunderbird and deleted global-messages-db.sqlite from my profile directory. When I started Thunderbird again I used Tools > Activity Manager to watch the database be rebuilt. When it was done I searched again from the search box and then it worked.

I'm using Yahoo Mail over IMAP.

Any news on this? Sirs, data integrity is the basis, is it possible that a serious bug like this goes dismissed for two years in a row??
Same story for Mork files, there are DECADES old bugs regarding them, EVERY time you move message threads between folders you get them corrupted and you have to manually delete them to force repair... this is the reason I switched to Evolution and Vivaldi two years now after being a loyal Mozilla user since Firefox 0.6 and Thunderbird 0.5.
C'mon Mozilla people... less bullshit and more substance please, admitted that you can ever do something productive given the CEO you got 8 years ago, very successful at destroying the foundation piece by piece.

(In reply to Anselmo Canfora from comment #23)

Any news on this? Sirs, data integrity is the basis, is it possible that a serious bug like this goes dismissed for two years in a row??
... C'mon Mozilla people... less bullshit and more substance please, admitted that you can ever do something productive given the CEO you got 8 years ago, very successful at destroying the foundation piece by piece.

Sir, this is free software. You are getting support in proportion to the licence fees you paid, which is (I imagine) zero.

I understand your frustration, but using angry and abusive language in this forum is not appropriate. If you really want this problem fixed, you know how: open up the code, fix it yourself, and contribute the fix back; or, hire someone to do the same. If you are unwilling or unable to do that, then either use the product and live with the problem, or don't use the product. In no case is abuse the right response.

Does this still reproduce?

Assignee: khushil324 → nobody
Flags: needinfo?(raysatiro)
Flags: needinfo?(psychonaut)

This has happened to me occasionally over the years. When I'm certain Thunderbird is not finding something I know is there then I will delete the database as I described above, wait for it to rebuild and then search again. There are also some things I search for that I guess just can't be searched for like numbers and stuff. Anyway I'm sorry to say I don't remember much, I've just learned to live with it.

Flags: needinfo?(raysatiro)

I think it's still happening. Try moving emails from IMAP accounts to a local folder and the system won't find them anymore in searches. Same as Ray I've been rebuilding the database every few months, so I'm not 100% sure that it's still happening but it should be easy to reproduce following the steps above.

Same for me. That is still happening.

Sometimes the symptom is that the email is shown in the list of mails, but when I click on it, it cannot be opened.

Not reproducible for me with Thunderbird 115.15.0 on openSUSE Tumbleweed.

Flags: needinfo?(psychonaut)

It's still happening. Thunderbird 128.5.2esr (64-bit) on Windows 11

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: