Open Bug 1205248 Opened 9 years ago Updated 1 year ago

global-messages-db.sqlite corrupted when mail/message is moved from IMAP folder to local folder, search doesn't find messages

Categories

(Thunderbird :: Search, defect, P3)

38 Branch
defect

Tracking

(thunderbird_esr68- affected)

Tracking Status
thunderbird_esr68 - affected

People

(Reporter: spam, Assigned: khushil324)

References

()

Details

(Keywords: dataloss)

User Story

https://support.mozilla.org/en-US/questions/1252397
https://support.mozilla.org/en-US/questions/1253814
https://support.mozilla.org/en-US/questions/1258666
https://support.mozilla.org/en-US/questions/1261826

Attachments

(1 file)

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0
Build ID: 20150826023504

Steps to reproduce:

- search for a text present in a known message contained on a IMAP folder
- click on the related item listed in search results
- the right mail is displayed
- now move the message to a local folder
- repeat the above search
- click on the item listed in new search results
- TB now displays a blank
- deleting global-messages-db.sqlite forcing TB to rebuild it clear the problem, until next message move...


Actual results:

TB does not refer the right mail anymore.


Expected results:

The correct message should be shown.
errata corrige: - TB now displays a blank *message*
I am unable to reproduce

1. local account/folders is not disabled from search indexing, correct?
2. does this happen for a message moved from *any* imap folder, i.e. not just inbox?
3. please install glodaquilla addon and use the additional message pane columns columns to ensure that glodaID for the message that has been moved is non-zero,
Flags: needinfo?(spam)
Hello,
- global indexing is active, I even don't know how to deactivate selectively for local folders
- it happens for messages moved from every folder
- it happens under Windows XP 32 bit with and IMAP account hosted by Exchange 2010, trying the same thing on Windows 8.1 64 bit with an IMAP account on a Cyrus server often works fine
Flags: needinfo?(spam)
please install glodaquilla addon. Enable the the glodaID column to see if the message that has been moved has a zero glodaID
Flags: needinfo?(spam)
Hello, here an example, no gloda id.
I believe I might be experiencing this problem also. I am running Thunderbird 52.5.0 (64-bit) on Mac OS X "El Capitan" 10.11.6.

I tried to install the GlodaQuilla plugin. Version 0.3.3, Released June 7, 2011, seems to be their latest version, and it "works with Thunderbird 3.0pre - 24.*". Thunderbird disables the "Add to Thunderbird" button for this add-on. However, I can install it from the "Extensions" tab of the Add-on Manager, using the "Install Add-on from file" menu item.

I just looked at one message which does not appear in the search results, but does appear when I search for it in its folder, via the Quick Filter bar. That message has a GlodaID of "b571".
(In reply to Wayne Mery (:wsmwk) from comment #4)
> please install glodaquilla addon. Enable the the glodaID column to see if
> the message that has been moved has a zero glodaID

Hello, no glodaID for the message with bug
Flags: needinfo?(spam)

Similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1254982 "Open message in list" is empty for imap messages archived to local folders [faceted search results] (still indexed on Inbox, not on archive folder, fixed by rebuilding gloda database)

I am affected by this bug. It is the most severe flaw in Thunderbird that I am currently experiencing.
In my experience, The problem is corrected by rebuilding the Gloda database. Some, but not all, messages which are in the Local Folders and which Gloda cannot find have an empty cell in the Glodaquilla plugin's Gloda ID column. I am using current Thunderbird (presently 60.7.0) on Mac OS X 10.11.6.

Workaround: select the local folder into which the disappeared message was archived. Enable Quick Filter. Type search text into Quick Filter search box. Desired message is found.

This is a serious search problem, and easily reproduced. Can we assign someone to it to fix?

related to bug 1217958?

Severity: normal → critical
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(mkmelin+mozilla)
Keywords: dataloss
Summary: global-messages-db.sqlite gets corrupted when mails are being moved from IMAP folder to local folder → global-messages-db.sqlite corrupted when mail/message is moved from IMAP folder to local folder, search doesn't find messages
User Story: (updated)

Sounds to me global-messages-db.sqlite isn't corrupted, it's just that it doesn't know about the message in question. Like perhaps we just don't run indexing on it.

Starting points for debugging could be in https://searchfox.org/comm-central/rev/cb71ec93ae822bce37fb00e321ef201cce24ff1e/mailnews/db/gloda/modules/index_msg.js#579 or indexer.js - but there's a lot of abstractions in gloda so it's often fairly hard to understand what goes on :/

Given the age of this bug I don't think we need to track it for 68 though we might uplift if we find the problem.
Khushil, putting this in your backlog.

Assignee: nobody → khushil324
Flags: needinfo?(mkmelin+mozilla)
Priority: -- → P3

Hey All,
Something interesting is going on here. This is not a bug actually. By default, all the local folders have indexingPriority set to -1. Because of this, TB does not index those folders in Global Search: https://searchfox.org/comm-central/rev/cb71ec93ae822bce37fb00e321ef201cce24ff1e/mailnews/db/gloda/modules/index_msg.js#1761
One can change this: Right click on the folder and go to properties, there will be a checkbox "Include messages in this folder in Global Search results". Tick that and you will see now messages coming up in the Global Search. The original author has given it as functionality which is very interesting.
This checkbox will be shown if there are some messages in the local folder.
Now, we can do one thing, set indexingPriority for all the local folders. Magnus and Wayne, what do you suggest?

Flags: needinfo?(vseerror)
Flags: needinfo?(mkmelin+mozilla)

I think the question is then, why is it set to -1, and where?
Unless there are reasons not to, we should just avoid that.

Flags: needinfo?(vseerror)
Flags: needinfo?(mkmelin+mozilla)

By default, all the local folders have indexingPriority set to -1

If it the default on your system then this is a different issue and definetly should be filed as a new bug.

But I don't see this on my system, so it's not the problem described in this bug

For SMTP like account folders:
Priority is set here: https://searchfox.org/comm-central/source/mailnews/db/gloda/modules/datastore.js#1929
For normal folders, indexingPriority will be a null string. In this case, new GlodaFolder() at time of creation will take default value which is 20. For Trash and SPAM like folders, stmt.getInt32(4) will give the value of -1 so indexingPriority will be set up as -1.

For Local Folders:
Priority is set here: https://searchfox.org/comm-central/source/mailnews/db/gloda/modules/datastore.js#2015. "stringPrio" is null for the new local folder which will be set up through JSM module and we are not assigning anything there. So, it will go to else statement: https://searchfox.org/comm-central/source/mailnews/db/gloda/modules/datastore.js#1961. But we are not passing the second argument in getDefaultIndexingPriority so it is setting up -1 for Trash and Junk and there is a comment not to add these folders into Global Search(Trash and Junk). And for other new local folders, it is setting up the default priority 20 which is the correct behavior. If folders have some special flags then the priority will be according to that.

My local folders, which demonstrate the problem that global search can't find results in them, have the checkbox "Include messages in this folder in Global Search results" ticked. So the absence of this checkbox tick can't be the full explanation of this bug.

Same problem is happening to me when moving messages from IMAP account folders to local folders. I've deleted global-messages-db.sqlite too but it is still happening and that file is not being created again.
I am with Thunderbid 60.9.0.
Messages in Local Folders are found in Global Search but some of them are empty when they are clicked. I've installed GlodaQuilla addon and these messages have a gloda id in local folder.
Actually, the only workaround that it is working for me is moving all messages from Local Folder to Inbox folder and then moving them back to local folder.
Nobody can fix that ??? This bug is present from 4 years ago :-(

I have the same problem since I switched from Pop3 to IMAP. I also have the checkbox "Include messages in this folder in Global Search results" ticked, so this is not the problem.

Please fix this, this is really a serious issue.

Thanks,
Klaus

I just had what I think is the problem described here. I moved some e-mails related to shopping to a local shopping folder. It was only several days ago. I think I was using 78.8.0 on Windows. Today (using 78.8.1) I searched for one of those e-mails using keyword target but only older e-mails from target appeared. The local folder has Include messages in global search results. When I searched via right-click > Search Messages it found the message.

I exited thunderbird and deleted global-messages-db.sqlite from my profile directory. When I started Thunderbird again I used Tools > Activity Manager to watch the database be rebuilt. When it was done I searched again from the search box and then it worked.

I'm using Yahoo Mail over IMAP.

Any news on this? Sirs, data integrity is the basis, is it possible that a serious bug like this goes dismissed for two years in a row??
Same story for Mork files, there are DECADES old bugs regarding them, EVERY time you move message threads between folders you get them corrupted and you have to manually delete them to force repair... this is the reason I switched to Evolution and Vivaldi two years now after being a loyal Mozilla user since Firefox 0.6 and Thunderbird 0.5.
C'mon Mozilla people... less bullshit and more substance please, admitted that you can ever do something productive given the CEO you got 8 years ago, very successful at destroying the foundation piece by piece.

(In reply to Anselmo Canfora from comment #23)

Any news on this? Sirs, data integrity is the basis, is it possible that a serious bug like this goes dismissed for two years in a row??
... C'mon Mozilla people... less bullshit and more substance please, admitted that you can ever do something productive given the CEO you got 8 years ago, very successful at destroying the foundation piece by piece.

Sir, this is free software. You are getting support in proportion to the licence fees you paid, which is (I imagine) zero.

I understand your frustration, but using angry and abusive language in this forum is not appropriate. If you really want this problem fixed, you know how: open up the code, fix it yourself, and contribute the fix back; or, hire someone to do the same. If you are unwilling or unable to do that, then either use the product and live with the problem, or don't use the product. In no case is abuse the right response.

(In reply to Jim DeLaHunt from comment #24)

(In reply to Anselmo Canfora from comment #23)

Any news on this? Sirs, data integrity is the basis, is it possible that a serious bug like this goes dismissed for two years in a row??
... C'mon Mozilla people... less bullshit and more substance please, admitted that you can ever do something productive given the CEO you got 8 years ago, very successful at destroying the foundation piece by piece.

Sir, this is free software. You are getting support in proportion to the licence fees you paid, which is (I imagine) zero.

I understand your frustration, but using angry and abusive language in this forum is not appropriate. If you really want this problem fixed, you know how: open up the code, fix it yourself, and contribute the fix back; or, hire someone to do the same. If you are unwilling or unable to do that, then either use the product and live with the problem, or don't use the product. In no case is abuse the right response.

My one is not "angry abusive language", I don't care about politically correct bullshit, I just remarked some facts: TB has a big problem with Mork files and with very very old and not scalable MBOX storage format. Also the fact that the foundation is on a steep dive since 2014 is a fact, just look at the numbers:

https://gs.statcounter.com/browser-version-partially-combined-market-share/desktop/worldwide/#monthly-201404-202209

https://en.wikipedia.org/wiki/Mitchell_Baker#Negative_salary-achievements_correlation_controversy

https://www.zdnet.com/article/programming-language-rust-mozilla-job-cuts-have-hit-us-badly-but-heres-how-well-survive/

but I admit I am going OT here, I should open a bug under "other" category titled "Mitchell Baker is the Mozilla CEO" but I have neither the time nor the desire to troll you. Perhaps Brendan Eich would have.

Your attitude "get it the way it is, or try to fix it yourself" is not very polite, nor in line with the true open source spirit. Actually your attitude, not mine, is subtly abusive of others. If you check Bugzilla you will actually find that I opened several well documented bugs across several years that helped to fix quite some defects, but you probably don't think this kind of userbase involvement is useful nor desired. And on the other hand the feeling I got when trying to interact with your developers via Bugzilla is that they don't care very much about users and serious bug reports, I found them actually not very cooperating, even (unjustifiably) arrogant, like the time they were all stating (with my horror) that MBOX is vastly superior and more scalable than Maildir, a statement that every sane developer would laugh at. This kind of arrogant attitude is what drives away users and contributors from Mozilla and what prevents for sure me from getting more involved with your products by submitting merge requests for products I don't even use anymore. But perhaps that is indeed the goal that someone has.

Cheers

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: