1498923 - Moving too many messages at once to Local Folders causes the IMAP Index file to recreate itself

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Judith Hellerstein]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:62.0) Gecko/20100101 Firefox/62.0

Steps to reproduce:

If Thunderbird is online and I try to move too many messages to an offline folder, this action appears to damage the index file for the inbox and so Thunderbird begins the process to create a new Index file.  The same thing happens when I tried to create folders so that Thunderbird could file messages into the right folder. 


Actual results:

This causes a rebuild of the Index file for the Inbox. This is a very lengthy process when Inbox is over 30,000 messages.  However, if mail was offline when the messages were moved the Index for the Inbox is not damaged and does not need to be rebuilt. 


Expected results:

Index for the Inbox should not have been corrupted forcing the creation of a new index file for my mailbox.

Comment 1

[gene smith]

(In reply to Judith Hellerstein from comment #0)
> 
> Steps to reproduce:

Want to make sure I understand your terminology...

> 
> If Thunderbird is online 

Ok, tb is connected to your imap server(s). Is there more than one imap server or account?

> and I try to move too many messages

How many are "too many"? 10, 1000, 30000?

> to an offline folder,

I assume you mean this folder has offline storage but tb is still online.

> this action appears to damage the index file for the inbox 

Are you moving messages from accountA Inbox to accountB Inbox? If not, how are you moving message? From which folder to which folder?

> and so Thunderbird begins the process to create a new Index file.  

Is this "new Index file" for the source or destination of of the move? By index file I assume you mean the ...<tb profile>/ImapMail/AccountX/FolderX.msf ?

> The same thing happens when I tried to create folders so that Thunderbird could file
> messages into the right folder.

I assume you mean you create a new folder and move messaages to it from Inbox and this causes Inbox.msf to rebuild or the new-folder.msf to be created and download all the messages from server again?
 
> Actual results:
> 
> This causes a rebuild of the Index file for the Inbox. This is a very
> lengthy process when Inbox is over 30,000 messages.  However, if mail was
> offline when the messages were moved the Index for the Inbox is not damaged
> and does not need to be rebuilt. 

So if you do the move with tb offline, it completes OK. Then when you go back online no download or index rebuild occurs?
 
> 
> Expected results:
> 
> Index for the Inbox should not have been corrupted forcing the creation of a
> new index file for my mailbox.

So you are saying when you move (or copy?) X messages from Inbox to another folder (or new folder?) on the same (or different?) account, that Inbox re-downloads all 30K messages and rebuilds the Inbox.msf file? But there is no problem if you take tb offline first?

Do you know the type of IMAP server you are connecting to? For example, maybe gmail, yahoo, dovecot etc?
If you could attach an IMAP log to show what is happening when you do the move it might help. See https://wiki.mozilla.org/MailNews:Logging

Comment 2

[Judith Hellerstein]

Replying inline.  With asterisks and ___ to show where my answer is

(In reply to gene smith from comment #1)
> (In reply to Judith Hellerstein from comment #0)
> > 
> > Steps to reproduce:
> 
> Want to make sure I understand your terminology...
> 
> > 
> > If Thunderbird is online 
> 
> Ok, tb is connected to your imap server(s). Is there more than one imap
> server or account?

_____ ****In offline mode, TB is not connected to the server. Turned TB offline.  There is only one IMAP account. Mail is stored in the mailbox titled Inbox.___________ 
> 
> > and I try to move too many messages
> 
> How many are "too many"? 10, 1000, 30000?

______****Sorry, too many appears to be anything over 100-150_____
> 
> > to an offline folder,
> 
> I assume you mean this folder has offline storage but tb is still online.

***_____ What I mean is when I try to move messages from my Inbox, to a folder in the Local Mail Folder area. So If my folder is called Forward and I am moving the messages that are from this newsletter titled Forward from my Inbox to an offline sotreage on my computer. My offline storage of messages are not accessible via IMAP. ****______

> 
> > this action appears to damage the index file for the inbox 
> 
> Are you moving messages from accountA Inbox to accountB Inbox? If not, how
> are you moving message? From which folder to which folder?

***_____Explained this above. so perhaps this is clearer now______
> 
> > and so Thunderbird begins the process to create a new Index file Inbox.msf. 

***___ in my profile among the many folders in my profile there is a file called Inbox, which stores my entire online iNbox and then there is a file called inbox.msf.  It is this file that appears to get corrupted.  So when I am deleting more than 150 messages from the server and moving them offline.  If I do this when my TB is online and connected to my server then it tends to cause my inbox.msf file to recreate it self and begin to download all the mail again from the server. However, If I instead take TB offline and then move these messages from the online Inbox to an offline folder and then take TB Online again the synching of the Inbox does not get messed up**____
> 
> Is this "new Index file" for the source or destination of of the move? By
> index file I assume you mean the ...<tb
> profile>/ImapMail/AccountX/FolderX.msf ?
> 
> > The same thing happens when I tried to create folders so that Thunderbird could file
> > messages into the right folder.
> 
> I assume you mean you create a new folder and move messaages to it from
> Inbox and this causes Inbox.msf to rebuild or the new-folder.msf to be
> created and download all the messages from server again?

***Yes that is what I mean***
>  
> > Actual results:
> > 
> > This causes a rebuild of the Index file for the Inbox. This is a very
> > lengthy process when Inbox is over 30,000 messages.  However, if mail was
> > offline when the messages were moved the Index for the Inbox is not damaged
> > and does not need to be rebuilt. 
> 
> So if you do the move with tb offline, it completes OK. Then when you go
> back online no download or index rebuild occurs?

***Yes that seems to be whatis happening. As such until I have time to manually move messages to my offline folders, I cannot use filters since filtering of messages can only be done when TB is online and communicating with the mail server.  My hosting company is Pair.com and they host my website and my mail. I will try and create a log file but not sure I willhave time this week or next to do it.****
>  
> > 
> > Expected results:
> > 
> > Index for the Inbox should not have been corrupted forcing the creation of a
> > new index file for my mailbox.
> 
> So you are saying when you move (or copy?) X messages from Inbox to another
> folder (or new folder?) on the same (or different?) account, that Inbox
> re-downloads all 30K messages and rebuilds the Inbox.msf file? But there is
> no problem if you take tb offline first?
> 
> Do you know the type of IMAP server you are connecting to? For example,
> maybe gmail, yahoo, dovecot etc?
> If you could attach an IMAP log to show what is happening when you do the
> move it might help. See https://wiki.mozilla.org/MailNews:Logging

Comment 3

[gene smith]

Ok, I userstand. You have tb online and you *move* 100 or more messages from Inbox to a folder under "Local Folder" it causes all the email in Inbox to be re-downloaded. If you take tb offline, do this again and then go back online then all is well (no re-download occurs).

On more question. I don't know if you can test this, but if you *copy* (instead of move) the emails from Inbox to Local Folders with tb online, does the same problem occur?

Comment 4

[gene smith]

Just moved 167 messages from imap Inbox to Local Folders/Test with tb online and saw no re-download of Inbox. But I don't have anywhere near 30K messages in Inbox so that may be a factor. 

Does this happen for you every time you manually try to move 100+ messages?

Also, you mention filtering is not possible because you are often offline. When you go online with tb and new messages are retrieved, they should be able to be filtered by tb into a destination mailbox (e.g., Local Folders/Test) at that time. Could this be used to avoid the manual move? Another possibility is "server-side filter" where you configure your Pair server (probably at the webmail page) to filter emails into a selected imap folder. Then when you go online with tb you select that folder and it "auto-syncs" the emails that Pair server has put there. Note: A server-side filter destination can't be in the Local Folders area.

Anyhow, when you get the chance, the imap log would be helpful.

Comment 5

[Judith Hellerstein]

Spent tonight cleaning up my mailbox and had been very successful in taking my TB mailbox offline and moving messages from the main mailbox to folders. I repeated this exercise for each letter of the alphabet. All Foders with A, B. C etc. Then bringing it back online and it was all working well.  Was down from 35k to below 24K before I must have done too much and the inbox.msf file gave up and started rebuilding itself. Somehow we are back at 27,720 adding an additional 4K of messages. Not really sure what these are as the Inbox is not downloading all these messages.  I will try and download the IMAP log and maybe you can tell what happened.  It all was going so well. But Then I must have down too much. Feeling very frustrated.

Comment 6

[Judith Hellerstein]

OK Addendum.  It seems that the nex Inbox.msf file resulted in many duplicates or triplicates of mails that had been deleted but somehow never got deleted from the server. This duplication happened each time the Inbox.msf file is recreated.  Also strange in that the TB Inbox is not synching well with my mail server as reading the mail via pine is showing about 1,000 more mails in my mailbox than TB is showing. Not really sure what is happening.  I also found out that when I deleted mails from my inbox via thunderbird it seems not to have always deleted them from the server.  Could this be the issue of why the Inbox.msf file is being recreated. I will look at how to create log files.

Comment 7

[gene smith]

Can you look on your Pair webmail site and see if the same duplicates occurs there? If so, tb will just show what is on the server.

Can you tell me the size (number of bytes) of the Inbox and Inbox.msf files? I doubt that you are bumping up against an OS size limit, but if you are running with an older filesystem, like FAT/FAT32, it is possible. (Just noticed you seem to be running OSX so it should be OK but still curious about the file sizes.)

You mention folders with A, B, C etc. Do you have that many folders or do you really mean messages with subjects that start with A, B, C etc?. If you really mean folders, please show approximately your folder structure. Also, are you still trying to move messages from Inbox into Local Folders?

When you move messages from Inbox to Local Folders with tb offline, tb has to "playback" what you did when it goes back online so it can tell the server which messages you moved. Some servers may just mark the message you moved as "deleted" but they are still present, but not visible, on the server. So it may help when you go back online with tb to right click your Inbox folder and do the "compact folder" command. This will completely delete the messages moved from the server. 

Since your Inbox seems to be confused right now and if your webmail shows how you want tb to show the messages (i.e., minimal dups, unwanted/moved messages not present, number of messages correct, etc.) then, with tb online I would recommend doing a right-click on the Inbox folder and selected "Folder Properties" and select the "Repair Folder" button. This will appear to do what you say is happening randomly and re-download Inbox, but it will cleanly rebuild you Inbox and Inbox.msf files to exactly match what is on the server.

One more suggestion after you have repaired the Inbox folder: When you move messages to Local Folders try doing it like this. Keep tb online and select a group of messages in Inbox (maybe your messages that start with D). Then use the "copy" function (not move!) to place them in Local Folders. (You can then select Local Folders to make sure the copy occurred to be sure.) Then select the messages you just copied in Inbox and delete them from Inbox. Right-click on Inbox and compact the folder to make sure they are gone from the server's Inbox and the local Inbox and Inbox.msf files. Not sure this will help but worth a try.

Comment 8

[Judith Hellerstein]

HI Gene,
Thanks for your suggestion. I have tried to do some logs but find the instructions too difficult to follow. I do not have bbedit but use Apple's text edit and it is not letting me add the .command extension so could you or another person provide more simple instructions on how to create log files using Mac OS 10.13.4

On folders, Yes it is the later. It is subjects starting with A, B, C, etc. I do understand the playback concept and that it was why it was moving messages to trash that function was done but was not shown well on the server so it was done again. Possibly that could be the answer for the duplication as not all message get duplicated when the Inbox.msf file is rebuilt.  I actually do not use the webmail version of my Inbox. Do not be shocked but if I want to see what they look like on the server side, I look at the messages using Pine so can only assume they got duplicated during the playback. It is now reflecting the accurate counts both in pine and in my TB Mailbox.
 Still does not tell me why this is happening but something is interfering with the playback. This possibly is the reason why the mail is more stable when Thunderbird is offline. 

I have no problem moving a small number of messages, less than 100, but larger sizes tend to lead to the corruption.
Inbox.msf is 14.8 Mb
Inbox is 1.02 Gb
But Trash is huge and is 1.52 gb.  Will empty this later.

Comment 9

[gene smith]

Attachment: tblog.command

I have attached a file called tblog.command for you containing this:

>#!/bin/sh
>export NSPR_LOG_MODULES=IMAP:5,timestamp
>export NSPR_LOG_FILE=$HOME/imap.log
>/Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin &

Shut down tb and save the attached file in your home directory. Then open a Terminal window and make sure the file is there by typing (without the > here and others below):

>ls tblog.command

If it's there, you will see "tblog.command" in response on the screen.

Next you have to make sure the file can be used to run tb. In the terminal type

>chmod a+x tblog.command

Then do

>ls tblog.command

again and double click on the response "tblog.command" in the terminal. Tb should then startup with IMAP logging enabled. 

Note: you can also use the OSX file manager program to double-click/run the tblog.command file to start tb with logging enabled. 

In tb, do the move of 100 or more messages and then, if it tries to redownload, let it finish and then shutdown tb and find the log file in your home directory,

>ls imap.log

It should be present. Use the buzilla "Attach File" link above to find and attach the file imap.log. The file won't contain any passwords but it will show other info such as folder names. So you may want to edit or redact things in the file before attaching.

Note: if the copy of 100+ messages works OK, you might shutdown tb and restart it using the tblog.command file again and copy another 100+ until it fails. This will keep the imap.log file from becoming too huge since imap.log is deleted and re-created on each tb startup.

I don't normally run OSX but I run linux which is similar so hopefully my instructions will work.

Comment 10

[gene smith]

I forgot to say that, for now, while recording the log do the moves with tb online.

Comment 11

[Wayne Mery (:wsmwk)]

Judith, is this resolved now when using a current version?

Comment 12

[Phoenix]

Resolved per whiteboard