Open Bug 1566717 Opened 5 years ago Updated 2 years ago

Emails moved to local folders all completely blank related to timeout

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Type:
defect

Tracking

(Not tracked)

People

(Reporter: marketing, Unassigned)

References

Details

(Whiteboard: [dupme?])

Attachments

(11 files)

Captura.PNG
23.60 KB, image/png
Details
2020-01-30 21_local_folders.png
23.86 KB, image/png
Details
2020-01-30 15_43_user_drag_drop.png
293.80 KB, image/png
Details
2020-01-31 12_08_47_local_to_imap.png
4.65 KB, image/png
Details
2020-02-01 10_46_31-scrambled code.png
62.80 KB, image/png
Details
2020-02-01 21_24_ - 5th download.png
107.60 KB, image/png
Details
2020-02-01 21_26_06-.png
246.67 KB, image/png
Details
2020-02-01 22_40 - messages 0.2kb.png
107.24 KB, image/png
Details
2020-02-01 23_03_50- error on 2nd try.png
93.15 KB, image/png
Details
2020-02-01 23_51_- source folder redownload.png
129.15 KB, image/png
Details
2020-02-01 23_56_- messages with blank duplicates.png
107.84 KB, image/png
Details
Attached image Captura.PNG —

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36

Steps to reproduce:

Cut and pasted a large number of emails across from a remote (IMAP) folder into a local folder.

Actual results:

A large number of emails are listed in the local folder, but each is completely blank. The original message were nonetheless removed.

Sample Message Source:
From - Tue Jul 16 10:43:46 2019
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:

Some messages have the blue arrow indicating that at some point I replied to them. They each have the date listed of the time they were copied. They are all listed identically.

Expected results:

They should have copied across with their original content! Further, perhaps as a safety measure, if messages are not moved properly there should be some way of retrieving the originals?

Have you tried to repair the local folder? Right-click, properties, "Repair Folder".

Yes, I have tried but it did not work. Thank you though!

Can you find the raw mailbox file in your file system (you can see the location also under Properties) and open it with a text editor. What do you see? If there isn't any private information, you could ZIP up the file and attach it here or send it to me in an e-mail for inspection.

Gene, moving IMAP to local folder can cause this?

Flags: needinfo?(gds)

There are many such reports, one being bug 1290669, which leads to bug 763390 where Ben is also willing to do some work to figure this out. I've copied him here.

See Also: → 775008

I just did a copy of all my bugzilla folder to Local Folders and they seem fine in Local Folders. I haven't check with imap log or wireshark, but a "move" from imap folder to local folder should just mark the messages in the imap folder as deleted. So if you set the delete mode in server setting to "just mark as deleted" you should still see the "moved" messages with red X and lines through them. Then you can undelete them. Note: if server is gmail by default any message marked \deleted is expunged so you won't see it with red X and line; however, it might still be in All Mail folder.

Edit: I just did a "move" (instead of copy) of multiple messages to local and I don't see a problem.

Further, perhaps as a safety measure, if messages are not moved properly there should be some way of retrieving the originals?

When I set the imap account to "Just mark as deleted" in server setting and look at the folder, the moved messages do have the red X and line through them and can be undeleted. So this acts as a sort of "safety measure". Then you can return to "move to trash" delete method.

Flags: needinfo?(gds)

Reporter, Do you see this only with a "large" number moved but not with a small number? If so can you define "large". Also, do the messages moved have large attachments? And curious if the source imap folder has offline storage set (this is default).

This has triggered for me on moving even individual messages, with plain text and no attachment.

(In reply to Ben Lerner from comment #7)

This has triggered for me on moving even individual messages, with plain text and no attachment.

Just moved your bug mail from imap to folder on Local Folders with no problem. Is this something you see just occasionally or all the time? If you can duplicate this maybe an IMAP:5 log would show something. If online storage exists for the moved message, I would think the only transaction with imap would be to mark the message as \deleted. If no online storage, tb would have to imap fetch the message first. Does your source imap folder have offline storage?

No, annoyingly, I've never been able to figure out a 100% reliable repro for this. If offline storage being set is the default, then yes, my imap folder has it set; I don't think I've tweaked that.

My school email is currently (about to change, annoyingly) a Zimbra setup that I connect to via IMAP in Thunderbird. When I try-and-fail to move a message from a school folder into Local folders, I recall having logged into Zimbra's web portal, and the message has completely vanished from there too. In Thunderbird's toolbar, I always set the View: dropdown to show "All", which I presume should display any \deleted messages also? I also have double-checked, and my account settings are set to move deleted messages to a trash folder, not to delete them immediately or or mark them as deleted, so I ought to be able to see the messages in that folder...but they never make it there either.

I don't know what an IMAP:5 log is -- if I do trigger this bug again, how could I collect such a log?

Apropos comment #5, note that I've never been able to trigger this behavior with a copy of messages from an IMAP folder to Local Folders -- it's only been a move that's caused the blank messages to appear. (Then presumably, the move operation claims its a success, and so deletes the message from the IMAP location, but (a) the move didn't really succeed, and (b) the delete is ignoring the "what should thunderbird do on deleting messages?" account preference above.)

Ben, It's probably safer to just do a copy and then if it works go back and manually delete messages from the source imap folder.

The way imap works is that when you "move" a message to local, it is really just copied and in the imap folder on the server the message is "marked" as \deleted. This means the message is still present on the server but not visible by default. You may not see it in zimbra webmail either since it is marked as \deleted on the server. In tb, you can still see the \deleted messages if you go to server setting and change the default "Move deleted mail to trash" to "just mark as deleted". (Not sure what you mean by the view-all dropdown, but not a way to see \deleted messages.) With this setting, messages marked \deleted will be visible with a red X and line though them, and they can be undeleted (really just removes the \deleted flag from the imap message on the server).

Most imap servers keep messages when they are marked \deleted by the client (ie, tb). However some have other ideas. By default, gmail expunges (permanently deletes) any message the client marks \deleted by default. You can change this at the gmail account site. I'm pretty sure that zimbra does not expunge messages marked as \deleted in the default setup. So to expunge messages (permanently remove messages marked \deleted) you have to right click and compact the folder in tb.

Oh yes, I understand two-phase commit protocols, and that copy+confirm+delete is much better/more reliable than move. It's just so much less convenient, and the UI (either drag-n-drop, or using the Nostalgy extension) makes it that much more of a foot-gun to do a simpler move instead. It seems like TB is missing a check: upon moving from IMAP to Local Folders, confirm that the message contents made it successfully, before marking the message as \deleted on the IMAP server.

(The View drop-down I'm talking about is just to the left of Quick Filter, and has All/Unread/Tags>/Custom views>/Save view as a Folder.../Customize... as its dropdown options.)

I see "Tags" button to left of Quick Filter button in toolbars. Anyhow, the \deleted flag/tag is not one you can just display. You have to set the account server to "just mark as deleted" to see \deleted messages still in the folder (will have the strikethrough and red X) This seems like a reasonable fix for the occasional failure to copy to local since tb at least tries to keep the "moved" emails (but can't control what some servers might do to \deleted emails).

To get the IMAP:5 log you have to be running tb with some env. vars set. Just running normally tb won't produce the log. The details are here: https://wiki.mozilla.org/MailNews:Logging . Edit the resulting log file for privacy issues and attach it above using the "Attach file" link.

If you still see a problem when using version 68 please see comment 13

Flags: needinfo?(benjamin.lerner)
Whiteboard: [closeme 2020-01-11]

We face the same problem here, multiple users, versions 68.3.1, 68.4.x. This becomes really frequent now as people archive 2019 stuff.
By drag and drop, move-to a local folder menu option, autoarchive extension - problem is quite similar. About 50-100 messages are blank when archived in bulk and everything else archived normally. After occuring once it is hard to reproduce. These are random users, logging each Tb will be hard, but the number of mails lost is quite high.

One user has blanks from between - 02-09.12.2019 (if fetch time is correct) so it could be Tb 60 too
I'm not sure if going back below 60 is going to help much. We have the mbox files affected, but it's hard to catch the bug in one IMAP:5 log, as it is hard to reproduce. Mails are lost in ove period and then everything is back to OK, like:

-Local folders
March - all ok
April - 50 blanks, rest OK
May - 50 blanks, rest OK
June - all OK and so on

Clients are Win10
Antivirus is Bitdefender Endpoint Security Tools Antymalware

kris, Are you archiving just to Local Folders or to other imap folders?
What is your imap server type?
If the original folder has not been compacted (i.e., imap expunged) and the server doesn't expunge on mark as deleted (e..g., gmail) the messages should still be present as I tried to describe above. Use "just mark as deleted" setting to maybe see them.
Does the problem occur for users without the "autoarchive" addon? (There is a warning about possible data loss on the "autoarchive-reloaded" addon page.)
Edit: Another question: Do you store/sync imap mails locally or keep them just on the server. You mention mbox files so I would assume you are referring to tb's mbox files where messages are stored.
Another edit: I assume when you try to open the "blank" messages that they are truly empty?

Archiving is done in most cases by drag and drop method, but yes there is one case where Autoarchive-reloaded blanked the emails for test user. (Only I and this particular user have it enabled for test purposes).

The mails are empty from periods we assume addon had done it's thing. For example, we see (all empty, the content mising from mbox file, fixing msf did not help at this time, there were only headers in mbox when checked in text editor):

02.12.2019, 12:15
02.12.2019, 12:15
02.12.2019, 12:15
02.12.2019, 12:15
05.12.2019, 16:17
05.12.2019, 16:17
05.12.2019, 16:17
05.12.2019, 16:17
09.12.2019, 07:52
09.12.2019, 07:52
09.12.2019, 07:52
09.12.2019, 07:52

placed in one mothly local folder and afterwards everything is back to normal, so if there are many emails archived daily, the issue can be observed when looking actively for those messages. But I agree, the addon is not the safest option, only a test.

But the same issue is with drag and drop in bulk from imap->local. I'll add another screenshot where it happend (not even archive method, just drag-and-drop to folder)

All imap folders 'Folder Properties' have option 'Select this folder for offline use' on so i presume they're cached as mbox just like local folders
The 'Retention policy' uses account settings, nothing is checked.

The mbox and msf files are in %profile%\ImapMail folder - and here im sure they're being compacted very rarely (only when an email is missing).

The workaround with 'mark as deleted only' option is helpful, but with such numbers of mails without subject it's really hard to find out what is missing and needs to be restored.

Is imap:5 log all that is needed to pinpoint the issue? I'm trying to reproduce the problem with logging on.

This evening as a test I copied about 17000+ emails from an imap folder to a Local Folder and don't see a problem. Not sure that an IMAP:5 log would help since copy to Local Folders just uses the local filesystem and not imap. Also, if you have local offline store for each folder, there would be no imap activity when they are moved to Local Folders. But if you can catch the problem while recording the IMAP:5 log, it might help, who knows.

The workaround with 'mark as deleted only' option is helpful, but with such numbers of mails without subject it's really hard to find out what is missing and needs to be restored.

I'm suggesting doing this in the source imap folder where the messages were copied (or moved) from. Those don't having missing subjects, right? If that doesn't find anything (messages crossed out) you can try looking in the mbox files for the source imap folders and see if the missing/corrupted messages are in there. The mbox files are just concatenated email files that are mostly readable unlike the .msf file that are totally cryptic to the casual reader.

One suggestion until we understand what is happening is to keep on archiving using drag/drop but when doing the drop into the local folder, hold down the control key so that a small '+' appears. This will ensure that a copy occurs and not a move. Then check that the copy is ok in Local Folders (subject and dates and all visible and messages readable). Then go back and delete the messages in the source imap folder.

I did a test (1600 messages uploaded to imap and redownloaded to local), but the log textfile of about 7GB in size. Many blanks produced. Not sure how to share it as it contains some private data.

2nd test 3004 messages from imap to local produced 3004 blanks plus around 600 readable, so the total didn't even match, as it was 3604 not 3004. The log size is 9.9 GB.

3rd test 1000 mails (from the same bunch, reuploaded and moved to local) test went ok. Log size is 1.5 GB.

will test more - 500 messages each time until all 3000 are downloaded, maybe there is some issue with some of the emails only...

just to note - 'mark as deleted' - deleting in TB marks messages to be deleted on imap
but moving does not - the messages are still taken down from imap and not available in web client anymore.

mails moved by 500 each time went ok

Next test - 3200 in one go from imap to local - everything OK.
But - reuploaded all to imap - now three are blank in imap, see screenshot

Well I'm perplexed. Let me repeat the question in comment 1 above: Have you tried to repair the Local Folder? Right-click, properties, "Repair Folder".
I just did another copy of 7k messages to a local folder with no problems seen. But I'm on linux and have no anti-virus running. Have you tried disabling the "Bitdefender Endpoint Security Tools Antymalware" and redo the test?

I tried the same thing on win10 with only "windows defender" and don't see a problem.

I compact the folders each time I'm prompted to, repairing did not help, two of the messages are bytes in size. One of them has body still in it. But it appears as empty, the code is not read properly.
It starts with header and some (I think it's Outlook) wrapping:

From - Fri Jan 31 12:13:43 2020
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:
X-Mozilla-Keys:
pan><o:p></o:p></p>
</td>
<td
style=3D"width:33.85pt;border-top:none;border-left:none;border-bottom:sol=
id
windowtext 1.0pt;border-right:solid windowtext
1.0pt;padding:0cm 3.5pt 0cm
3.5pt;height:15.0pt" width=3D"45"
valign=3D"bottom" nowrap=3D"nowrap">
<p class=3D"MsoNormal"><span
style=3D"mso-fareast-language:PL">=C2=A0</s=
pan><o:p></o:p></p>

but what is displayed is only an inline base64 code (see screenshot). The two others are just headers.

Anyway, I'll try with Windows Defender only today and post results! Thanks for advice do far!

Attachment #9123733 - Attachment description: 2020-01-30 21_09_45-2019-05 - Lokalne foldery – Mozilla Thunderbird.png → 2020-01-30 21_local_folders.png
Attachment #9123733 - Attachment filename: 2020-01-30 21_09_45-2019-05 - Lokalne foldery – Mozilla Thunderbird.png → 2020-01-30 21_local_folders.png
Attachment #9123811 - Attachment description: 2020-01-31 12_08_47-JW - krzysztof.szlesinski@naprzod.pl – Mozilla Thunderbird.png → 2020-01-31 12_08_47_local_to_imap.png
Attachment #9123811 - Attachment filename: 2020-01-31 12_08_47-JW - krzysztof.szlesinski@naprzod.pl – Mozilla Thunderbird.png → 2020-01-31 12_08_47_local_to_imap.png
Attachment #9123776 - Attachment description: 2020-01-30 15_43_14-TeamViewer.png → 2020-01-30 15_43_user_drag_drop.png
Attachment #9123776 - Attachment filename: 2020-01-30 15_43_14-TeamViewer.png → 2020-01-30 15_43_user_drag_drop.png

(In reply to kris.szlesinski from comment #28)

I compact the folders each time I'm prompted to, repairing did not help,

Just to sure, I'm referring to repairing the destination "Local Folder" and not the source imap folder. Also, when you move a bunch of message from imap folder to local folders, you may get the prompt to compact the folders a bit later. It may be better to skip the compacting since it will permanently remove the copied messages from the source imap folders so the only fix, if they are somehow corrupted in Local Folders, is to restore them from a backup.

two of the messages are bytes in size. One of them has body still in it. But it appears as empty, the code is not read properly.
It starts with header and some (I think it's Outlook) wrapping:

Can't tell much about this. If you could save and attach the full email source I might be able to see what the problem is.

Anyway, I'll try with Windows Defender only today and post results! Thanks for advice do far!

You're welcome.

I did the tests - and I think I've found something -
5 times downloaded 3211 messages in Windows Defender environment - all good
4 times - back, where we use Bitdefender Endpoints and Firewalls - ok. But the 5th time
messages were downloaded and visible in folder, but TB was doing some activity it couldn't finish.
See screenshot:

Operation was in endless loop so I had to restart TB and here is the result:
181 readable, rest blank (see screenshot):

Not sure what this might be, but i will rerun the test couple times with addons off

Ok, I can't tell much from the error log except the autoarchive is running. So yes, run the test with TB in "safe mode" which disables all possible addons/extensions. It's under Help/Restart with addons disabled... which you probably know.

OK, so I disabled all addons, emptied the local folders and went again. on the first try i ran into an error.
The content is downloaded, but the operation is not completed
Activity cannot finish for some reason, nothing to see in debug log and all the messages are 0.2kB in size.
Here is the pic:

So messages that fail to copy always have a size of 0,2kB (or about 200 byte I think)? Also, did tb hang up and need restarting or can you still do other stuff after this occurs? Are you saying every message is corrupted in your latest run?
Also, please explain the exact steps you are doing to cause this. I assume you are selecting all the messages in the source folder and then using right-click to copy them to the destination Local Folder? That's what I've been doing and haven't seen a problem. I'm using tb 68.4.2 (32-bit) on win10. Maybe you are using a different version or possibly 64-bit. Maybe I need to change over to 64-bit but that's just what was on the laptop already.

I run the test again - cleaned first folder, compacted thrash, first try all good.
No prompt for cleanup, so i restarted TB, fixed folder 1 just in case and started downloading into folder2.
Again, same situation, TB cannot finish some task, activity goes on, message size is 0.2 KB(also the attachments gone, and 'Re:' from subjects of returned messages).

Last thing before the loop happened was TB trying to update source imap folder while copying its content,
it did sync the source folder, but now it's in the loop, and after restart timestamps and subjects will be gone too :(
See pic-

I have been copying a gmail folder of 7653 messages to LFs many times with no issue. The gmail folder doesn't have local storage so tb must first fetch the message from gmail server before doing the copy. Still not sure if you are doing a copy or a move from imap folder. It shouldn't matter but to duplicate the problem I need to know for sure what you are doing. If you are doing move, try doing copy instead and see if it matters.
Also, one more info: what is your imap server type? (Sorry if you told me and I forgot.)

didn't see your answer before posting - it 64-bit 68.2.4. It's operable when it happens, I can still do stuff like sending and receiving.
Interesting is that after restart all 3211 are blank, and at the very bottom of the folder there are 4 that are readable. I once mentioned it above - once its 5, once 100, once 500 messges that are in the local folder ON TOP of the blank messages that are fully readable.

Between the downloads to local I do fix folder step for folders that have changed due to deletion or addition, but all sizes seem OK, so there is no change. I even restart TB between tests so it may prompt me for compacting, but TB thinks all space is correctly allocated, and there is no prompt, but I have the feeling there should be one. And then comes the mentioned hang and error.

I do a copy Ctrl+drag and drop - as you asked previously - i have made five folders for five separate tests to which i copy the content from 'JW' imap folder.
The server is linux based postfix one. What information you need? I don't have too much access to it, but I can ask around, no problem.

Ok, ctrl d/d should do a copy. With copy there should be no compact prompt since nothing is marked deleted in the source folder. Maybe compact applies to LFs, not sure.

Another thing to try: if you have local storage for the source imap folder, try doing the copy with tb set offline. Don't think it would matter but who knows...
Also, when you do the copy/move as a test, how long does it take? Mine takes maybe 2 minutes but I haven't timed it.

I think postfix is for sending mail, not sure. If on linux the imap server is probably dovecot which is pretty reliable.

it 64-bit 68.2.4

Typo? the latest is 68.4.2 I think.

Between the downloads to local I do fix folder step for folders that have changed due to deletion or addition, but all sizes seem OK, so there is no change.

Don't understand what you mean by "fix folder step...".

You're right it's dovecot, it takes about two minutes to download. Size is approx 1.1 GB.
When downloading offline, the progress bar seems stuck on message 1. But it finishes eventually, and all seems correct.

(In reply to gene smith from comment #48)

it 64-bit 68.2.4

Typo? the latest is 68.4.2 I think.

Between the downloads to local I do fix folder step for folders that have changed due to deletion or addition, but all sizes seem OK, so there is no change.

Don't understand what you mean by "fix folder step...".

Pardon me, i meant 68.4.2
Fix folder = Repair Folder

Sorry for mistakes it's getting late here, and if there is anything I may check for you I'll get it done first thing in the morning
Thank you for your help so far I really appreciate it -
we'd really like to have the problem cleared as soon as we can.

Kris

Yes, I set the source folder to have local storage (also called storage for offline use) and did the copy. You're right, it does seems stuck at 1 but the copy completes much faster when offline. This may mean when you did the online copies, the messages are being fetched from imap first. This should indicate that you did not have local storage for the source imap folder. Before you did the offline copy, did you set the source folder to have offline (local) storage and then download the folder? This can be done using the right-click properties menu under synchronization or under sync setting for the account.
Anyhow, maybe an IMAP:5 log would be helpful if the corrupted messages are coming from the imap server which it is beginning to sound like.

Yes, tomorrow maybe an imap log would help: see https://wiki.mozilla.org/MailNews:Logging

In the picture with comment 46 it shows the messages being download. I see the same thing when I copy an imap folder without local/offline storage to LFs. That indicates that the source folder doesn't have offline storage and the messages only reside on the imap server. Is this unusual? In an earlier comment 20 you said:

All imap folders 'Folder Properties' have option 'Select this folder for offline use' on so i presume they're cached as mbox just like local folders
The 'Retention policy' uses account settings, nothing is checked.

So there should be no need for tb to download messages for typical folders if the above statement is correct. The copy should occur from the imap folder's mbox file directly to the LFs mbox file using only window filesystem. But maybe there are exceptions in your user's accounts.

Then in comment 47 you mention duplicate messages. I see duplicates if I copy to the same LF without first deleting them from the LF. This is presently expected behavior. Of course, corrupted messages are not!

Anyhow, try to produce an IMAP:5 log while doing a defective copy. You can leave off the "timestamp" item which will make the file some shorter. If you do the copy and it works OK, you should probably restart tb so the log file is completely deleted and re-written and the size doesn't become even bigger. Then again, if the source folder does have offline storage, the imap log should be fairly short since no fetch from the imap server is needed.

One more possible issue: When the copy from imap folder to LF folder is in progress, could new messages appear the the source imap folder? If the source folder is Inbox this is definitely possible. (However your tests don't use Inbox.) New messages might appear in the source folder due to "server side" filters or possibly tb local filters. I don't think this should really cause a problem but not 100% sure. Anyhow, doing the copy with tb offline and with the source folder having local/offline storage would prevent this potential problem.

(In reply to gene smith from comment #52)

In the picture with comment 46 it shows the messages being download. I see the same thing when I copy an imap folder without local/offline storage to LFs. That indicates that the source folder doesn't have offline storage and the messages only reside on the imap server. Is this unusual? In an earlier comment 20 you said:

All imap folders 'Folder Properties' have option 'Select this folder for offline use' on so i presume they're cached as mbox just like local folders
The 'Retention policy' uses account settings, nothing is checked.

So there should be no need for tb to download messages for typical folders if the above statement is correct. The copy should occur from the imap folder's mbox file directly to the LFs mbox file using only window filesystem. But maybe there are exceptions in your user's accounts.

but we have Bug 505456 - move/copying multiple imap messages to local folder bypasses offline store and redownloads messages. Need to preflight the move/copy.

https://mzl.la/3b5S89N lists some bugs related to copying from imap to local

Flags: needinfo?(benjamin.lerner)

Thanks Wayne, looks like I've sort of been "re-inventing the wheel" and or "beating a dead horse". My old comment at bug 505456 comment 4 sums it up pretty well. Also bug 1525033 comment 5 describes almost exactly what Kris sees, even the 0.2kb message sizes.

<some time passes>

Ok, I have duplicated the problem. If I pull the network cable for 5-6 sec while copying to LFs and plug it in back in the copy stops. It tries to start back when I plug it back but shortly aborts the copy. One time I looked at the messages in LFs and they were OK, just fewer than expected. I tried the copy again and this time a bunch of them in LF were 0.2kb with subjects and dates but TB was pretty much hung (spinning circle when accessing LFs). On tb restart, for the size 0.2kb message I only see the time, all 11:05PM (i.e., now). Down below I see good messages. So this is exactly what Kris sees.

Apparently something on Kris's network is causing a similar interruption. So until we can fix this I think the only work around is to make sure the offline store is present for any folder that are copied to LFs and then go offline to do the copy to LFs. Move would probably work OK in this mode but I still recommend just doing a copy and then go back and delete the copied messages in the source imap folder.

Status: UNCONFIRMED → NEW
Ever confirmed: true

I've sent IMAP:5 to gds@chartertn.net via WeTransfer.

Thanks kris. I will download an look at the log a bit later on my desktop computer that has more disk space. I suspect it will show some loss of connection to the server but the imap log is often not clear as to what causes the loss.
I was maybe going to tell you that a log was no longer necessary since I can now duplicate the problem, but decided it was OK so I have something to compare my results to.

I was able to modify the tb code so that copies from folders with offline store to Local Folders now just obtains the data from offline store rather than going to the imap server and fetching it. This addresses the issue in old Bug 505456.
However, the changed code is not real clear and has stuff in it about "news" messages as well as imap. So more testing is needed before I submit a patch.
I still haven't downloaded kris's log but still plan to look at it in detail and determine why connection glitches seem to corrupt the data.

kris, I was able to read the logs using "less" and search for things. What I see is as you describe: I see one complete fetch of all the messages that works fine. Then you do the copy again and this time there is a tcp timeout error. This will occur if the server takes more than 100 seconds to respond to anything tb sends to it. It looks likes the server sends a 64k size packet of message data and tb ACKs the response but the server doesn't send another packet for over 100 seconds. Tb then closes the tcp connection, opens a new connection and requests the data again and eventually the same thing happens (tb detect a server timeout). Tb only does 1 retry so nothing more happens.

I can't tell exactly what is happening since you need something like wireshark to see all the tcp transactions. I tried to duplicate this on my system by slowing down the network interface some but never saw a "timeout". I did see one other problem: with a slow network all the messages are fetched OK and it takes more than 10m. When the fetch completes, tb then does an imap "check" command but the "stream" got "closed" and the check fails. (Tb does imap check on 10 min interval.) This causes tb to retry the fetches and it fails again due to the "check". This also results in "blank" messages in LFs like you see. But the blank messages are really duplicates due to the retry. (I suspect your blanks are duplicates due to retries too but not 100% sure.) I can fix this by changing the hardcoded imap check interval from 10m to something bigger, but not a good solution so need to look closer at this too.

So you might try your copy again with mailnews.tcptimeout set to something bigger (use the config editor in advanced settings). Maybe 1000 seconds would be OK. Be sure to restart tb if you change this so it takes effect. Then again, I'm not 100% sure your server will ever answer or respond again so this might not help.

Also, I have made a modified ESR 68 build that you can run and it addresses the issue described in comment 52 so that if the source folder has offline store it is used rather than having to fetch it from the server. It is here:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&collapsedPushes=525525&revision=8e3022b16b5eb4b314546b11393bf9455f6f5455&selectedJob=287405213
Click on the green "B" next to Windows 2012 and you will see items appear below. Then click on "Job details" and there are various items you can select. I think the one you want is called "target.installer.exe". This will install the 68 version with my small change. If you run this you should set the mailnew.tcptimeout back to default 100. Here's a direct link to target.installer.exe: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZhVdaIpISRKv3qgXzGnvmQ/runs/0/artifacts/public/build/install/sea/target.installer.exe So if you can let me know if this works I would greatly appreciate it.

I'm getting netowrk timeouts trying to get the exe.
I will modify the tcptimeout pref to 1000 and test today and post some results when ready!

(In reply to kris.szlesinski from comment #59)

I'm getting netowrk timeouts trying to get the exe.

The .exe downloads almost instantly from here. Guess I'm a lot closer to the mozilla server.

I will modify the tcptimeout pref to 1000 and test today and post some results when ready.

I have my doubts that will fix it but maybe it will. But probably not exactly the same problems but it helped for this: https://bugzilla.mozilla.org/show_bug.cgi?id=538375#c135 . Hopefully you can eventually get the .exe and try it too.

It was a firewall issue, I have downloaded the exe from edge device successfully. It does bypass imap redownload, thus solving our problem with copying/moving.
I tested also moving mails with Archive function and it runs smoothly so far too.

If I get a chance I'll do some more tests with weaker connections, hotspots as well, but I am sure the solution is already covered by your modified version.

Hoping for mainstream release in the future!

It was a firewall issue, I have downloaded the exe from edge device successfully.

I wonder if possibly the timeouts I see in the log are also due to the firewall? Maybe the imap server is outside of the firewall so you sometimes see the timeouts when downloading from the dovecot imap server too?

Anyhow, glad to hear that the exe seems to fix the issue with tb not using the local store. I will submit the patch for formal approval.

I am submitting a patch described in comment 57 and partially confirmed to work in comment 61. The patch is over in Bug 505456. It almost fixes this bug but does not address the blank messages that become present in the destination Local Folder that can occur when a retry occurs due to network imap fetching errors. Therefore, I don't consider this a true duplicate of Bug 505456.

See Also: → 505456
Component: Folder and Message Lists → Networking: IMAP
Depends on: 505456
Product: Thunderbird → MailNews Core
Summary: Emails moved to local folders all completely blank → Emails moved to local folders all completely blank related to timeout
Whiteboard: [closeme 2020-01-11] → [dupme?]
See Also: → 462156
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: