1550929 - Moving lots of messages in offline mode results in garbage uploads after switching back to online mode

Status: NEW

(MailNews Core :: Networking: IMAP, defect)

Description

[Dave Miller [:justdave]]

Thunderbird 60.6.1 (64-bit) (Mac OS X 10.14.4)

Steps to reproduce:

1. Copy 20,000 messages from a Local Folder to an IMAP folder while in offline mode.
2. Switch to online mode
3. Watch as Thunderbird starts uploading the messages.

Expected results:

1. The messages should upload

Actual results:

1. After a minute or so, Thunderbird downloads the contents of the folder from the server, wiping out all of the messages you were trying to upload.
2. Thunderbird continues trying to upload anyway. The resulting messages on the server once it finishes are all gibberish.

Comment 1

[Wayne Mery (:wsmwk)]

Sounds like bug 568167 :(

Comment 2

[gene smith]

Is the number of message 20k significant? Would this happen if only one or two messages were copied? What do you mean by gibberish exactly? Also, are you running the destination imap mailbox having local storage sync'd to server or just storing headers?

Edited addition: Test with imap folder having local storage and copied 1 message while offline and worked OK after going back online. Then tried same test with 5 messages with same good result. Haven't tried 20k messages.
Reporter, how many messages were in the destination folder originally before you did the copy. Also, how are you observing the "upload" and "download"? Do you have an imap log or network trace of the activities?

Comment 3

[gene smith]

(In reply to Wayne Mery (:wsmwk) from comment #1)

Sounds like bug 568167 :(

I just verified that critical/dataloss bug 568167 is valid. However, the STR must be pretty unusual or it would be reported a lot more. I don't see it as a dup of this bug.

Comment 4

[Dave Miller [:justdave]]

By gibberish I mean there were several thousand copies each of 3 or 4 different messages, without any headers or with corrupted headers. Thunderbird was set for offline caching of message content. I'm not sure the 20k is relevant, it's how many we did. The mailbox visibly updated to remove all the messages when it was around 4000 or so into the upload process. (however, only a few dozen messages actually made it to the server intact).

Comment 5

[gene smith]

How many message were in the destination imap folder before you did the copy?

Where are you actually seeing messages w/o headers or corrupted headers or duplicates of the 3 or 4? Are you looking on the server itself or in tb or in the profile ImapMail/ file that tb puts the messages into?

When tb completed the sync, were all the original messages in the destination folder corrupted or missing?

Which messages "made it to the server", the copied files or the original files or a combination? So only 12n out of the 20k and whatever was originally in the mailbox were OK? Were the 12n that made it original emails or copied in emails. (Where n is small number.)

Did the emails in source folder in Local Folders remained OK after all this?

What is the server type, i.e.,dovecot, cyrus, gmail etc?

Why did you do this while offline (just curious)?

Did the copy from Local Folder to imap folder finish before you went online? Did you check that the copied in message were present before switching to online (you can't read the emails but the 20k emails should be in the thread list).

Just trying to understand the details in order to duplicate the problem.

Comment 6

[Dave Miller [:justdave]]

(In reply to gene smith from comment #5)

How many message were in the destination imap folder before you did the copy?

About 15.

Where are you actually seeing messages w/o headers or corrupted headers or duplicates of the 3 or 4? Are you looking on the server itself or in tb or in the profile ImapMail/ file that tb puts the messages into?

In TB itself, in the IMAP folder, both before and after doing a mailbox rebuild from the folder properties.

When tb completed the sync, were all the original messages in the destination folder corrupted or missing?

No, only the ones that had been uploaded, the original 15 that were there to begin with were still intact.

Which messages "made it to the server", the copied files or the original files or a combination? So only 12n out of the 20k and whatever was originally in the mailbox were OK? Were the 12n that made it original emails or copied in emails. (Where n is small number.)

I'm not sure any of them did, I think the only ones intact were the original messages that were already there.

Did the emails in source folder in Local Folders remained OK after all this?

Yes.

What is the server type, i.e.,dovecot, cyrus, gmail etc?

Whatever Dreamhost is running. It was mail.dreamhost.com

Why did you do this while offline (just curious)?

Obviously mistaken impression that it would upload faster if done in a batch (the offline copy from Local Folders certainly went much faster)

Did the copy from Local Folder to imap folder finish before you went online?

Yes, we watched the progress bar at the bottom and didn't touch anything until it was done.

Did you check that the copied in message were present before switching to online (you can't read the emails but the 20k emails should be in the thread list).

Yes.

Just trying to understand the details in order to duplicate the problem.

The actual motivation for doing this was trying to recover lost messages. A family member had gotten a new laptop, and used the migration utilities to copy all of the data from the old laptop to the new one. Another family member was trying to clean the old computer off before sending it in to be recycled and didn't realize that all they had to do was reformat the disk, so they were browsing around the computer deleting files, and ran Thunderbird and started deleting email, not realizing they were also deleting it off the server while doing so. We got the mail back by restoring a time machine backup from prior to the incident, then running Thunderbird in offline mode so it couldn't sync with the server, then copying all of the offline-cached email from the IMAP folders to Local Folders. Then going online, letting it sync (thus re-deleting all the mail that was nuked as it caught up to what was on the server), then copying the mail back to the server from Local Folders. After discovering what was reported in this bug, the copy was tried again with it in online mode, and it succeeded (though took a bit longer to upload that way).

Comment 7

[gene smith]

Ok, thanks for the details. I will look into to this asap with more than just a few files. Based on the other bug that Wayne pointed to, it seems that nothing is really copied, except maybe headers, when a copy from Local Folders to an offline imap folder occurs. Anyhow, glad you didn't lose any emails and got everything back again.