1741748 - When copying a message (imap APPEND) to Zimbra imap server, if first line is "From " it confuses Zimbra

Status: NEW

(Thunderbird :: General, defect)

Description

[gene smith]

For testing of copying a huge folder between servers, a few years ago a tb user sent me a very large mbox file saved from tb that I imported into Local Folders and then copied to gmail imap server. Everything looked ok on gmail at that point. I've copied this email set to several imap servers from gmail and, once I got a successful copy, the messages all looked correct at the destination. Now recently I tried copying the same messages from gmail to a locally running Zimbra imap server. The messages all seem to arrive OK and are all present but some the Date values in the summary list for the Zimbra account show only the time of the copy and no day/month/year of the actual message.

In addition, this seems to cause Zimbra to skip UIDs which are normally consecutive and contiguous numbers assigned to messages when they are appended.

I think the lack of complete Date and the UID issue has something to do with the "From " and "X-Mozilla-Keys" headers that are somehow present in the gmail source messages derived from Local Folders. I only see these on gmail messages that I copied from Local Folders and not on gmail messages that come in normally.

After copying 200 gmail messages from this set to the local zimbra server, to obtain the information for the email summary, tb asks Zimbra for a set of message headers that includes the DATE header and zimbra responds like this for each of the 200 message with multiple lines of data:

Zimbra: * 43 FETCH (UID 33384 RFC822.SIZE 11410 BODY[HEADER.FIELDS (FROM TO CC BCC SUBJECT DATE MESSAGE-ID PRIORITY X-PRIORITY REFERENCES NEWSGROUPS IN-REPLY-TO CONTENT-TYPE REPLY-TO)] {404}
Zimbra: From 
Zimbra: Date: Thu, 16 Jun 2016 17:50:13 -0400
Zimbra: (Other requested headers sent...)
:

Note that zimbra always sends the not-requested "From " line first. The 2nd line returned above is the correct "Date" header but for other messages the order of the returned headers varies and the 2nd one is usually not "Date" but something else with "Date" header later on. It appears that the date displayed in the summary is only shown as the current time when Zimbra returns the "Date" header right after the bogus "From ".

For example this works OK and doesn't corrupt the summary:

Zimbra: * 3 FETCH (UID 33344 RFC822.SIZE 16163 BODY[HEADER.FIELDS (FROM TO CC BCC SUBJECT DATE MESSAGE-ID PRIORITY X-PRIORITY REFERENCES NEWSGROUPS IN-REPLY-TO CONTENT-TYPE REPLY-TO)] {415}
Zimbra: From 
Zimbra: From: "Guy Lombardo" <[e-mail]>
Zimbra To: "[e-mail]" <[e-mail]>
Zimbra: Date: Wed, 17 Apr 2019 16:20:59 +0000
:

In this case, the bogus "From " is still transmitted by Zimbra but the "Date" header is not right after the "From " and the message summary displays OK with the correct date. The "From" in the summary is also OK even though it was transmitted right after the bogus "From ".

Comment 1

[gene smith]

Attachment: From-fix.diff

I found a possible fix for this that still allows imap APPEND of messages that start with the
From delimiter that got left in after a copy from tb Local Folders. It seems that only Zimbra server doesn't like this but zimbra does not flag an error but just silently ignores the "Date:" header and also skips UIDs when multiple messages are copied/APPENDed to a zimbra mailbox.

All this does is change the first line of the transmitted message if it starts with From to X-X: which is a header of the same length that will be ignored. I first tried to just strip off the "From " line and not send it. But the message length, which is already calculated, encoded and sent, was then wrong so the Zimbra server rejected the APPEND. So I just changed the bogus "From " into a legal header without changing the message length and it fixes the problems, i.e., no more bad dates in the summary and skipped UIDs.

I still think Zimbra should tolerate the leading "From " line like other servers, so I'll report this issue to them.

Comment 2

[gene smith]

This also affects icloud when messages are appended with a leading "From " line. It causes no header to be returned when fetched after new messages with the leading "From " have been appended and detected. This results in empty summary line showing only the current time/date. The message body and header appear OK when the empty summary is line is opened into preview pane but summary line remains blank even after folder repair.
The From-fix.diff patch above also fixes this for icloud imap.

Comment 3

[Rafael Rivera]

Proton also appears to have a RFC2822 strict IMAP server, rejecting Thunderbird messages w/ mbox From lines. Seems like this should be cleaned up in Thunderbird and not compliant IMAP servers right?