Forwarding out of filter rules result in SMTP timeout if email doesn't end with <CR><LF>

UNCONFIRMED
Unassigned

Status

defect
UNCONFIRMED
4 years ago
Last month

People

(Reporter: c.nickel, Unassigned)

Tracking

42 Branch
x86_64
Windows 7

Firefox Tracking Flags

(Not tracked)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0
Build ID: 20151029151421

Steps to reproduce:

I created a filter rule, that forward an email to another recipient.
The email body is base64 encoded.



Actual results:

Thunderbird tries to send the email, but runs into a SMTP timeout.

With a sniffer tool, I see that thunderbird tries to forward the email decoded.
To signal the end of the DATA command, thunderbird only gives a ".<CR><LF>", but it should be "<CR><LF>.<CR><LF>". So the mail server gives a timeout, because it is still waiting for "end of data".


Expected results:

Email should be forwarded.
OS: Unspecified → Windows 7
Hardware: Unspecified → x86_64
Are you suggesting this is a regession?
If so, can you find the one day regression range?
Flags: needinfo?(c.nickel)
I don't know, if it is a recession.

But I can give you some futher information:

Thunderbird seems to decodes the base64 coded mail for filtering the body.

The problem occurs, when the original email source code, generated by the sender, doesn't end with <CR><LF>. This seems to happen with some automate generated emails like paypal.
The generated email source code will be base 64 encoded and the email will be send to the recipient.

If I forward this email manually, there will be no problem, because the email is base64 encoded.

Now, I try to filter this email. The email body will be decoded and Thunderbird tries to forward the email. Because of the missing <CR><LF> at the end of the decoded email, the SMTP server runs into a timeout.

In my opinion, there are two solutions.

Solution a):
Thunderbird attaches <CR><LF>.<CR><LF> at every email. If the decoded email doesn't end with <CR><LF> the email could be forwarded anyway.

Solution b):
After filtering the body, Thunderbird encodes the email again and forward it to the recipient.
Flags: needinfo?(c.nickel)
I confirm the continuing existence of this bug in Thunderbird 52.3.0.
Summary: Forwarding out of filter rules result in SMTP timeout → Forwarding out of filter rules result in SMTP timeout if email doesn't end with <CR><LF>
This looks like the root cause of the problem I am having with Thunderbird 52.6.0 (64 bit) on macOS 10.11.6

The emails in question are notifications from 23andme which are base64 encoded.
I can manually forward them without a problem, but a filter rule does not forward them. After a period of time there is a timeout message from the smtp server. No sent message appears in the Sent folder.

A workaround is to change mail.forward_message_mode = 0
(i.e. forward messages as an .eml attachment, not inline)
However, this is not satisfactory for me due to unreliability in being able to read such .eml attachments.

This is still a valid bug as of TB 60.4.0 on Ubuntu 16.04.

TB 60.6.1(32bit on Windows10) still has this bug. And bgz's workaround [mail.forward_message_mode = 0] still working.

(In reply to hirameki from comment #6)

TB 60.6.1(32bit on Windows10) still has this bug. And bgz's workaround [mail.forward_message_mode = 0] still working.

sorry. not TB 60.6.1. TB 60.5.1 is correct.

(In reply to hirameki from comment #7)

(In reply to hirameki from comment #6)

TB 60.6.1(32bit on Windows10) still has this bug. And bgz's workaround [mail.forward_message_mode = 0] still working.

sorry. not TB 60.6.1. TB 60.5.1 is correct.

hirameki, Can you or other reporters on this bug attach a minimal sized email that has this problem. That will make it a lot easier to debug. Or you can just send it to me if you prefer. Thanks.

(In reply to bgz from comment #4)

The emails in question are notifications from 23andme which are base64
encoded.
I can manually forward them without a problem, but a filter rule does not
forward them. After a period of time there is a timeout message from the
smtp server. No sent message appears in the Sent folder.

Thanks, I received the example .eml files via email. It appears every line ends in LF instead of the rfc2822 standard CRLF. Not sure if base64 encoding should really matter since it's all text, just unreadable. Anyhow, I will try to forward these via a filter and see what happens.

I haven't been able to duplicate the problem yet. I think the mail maybe has to come from 23&me or paypal or whatever with just /n for end of lines and fetched as-is into a mailbox. When I append the .eml to a mailbox, tb seems to "repair" the file and put /r/n on the lines so when it is forwarded or sent to another account all is OK, even via a filter.

Assuming the problem you see is due to line ending /n instead of /r/n, it may depend on the imap server containing the mailbox (folder) where you are doing the forward from. Can you tell my the type of imap and smtp server you are seeing the problem on? For example, dovecot, gmail, yahoo, or whatever. (You might be able to tell by looking at an IMAP or SMTP log, see below.)

Also, assuming the problem is line endings, tb just sends the email file as-is for the most part to the smtp server. It assumes that the last line of the email will end with /r/n so, after the file is sent to the smtp server, tb just sends a terminating "./r/n". So the effective terminating sequence might be just "\n.\r\n" instead of the correct "\r\n.\r\n".

Again, assuming the problem is the /n, tb can be changed so it sends the full terminating sequence "\r\n.\r\n" and not assume that the last line of the file already ends in \r\n. I didn't see any ill effects when I made this code change when I sent emails. (It may cause an extra blank line at the end of some text only emails, not sure.)

Another thing that might help would be for you to produce and attach at the link above an smtp log that records the activities when you attempt to forward the email and it fails. This may show where in the send sequence the timeout occurs. We are assuming it is due to the ending sequence but maybe not. See https://wiki.mozilla.org/MailNews:Logging. Like this for windows:

set MOZ_LOG=SMTP:5,timestamp
set MOZ_LOG_FILE=%USERPROFILE%\Desktop\smtp.log
"%ProgramFiles(x86)%\Mozilla Thunderbird\thunderbird.exe"

Note: You can record imap by changing SMTP to IMAP above, or you can record both together into the same log like this:
set MOZ_LOG=SMTP:5,timestamp,IMAP:5
Imap log data tends to be a bit more verbose.

bgz, I got your email with the smtp log. Looks like tb tries to send the terminating period but nothing happens after that. Do you see an error pop up or anything listed in activity manager about send failure?

So if there's a timeout, it is after the period is sent, it seems. So apparently the smtp server is waiting for more data.

I don't see anything in the smtp log about server type. But I am interested in knowing the IMAP server type since the problem might be due to the mailbox the filter is forwarding from, probably not the smtp process itself. You might be able to tell the imap server type by looking at the imap log for responses to imap command ID or possibly CAPABILITY responses (toward the end of the capability list).

There is another recent bug where failure to send the QUIT command after the period was reported. I haven't been able to duplicate that either: Bug 1529009. But no more info received from the reporter.

I will look closer at the file with base64 since you say only it gives you problems.

I get the following pop-up after about 2-3 minutes from running the filter rule:

Send Message Error
Sending of the message failed.
The message could not be sent because the connection to Outgoing server (SMTP) mail.gol.com
timed out. Try again.

I'll email you with some more info from logs but it looks like the IMAP servers are IMAP4rev1

I'm looking for something more like this; Tb sends the server its version ID and then the server, if it's nice, responds with its (lines may not be contiguous in log):

[26398:Unnamed thread 0x7270fdf7d700]: I/IMAP 0x7270fba8b000:mobile.charter.net:A:SendData: 5 ID ("name" "Thunderbird" "version" "60.4.0")
[26398:Unnamed thread 0x7270fdf7d700]: I/IMAP 0x7270fba8b000:mobile.charter.net:A:CreateNewLineFromSocket: * ID ("name" "Email Mx" "version" "M.9.00.023.01.5 201-2473-194" "vendor" "Openwave Messaging" "support-url" "http://support.openwave.com" "date" "Wed Nov 14 12:28:00 EST 2018")
[26398:Unnamed thread 0x7270fdf7d700]: I/IMAP 0x7270fba8b000:mobile.charter.net:A:CreateNewLineFromSocket: 5 OK ID completed

The IMAP4rev1 is just the imap RFC version that the server conforms to. Almost all say that. The server type also sometimes appears after the capability string is sent, but not always. The ID response is better but not all servers support ID command (not nice).

I sent you the full IMAP log as an email attachment.
I can see Thunderbird announcing it's version number but no version information coming back.

@gene smith were you able to decipher this from @bgz 's log?

(In reply to bgz from comment #14)

I sent you the full IMAP log as an email attachment.
I can see Thunderbird announcing it's version number but no version information coming back.

@gene smith were you able to decipher this from @bgz 's log?

I can't find the imap log sent via email from reporter bgz. I did find a message dated 2019-03-02 from bgz but it was empty (edit the next day: now see it, must have been a network problem.). Anyhow, not sure how helpful the imap log would be anyhow. bgz, if you still have the imap log you sent, it is probably better to just attach it using the link above on this bugzilla page.

I did work on this some back in late Feb and early March but other thing came up and haven't gotten back to it. I will mark this as "need info" from myself as a reminder. I do have the sample emails from bgz that have the problem.

I suspect that the problem is due to email lines not ending in the required /r/n. However, tb should probably be more tolerant of malformed emails when trying to send/forward them.

Flags: needinfo?(gds)

FYI I am still here. I believe I had reproduced the issue on Windows 7 with a different mail server. I didn't upload the logs here because I am not very clear what information is buried in them - thinking of privacy/security aspect.
I had put one of the logs on OneDrive and given a link at the bottom of my email of 2-Mar-19 05:34 (GMT)... that link is still active.

bgz, I now see the 3-2-2019 email today. Also the link to the imap log (labeled smtp.log) works too. But I don't see anything about the actual imap server type or version in the CAPABILITY or ID imap responses. But that's OK.

In the 3-2 email you said you were forwarding a message from Local Folders using a filter rule. If you haven't already send me the .eml that caused the problem, please do. Via the OneDrive link is fine now that I have the link. Thanks.

gene,

In an email of 22-Feb-19 I attached two .eml files. One was from macos and one from Windows 7. At least one of those is believed to cause the problem (at some point I think 23andme changed the formatting of their emails such that I think the problem no longer occurs with more recent emails).

In case you can't find that email I have put the two .eml files in the same folder as the log on OneDrive (I guess you can see the entire "Public Videos" folder). If you can't get them from there then please let me know.

I also put a file "About the eml files.txt" in that folder which contains the text from my email of 22-Feb.

bgz, Yes, I see the .eml files on the onedrive link. So are you saying that the problem no longer occurs for you since the 23/me has changed their format?

But if still a problem let me ask some more details:

  1. Problem only occurs for (old format) 23/me emails?
  2. When email originally received via imap, I assume it is received in INBOX. If not, which folder.
  3. Is the email moved or copied via filter or manually form inbox to another folder? Imap folder, or Local Folder?
  4. When you manually forward the message and it works OK, which folder is it in? Imap or local?
  5. When you forward using a rule, which folder is the message in? Again, imap or local folder?
  6. Are you forwarding to a remote person or to another account you have setup in tb?
  7. Any other details that might be helpful?

Note: If forwarding to a another imap account in tb, it might be better to just MOVE the message using the filter rule. This will avoid using SMTP and will instead will just do an imap APPEND to the destination mailbox and it is less sensitive to malformed emails. Of course, this is not relevant if forwarding to a remote account.

Flags: needinfo?(gds)

(In reply to Christoph Nickel from comment #2)

The Bug is confirmed on :

  • 60.8.0 (32 bits) on Windows 10 Pro 1903 (64bits)
  • 68.0b4 (32 bits) on Windows 10 Pro 1903 (64bits)

If I send this mail as Text (from WebMail or TB), thunderbird will FAIL to forward it.
If I send this mail as HTML (from WebMail or TB), thunderbird will forward it.

I think it's linked with https://bugzilla.mozilla.org/show_bug.cgi?id=161806

(In reply to landsteph from comment #21)

(In reply to Christoph Nickel from comment #2)

The Bug is confirmed on :

  • 60.8.0 (32 bits) on Windows 10 Pro 1903 (64bits)
  • 68.0b4 (32 bits) on Windows 10 Pro 1903 (64bits)

If I send this mail as Text (from WebMail or TB), thunderbird will FAIL to forward it.
If I send this mail as HTML (from WebMail or TB), thunderbird will forward it.

Can you attach, provide a link to or maybe send me the .eml file you refer to as "this" mail? Thanks.

I think it's linked with https://bugzilla.mozilla.org/show_bug.cgi?id=161806

Maybe.

@ gene smith
Wrong syntax, my mistake.
A simple mail with just "Test" as Subject, and "Test" as body will do the job...
The first one as plaintext => FAIL
The second one written in color (html) => OK

the workaround above : "mail.forward_message_mode = 0" is still working, but is not satisfying.

FWIW the linked workaround from the linked bug fixed the issue for me: https://bugzilla.mozilla.org/show_bug.cgi?id=161806#c6

" As a workaround, set "mail.file_attach_binary = true" "

(In reply to landsteph from comment #23)

@ gene smith
Wrong syntax, my mistake.
A simple mail with just "Test" as Subject, and "Test" as body will do the job...
The first one as plaintext => FAIL

I just forwarded in tb a text-only message and it worked. Are you saying the problem is windows specific?

(In reply to IV2KBMoFxYIA from comment #24)

FWIW the linked workaround from the linked bug fixed the issue for me: https://bugzilla.mozilla.org/show_bug.cgi?id=161806#c6

" As a workaround, set "mail.file_attach_binary = true" "

I think this will cause the "bad" email to be forwarded as an attachment and not try to send it inline. May not be "satisfying" either.

Once I did the mail.file_attach_binary = true setting I also changed the forwarding to be inline. Now it does the inline forwarding without the error message. For my purposes I have nothing else that this ticket would accomplish for me. Give it a try. It may sufficient for your needs too.

You need to log in before you can comment on or make changes to this bug.