237828 - CRLF missing on the last line if body gets encoded with base64

Status: RESOLVED DUPLICATE of bug 242718

(MailNews Core :: Backend, defect)

Description

[JongAm Park]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/124 (KHTML, like Gecko) Safari/125.1
Build Identifier: Mozilla Thunderbird 0.5+ (20040313)

The heading of the bug 147383 is somewhat misleading. 
I added what I found to the  bug 14738,but it doesn't seem to get people's attention.

When you send email to a person whose email address is hotmail, hanmail.net, and yahoo, and if you 
write somewhat long text in Korean, probably CJK, it just hangs by displaying "delivering" dialog box.

On Windows platform, some nightlies didn't have this problem, recently. Generally speaking, the 
Thunderbird before 1.6 code of the Mozilla didn't have this problem.
Sometimes, on Windows, with some nightlies this problem is solved unintentionally, I think, because 
there is no mentioning about this problem in fixed list.

However this problem still exists, and it is very annoying problem.

Please don't say that Hotmail has their own mechanism, or "you can't send with hotmail's SMTP server" 
or "you can't receive on Hotmail account", thing.

This is a bug, "You can't send email to their account using your own ISP's smtp server.".

I have no idea if the Mozilla Suite has this problem nowadays. Because I only use the thunderbird.
However, I think that the Thunderbird shares the code.

FYI, Apple's Mail, and MS's Outlook express don't have problem in sending to hotmail, hanmail, yahoo 
accounts.

 

Reproducible: Always
Steps to Reproduce:
1. Choose a person's email address, which ends with @hotmail.com, @yahoo.com, @yahoo.co.kr, 
@hanmail.net. (hanmail is very big portal in Korea. Daum is another name of the hanmail. )
2. Write somewhat long email. Please read bug 147383 about this.
3. Try to send email
Actual Results:  
"Delivering" Dialog box appears and just hang there.


Expected Results:  
It should send email nicely.

Comment 1

[Christian Eyrich]

The SMTP server you're using is Yahoo SBC? Didn't you try any other?
Occurs this problem only with recipients addresses on any of the named domains,
really no other?

Please attach a log of a failed and one of a succeeded sending action (by using
"Create a New Attachment" above). Please see
http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#smtp how to
create such a log.

Comment 2

[JongAm Park]

(In reply to comment #1)
> The SMTP server you're using is Yahoo SBC? Didn't you try any other?
> Occurs this problem only with recipients addresses on any of the named domains,
> really no other?
> 

The SMTP servers I use are :

  - smtp.sbcglobal.yahoo.com
  - smtp.usc.edu
  - smtp.myrealbox.com

I have three accounts for the three smtp servers.

FYI, I have set every authentication setting in right way.
Other email clients have no problem in sending to hotmail, hanmail, yahoo accounts.

> Please attach a log of a failed and one of a succeeded sending action (by using
> "Create a New Attachment" above). Please see
> http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#smtp how to
> create such a log.

I'm sorry. I tried it before.  With Windows version of the Thunderbird, the log can be captured.
However on Macintosh it was not possible.
The explanation about MacOS X seems to be wrong. You can't drag & drop the shell script onto the 
Thunderbird, or open the shell script by the Thunderbird.

OK. I will add a log file which is generated on the Windows platform.
Please wait for a few hours.

Comment 3

[JongAm Park]

Attachment: Working correctly on Windows (again!)

With March.18 version of Thunderbird nightlies, it starts to work again on
Windows.

So, I attach the normal log file which is for when it is working correctly.

Comment 4

[JongAm Park]

Please let me know how to store the log on MacOS X.

The explanation page you gave me can't be helpful.
I tried the Unix way and the Mac way. Both can't save the log file.

On Windows, it seems to work from time to time. However, on MacOS X,
it doesn't.

Comment 5

[Christian Eyrich]

Hm, I can't say anything about MacOS X. It might be that the D&D doesn't work
with TB, just with the Suite.

What would be more useful than a Moz log is a tcpdump log (tcpdump -s 0 -w
logfile), this tool is available for (and maybe even included in) MacOS X.

Comment 6

[JongAm Park]

Attachment: tcpdump on MacOS X

tcpdump -s 0 -w thunderbird.log -i ppp0 dst host 67.127.99.106 and tcp port 25
or udp port 25

Comment 7

[JongAm Park]

Attachment: Another TCPdump from/to

sudo tcpdump -s 0 -w thunderbird2.log -i ppp0 host 67.127.99.106 and tcp port
25 or udp port 25

Comment 8

[Christian Eyrich]

Thanks for the logs though the first one is quite useless because of "dst host",
so it don't show what you sent out.

But the problem might be the same as in the second log, there's an CRLF missing
before the final period in the message body. This CRLF.CRLF sequence is needed
by the SMTP server to recognize the mail end.

Up to Mozilla 1.5 we added an additional CRLF to each mail before the period.
But since the message body should contain an CRLF itself, this was superfluous
and I removed it in bug 163783.

So maybe you ran into an situation were we don't have a final CRLF at the
message bodies end.
The question now is, what data do you send? Is there some pattern when the error
occurs and when not?
And why does this only happen if the recipient is at hotmail, hanmail.net, and
yahoo (to repeat my question regarding this from comment #1, it's only with
recipients with this domains?).

Comment 9

[JongAm Park]

(In reply to comment #8)

> But the problem might be the same as in the second log, there's an CRLF missing
> before the final period in the message body. This CRLF.CRLF sequence is needed
> by the SMTP server to recognize the mail end.
> 


OK. Then I will try recapturing it again. However it probably is not my fault.
The Thunderbird just hangs while it try to send email to the 3 accounts.
After sometime, it displays "time out" error message. Because I didn't want to
wait for a long time, I just cancel the delivery. I waited it for 1~2 minutes, anyway.
So, the log was for that duration.

What I sent it just normal Korean message, which can be sent without problem with Apple's mail, MS's 
Outlook express. That same messages can be sent with the Thunderbird with pre-Mozilla 1.6 codes.
It can be sent with current version of Thunderbird on Windows.
However I said that even on windows, it can be sent and can't be sent depending on nightlies.

When you write the message text a little longer than a threshold length, then it sees to fail in sending 
email. 

How can a user strip off the CRLF? I don't use any filter.  Users just push return keys when they need
new line. So, probably the thunderbird treats the CRLF in wrong way.

If I remember this correctly, this bugs started to appear with Mozilla 1.6 codes, and at that time for 
increasing performance, some "string" thing was improved.

OK. I will recapture the tcpdump log, and at this time I will wait until the timeout error message is 
displayed.

Comment 10

[JongAm Park]

Attachment: Time out error dialog box

This is what you get when you wait for a long time.

Comment 11

[Christian Eyrich]

I didn't assert it's the users fault. There must be a situation/a content that
causes Mozilla/TB to produce a mail body without a CRLF at behind the last line.

If you just send normal text (using the plain editor), you should be able to
work around the problem by just type one, two newlines after the last line.
But if you're attaching files (resp. Mozilla encoded your text in one) or using
the HTML editor that doesn't work.

As I wrote, we inserted an additional CRLF at the message end up to and
including 1.5, so the problem doesn't occur in this versions (and TB based on it).

I asked for the data you're sending because thousands of users don't have the
problem, so I think there must be anything special you're doing.
And because the message body in your second log only consists of an attachment.
Does your text always get encoded and sent as an attachment? You can see this if
you inspect some other mails source code.
I'm interested if
1. all the problematic mails have an attachment
2 [details] [diff] [review]. the others (i.e. mails with text length under the threshold) don't
3. mails composed with Apple's mail and M$ OE have and attachment (though have
the same content as the ones composed in TB)

I don't want you to fix the problem, developers will do that. But to do so, we
need some more informations to see something like a pattern. Searching hundrets
of function for a mysterious error doesn't work.


The timeout occurs because the server doesn't get the end of message CRLF.CRLF
and waits and waits.

Comment 12

[JongAm Park]

(In reply to comment #11)
> I didn't assert it's the users fault. 

:) I'm sorry if I made you think that you said that it was the users' fault.
I didn't mean it. 

> If you just send normal text (using the plain editor), you should be able to
> work around the problem by just type one, two newlines after the last line.
> But if you're attaching files (resp. Mozilla encoded your text in one) or using
> the HTML editor that doesn't work.
> 

OK. I will try it when I go back home. 
Probably my signature doesn't contain CRLF after its last character.
I didn't attach any files. Probably messages written in CJK may sent as attached
files.
(MIME?) I would like to check it also.

However if it is due to my signature file, it is also a problem. Whether a
signature file
contains the CRLF or not at the end of the signature, the emailer should attach
CRLF to the
end of the message, which include the signature.

> As I wrote, we inserted an additional CRLF at the message end up to and
> including 1.5, so the problem doesn't occur in this versions (and TB based on it).
> 

Oh.. You did? However I noticed this problem from the Thunderbird with Mozilla
1.6 codes.
Not alpha.. but beta something.

> I asked for the data you're sending because thousands of users don't have the
> problem, so I think there must be anything special you're doing.
> And because the message body in your second log only consists of an attachment.
> Does your text always get encoded and sent as an attachment? You can see this if
> you inspect some other mails source code.

OK...

> I'm interested if
> 1. all the problematic mails have an attachment
> 2. the others (i.e. mails with text length under the threshold) don't
> 3. mails composed with Apple's mail and M$ OE have and attachment (though have
> the same content as the ones composed in TB)
> 

OK. I will check them also.

I hope this could make the Thunderbird better! :)

However today I should write an audio codec. So.... Probably I could test them
on Sunday.
Have a nice weekend!!! 

> I don't want you to fix the problem, developers will do that. But to do so, we
> need some more informations to see something like a pattern. Searching hundrets
> of function for a mysterious error doesn't work.
> 
> 
> The timeout occurs because the server doesn't get the end of message CRLF.CRLF
> and waits and waits.

Comment 13

[Christian Eyrich]

> I didn't attach any files. Probably messages written in CJK may
> sent as attached files.
> (MIME?) I would like to check it also.

Yep, that would be interesting.

> However if it is due to my signature file, it is also a problem.
> Whether a signature file contains the CRLF or not at the end of
> the signature, the emailer should attach CRLF to the end of the
> message, which include the signature.

Of course. And Mozilla does.
I only meant that you can't work around the problem by manually adding a newline
at the message if you have a signature after it.
But I missed that you can of course manually add a newline to the signature
after it has beens pasted in.

Anyway, I don't think normal text in the editor isn't a problem, so this
shouldn't be necessary.

> However I noticed this problem from the Thunderbird with Mozilla
> 1.6 codes.
> Not alpha.. but beta something.

That can be. The patch was checked in 2004-10-16, the 1.6a alpha tree might have
been branched at this time and so the alpha still had the extra CRLF.

> However today I should write an audio codec.

Oh, sounds more interesting than hunting a missing CRLF. :)
Take the time you need for the checks. Better well done than quick.

Comment 14

[JongAm Park]

How do you do? Finally, I send another tcpdump.
I captured it until the thunderbird displayed 'can't deliver email' dialog.

> There must be a situation/a content that
> causes Mozilla/TB to produce a mail body without a CRLF at behind the last line.

Well.. I used the same signature file on Windows and Mac.
However on Windows, the current Thunderbird sends the message well, while on
Mac, it is not.

The signature file has CRLF at the last line of the file.
The file is saved as a DOS format. So, the new line is CRLF, not CR or LF.d


> If you just send normal text (using the plain editor), you should be able to
> work around the problem by just type one, two newlines after the last line.
> But if you're attaching files (resp. Mozilla encoded your text in one) or using
> the HTML editor that doesn't work.

I didn't use the built-in HTML editor or the messages was not sent as
an attached file. ( I should state that sometimes it sends! )
What is strange is that although the signature file contains CRLF, the Thunderbird
doesn't add CRLF to the end of the message. it strips off the CRLF. 

Now, I put new line by pressing "Enter key" at the last line of the signature
part in the email message. But it can't send the message also.

And here is my answer to the each question.

> 1. all the problematic mails have an attachment

NO. They don't contain attachment.

> 2. the others (i.e. mails with text length under the threshold) don't

No. They don't.

> 3. mails composed with Apple's mail and M$ OE have and attachment (though have
     the same content as the ones composed in TB)

NO. They don't. They are just encoded.

I will attach some screen captures for helping your understanding. ( I hope. :) )

Comment 15

[JongAm Park]

Attachment: RAW email source -- begining part

Comment 16

[JongAm Park]

Attachment: encoded mail ending part

Comment 17

[JongAm Park]

Attachment: Apple Mail Raw Source

Comment 18

[JongAm Park]

Attachment: message used for testing now.

Comment 19

[JongAm Park]

Attachment: new tcpdump log - waited until it displays "can't send the message" dialog box.

I think this log contains all message the thunderbird sent.
I waitd until it displayed the "can't send email" dialog box.
Hope it helps.

Comment 20

[Christian Eyrich]

Strange, very strange. Potentially DOS line endings can cause trouble on Mac.
But since the problem also shows if the complete text including attachment is
base64 coded, I don't think, that's it.

Your newly attached tcpdump log is nearly empty (just contains a few bytes
garbage). Would you please reattach it?

Comment 21

[JongAm Park]

(In reply to comment #20)
> Strange, very strange. Potentially DOS line endings can cause trouble on Mac.
> But since the problem also shows if the complete text including attachment is
> base64 coded, I don't think, that's it.
>

So, I tried save it as Unix format also. But the result of the test was same. It
couldn't send email.

FYI, I should say this. Email accounts other than the 3 email service are OK.
I can send any email to whoever has an account on their email service provider.
However only the threes are in problem.
 
> Your newly attached tcpdump log is nearly empty (just contains a few bytes
> garbage). Would you please reattach it?

Oh.. Really? I can't understand why it happened.
By the way, what tcpdump log analyser do you use?

I will try to attach a new one when I go back home.

Comment 22

[Christian Eyrich]

> FYI, I should say this. Email accounts other than the 3 email service are OK.

Oh, I feared that. Makes it only more strange.

Er, do you always attach your signature? If yes, could you please try without?
I know that the sig shouldn't be the cause if sending to other email services
works. But I want to try everything.

> By the way, what tcpdump log analyser do you use?

Ethereal to show and filter. Analyzing is done in my head. :)

Comment 23

[JongAm Park]

Attachment: tcpdump -s 0 -w thunderbird3.log -i ppp0 <IP> and tcp port 25 or udp port 25

For the log, I sent not-so-long message to my friend. It also included the text
that I received from her. I removed my signature to test what you mentioned.
I also waited until the 'can't send message" dialog box appears.

>> FYI, I should say this. Email accounts other than the 3 email service are
OK.

>Oh, I feared that. Makes it only more strange.

Yes. it is strange. More specifically, it seems to be related to CJK characters
in some way. Because if I write messages in English, the problem is minimal.


> > By the way, what tcpdump log analyser do you use?

> Ethereal to show and filter. Analyzing is done in my head. :)

Well.. yes.. Ethereal is better than pure "thinking" in your head. but..
isn't it difficult to understand the packet information?
Unfortunately, good analysers are mostly commercial programs and expensive.
There seems to be no pure "tcpdump" analyser at freshmeat.net or the
sourceforge.net.
Next time, shall I write some Korean text and give it to you?
Do you think it is a good idea? Then you may test with the text in the file for
yourself.

Always appreciate your devotion to Mozilla project. :) I think Thunderbird
sould replace the outlook express and the Apple mail! :)

Comment 24

[JongAm Park]

I'm seriously suspect "EUC-KR" coding and the misbehaviour.

Today, I sent somewhat long email message written in English to my hotmail
account. There was no problem in sending it at all.

So.. Could you check "EUC-KR" coding if it can disturb the misbehaviour?
As I said before the tendency is more obvious if you write message ( in Korean)
with certain length which exceeds some certain length.

Comment 25

[Christian Eyrich]

I already tried sending a message in EUC-KR myself (I used the text extracted
from your first tcpdump log, inserted in my MailDB and chose "Edit As New") and
also did again now (with text from your last log).

I have no problem sending this. The difference between your mails and mine is,
that the text of yours has Content-Transfer-Encoding: base64 coded, mine goes
out as quoted-printable or 8Bit.

I don't think the charset itself is the problem, it's more the base64 encoding.
Ok, binary attachments aren't affected, but these get a boundary delimiter after
the encoded data.

I know I asked before in comment #11, but please write a mail on Mac with less
than the critical length, test if it goes through and look what
Content-Transfer-Encoding: it has. If you've a non-affected Windows nightly
available, please try the same.

Comment 26

[JongAm Park]

> I have no problem sending this. The difference between your mails and mine is,
> that the text of yours has Content-Transfer-Encoding: base64 coded, mine goes
> out as quoted-printable or 8Bit.
> 
> I don't think the charset itself is the problem, it's more the base64 encoding.

Probably... 

> Ok, binary attachments aren't affected, but these get a boundary delimiter after
> the encoded data.
> 
> I know I asked before in comment #11, but please write a mail on Mac with less
> than the critical length, test if it goes through and look what
> Content-Transfer-Encoding: it has. If you've a non-affected Windows nightly
> available, please try the same.

Well.. I did it before. On a Windows system, I also use MIME base64 message, not
quoted-printable. There is no problem.

Sorry that I can't be helpful.

Comment 27

[JongAm Park]

OOPs! Sorry. On a Windows system, "for messages that
contain 8bit characters, use 'quoted printable' MIME encoding.
Leave unchecked to send the message as is" is unchecked.

However on the MacOS X, it was checked.
So, I unchecked it on the MacOS X, and tested.
It can successfully send messages!

OK. So, it is not the EUC-KR encoding problem.
So, this bug should be "MIME encoding" error?

Comment 28

[JongAm Park]

Yes. Even with the windows version that worked, when I check the 'quotable
print' MIME encoding option, it can't send email to the accounts.

Comment 29

[Christian Eyrich]

Well, I looked up the source the last hour and I have narrowed it down to two
places where I think this happens.
End of nsMsgSendPart::Write() or end of nsMsgComposeAndSend::Clear().

But to verify and to test, I have to be able to generate an base64 coded body,
but I don't get it managed, neither on Windows nor on Linux.

Comment 30

[JongAm Park]

(In reply to comment #29)
> Well, I looked up the source the last hour and I have narrowed it down to two
> places where I think this happens.
> End of nsMsgSendPart::Write() or end of nsMsgComposeAndSend::Clear().
> 
> But to verify and to test, I have to be able to generate an base64 coded body,
> but I don't get it managed, neither on Windows nor on Linux.

I check the preference option, "For messages that contain 8-bit characters, use
'quoted printable' MIME encoding. Leave unchecked to send the message as is." in
"compose" section for the encoding. 

Comment 31

[Christian Eyrich]

Yes of course, that's what you already mentioned. But this option does only what
it should on my system, no base64 in either one. :(
Since you also get base64 on Windows it can't be a MacOS X speciality. As I
wrote by mail, there may be another pref that causes it in association with qp
encoded characters.

Comment 32

[JongAm Park]

Attachment: Send As it is option is applied ( it means that the checkbox is unchecked )

Although the "Leave unchecked to send the message asis" is unchecked, the
thundirbird can't send email.

( On MacOS X )

Comment 33

[David :Bienvenu]

dup of bug 242718, which is fixed.

*** This bug has been marked as a duplicate of 242718 ***