583192 - fail to send attachments in Windows 7 64 bit

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: SMTP, defect)

Description

[Alexey]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.7) Gecko/20100713 Thunderbird/3.1.1

In GUI during delivering mail you will see that Thunderbird hangs for about 2 minutes on status "delivering mail" , progress 98 % . After it you will get message "Sending of message failed.
The message could not be sent because the connection to SMTP server <mail server name> timed out. Try again or contact your network administrator."

During sending mail attachments Thinderbird forget to send CR\LF. 


This is exaple what send Thunderbird 

      0x0090:  447a 3765 5167 374f 5874 2f69 446f 342f  Dz7eQg7OXt/iDo4/
        0x00a0:  4437 4943 6869 6553 4253 5a57 356b 5a58  D7IChieSBSZW5kZX
        0x00b0:  4a48 4b53 344e 4367 3d3d 0d0a 2d2d 2d2d  JHKS4NCg==..----
        0x00c0:  2d2d 2d2d 2d2d 2d2d 2d2d 3032 3036 3031  ----------020601
        0x00d0:  3031 3038 3034 3037 3034 3031 3036 3032  0108040704010602
        0x00e0:  3038 2d2d 2e0d 0a                        08--...

If you will see on 0x00e0:  3038 2d2d 2e0d 0a you will understand that between 

0x2d 0x2d and 0x2e need to be 0x0d 0x0a. If it not exist mail server fail to detect finishing of mail. Check example which i got using The Bat for same attachment. 

This is example what send The Bat for same attachment. 
        0x0090:  4437 4943 6869 6553 4253 5a57 356b 5a58  D7IChieSBSZW5kZX
        0x00a0:  4a48 4b53 344e 4367 3d3d 0d0a 2d2d 2d2d  JHKS4NCg==..----
        0x00b0:  2d2d 2d2d 2d2d 2d2d 4637 3145 4631 4632  --------F71EF1F2
        0x00c0:  4345 3541 3038 432d 2d0d 0a2e 0d0a       CE5A08C--.....

0x00c0:  4345 3541 3038 432d 2d0d 0a2e 0d0a       CE5A08C--.....

Between 0x2d 0x2d and 0x2e exist 0x0d 0x0a.




Reproducible: Didn't try

Comment 1

[WADA:World Anti-bad-Duping Agency]

>         0x00e0:  3038 2d2d 2e0d 0a                        08--...
> If you will see on 0x00e0:  3038 2d2d 2e0d 0a, you will understand that
> between 0x2d 0x2d and 0x2e need to be 0x0d 0x0a.

How the part is atached to the mail?
Is last [CRLF] written after last close boundary by Tb if the mail is saved by "Send Later"?
What is mail size? (See bug 437494, please)
> MAIL FROM:<[e-mail]> SIZE=??????

Comment 2

[WADA:World Anti-bad-Duping Agency]

Bug 580842 may be relevant.

Comment 3

[Alexey]

(In reply to comment #1)
> >         0x00e0:  3038 2d2d 2e0d 0a                        08--...
> What is mail size? (See bug 437494, please)
> > MAIL FROM:<[e-mail]> SIZE=??????

Tried different sizes. Did experiments with different sizes from 1kb to 200kb of attachment.

Comment 4

[Alexey]

(In reply to comment #2)
> Bug 580842 may be relevant.

Not relevant because i tried just now to send completely new email using button "Write"

Comment 5

[Michael A. Pasek]

Additional data from the failed send would be helpful, particularly the "MAIL FROM:" line and some packets (from both client and server) surrounding the one you included in the description.  

If the send fails on ALL mails with attachments (as implied by comment #3), then additional packet capture data and "MAIL FROM" lines from those attempts would help to determine the pattern of the failures and isolate the cause (and correct it).

Comment 6

[Alexey]

(In reply to comment #1)
> >         0x00e0:  3038 2d2d 2e0d 0a                        08--...
> > If you will see on 0x00e0:  3038 2d2d 2e0d 0a, you will understand that
> > between 0x2d 0x2d and 0x2e need to be 0x0d 0x0a.
> 
> How the part is atached to the mail?
> Is last [CRLF] written after last close boundary by Tb if the mail is saved by
> "Send Later"?
> What is mail size? (See bug 437494, please)
> > MAIL FROM:<[e-mail]> SIZE=??????

mail size 3093 bytes

Comment 7

[Alexey]

(In reply to comment #5)
> Additional data from the failed send would be helpful, particularly the "MAIL
> FROM:" line and some packets (from both client and server) surrounding the one
> you included in the description.  
> 
> If the send fails on ALL mails with attachments (as implied by comment #3),
> then additional packet capture data and "MAIL FROM" lines from those attempts
> would help to determine the pattern of the failures and isolate the cause (and
> correct it).

Mail fails with any attachment 
Data which you requested : 

      0x0000:  4500 0054 6260 4000 7806 3844 4d3d d7bb  E..Tb`@.x.8DM=..
        0x0010:  c143 81c3 ed61 0019 2912 a346 f2df 8f65  .C...a..)..F...e
        0x0020:  5018 fcfc 640d 0000 4d41 494c 2046 524f  P...d...MAIL.FRO
        0x0030:  4d3a 3c65 7269 6461 6e40 3132 636f 6e6e  M:<eridan@12conn
        0x0040:  6563 742e 636f 6d3e 2053 495a 453d 3239  ect.com>.SIZE=29
        0x0050:  3732 0d0a                                72..

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> Mail fails with any attachment 

Within 4KB, so it can't be buffer boundary related issue. And, if your problem occurs on any Tb 3.1.1 user, no one can send mail via SMTP using Tb 3.1.1.  

Do you use addons?
If yes, can you reproduce your problem with thunderbird.exe -safe-mode?
Do you enable outbound mail scanning of your anti-virus software?
Do you use SMTP proxy on your PC or in your environment?

Comment 9

[Alexey]

(In reply to comment #8)
> (In reply to comment #7)
> > Mail fails with any attachment 
> 
> Within 4KB, so it can't be buffer boundary related issue. And, if your problem
> occurs on any Tb 3.1.1 user, no one can send mail via SMTP using Tb 3.1.1.  
> 
> Do you use addons?
No i not use any addons. But for any case i tried to send using "thunderbird.exe -safe-mode" . No changes in behaviour.
> If yes, can you reproduce your problem with thunderbird.exe -safe-mode?
Yes, got same results
> Do you enable outbound mail scanning of your anti-virus software?
Completely removed antivirus from computer and disabled firewall.
> Do you use SMTP proxy on your PC or in your environment?

No i not have. Tested following configurations : 

PC -> NAT gateway -> SMTP server
PC -> SMTP server 

Want to try to downgrade thunderbird to some previous versions and compare results.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Excess two bytes are inserted at somewhere then last [CRLF] is lost?
Can you attach mail data? (Send Later, save in .eml, attach to this bug, not paste)
Can you reproduce problem with "Send Later" + "Send Unsent Messages"?

Comment 11

[Alexey]

Attachment: mail which fail

Comment 12

[Alexey]

(In reply to comment #10)
> Excess two bytes are inserted at somewhere then last [CRLF] is lost?

If i add somewhere 1 or 2 chars behaviour same. 

> Can you attach mail data? (Send Later, save in .eml, attach to this bug, not
> paste)
> Can you reproduce problem with "Send Later" + "Send Unsent Messages"?

Yes. Tried just now. And got same results.

Comment 13

[Alexey]

Attachment: small tpdump file

This file created on SMTP server. using command line parameters for tcpdump 
./tcpdump -n -s 1500 -w /mailbug

Comment 14

[Michael A. Pasek]

'tcpdump' seems to indicate that TB is not ensuring the "period on a line by itself" is on a line by itself.  That is, the closing "CRLF" for the attachment "boundary" is missing, so the "2E 0D 0A" isn't on a line by itself.

From RFC2046, section 5.1.1:
   The Content-Type field for multipart entities requires one parameter,
   "boundary". The boundary delimiter line is then defined as a line
   consisting entirely of two hyphen characters ("-", decimal value 45)
   followed by the boundary parameter value from the Content-Type header
   field, optional linear whitespace, and a terminating CRLF.

Alexey: Can you get simultaneous 'tcpdump' or Wireshark captures from the Thunderbird client and the SMTP server ?  It's possible that the "CRLF" for the boundary is being inadvertently "stripped" by some intervening equipment/process -- possibly while converting from one encoding (e.g., "binary") to another (e.g., "base64").

As WADA noted, there is something unique in your environment, or NOBODY would be able to send attachments.

Comment 15

[Michael A. Pasek]

The problem appears to be that described in Bug 463129.  WADA, do you concur ?

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> The problem appears to be that described in Bug 463129.  WADA, do you concur ?

I don't think so.
As seen in attached tcp dump, mail is very simple multipart/mixed mail.
  text/plain part with null lines only.
  small application/octet-stream part with base64 encoded.
I also think TCP dump at client side is mandatory data for diagnosis.

Alexey(bug opener):
If mail data is saved by "Send Later", Tb 3.1.1 generates next lines at bottom.
> CmZzZA0KZg0Kc2QNCmYNCmJpbgAEAAAA[CRLF]
> --------------090309030807000208070803--[CRLF]
Can you check with next crafted mail data?
> CmZzZA0KZg0Kc2QNCmYNCmJpbgAEAAAA[CRLF]
> --------------090309030807000208070803--[CRLF]
> [CRLF]
> abcdefghijklmn[CRLF]
> [CRLF]
> [CRLF]
(1) Compose mail, Send Later, Terminate Tb
(2) Edit "Unsent Messages" under "Local Folders", add [CRLF] etc. at bottom
(3) Delete "Unsent Messages.msf", Restart Tb, View mail in Outbox folder
(4) Send Unsent Messages

Comment 17

[Michael A. Pasek]

(In reply to comment #16)
> I don't think so.
> As seen in attached tcp dump, mail is very simple multipart/mixed mail.
>   text/plain part with null lines only.
>   small application/octet-stream part with base64 encoded.

But the base64 part, like Bug 463129, Bug 326303, and bug 523796, does not have a trailing CRLF on the "close-delimiter" -- like the 'tcpdump' data attached to this bug report.

> If mail data is saved by "Send Later", Tb 3.1.1 generates next lines at bottom.
> > CmZzZA0KZg0Kc2QNCmYNCmJpbgAEAAAA[CRLF]
> > --------------090309030807000208070803--[CRLF]

The data may be SAVED with the trailing CRLF, but it is not transmitted with the trailing CRLF:
Original (description) tcpdump:
>         0x00c0:  4345 3541 3038 432d 2d0d 0a2e 0d0a       CE5A08C--.....
Attached tcpdump:
>         0x00b0:  3038 3037 3038 3033 2d2d 2e0d 0a         08070803--...

In both cases, there is no trailing CRLF for the "close-delimiter", so TB's "period on a line by itself" isn't on a line by itself -- and the timeout occurs while the receiver waits for the end of data.

Comment 18

[Michael A. Pasek]

(In reply to comment #17)
> Original (description) tcpdump:
> >         0x00c0:  4345 3541 3038 432d 2d0d 0a2e 0d0a       CE5A08C--.....

Oooops! Pasted wrong line -- that was from "The Bat" (correct) transmission; TB's was:
>        0x00e0:  3038 2d2d 2e0d 0a                        08--...

Comment 19

[Alexey]

> Can you check with next crafted mail data?
> > CmZzZA0KZg0Kc2QNCmYNCmJpbgAEAAAA[CRLF]
> > --------------090309030807000208070803--[CRLF]
> > [CRLF]
> > abcdefghijklmn[CRLF]
> > [CRLF]
> > [CRLF]
> (1) Compose mail, Send Later, Terminate Tb
> (2) Edit "Unsent Messages" under "Local Folders", add [CRLF] etc. at bottom
> (3) Delete "Unsent Messages.msf", Restart Tb, View mail in Outbox folder
> (4) Send Unsent Messages

I am not sure is i did all correct. But this mail successfully delivered. Check in attachment tcpdump session.

Comment 20

[Alexey]

Attachment: krafted mail success

This is krafter success mail.
Done with following steps : 

1) write empty mail.  Filled address To:. Added empty attachment. Menu File-> Send Later.
2) Closed Thunderbird
3) In ....Thunderbird/Profiles/....Local Folders/. 
Opened for edit file Unsent Messages . Added at the end :
[CRLF]
abcdefghijklmn[CRLF]
[CRLF]
[CRLF]
4) Saved. Remove file Unsent Messages.msf.
5) Started Tb. opened for view in Outbox. Send unsend messages.

Comment 21

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #20)
> krafted mail success

Last data in tcp dump.
> --------------040404040303080803080206--[CRLF]
> [CRLF]
> abcdefghijklmn[CRLF]
> .[CRLF]
Last two consecutive [CRLF]s in Outbox mail folder is lost when mail data arrived at your SMTP server.
If Tb 3.1.1 doesn't send last [CRLF] of last next line in Outbox folder,
> --------------040404040303080803080206--[CRLF]
any user of Tb 3.1.1 should not be able to send mail via any SMTP server.
It looks for me that someone eats [CRLF](s) in your environment.

Can you check with Gmail's SMTP, wiith "Send Later+Send Unsent Messages"?
Get an free Gmail account, add Gmail's SMTP to your Tb's SMTP definitions, and add an identity for Gmail's mail address to any account, please.

Comment 22

[Michael A. Pasek]

"Crafted" message (and success) differs from failed message (and from bugs mentioned in Comment #17) because it does not have a base64-encoded part, and that base64-encoded part is not at the end of the message.

If you make a "crafted" message like that of Comment #20 (only text/plain parts), but with only a single CRLF following "..klmn", is that message successfully sent, or is final CRLF "eaten" (and timeout occurs) ?

Was tcpdump (attachment 462337 [details]) taken at SMTP server or at TB client ?  If tcpdump taken at TB client, no need to do GMail account setup and test because the CRLF is being stripped at client; GMail test would show same data as previous test.

Comment 23

[Alexey]

> Can you check with Gmail's SMTP, wiith "Send Later+Send Unsent Messages"?
> Get an free Gmail account, add Gmail's SMTP to your Tb's SMTP definitions, and
> add an identity for Gmail's mail address to any account, please.

Checked with gmail smtp. With gmail mail server it works. But i cannot do tcpdump because it use TLS.

Comment 24

[Alexey]

(In reply to comment #22)
> "Crafted" message (and success) differs from failed message (and from bugs
> mentioned in Comment #17) because it does not have a base64-encoded part, and
> that base64-encoded part is not at the end of the message.
> 
> If you make a "crafted" message like that of Comment #20 (only text/plain
> parts), but with only a single CRLF following "..klmn", is that message
> successfully sent, or is final CRLF "eaten" (and timeout occurs) ?
> 

        0x0310:  2d2d 2d2d 3031 3035 3038 3034 3039 3032  ----010508040902
        0x0320:  3030 3036 3037 3030 3036 3032 2d2d 0d0a  000607000602--..
        0x0330:  0d0a 6162 6364 6566 6768 696a 6b6c 6d6e  ..abcdefghijklmn
        0x0340:  0d0a 2e0d 0a         

Look like it forget to place [crlf] between -- and . only in case if attachment is finishing part of mail.

> Was tcpdump (attachment 462337 [details]) taken at SMTP server or at TB client ?  If
> tcpdump taken at TB client, no need to do GMail account setup and test because
> the CRLF is being stripped at client; GMail test would show same data as
> previous test.

In capture on client side . Same incorrect behavior of thunderbird. 
0   41 41 0d 0a 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d  AA..------------
0610   2d 2d 30 38 30 36 30 30 30 37 30 36 30 31 30 31  --08060007060101
0620   30 32 30 31 30 30 30 37 30 39 2d 2d 2e 0d 0a     0201000709--...

Comment 25

[Alexey]

Attachment: succes dump with base64 transfer encoding

same steps but binary attachment

[CRLF]
abcd.....
[CRLF]
[CRLF]

Sent success

Comment 26

[Alexey]

Attachment: succes dump with base64 transfer encoding(1crlf)

same steps but binary attachment

[CRLF]
abcd.....
[CRLF]

Sent success

Comment 27

[Alexey]

Attachment: Failed mail for control with same attachment(no modifications)

Control mail with same attachment.

Comment 28

[Alexey]

Attachment: Failed mail without 1st CRLF

This mail created with same attachment, but with following modification. 

abcd[CRLF]

Send fails.

Comment 29

[Alexey]

Attachment: succes dump with base64 transfer encoding(extra finish crlf)

This dump created with following modification: 

abcd.....[CRLF]
[CRLF]

Send success.

Comment 30

[Michael A. Pasek]

I expected all of those results EXCEPT the "Failed mail without 1st CRLF".  Why would not having 2 CRLF's between the base64 attachment and the "epilogue" cause TB to strip off the trailing CRLF of the epilogue ???

I tried to figure out where data for that case would be processed, but the code I've been able to find handles an entire file -- it could care less about MIME, boundaries, etc.,  and would send the trailing "CRLF" of the saved "Send Later" message if it were in the file.  It does not, however, ensure that the entire message ends with CRLF; it just calls PostDataFinished (if that's the function that's handling this data) to send the ".CRLF".

This still looks like 463129, et al., to me.

Questions:
1) Q: Should an RFC5322 message end with CRLF ?
   A: Yes, if it is to be transmitted using SMTP (RFC5321, 2.3.7).
      Although RFC5322 refers to "lines", it does not define that term
      as explicitly as RFC5321.
2) Q: Does an RFC5322 message containing RFC2046-encapsulated parts
      require a CRLF as part of the "close-delimiter" ?
   A: This is ambiguous in RFC2046 (text vs BNF).

Comment 31

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

Some additional questions and some comments:
1. Are you using the official 32 bit Windows build from mozillamessaging.com? there are no official 64 bit Windows builds  which means unofficial 64 bit builds are not supported. I am 99% sure you are using the official 32 bit build but have to be 100% certain :-)
2. I tested "mail which fail" i.e. https://bugzilla.mozilla.org/attachment.cgi?id=462046 and it worked fine from Windows 7 32 bit
3. Is there any other testing on Windows 7 32 bit you would like me to do? If so, please let me know!

Comment 32

[WADA:World Anti-bad-Duping Agency]

Alexey(bug opener):

(Q1) Mail data stream passed to SMTP by Tb is held in %temp%\nsemail.eml(or nsemail-N.eml) while Tb is sending(deleted after Cancel of sending).
Copy %temp%\nsemail.eml while sending doesn't complete.
Lack of last [CRLF] is seen in this file?

(Q2) Can you reproduce your problem with new profile, with clean Tb code?
0. Download thunderbird-3.1.3pre.en-US.win32.zip, and UNZIP it.
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2010-08-04-00-comm-1.9.2/
1. Start Tb 3.1.3pre with thunderbird.exe -p, and create a new profile.
2. Start Tb 3.1.3pre with the new profile, change unwanted settings for test.
   Disable Gloda, Disable update check, ...
3. Define an dummy POP3 account. (Stop immediately, "Manual Config")
4. Define two SMTPs. Your SMTP(o SSL), Gmail's SMTP(SSL)
5. Define two identities with your mail address.
   A : use your SMTP, B: use Gmail's SMTP
6. Send mail to yourself using identity A and B
To see SIZE= on MAIL FROM, get SMTP log.
> https://wiki.mozilla.org/MailNews:Logging
> SET NSPR_LOG_MODULES=timestamp,smtp:5
If you use DebugView, you can check SMTP log ad-hoc(see bug 402793 comment #6.)

Check with new profile/unzipped Tb 3 is to surely rule out add-on's bug and problem due to your special settings around mail sending(auto BCC:, signature, wrap length, ...).
As for mail data stream(SMTP level) passed to SMTP server by Tb's SMTP code, there is no difference between your SMTP case and Gmail's SMTP case. So, if Tb's SMTP code doesn't send last [CRLF], your problem should occur with Gmail's SMTP server which uses SSL too.

Comment 33

[Alexey]

(In reply to comment #31)
> Some additional questions and some comments:
> 1. Are you using the official 32 bit Windows build from mozillamessaging.com?
> there are no official 64 bit Windows builds  which means unofficial 64 bit
> builds are not supported. I am 99% sure you are using the official 32 bit build

I am using official 32-bit Thunderbird under Windows 7 64-bit. 

> but have to be 100% certain :-)
> 2. I tested "mail which fail" i.e.
> https://bugzilla.mozilla.org/attachment.cgi?id=462046 and it worked fine from
> Windows 7 32 bit
may be bug related to microsoft implementation for running 32-bit in 64-bit environment ? 
> 3. Is there any other testing on Windows 7 32 bit you would like me to do? If
> so, please let me know!

Comment 34

[Alexey]

Attachment: This is nsemail file(tb 3.1.1)

But in tcpdump i see 

        0x0320:  2d2d 2d2d 2d2d 2d2d 2d2d 2d30 3030 3830  -----------00080
        0x0330:  3430 3230 3530 3030 3030 3130 3930 3730  4020500000109070
        0x0340:  3230 362d 2d2e 0d0a                      206--...

Comment 35

[Alexey]

(In reply to comment #32)
> Alexey(bug opener):

> (Q2) Can you reproduce your problem with new profile, with clean Tb code?
> 0. Download thunderbird-3.1.3pre.en-US.win32.zip, and UNZIP it.
> > http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2010-08-04-00-comm-1.9.2/
> 1. Start Tb 3.1.3pre with thunderbird.exe -p, and create a new profile.
> 2. Start Tb 3.1.3pre with the new profile, change unwanted settings for test.
>    Disable Gloda, Disable update check, ...
> 3. Define an dummy POP3 account. (Stop immediately, "Manual Config")
> 4. Define two SMTPs. Your SMTP(o SSL), Gmail's SMTP(SSL)
> 5. Define two identities with your mail address.
>    A : use your SMTP, B: use Gmail's SMTP
> 6. Send mail to yourself using identity A and B
> To see SIZE= on MAIL FROM, get SMTP log.
> > https://wiki.mozilla.org/MailNews:Logging
> > SET NSPR_LOG_MODULES=timestamp,smtp:5
> If you use DebugView, you can check SMTP log ad-hoc(see bug 402793 comment #6.)
> 
Reproduced. Got same results. See next attachment with smtp.log

> Check with new profile/unzipped Tb 3 is to surely rule out add-on's bug and
> problem due to your special settings around mail sending(auto BCC:, signature,
> wrap length, ...).
> As for mail data stream(SMTP level) passed to SMTP server by Tb's SMTP code,
> there is no difference between your SMTP case and Gmail's SMTP case. So, if
> Tb's SMTP code doesn't send last [CRLF], your problem should occur with Gmail's

> SMTP server which uses SSL too.

Not sure. May be Gmail smtp server not strictly require [CRLF].[CRLF]. Due to encryption not possible to check.

Comment 36

[Alexey]

Attachment: smtp.log

Comment 37

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #34)
> This is nsemail file(tb 3.1.1)

File size=799 bytes, and date/from is next;
Date: Thu, 05 Aug 2010 09:11:45 +0200
From: Alexey <[e-mail]>

(In reply to comment #36)
> smtp.log

2010-08-05 07:54:03.900000 UTC - 0[1029140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=790

Above are data of same send test?

Comment 38

[WADA:World Anti-bad-Duping Agency]

Two mail send existed in smtp log.
(1) to smtp.gmail.com
> 2010-08-05 07:52:53.949000 UTC - 0[1029140]: SMTP Connecting to: smtp.gmail.com
> 2010-08-05 07:52:54.105000 UTC - 0[1029140]: SMTP Response: 250-mx.google.com at your service, [77.61.215.187]
> 2010-08-05 07:53:00.860000 UTC - 0[1029140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=791
(2) to 193.67.129.195
> 2010-08-05 07:53:37.739000 UTC - 0[1029140]: SMTP Connecting to: 193.67.129.195
> 2010-08-05 07:54:03.869000 UTC - 0[1029140]: SMTP Response: 220 stun.12connect.com ESMTP
> 2010-08-05 07:54:03.900000 UTC - 0[1029140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=790

Same mail data is used in above two mail send? Can you check copy in Sent mail folder after above test?
If same sent mail in Sent(==nsemail.eml+BCC: etc.), why SIZE is different in two cases?

Comment 39

[Alexey]

(In reply to comment #38)
> Two mail send existed in smtp log.
> (1) to smtp.gmail.com
> > 2010-08-05 07:52:53.949000 UTC - 0[1029140]: SMTP Connecting to: smtp.gmail.com
> > 2010-08-05 07:52:54.105000 UTC - 0[1029140]: SMTP Response: 250-mx.google.com at your service, [77.61.215.187]
> > 2010-08-05 07:53:00.860000 UTC - 0[1029140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=791
> (2) to 193.67.129.195
> > 2010-08-05 07:53:37.739000 UTC - 0[1029140]: SMTP Connecting to: 193.67.129.195
> > 2010-08-05 07:54:03.869000 UTC - 0[1029140]: SMTP Response: 220 stun.12connect.com ESMTP
> > 2010-08-05 07:54:03.900000 UTC - 0[1029140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=790
> 
> Same mail data is used in above two mail send? Can you check copy in Sent mail
> folder after above test?
> If same sent mail in Sent(==nsemail.eml+BCC: etc.), why SIZE is different in
> two cases?

I not know why it is different. It is exactly same mail. used same POP3 identify . But different SMTP identifies. 1) SMTP is GMAIL SMTP with SSL 2) normal smtp server. I only switched what smtp identify to use. 
I did extra check to be 100 % sure and updated nsemail.eml and smtp.log. See next attachemnt.

Comment 40

[Alexey]

Attachment: new smtp.log

Comment 41

[Alexey]

Attachment: new nsmail.eml

Comment 42

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #41)
> new nsmail.eml

File size=868 bytes
> Date: Thu, 05 Aug 2010 09:42:18 +0200
> X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.9pre) Gecko/20100804 Lanikai/3.1.3pre

nsemail(-N).eml in %temp% when sending is in progress(not complete)?
Did Tb send X-Mozilla-Draft-Info: to SMTP server?

(In reply to comment #40)
> new smtp.log

(1) Your SMTP
> 2010-08-05 11:18:36.664000 UTC - 0[3029140]: SMTP Connecting to: 193.67.129.195
> 2010-08-05 11:19:02.763000 UTC - 0[3029140]: SMTP Response: 220 stun.12connect.com ESMTP
> 2010-08-05 11:19:02.794000 UTC - 0[3029140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=795
(2) Gmail's SMTP
> 2010-08-05 11:20:48.106000 UTC - 0[3029140]: SMTP Connecting to: smtp.gmail.com
> 2010-08-05 11:20:48.200000 UTC - 0[3029140]: SMTP Response: 220 mx.google.com ESMTP z55sm94911eeh.9
> 2010-08-05 11:20:52.350000 UTC - 0[3029140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=793

Please attach next in addition to smtp log for SIZE in MAIL FROM: check.
(1) nsemail(-N).eml in %temp% while send doesn't complete with your SMTP.
(2) Sent mail copy in local Sent folder after mail send with Gmail's SMTP.
    (Save as .eml, and attach, please)

Please note that we are not checking mail structure, string in Subject:, data in mail body or attachment and so on. We are checking mail data length and transfered mail data around last [CRLF]. Please keep same data/same length including [CRLF]. If you prepare a clean test mail, you can easiliy send same data/same length, by adjusting next headers, unless you touch Subject:. mail body text, attachment data, by "Edit As New", From: selection, and "Send".
> Message-ID: <[e-mail]>
> Date: Thu, 05 Aug 2010 09:42:18 +0200
> From: ppppp <[e-mail]>
> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.9pre) Gecko/20100804 Lanikai/3.1.3pre
> To: [e-mail]
Length of Date: is not changed.
Length of Message-ID depends on @xxx.yyy.zzz of From: you selected upon send.
  As two identities for same your mail address, length is not changed
  by SMTP server switch to test.
Length of To: depends on recipient's mail addr.
  As same recipient is used in test, length is always same in your test.
Length of From: depends on pppp, ep before @, xxx.yyy.xxx after @.
  As two identities for same your mail address, length is not changed
  by SMTP server switch to test, unless you change length of ppppp part.
Length of User-Agent: depends on Tb's version.
  As Tb 3.1.1 and Tb 3.1.2pre generates different length,
  mail data length need to be adjusted, if absolutely same mail data is
  required in test, by changing ppppp part or by changing line length of a
  mail body text line.
  Clean mail data for Tb 3.1.1 and Tb 3.1.2pre are better prepared separately.

Comment 43

[Alexey]

Attachment: arch with files

In this archive : 

nsemail.eml - failed mail through normal smtp 
stmp.log - log of both attempts
testmail_template.eml - Saved mail from Sent folder

Comment 44

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #43)
> arch with files
> In this archive : 
> nsemail.eml - failed mail through normal smtp 
> stmp.log - log of both attempts
> testmail_template.eml - Saved mail from Sent folder

Both are sent with SIZE=809, and size of both mail data file is 809 bytes.

stmp.log:
> 2010-08-05 13:06:33.165000 UTC - 0[2e2d140]: SMTP entering state: 3[CRLF]
> 2010-08-05 13:06:33.165000 UTC - 0[2e2d140]: SMTP Send: MAIL FROM:<[e-mail]> SIZE=809[CR][CRLF]
> 2010-08-05 13:06:33.196000 UTC - 0[2e2d140]: SMTP Send: RCPT TO:<[e-mail]>[CR][CRLF]
> 2010-08-05 13:06:33.227000 UTC - 0[2e2d140]: SMTP Send: DATA[CR][CRLF]
> 2010-08-05 13:06:33.258000 UTC - 0[2e2d140]: SMTP Send: .[CR][CRLF]
All log of "SMTP Send:" ends with [CR][CRLF].
[CR] may be first byte of really sent [CRLF], which is generated by SMTP logging. But the [CR] may be really passed data by "SMTP Send:" to other module.
Newline character may depend on server(Tb may send newline which server sent).
If latter, it may be a cause of lack of [CRLF] with your SMTP server.

Comment 45

[Alexey]

(In reply to comment #44)
> All log of "SMTP Send:" ends with [CR][CRLF].
> [CR] may be first byte of really sent [CRLF], which is generated by SMTP
> logging. But the [CR] may be really passed data by "SMTP Send:" to other
> module.
> Newline character may depend on server(Tb may send newline which server sent).
> If latter, it may be a cause of lack of [CRLF] with your SMTP server.

So it is bug or not bug ? And what extra information you need ?

Comment 46

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #45)
> So it is bug or not bug ? And what extra information you need ?

Stand alone [CRLF] followed by [CRLF] was seen in SMTP log I took on 2007/08/11 and in other recent IMAP logs, even though really sent data should ends with [CRLF].
> Note: smtp.ops.dti.ne.jp is non-SSL server I use.
> 0[2c48f8]: SMTP Connecting to: smtp.ops.dti.ne.jp[CRLF]
> 0[2c48f8]: SMTP entering state: 0[CRLF]
> 0[2c48f8]: SMTP Response: 220 smtp11.dti.ne.jp ESMTP DTImail 3.11s; Sat, 11 Aug 2007 17:51:14 +0900 (JST)[CRLF]
> 0[2c48f8]: SMTP entering state: 15[CRLF]
> 0[2c48f8]: SMTP Send: EHLO [192.168.0.10][CR][CRLF]
So, I think the stand alone [CRLF] followed by [CRLF] for "SMTP Send:" log is merely a bug of logging of Tb(Tb/NSPR failed to write last [LF] in sent data), because 0x0d0a is correctly logged in your TCP dumps and other TCP dumps attached to other bugs.
However, I'm not sure that Tb's SMTP code correctly passes final [CRLF] to other modules such as Socket layer and that the stand alone [CRLF] in NSPR log is a bug of logging code instead of problem in Tb's SMTP code or non-SSL socket code.

Can you check with official 32bits Tb 3.1.1 on 32 bits MS Windows?

Can you check with 64bits Tb 3.x on current 64 bits MS Windows 7?
> http://getsatisfaction.com/mozilla_messaging/topics/thunderbird_keeps_reopening_windows_7_pro_64_bit
> http://www.mozilla-x86-64.com/
> http://wiki.mozilla-x86-64.com/Thunderbird:Download
Note:
Only installer build is available. Please be careful not to corrupt your current Tb 3.1.1 environment for daily use.

Comment 47

[WADA:World Anti-bad-Duping Agency]

FYI.
Following is web page found by google search for "windows 7" crlf cr lf.
> http://www.codinghorror.com/blog/2010/01/the-great-newline-schism.html
According to the page, there is "Unicode Line Separator"(LS=U+2028)) in addition to tradissional crlf/lf/cr for newline character.
> http://www.fileformat.info/info/unicode/char/2028/index.htm
Following is an example of API of Java for LineSeparator conversion.
http://bits.netbeans.org/dev/javadoc/org-netbeans-modules-editor-util/org/netbeans/lib/editor/util/CharacterConversions.html

"Unicode Line Separator"(LS=U+2028) is used by nsBidiPresUtils.cpp. 
> http://mxr.mozilla.org/comm-central/source/mozilla/layout/base/nsBidiPresUtils.cpp#736
If last [CRLF] handling by Tb is affected by Bidi support, and if MS Win7's LineSeparator handling around U+2028 is affected by API used by application and settings of OS(such as Locale), this bug's problem only in your environment may be explained, although I'm absolutely uncertain.

Can you check additional test case with Tb 3.1.1, with your SMTP?
  Is your problem affected by html5.enable=false(Tb 3.1's default)/true?
No need of data. I don't think relevant to your problem, but please rule out it.

By the way, current my questions are.
(1) Why problem occurs in your environment only?
    If problem occurs on many users, the many users can't send any mail via
    non-SSL SMTP, unless multiple [CRLF]s are put at bottom of mail data.
(2) Why with your SMTP only(non-SSL)? (no problem with Gmail's SMTP of SSL)
    I'm still suspecting issue by "outbound mail scanning or proxy", 
    because of "problem only on non-SSL, no problem on SSL".
(3) What is partiqularity of your environment?
    64bits Win7? Win7? Locale of your Win7? 32bits Tb3 on 64bits Win7? 
Please note that your problem can't be seen in Bug 437494. In any report of Bug 437494, last [CRLF] in mail data is sent as expected. Problem is loss of a part of final .[CRLF](0x2d0d0a) only, when there is no room for final .[CRLF] in last 4KB buffer. There is no difference between non-SSL and SSL.

Questions.
When did your problem start to occur?
(upgrade of Tb, install of add-on, patch apply on Win7, ...)
The official 32bits Tb 3.1.1 is your first use of Tb on your 64bits Win7?
Can you check with other on-SSL SMTP server?

Comment 48

[WADA:World Anti-bad-Duping Agency]

FYI.
GMX provides free mail service too. (by Google search for "free smtp server non-ssl").
> http://www.iopus.com/guides/bestpopsmtp.htm
According to next page, SMTP of GMX doesn't seem to support SSL.
> http://forum.gmx.com/forum/posts/list/80.page#2448

Comment 49

[Alexey]

(In reply to comment #47)
> Can you check additional test case with Tb 3.1.1, with your SMTP?
>   Is your problem affected by html5.enable=false(Tb 3.1's default)/true?
> No need of data. I don't think relevant to your problem, but please rule out
> it.

Tested with html5.enable=false and html5.enable=true. Not see any changes. Btw upgraded to 3.1.2 TB. Same behaviour.
> 
> By the way, current my questions are.
> (1) Why problem occurs in your environment only?
>     If problem occurs on many users, the many users can't send any mail via
>     non-SSL SMTP, unless multiple [CRLF]s are put at bottom of mail data.
> (2) Why with your SMTP only(non-SSL)? (no problem with Gmail's SMTP of SSL)
>     I'm still suspecting issue by "outbound mail scanning or proxy", 
>     because of "problem only on non-SSL, no problem on SSL".

Configured on my mail server SSL. Mails still no go through with same sympthoms. Look like that gmail server not require [CRLF].[CRLF]
> (3) What is partiqularity of your environment?
>     64bits Win7? Win7? Locale of your Win7? 32bits Tb3 on 64bits Win7?

My environven Win 7 64 bit English. 32 Bit official TB 3.1.2. Locale for non unicode - Russian(Russia). Format : Russian(Russia) . Location : Russia . Problem started from version 3.0.x. 
 
> Please note that your problem can't be seen in Bug 437494. In any report of Bug
> 437494, last [CRLF] in mail data is sent as expected. Problem is loss of a part
> of final .[CRLF](0x2d0d0a) only, when there is no room for final .[CRLF] in
> last 4KB buffer. There is no difference between non-SSL and SSL.
> 
As i tested. In my case also no diffrence between SSL or no SSL. 

> Questions.
> When did your problem start to occur?
> (upgrade of Tb, install of add-on, patch apply on Win7, ...)
> The official 32bits Tb 3.1.1 is your first use of Tb on your 64bits Win7?

I use TB on windows x64 already more than year. I even not remember from what version i started .

> Can you check with other on-SSL SMTP server?

Checking now.

Comment 50

[Alexey]

(In reply to comment #48)
> FYI.
> GMX provides free mail service too. (by Google search for "free smtp server
> non-ssl").
> > http://www.iopus.com/guides/bestpopsmtp.htm
> According to next page, SMTP of GMX doesn't seem to support SSL.
> > http://forum.gmx.com/forum/posts/list/80.page#2448

Just now registered and configured email account on gmx.com. You were right it is possible to use non-SSL smtp (ie port 25). So i got same behaviour as i got with any another smtp servers. 
So for now only GMAIL smtp server can accept such mails.

Comment 51

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #49)
> Configured on my mail server SSL. Mails still no go through with same
> sympthoms. Look like that gmail server not require [CRLF].[CRLF]

> As i tested. In my case also no diffrence between SSL or no SSL. 

It's first report about SSL SMTP server other than Gmail's SMTP by you.
According to next page, Yahoo's SMTP server requests SSL.
> http://techblissonline.com/yahoo-pop3-and-smtp-settings/
Can you check with Yahoo's SMTP server who requests SSL?

Comment 52

[WADA:World Anti-bad-Duping Agency]

Can you check Process Monitor log for file I/O by Tb to nsemail(-N).eml?
> http://technet.microsoft.com/en-us/sysinternals/bb545027.aspx
> http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
> Filter : Process Is thunderbird.exe, Include
>          Path End with .eml, Include
> Select approarite column via Options/Select Column
Does Tb reads all data Tb wrote to nsemail(-N).eml?
Does problem occur even when size of nsemail(-N).eml is greater than 4096?
(I could reproduce Bug 437494 with 8190=4096+4094, but couldn't reproduce with 4094.)

Comment 53

[Michael A. Pasek]

Just found this snippet in RFC5321 (discussing the "period on a line by itself"):
  An extra <CRLF> MUST NOT be added, as that would cause an empty line to 
  be added to the message.  The only exception to this rule would arise if the 
  message body were passed to the originating SMTP-sender with a final "line"
  that did not end in <CRLF>; in that case, the originating SMTP system MUST 
  either reject the message as invalid or add <CRLF> in order to have the 
  receiving SMTP server recognize the "end of data" condition.

Thus, if the "base64" attachment IS missing the final CRLF, the SMTP layer MUST add a CRLF -- although it's probably TB that's stripping it off.  There should probably be a check -- before message transmission begins -- that the message DOES end with a CRLF.

Comment 54

[WADA:World Anti-bad-Duping Agency]

Alexey(bug opener), can you get SMTP log and Socket log?
> https://wiki.mozilla.org/MailNews:Logging
> https://developer.mozilla.org/en/HTTP_Logging
> SET NSPR_LOG_MODULES=timestamp,smtp:5,nsSocketTransport:5,nsHostResolver:5
> For mail of same SIZE=, with non-SSL SMTP, with SSL SMTP(Gmail).

Tb's SMTP and socket layer code requests send of mail data using 4KB buffer after receive of '354 Enter mail, end with "." on a line by itself'
In test o Bug 437494, following can be observed.
(1) Size=8190=4096+4094: 2*4096 bytes is sent. last LF in .CRLF is lost
    calling PR_Write [count=4096], PR_Write returned [n=4096]
    calling PR_Write [count=4096], PR_Write returned [n=4096]
(2) Size=8191=4096+4095: 2*4096 bytes is sent. last CRLF in .CRLF is lost
    calling PR_Write [count=4096], PR_Write returned [n=4096]
    calling PR_Write [count=4096], PR_Write returned [n=4096]
(3) Size=8192=4096+4096: 2*4096 bytes is sent. last .CRLF is lost
    calling PR_Write [count=4096], PR_Write returned [n=4096]
    calling PR_Write [count=4096], PR_Write returned [n=4096]
(4) Size=8193=4096+4097: 2*4096 bytes + 4 bytes is sent. final .CRLF is sent
    calling PR_Write [count=4096], PR_Write returned [n=4096]
    calling PR_Write [count=4096], PR_Write returned [n=4096]
    calling PR_Write [count=4],    PR_Write returned [n=4]
Total bytes of PR_Write should be Size + 3 bytes(for added .CRLF).

In your case, Total bytes of PR_Write is Size+3? Or Size+1?
If Size+3, Tb/Socket passes CRLF before .CRLF to TCP normally.
If Size+1, Tb/Socket doesn't pass CRLF before .CRLF to TCP.

Comment 55

[Bernardo Boncompagni]

I have the same exact symptoms: when I try to send an e-mail with an attachment it times out with message

"Sending of message failed.
The message could not be sent because the connection to SMTP server <mail
server name> timed out. Try again or contact your network administrator."

after the bar reaches 98% (for larger attachments, the bar reaches smaller percentages). This happens both on SSL (tested on gmail) and non-SSL servers (my server). I can send e-mails without attachments. I am on Windows 7 Pro 64 bit, fresh install.

Comment 56

[Wayne Mery (:wsmwk)]

Alexey, Bernardo are you seeing this with current version 13 or newer?

Comment 57

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #56)
> Alexey, Bernardo are you seeing this with current version 13 or newer?

Comment 58

[Bill Dee]

This is STILL occurring, although seemingly somewhat randomly. (?)
Sometimes it works, and sometimes it doesn't.
I just added two images to my signature, but they are web images,
so I don't know why Tbird would try to attach them.
But, if I remove the images, it sends no problem.

I tried the fix someone suggested, setting network.tcp.sendbuffer to 65536.
That didn't fix it. So it must be internal to Thunderbird.

Isn't anyone going to fix this???

Windows 7 Home Premium 64bit SP1
Thunderbird 17.0

Comment 59

[Alexey]

On version 17.0.4 not see that issue anymore.

Comment 60

[David Lechner (:dlech)]

I just ran into these same symptoms. It turned out to be a networking problem as suggested by the getsatisfaction link above and not a problem with Thunderbird. My network was working, but much slower than it should have been and dropping out randomly. I upgraded my network driver and the problem went away.

Comment 61

[Wayne Mery (:wsmwk)]

WFM per comment 59

Comment 62

[Bill Dee]

Nice for you that it apparently works for you. Although I wouldn't trust anything you say.

WHY is someone who obviously doesn't know what the hell they're talking about or doing allowed to set the status as "Resolved - Works for Me"???

That's the most ridiculous, absurd, and asinine thing I've EVER seen when it comes to support and development.

If we tried to get away with that in the private sector, we would be laughed at, scorned, admonished, and possibly even fired.

No wonder TBird and Ffx have SO MANY issues and bugs and they too-rarely get corrected.

WTF???

Comment 63

[Wayne Mery (:wsmwk)]

I'll accept that you are frustrated. But don't be surprised when others don't tolerate your attitude. 

Alexsey is the original reporter. He reports HE no longer sees the problem, based on HIS original bug report. When that occurs, in the absence of material evidence in the code contrary to the reporter's finding, we tend to close the bug.

That doesn't mean there are not other problems still occurring for other people. File a well described new bug or support topic please.
- http://getsatisfaction.com/mozilla_messaging/topics/
- https://bugzilla.mozilla.org/enter_bug.cgi?product=Thunderbird