402759 - unable to copy to sent folder ("Message contains bare newlines" response to append of IMAP)

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Henrik Gemal]

when I sent a mail with two attachments I got the "unable to copy to sent folder" and the log said:

3188[32a7260]: 341b858:imap.gmail.com:A:CreateNewLineFromSocket: * STATUS [Gmail]/Sent Mail (MESSAGES 1462 RECENT 0 UIDNEXT 1463 UNSEEN 0)
3188[32a7260]: 341b858:imap.gmail.com:A:PARSER:Internal Syntax Error on line: %s: * STATUS [Gmail]/Sent Mail (MESSAGES 1462 RECENT 0 UIDNEXT 1463 UNSEEN 0)

I've got the error loads of times before :( 

Can I provide you with more information?

Comment 1

[David :Bienvenu]

maybe a bit more of the log around the error would be helpful. I wonder why we're doing a STATUS on the SENT folder - do you have things set to check all folders for new messages?

I wonder if gmail should be putting " " around the folder name.

Comment 2

[Henrik Gemal]

(In reply to comment #1)
> maybe a bit more of the log around the error would be helpful. I wonder why
> we're doing a STATUS on the SENT folder - do you have things set to check all
> folders for new messages?
> 

Yes I have check all folders on.

I'll try to see if I can get the full log

Comment 3

[Henrik Gemal]

Forgot to say that I'm running:
Thunderbird version 3.0a1pre (2007110604)

and it always happens when I try to sent mail with attachments. Not that it always happens, only that I only see the error on mails with attachments.

I've also seen the error on another host than gmail. I've seen it on Cyrus IMAP4

Comment 4

[Henrik Gemal]

I've create a new bug report about the status error, since that's was not what was causing my sent folder problem.
I'll try to reproduce the send folder problem again and will post the log when I get it.

Comment 5

[WADA:World Anti-bad-Duping Agency]

To Henrik Gemal(bug opener):
Is there any other issue than problem of Bug 402772(your the new bug, not quoted space-included-folder-name in STATUS response by Gmail IMAP) in phenomena of this bug?

Comment 6

[Henrik Gemal]

Looking at the log I see:
348[31911b0]: 3191a88:m00.opasia.dk:A:SendData: 9 append "INBOX.Sent" (\Seen) {1778194+}
348[31911b0]: 3191a88:m00.opasia.dk:A:CreateNewLineFromSocket: 9 NO Message contains bare newlines

dupe of bug 301010?

Comment 8

[WADA:World Anti-bad-Duping Agency]

> m00.opasia.dk
> append "INBOX.Sent" (\Seen) {1778194+}
> NO Message contains bare newlines

Original problem of bug 301010 is standalone "LF" in data when copy of a mail from Exchange server to Cyrus IMAP server.
1. FETCH from Exchange server, and "LF" is returned by Exchange server
2. APPEND to Cyrus server, needless to say in pass through mode, then Cyrus complains

Is "Gmail IMAP" involved in your "bare newlines" problem with m00.opasia.dk?

Is the sent mail generated by Thunderbird?
Where the attachments are obtained from? Other IMAP server? Or local file?
If obtained from IMAP server, see IMAP protocol log with editor or tools which can identify/isolate LF, CR, CRLF, LFCR.
If stand-alone LF is included in data, IMAP log will probably becomes as following.
> Log_data_1[LF]Log_data_2[CRLF]  (may be when MS Win only) 
Notepad.exe is possibly useful for this purpose, because note.pad exe supports CR+LF only as newline and above data is displyed in single line, although many text editors display it as two lines.

Comment 9

[Henrik Gemal]

The error occors when I sent a mail and tbrd tried to add the mail to the sent folder on my
"m00.opasia.dk Cyrus IMAP4 v2.2.12-Invoca-RPM-2.2.12-16"
server

It's a normal mail with attachments from the local file system.

Gmail IMAp is NOT involved

Looking at the log I see:
-----------------------

qikofNucCzDSK06L9uZZVsDOaildPj4NCnN0YXJ0eHJlZg0KMjczNTc1DQolJUVPRg0K
--------------010107040407080502000502--


3476[4293750]: 403d038:m00.opasia.dk:A:SendData: 
3476[4293750]: ReadNextLine [stream=3f7f568 nb=37 needmore=0]
3476[4293750]: 403d038:m00.opasia.dk:A:CreateNewLineFromSocket: 6 NO Message contains bare newlines

------------------------------

is the problem the empty "403d038:m00.opasia.dk:A:SendData:" ?

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #9)
> is the problem the empty "403d038:m00.opasia.dk:A:SendData:" ?

I think so.
Can you check sent binary and newline char(s) of the line of log data?

> The error occors when I sent a mail and tbrd tried to add the mail to the sent folder
> It's a normal mail with attachments from the local file system.

> --------------010107040407080502000502--
>
> 
> 3476[4293750]: 403d038:m00.opasia.dk:A:SendData: 

It looks that two null lines after close boundary were sent, and one null line was sent by next SendData.
(Q1) Can problem be re-produced by "Send Later" and "Send Unsent Messages"?
(Q2) Null lines after close boudary are CRLF? LF? CR? Or mixture of them?
(Q3) Will phenomenon be altered by increasing or decreasing of number of null lines after close boundary?
 1. Compose mail, and "Send Later" 
 2. Click mail folder other than "Unsent Messages" folder of "Local Folders"
 3. Edit "Unsent Messages" by text editor who can keep new line character
    used in "Unsent Messages" by Tb.
    ("Sakura editor" which I use can do it)
    And add or remove null line(s) after close boundary line.
 4. "Rebuild Index" of "Unsent Messages" folder
 5. "Send Unsent Messages"

Comment 11

[WADA:World Anti-bad-Duping Agency]

Problem of comment #9 looks to be different from Bug 301010, because all data involved in the problem seems to be generated by Tb ("LF" sent by IMAP server is not involved).
Re-opening.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> imap.gmail.com:A:PARSER:Internal Syntax Error on line:
> %s: * STATUS [Gmail]/Sent Mail (MESSAGES 1462 RECENT 0 UIDNEXT 1463 UNSEEN 0)

Gmail IMAP's issue(folder name with space is not quoted) looks to be completely different problem from this bug's problem of "unable to copy to sent folder".
Removing Bug 402793 from Blocks: field. 

By the way, there is no problem with "[Gmail]/All Mail" & "[Gmail]/Sent Mail" currently. "Not quoted" issue of Gmail IMAP seems to be already resolved by Gmail.

Comment 13

[WADA:World Anti-bad-Duping Agency]

To Henrik Gemal(bug opener):

Does your problem still remain?
If yes, could you please attach complete IMAP protocol log data, instead of very very partial log only. 
If problem doesn't remain, close this bug as INVALID or WORKSFORME, please.

Comment 14

[Andy Boze]

Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.9pre) Gecko/2008042202 SeaMonkey/2.0a1pre

I've been having the same problem for several months, most recently today. The IMAP server in use here is SendMail.

The problem never seems to occur unless I'm sending an e-mail with a large attachment. I compose the e-mail in plain text mode and attach the file, usually a PDF or .xls. The e-mail sends without difficulty, but when SeaMonkey tries to copy to the Sent folder on the IMAP server, I get the following error message:

"The current command did not succeed, The mail server responded: Message contains bare newlines."

I take this to mean that in putting the message together, SeaMonkey terminated one or more lines (of the attachment?) with just a LF instead of the required CRLF.

Comment 15

[WADA:World Anti-bad-Duping Agency]

It's possibly one of following cases, because problem occurs at null-lines(multiple [CRLF]s) after close-boundary of multi-part data.

> --------------010107040407080502000502--[CRLF]  (<== close boundary)
> [CRLF][CRLF]
> [CR] (here is buffer boundary, last [CRLF] is split to [CR] and [LF])
> 3476[4293750]: 403d038:m00.opasia.dk:A:SendData: [LF]
> 3476[4293750]: 403d038:m00.opasia.dk:A:CreateNewLineFromSocket: 6 NO Message contains bare newlines

> --------------010107040407080502000502--[CRLF]  (<== close boundary)
> [CRLF][CRLF] (here is buffer boundary, last [CRLF] remains)
> 3476[4293750]: 403d038:m00.opasia.dk:A:SendData: [CRLF] (<= last [CRLF])
> 3476[4293750]: 403d038:m00.opasia.dk:A:CreateNewLineFromSocket: 6 NO Message contains bare newlines

To Henrik Gemal:
Check your log file to see what([CRLF], [CR], or [LF]) is really sent, please.
Or attach NSPR log file to this bug, if file is still avaiable.

Comment 16

[Dave Murray]

Hi!  I'm also seeing a nasty version of this problem since switching from TB2 to TB3: *any* sent message will fail to save.  I see the following:

  * Sent messages fail to copy to the sent folder.
  * Draft messages and templates also fail to copy to their respective folders.
  * The "message contains bare newlines" dialogue is shown.
  * Messages *do not* need attachments to cause the problem; simply opening a compose window and then saving as draft is sufficient.
  * The sent/draft/template folders are on a Cyrus v2.2.3/FreeBSD IMAP server.
  * Setting the sent/draft/template destinations to be in local folders suppresses the problem; the problematic messages can then be successfully moved from the local to the corresponding IMAP folder.
  * Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b4pre) Gecko/20090419 Shredder/3.0b3pre ID:20090419033223 (all TB3 versions I've tried have behaved the same way).

I've attached an example failed save-as-draft NSPR_LOG_MODULES=IMAP:5 log.  I've preserved the line endings; there's odd combinations of CRs and CR/LFs.  There does appear to be an empty SendData (with terminating "bare" CR), implicated in one of the above comments.

This is a very apparent bug if you have it (sending mail appears to fail), so I guess it must be quite rare - maybe only some versions of the Cyrus server provoke it.  If anyone's looking into it and wants more info, please ask!

Comment 17

[Dave Murray]

Attachment: Log of failed IMAP save-as-draft.

Comment 18

[David :Bienvenu]

Dave, is it possible to get access to an account on that server to debug?

Comment 19

[Dave Murray]

Hi David, yes.  I created an account for you, but - wait for it - it works!  I'm just comparing the settings of my non-working and your working account in both Thunderbird and Cyrus, and will report back.

Comment 20

[Dave Murray]

Okay, it's something in my profile.  If I create a new profile (should have done this first, of course; ahem) drafts messages save just fine to the drafts folder in the problematic IMAP account (I presume sent/template messages will work also).  I'll try to see if I can identify what breaks it in my normal profile.

Comment 21

[David :Bienvenu]

comparing protocol logs when a draft is appended might be interesting. All I can think is that we're generating slightly different protocol in the two different profiles.

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> Log of failed IMAP save-as-draft.

Log for "append to Drafts is as follows. (<CRLF>==0x0D0A, <LF>=0x0A)
> ReadNextLine [stream=96e5568 nb=16 needmore=0]<CRLF>
> S-Drafts:CreateNewLineFromSocket: 9 OK Completed<LF>
> S-Drafts:SendData: 10 append "Drafts" (\Seen \Draft) {888+}<LF>
> S-Drafts:SendData: FCC: mailbox://nobody@Local%20Folders/Sent<LF>
> X-Identity-Key: id14<LF>
> (snip)
> MIME-Version: 1.0<LF>
> S-Drafts:SendData: <LF>
> ReadNextLine [stream=96e5568 nb=38 needmore=0]<CRLF>
> S-Drafts:CreateNewLineFromSocket: 10 NO Message contains bare newlines<LF>
> S-Drafts:SendData: 11 IDLE<LF>
> ImapThreadMainLoop leaving [this=97c7000]<CRLF>
> S-INBOX:SendData: DONE<LF>
> ReadNextLine [stream=6aceba8 nb=17 needmore=0]<CRLF>

Tb sends <LF> instead of <CRLF>, if server uses <LF> instead of <CRLF>?
Or Tb sends <LF> instead of <CRLF>, if linebreak character of mail data in local mail folder is <LF> instead of <CRLF>?

To Dave Murray:

Which is the draft mail?
 (a) Newly composed mail by Tb
 (b) Mail by edit of draft, edit as new, reply/forward for;
     (b-1) Imported mail to local mail folder(usually folder of "Local Folders")
     (b-2) Downloaded/copied mail in local mail folder(POP3, "Local Folders") 
     (b-3) Mail in IMAP folder
     (c) If (b-2/b-3), original mail data is retrieved from;
         (c-1) POP3 server
         (c-2) IMAP server
     (d) If (b-3, IMAP folder), offline-use=Yes?  
     (e) If Edit of draft, who is original creator of the draft mail?

Comment 23

[Dave Murray]

David, I'll compare logs and report back.

Wada, it's (a), a newly-composed message; I just need to create a new message and save it as draft to cause the problem.

Comment 24

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #23)
> it's (a), a newly-composed message;
Do you use add-on(extension)? (new profile==no add-on usually, and it's OK...)

Comment 25

[Dave Murray]

Attachment: Log of working IMAP save-as-draft.

Log of a working save as draft, from a newly-created profile, but to the same IMAP folder.

Comment 26

[Dave Murray]

Wada, I disabled all extensions (presuming this is good enough) and it didn't make any difference.  I also tried removing my user.js.

Removing the whole IMAP account from TB, moving aside its directory under ImapMail in my profile, then re-creating the account didn't help either.

prefs.js in my profile has a lot of differences to the prefs.js in a the profile, but I'll try to take a look through for anything suspicious.

Comment 27

[WADA:World Anti-bad-Duping Agency]

First log("bare new lines" occurs), with current profile.
  STARTTLS is issued, capability command is issued (obtained by Tb 2?)
Second log("bare new lines" doesn't occur), with new profile.
  STARTTLS is NOT issued, capability command is NOT issued (obtained by Tb 3?)

There is no difference in <LF>/<CRLF> usage between logs.
Server's behaviour with STARTTLS looks to be different from one without STARTTLS.

Do you set "security setting=None" in the new profile?
Can problem be reproduced with "security setting=STARTTLS", using Tb 3/the newly created profile?
Does your server supports SSL for IMAP connection? If yes, does problem occur when "security setting=SSL/TLS(SSL if Tb 2)" (any profile, any Tb version)?

> Shredder/3.0b3pre ID:20090419033223

Bug 470650 is fixed on 3/24. It probably doesn't relate to this bug's issue in your case, but test with newer build please, to avoid unwanted problems during testing.

Comment 28

[David :Bienvenu]

The 04/19 build should have the fix for bug 470650 - I wonder if Dave configured the imap account settings to use STARTTLS  in the profile that worked.

I'm not sure I trust the line endings in the log to reflect the line endings we're actually sending.

Comment 29

[Dave Murray]

Sorry for the delay, but I did eventually get to the bottom of this.  Rather embarrassingly, it turned out to be a problem of my own making, so this issue is "fixed - operator error" as far as I'm concerned.

For posterity, the problem was that I had a:

    user_pref("mail.identity.id1.header.xface", "X-Face: 9=rCbaW3...");

in my user.js and in the the second string I had embedded '\n's.  As per above, I did try removing my user.js, but the gotcha was that somehow I must have managed to duplicate this line into my prefs.js too.  Removing it from there fixes the problem, of course.  (Something must have changed between TB2 and TB3, as it used to work, but I don't consider that a bug.)

Apologies for the noise - I guess I'm owe a couple of people a beer!

Comment 30

[David :Bienvenu]

Dave, is there some extension that looks for xface? How come those headers didn't show up in the protocol log?

Oh, it wasn't necessarily that there were bare newlines, but rather that the x-face header that was getting added wasn't rfc822-compliant...

Comment 31

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #29)
> For posterity, the problem was that I had a:
>     user_pref("mail.identity.id1.header.xface", "X-Face: 9=rCbaW3...");
> in my user.js and in the the second string I had embedded '\n's. 

Problem you previously reported with attaching log is absolutely irrelevant to X-Face: header. Problem occurred at "null line" as separator of mail headers(no X-Face: header in log) and mail payload(mail body), and STARTTLS was not issued in the case.
Dave Murray, have you changed "connection security" setting?
> mail.server.serverN.socketType : 0=None, 2=STARTTLS, 3=SSL/TSL (1="TLS, if avail" of Tb 2)

Comment 32

[Dave Murray]

Hi David,

(In reply to comment #30)
> Dave, is there some extension that looks for xface?

I had been using MessageFaces

    https://addons.mozilla.org/en-US/thunderbird/addon/393

although by the time I was seeing this issue I had disabled it (due to it causing TB to hang on opening an incoming message with a read receipt request).

Really, I'm not that bothered about xface!


> How come those headers didn't show up in the protocol log?

Good question, as it would have made the source of my problem immediately obvious.  I don't know, without digging into the source code.  Maybe headers inserted via the pref have slipped through the logging net?

 
> Oh, it wasn't necessarily that there were bare newlines, but rather that the x-face header that was getting added wasn't rfc822-compliant...

Maybe, although, ironically, I'd added the '\n's to try to wrap the header within the 78-char line length limit!

Comment 33

[Dave Murray]

Hi Wada,

(In reply to comment #31)
> Problem you previously reported with attaching log is absolutely irrelevant to
> X-Face: header. 

> Dave Murray, have you changed "connection security" setting?

Respectfully, I strongly suspect it is related to the x-face header; the problem went away immediately after I removed just that line from my prefs.js.  Moreover, the header string I was trying to insert contained '\n's, which you could easily imagine resulting in bare newlines.

The problem occurred both with and without TLS.

Comment 34

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #33)
> The problem occurred both with and without TLS.

I see.

We are asking whether your original case(log attached to Comment #17) was really caused by id14.header.xxx="...\n\n..." or not.
In original case, Bad log=X-Identity-Key: id14, Good log=X-Identity-Key: id1.
And log line for "X-Face:" header is not seen in log file.
We are wanting to know whether all of next is true or not.
  a) id14 had mail.identity.id14.header.xxx="...\n\n...",
     but id1 didn't have optional setting or setting doesn't have "\n" in string.
  b-1) If ...idN.header.xxx="...\n\n..." is set, problem always occurs,
  b-2) but there is no log line for "xxx: ..." in NSPR log header always.
      (IMAP logging probably writes log of pre-defined header set only)
      (for IMAP append, but we are not sure.)
Dave Murray, all of a), b-1), b-2) was true?

Comment 35

[Dave Murray]

Hi Wada,

> We are asking whether your original case(log attached to Comment #17) was
> really caused by id14.header.xxx="...\n\n..." or not.

Thanks for continuing to look into this.  However, just to be clear, I'm happy this issue is resolved by removing the offending x-face pref, so please don't expend any effort on this on my behalf.


> Dave Murray, all of a), b-1), b-2) was true?

Yes.  My account IDs all changed, because I created a new profile, but the summary is that if you have the following lines in your user.js:

  user_pref("mail.identity.id1.headers", "xface");
  user_pref("mail.identity.id1.header.xface", "X-Face: line 1\n  line 2");

you get the "bare newlines" error when you try to save a sent/draft/template message to a Cyrus IMAP server.  The header lines don't appear in the NSPR log.

In passing, I notice TB copies these lines from user.js to prefs.js, but then doesn't remove them from prefs.js when I you them from user.js, so I don't feel so bad about not realising they were still in effect.

Comment 36

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #35)
> > all of a), b-1), b-2) was true?
> Yes.

Your NSPR log has been explained by your answer. Now I can sleep well :-)

Comment 37

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #35)
> In passing, I notice TB copies these lines from user.js to prefs.js, but then
> doesn't remove them from prefs.js when I you them from user.js, so I don't feel
> so bad about not realising they were still in effect.

I've opened Bug 493514 for you(and for me).

Comment 38

[Wayne Mery (:wsmwk)]

this bug can be closed, no?

Comment 39

[Dave Murray]

(In reply to comment #38)
> this bug can be closed, no?

As far as I'm concerned, yes.  (But I'm not the original poster.)

Comment 40

[Wayne Mery (:wsmwk)]

wada, regarding comment 13 - do you suspect this problem should be gone in current releases?

Comment 41

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #40)
> wada, regarding comment 13 - do you suspect this problem should be gone in
> current releases?

I guess "LF only line generated by Tb himself" is already resolved by some other bugs, but I'm not sure.
If this bug still occurs in original poster's environment, problem is apparently not resolved yet.

Comment 42

[Wayne Mery (:wsmwk)]

bienvenu, I'm closing on basis of comment 39. Please repoen if you feel Henrik's original problem is not fixed.