83396 - Alert "message contains bare newlines" displays and lost messages when copying msgs from Local(or other IMAP account)folder to an IMAP account folder

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect, P2)

Description

[Karen Huang]

Used 05-30-06-trunk build

Got an abnormal alert when trying to copy local msgs to an IMAP folder

1) Copy about 200 msgs from local to an IMAP folder
2) Actual Results: didn't copy completely and got an abnormal alert: "message 
contains bare newlines"(see following attach screen shot)

Expected results: Should copy successfully & completely without this alert

Comment 1

[Karen Huang]

Attachment: Attach this alert info

Comment 2

[Navin Gupta]

This again is not reproducible always. cc sheela, who has seen this 
come and disappear for the same message.

Comment 3

[Karen Huang]

Nominating nsbeta1 since it seems that many peoples is hitting this problem.....

Comment 4

[Karen Huang]

*** Bug 83831 has been marked as a duplicate of this bug. ***

Comment 5

[Karen Huang]

changing summary as David mentioned on bug 83831 - this only happens on some 
messages....

Comment 6

[David :Bienvenu]

so this isn't a true dup of the other bug - the error message I was seeing had
to do with an invalid message header because we're copying the From - envelope
header. This is a similar bug, however, and I've seen it as well on different
messages than the ones that generate the above error message.

Comment 7

[Karen Huang]

Looks like there are different problems occurring for copying local msgs to IMAP 
folder now....restoring this summary back based on David's comments.

Comment 8

[scottputterman]

reassigning to naving.  It sounds like we need to find the message that causes
this.  Sheela or Karen since you've both seen this, can you try finding a message?

Comment 9

[Karen Huang]

From 200 msgs??....I need to investigate which one!!....

Comment 10

[Karen Huang]

But, I did have the IMAP log saved at that time when copying the 200 msgs.
I will attach here, hope it help....

Comment 11

[Karen Huang]

Attachment: attach the IMAP log when coping the 200 msg from local folder to IMAP folder

Comment 12

[scottputterman]

adding pdt+

Comment 13

[Karen Huang]

It seems that I losted the messages duing copying these 200 messages.
I only have 54 messages left and I am still couldn't finding the specific 
message which causing this alert and losting messages. Updating summary to 
include losting messages.

Comment 14

[Navin Gupta]

Try deleting the msf file. They may come back. If you are copying messages, how
can you lose them ?

Comment 15

[Karen Huang]

I used drag/drop for copying these messages, it probably moving the messages 
instead of copying messages...I just couldn't find them...but I willl keep 
finding them.... :-(

Comment 16

[Navin Gupta]

I cannot find any such message. Also only that message would not be copied but 
the rest of messages (multiple copy) should be copied after bug 83831 gets 
checked in.

Comment 17

[scottputterman]

moving to 0.9.3

Comment 18

[lchiang]

Does the message generating this alert disappear, causing data loss?

Comment 19

[Sheela Ravindran]

I have been able to reproduce this problem on linux machine with yesterday's
trunk build. I have this message that causes the bare lines alert dialog trying
to move it back to an imap folder.
The following are the steps to produce this problem:

create a new message with a jpg attachment, some text and send it to yourself
After receiving it in the imap account
Drag and drop it to pop account folder
Go to your pop account folder where you moved the message
select that message and click on the file button and try to move it back to an
imap folder
Results in the alert dialog where it fails to move the message back to imap folder 

Navin, 
If you are not able to reproduce using steps above stop by I can reproduce this
on my machine.  

Comment 20

[Navin Gupta]

There is no dataloss. The only bad thing is the messages does not copied. 

Comment 21

[Sheela Ravindran]

Navin,
I deleted the .msf file of the local folder where I was trying to move that
message and relaunched the application and tried to move again and resulted with
the same error still. So deleting .msf file did not do any good to get rid of
the problem.  

Comment 22

[Sheela Ravindran]

That's right, I was not able to just move the message from the local folder back
to an imap folder.  

Comment 23

[Scott MacGregor]

Mail news triage meeting --> .9.5

Comment 24

[Karen Huang]

I can reproduce this probelm by using the attach msg as following.

Comment 25

[Karen Huang]

Attachment: attach this mail msg that displays alert "Message contain bare newlines" when copying this msg from Local Folder to IMAP Folder

Comment 26

[scottputterman]

moving to 0.9.6

Comment 27

[scottputterman]

moving to 0.9.7

Comment 28

[Heiki Kask]

I migrated several users from POP to IMAP accounts and noticed that copying from
local to IMAP failed on two cases:

- message had only headers and no body. Didn't check was there any empty lines
or was the message body just empty. I guess that the error message was "message
contains bare newlines"

- message header had first line as "From - ..." used as e-mail separator at some
mailboxes. There was another error thrown out, unfortunately I don't remember
exact wording.

I assumed that this was Cyrus IMAP server issue, but since you are talking about
it here....

Comment 29

[scottputterman]

moving to 1.0.1

Comment 30

[Karen Huang]

*** Bug 142620 has been marked as a duplicate of this bug. ***

Comment 31

[Karen Huang]

Olga seeing this problem when copying messages from other IMAP account to NMS 
6.0 (tintin.mcom.com) account. So, this seems is also occurring for copying IMAP 
account messages to IMAP account messages.....

Comment 32

[Olga]

It was Web Mail Account.

Comment 33

[Karen Huang]

Olga, did you copy from Webmail IMAP to NMS 6.0 (tintin.mcom.com)IMAP?
Can you attach that problem message here? Thanks.

Comment 34

[Karen Huang]

Olga, did you lost this message during the copy?
We may need to nominating nsbeta1 again if msgs have been lost during the 
copy......

Comment 35

[Olga]

This morning after re-starting NO problem with copy/move neither from Web Mail
nor from another POP, IMAP. Yesterday I saw the same alert mentioned in comment #1.

Comment 36

[Henrik Gemal]

I can now get this 100% when trying to move a message that contains text and a
attached jpg image from a Exchange IMAP server to a Cyrus IMAP server...:(

Comment 37

[Karen Huang]

*** Bug 159323 has been marked as a duplicate of this bug. ***

Comment 38

[Dan Mosedale (:dmosedale, :dmose)]

I'm seeing this also copying IMAP -> IMAP on Linux.

Comment 39

[Dan Mosedale (:dmosedale, :dmose)]

In particular, I'm copying from a UW IMAP server to a Netscape Mail Server.

Comment 40

[David :Bienvenu]

cross-imap server move/copy copies the msg down to a local tmp folder and then
up to the destination imap server. The problem is that some servers (like our
NMS server) are very picky about the format of the messages they allow to be
appended to a folder.

Comment 41

[Boris Folgmann]

I have the same problem with mozilla (1.0 and 1.1) on linux when trying to move
some local mails to my Cyrus IMAPD server. I hope to see a quick fix rsn so that
mozilla skips the mails with bare newlines and continues the copy/move operation.

Is ist now clear if mozilla has problems or if the imap servers have problems?
Reading this thread I'm not sure about that.

Comment 42

[Henrik Gemal]

*** Bug 133377 has been marked as a duplicate of this bug. ***

Comment 43

[Markus Gerstel]

Check
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=5629
for a statement about this bug/feature/enforcement in Cyrus IMAPD.

Comment 44

[(not reading, please use [e-mail] instead)]

mass re-assign.

Comment 45

[vongillus]

I try to set up a Cyrus server, and the problem is still there in mozilla 1.5
Linux. A google search tend to say that it's a problem with mozilla and not Cyrus.

The problem is always reproductible when I create a mail with big attachment, I
save it, and I try to move it to the Imap server.

Hope it will be solve soon...

Comment 46

[David :Bienvenu]

Is this problem more easily reproducible on Linux?

Comment 47

[Dan Mosedale (:dmosedale, :dmose)]

I used to see this from time to time on Linux, and I don't think I've ever seen
it on Windows, so I suspect it may in fact be platform-specific.

Comment 48

[Tashfeen Bhimdi]

It happened to me in Windows when I copied a large number of messages from one
IMAP folder to another folder on the same server.  This was back in the 1.3 days
though, now I usually just use a web interface for my IMAP provider to move a
large number of messages.

Comment 49

[David :Bienvenu]

from one imap folder to another on the same server does not involve the client,
or at least its line endings - it's simply a copy command sent to the server,
just like the web client would do.

Comment 50

[Tashfeen Bhimdi]

My mistake, you are correct.  The message copies I made were from one IMAP
server to another, MyRealBox -> Fastmail I believe.  I remember being frustrated
with the "bare newlines" error and came across a "Migrate IMAP" feature within
Fastmail's website and used that instead of Mozilla.

Comment 51

[Tarjei Huse]

I would like to add that I'm seeing this bug in Mozilla 1.5 and Thunderbird 0.7
on a windows XP home ed machine when moving mails from the local folders to an
Cyrus-IMAPD account. Any Ideas?

Tarjei

Comment 52

[Flo Ledermann]

... this bug is around for a really long time and annoying quite some people.
migrating from an archive of local pop-messages to an imap solution is not so
uncommon today, i guess. however, nobody seems to have even looked at any code
that could cause this problem...

can we increase the priority of this an get somebody to take a look at it? imho
this is one of the most visible mail bugs!

Comment 53

[Christian Eyrich]

I fear that's the classic problem that the person knowing the code can't
reproduce the problem and vice versa.
David, would a tcpdump log help you more than a Mozilla log?

Comment 54

[Flo Ledermann]

sorry for being a bit nervous in my last post... i did some research on this,
with the following results:

1.) AFAIK, this problem happens only with the Cyrus IMAP server. The Cyrus folks
follow a zero-tolerance policy concerning RFC violations, and reject malformed
mails submitted with the IMAP APPEND command.

2.) The actual bug is found in the e-mail clients sending non-RFC-compliant
mails (e.g. Outlook up to version 5). If such a mail is received by mozilla
through a POP connection, it is stored on disk (as is) and displayed correctly.
If such a mail is received by Cyrus IMAP by a local sendmail daemon, it is
(AFAIK) corrected and stored on disk in the correct format.

3.) The problem occurs when you try to move such a mail stored by mozilla
locally to the Cyrus IMAP server - mozilla delivers the mail in the original
(non-compliant) format, and Cyrus thinks it is talking to a non-compliant client
and rejects the mail.

the problem is that we have a deadlock situation here - IMHO mozilla is right to
store the mails in the original form, and cyrus is right to reject
non-RFC-compliant mails. and it makes no sense to blame "the other" clients,
since even if they fix this problem in future releases, this won't change the
messages already received by mozilla.

in addition, since sendmail usually corrects malformed mails, this problem is
not very visible to end-users of such buggy email clients - usually their
messages are delivered without problems (even to Cyrus). this problem only
occurs when such a malformed message is "replayed" by mozilla.

the possible fixes to this problems that i can see are:

1.) talk to the cyrus folks about theirt policy. i am not sure if they are aware
of this scenario when they argue in favour of their zero-tolerance policy.

2.) make mozilla correct the stored emails when sending to an IMAP server. this
is possibly not trivial(?). the three main causes for these kinds of problems
are: LF without CR in the mail body, NUL characters in the mail body, or
malformed headers. (the relevant RFCs are RFC821 (SMTP), RFC2060 (IMAP4), and
RFC1939 (POP3), i guess)

3.) offer end-users a link to an external program, that is able to fix their
mailboxes when needed. unfortunately, i haven't found such a software by now...
or build a hack into mozilla, to either re-send these mails via SMTP to the
account (with the original date), or something like that (probably least
preferred by all)

fact is, that i and other people are completely stuck in the process of
migrating our mail archives to IMAP. some people just deleted the messages, but
i have a large archive reaching back to 1996 and i would loose 1/3rd of my
historic messages - completely unacceptable for me.

if anyone can think of a hack to allow me to move my mails, i would be very
happy for now.

Comment 55

[Flo Ledermann]

p.s. the most relevant post from the cyrus mailing list concerning this issue:

http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=5639
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=5641
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=7038
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=21465
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=21835
http://asg.web.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&msg=21877

Comment 56

[Christian Eyrich]

Flo, to follow strictly the RFC is good in my opinion. I know about the "be
tolerant when reading/recieving" theory, but I don't agree with it.

The problem that made me look into this bug can indeed be caused by imported
Outlook mails. But 1. this would mean a bug in the import code, and 2. doesn't
absolve the code that writes to IMAP from responsibility.
To this mails on Unix/Linux are also stored with LF only line endings in the
mbox (and CR only on MAC).

In result 2.) you don't write about importet mails from non-RFC-compliant
clients but received through POP.
That confuses me - do you know a mail with LF is written with LF (on
not-Unix-systems) or do you just think so?
AFAIK the code in RetrResponse() and ReadNextLine(), line endings are stripped
away uppon receiving a line and the gets a new one depending on the platform. So
we end up with mboxes with different line endings on different operating systems.

I think your conclusion 2.) is the right one.
David, don't we really convert the line endings when appending to IMAP?

Comment 57

[Flo Ledermann]

FYI: i am sitting on a windows XP box.

and now that you mention it: i couldn't actually find the LF characters that
Cyrus complains about in my mbox files... that was my first try, to fix the mbox
files manually. so maybe it is really just a bug in the mozilla IMAP code. but
sorry i can't help you here, i am not a mozilla hacker (and don't currently plan
to become one, due to time constraints)

the reason why i thought it is the original source of the mail that couses the
troubles, is, that this bug occurs only for some mails. and for me, there are
some senders (using for example outlook) that are more frequently the source of
such mails than others. but, you are right, this is just speculation...

of course strictness with respect to RFCs is good. i just thought that this was
a scenario that the RFCs didn't fully consider.

Comment 58

[David :Bienvenu]

Flo, that was a very accurate summary. This problem also happens with the
Netscape IMAP server, since it was developed by the same person as Cyrus.

We could fix the imap append code to repair naked newlines, and we could replace
embedded nulls with spaces - malformed headers are a bit trickier.

I'll try to look into it today.

Comment 59

[Flo Ledermann]

just one more comment: it would be great if mozilla would, if you move messages,
at least delete those messages that have been moved succesfully. at the moment,
if you move multiple messages at once, and the error occurs in one message, the
messages that have been copied successfully are not deleted from the original
folder, resulting in duplicate messages that are very hard to sort out. one has
no clue which messages have been copied and which not, and which message caused
the error. this is also what the reporter of this bug pointed out, i think.

or maybe open a new bug "remove successfully moved messages from original folder
even if an error occurs"? i haven't looked if this happens in other cases too...

Comment 60

[Boris Folgmann]

Flo, the duplicate mail problem is indeed annoying. Shouldn't it be possible to
use formail to remove the duplicate mails from the local folder?

Comment 61

[Flo Ledermann]

unfortunately, the duplicate mails are now on my imap server. is it possible to
use formail on the mailboxes on the server? i could theoretically have access to
the server to fix my mails.

could formail be used to fix invalid headers and the other problems mentioned
above? i have never used this tool...

Comment 62

[Boris Folgmann]

The formail command is part of the procmail package found on most Linux systems.
It can split mailboxfiles into seperate mails and can also manipulate singe mails.
But it may break something if you directly let it work on cyrus' mail files.
Even when you turn cyrus off. May be the db files of cyrus get out of sync.
Surely they will, if you delete some mails with a simple rm.
On the other hand formail is perfect for header manipulation. Check man formail.

Comment 63

[David :Bienvenu]

one tricky thing about fixing this is that we need to fix it when we create the
temp file who's contents we append to the imap folder, because the append
command requires us to specify the message size up front, and we use the size of
the temp file for that. Thus, we can't be inserting extra cr's or lf's to fix
bare newlines at append time; we have to do it when we generate the temp file.

Comment 64

[Christian Eyrich]

But that alone wouldn't be so hard. Or is it that this temp file generation is
also used for other purposes where this conversion would be wrong?

Comment 65

[David :Bienvenu]

yes, I think the code is used for other purposes - I'll probably need a flag
that says to sanitize the copied message.

Comment 66

[David :Bienvenu]

actually, the code looks to me like it handles this. Apparently it doesn't work,
but when creating the temp file, it definitely scans for a CR or LF when parsing
for lines, and writes a CRLF instead of a single CR or LF terminator. I'll have
to try the test case.

Comment 67

[David :Bienvenu]

Attachment: proposed fix

the code was only figuring out the linebreak length once, which is wrong - it
needs to figure it out for every line. This causes a slight amount of code
duplication, but I couldn't figure out an easy way to just have one instance of
the code that determines the linebreak length

Comment 68

[David :Bienvenu]

fix checked in.

Comment 69

[Flo Ledermann]

i had only now time to test this with a current nightly.. and unfortunately, i
have to say, the bug seems to be not fully fixed.

i was able to move approx. 2/3rds of the massages that would trigger the error
message to my IMAP server - however, there are still messages remaining that
trigger the error as before.

nearly all of these messages have large attachments >200k in size. there is only
a single message that triggers the error for me that is does not have an
attachment, but is a 15k multipart html message. i will attach the source of
this message to the bug.

sorry for the bad news, otherwise thanks and respect for the quick help!

using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a) Gecko/20040421

Comment 70

[Flo Ledermann]

Attachment: testcase (saved as .eml)

see previous comment. this is an example for a message still not working for
me.

Comment 71

[David :Bienvenu]

sorry it's not completely fixed for you. I don't think that test case helps me,
because the line endings are correct in it, and I can copy it to a cyrus server
w/o a problem. Would it be possible for you to save off the actual message into
a text file from your mailbox, using a text editor that maintains the line
endings, and e-mail that to me? And maybe one of the other messages with the big
attachment as well? thx...

Comment 72

[David :Bienvenu]

after a little debugging, I think the original message is OK, but we're messing
up the line ending in the process of saving the message to the temp file, prior
to uploading the message. When data is streamed to us, the stream might break on
a CRLF boundary, and in that case, we'll write an extra blank line, I think...

Comment 73

[David :Bienvenu]

Attachment: proposed fix

handle case where crlf are split by block boundary. Thx very much for the test
cases, Flo.

Comment 74

[David :Bienvenu]

fixed

Comment 75

[Boris Folgmann]

Sounds great! Will the fix be in release 1.7?

Comment 76

[Flo Ledermann]

amazing! tried it out in the current nightly, and everything works! thanks a LOT
for fixing this so fast.

there are two bugs that should be spun off from this one: invalid headers in
mail messages (same problem as with this bug, but probably harder to fix as
indicated in comment #58), and the general problem that if a problem occurs
during moving of multiple mails, you end up with duplicate mails since the mails
that have been copied successfully are not removed from the source folder.

how should we proceed with these two bugs? i didn't query yet if they are
already there...

Comment 77

[David :Bienvenu]

Comment on attachment 146879 [details] [diff] [review]
proposed fix

this change affects the copying of local messages to imap folders (or messages
from other imap servers) - it actually fixes a data corruption problem, so it's
a good one to get in. Usually, an extra blank line doesn't cause any harm, but
it can be a bad thing...

Comment 78

[Mike Kaply [:mkaply]]

Comment on attachment 146879 [details] [diff] [review]
proposed fix

a=mkaply for 1.7

Comment 79

[David :Bienvenu]

Attachment: cumulative fix as checked into 1.7 branch

Comment 80

[Scott MacGregor]

FYI, David fixed this on the tbird 0.6 branch too

Comment 81

[Christian Eyrich]

A user I've contact to still experiences this problem with TB 2004-04-28-00-trunk.
I've a log here that shows this and also the mail that generates this.
Unfortunately I can't reproduce the problem with my own Cyrus IMAPd and the
messages, so it's difficult to help here.

David, do you think there's any chance to find another bug in the move-code
without being able to reproduce it yourself? I could supply you with log and
mails too.

Comment 82

[Flo Ledermann]

FYI: unfortunately i cannot assist you with this bug anymore, because my
organisation has migrated to a different imap server. for me everything worked
fine, but i was told only afterwards that the server software had changed ;)

Comment 83

[Christian Eyrich]

Erm, you mean your comment #76 "everything works" was made after testing on the
new server (without your knowledge)? So the bug you discovered might also still
be active.

Comment 84

[David :Bienvenu]

Christian, sure, send me the mail and the logs

Comment 85

[Flo Ledermann]

i _think_ i verified the bug while the cyrus server was still active. but i am
not sure. sorry about this, i only was told about the switch two days ago, but
it happened earlier...

Comment 86

[David :Bienvenu]

I'm reasonably confident that the last fix actually fixed your problem, at least
for the sample messages you sent me. But I'll have a better idea when I get the
mail messages from Christian.

Comment 87

[Karen Huang]

By using the original msg that I attached on #25:
http://bugzilla.mozilla.org/show_bug.cgi?id=83396#c25

I am not able to reproduce this problem when copy the 
Verified as fix 

Comment 88

[Karen Huang]

By using the original msg that I attached on #25:
http://bugzilla.mozilla.org/show_bug.cgi?id=83396#c25

I am not able to reproduce this problem anymore when copy this message from 
Local to IMAP on latest 1.7 branch 06-24 build.

Changing keywords from fixed1.7 to verified1.7.
Leave this bug status "as is" until this bug be verified on trunk again...

Comment 89

[Karen Huang]

By using the original msg that I attached on #25:
http://bugzilla.mozilla.org/show_bug.cgi?id=83396#c25

I am not able to reproduce this problem anymore when copy this message from 
Local to IMAP on latest 1.7 branch 06-24 build.

Changing keywords from fixed1.7 to verified1.7.
Leave this bug status "as is" until this bug be verified on trunk again...

Comment 90

[Boris Folgmann]

Ok, I upraded to the mozilla-1.7-0.3.2 rpm from the FC2 devel-branch. Copying
mails from local to cyrus-imapd that previously triggered this bare lines error
seems to be better now. But I came across an other problem, mozilla refuses to
copy a certain email with this error message:

current command not successful: mail-server responded: message contains
invalid header.
 
I translated this back from the german locale to english. I'm going to attach
the mail for you.

Comment 91

[Boris Folgmann]

Attachment: .eml triggering the invalid header problem

Comment 92

[Andreas Heinz]

hi

i also came across this problem and hopefully can give some additional infos.
i dont think this "bug" is resolved yet.

i tried converting my mails having of old pop times by saving them via "the 
bat's" export function saving them to mbox files.

after your post, that some programs just save the mails "as is" without 
converting them, i thought i should save some mail coming from an cyrus imap 
server.

so i did this by using the bat, copied this mail to thunderbird version 0.9 
(20041103) and tried to move this mail over to another cyrus mail server.

this ended with the "Message contains invalid headers" dialog.

hope this can be fixed. thx :)

Comment 93

[Andreas Heinz]

additional information:
when you try to put mails, imported from the same mbox files to the imap server 
with evolution it works very well.

Comment 94

[Bill Sheppard]

I see the problem frequently when moving messages from my local Sent folder to
my Imap sent-messages folder on Mozilla 1.7.1 running on Linux to the Sun ONE /
iPlanet Messaging Server V5.2.  To get around it I take move each of the
messages which won't copy to a local folder, then load the folder into emacs,
then save it back out.  Whatever line-endings Mozilla and/or the IMAP server are
complaining about get worked out when emacs resaves the file.

Comment 95

[David :Bienvenu]

my guess is that it's unhappy about the received-by header(s). I'll see if I can
find a cyrus server to test against...

Comment 96

[Bookholt]

AFAIK this bug is not fixed.  

I am seeing the "bare newlines" bug on TB 1.0 (20041206).  I have observed this
bug since the very early releases of TB as well as Mozilla Mail on Linux and
Windows.  Most recently it occurs when moving messages from one Cyrus IMAP
server to another Cyrus IMAP server.  

I agree with Flo's description that this bug seems to occur more often with
certain senders and the offending messages usually include attachment(s).  

I can move the problematic message(s) to different folders within the original
IMAP server, but when I try to move them to a different IMAP server it complains
about bare newlines.  

Some times the same message will not trigger the error and will be moved
correctly.  Usually I keep trying to move the problematic message(s) for a few
days and eventually the move succeeds.  

The server I'm trying to move mail to is running cyrus-imapd-2.2.8-1 built from
an SRPM.   I cannot (or at least I don't know how to) get version information
from the other IMAP server.  

I can attach the Cyrus logs (from the server to which I'm moving mail), a sample
problematic message, and a tcpdump if necessary.  

Comment 97

[David :Bienvenu]

a sample problematic message would be useful (zipped, perhaps, to save line endings)

Comment 98

[Bookholt]

I just tried attaching a message I was originally unable to move, but even when
zipped it is >300k so I'm prevented from uploading it.  

However, after ~24 hours and a reboot or two I was able to move said message.  

Usually I just read my messages from the preview pane, but immediately before
the message was successfully moved, I double-clicked the attached message to
view it independently and also viewed its source.  Maybe this had something to
do with it?  

Comment 99

[Bookholt]

(In reply to comment #98)
> Usually I just read my messages from the preview pane, but immediately before
> the message was successfully moved, I double-clicked the attached message to
> view it independently and also viewed its source.  Maybe this had something to
> do with it?  

Apologies.  The above is less than clear.  I will try to rephrase: 

Immediately before I was able to move the message I opened it by double-clicking
on it in my inbox.  Once the message window was open I viewed its source.  I
closed the message window (but left the source window open) and was able to
successfully move the message from one IMAP server to the other IMAP server via
drag and drop.  

Perhaps by opening the message the newlines felt they needed to get properly
dressed (or a parsing function was activated that cleaned them up).  

Comment 100

[David :Bienvenu]

Attachment: fix for eudora and outlook import causing these messages

Eudora (and I think Outlook) import can also cause us to get corrupt messages that can't be copied to a picky imap server. This fixes that - the problem happened when a CRLF pair spanned 4K blocks - we would write a gargage char in between the CRLF because we weren't checking that the block ended in CRLF as opposed to just CR.
There are probably bugs out there about Eudora and Outlook messages w/ attachments getting corrupted when imported.

Comment 101

[Scott MacGregor]

Comment on attachment 209013 [details] [diff] [review]
fix for eudora and outlook import causing these messages

approving for the 1.8.1 branch

Comment 102

[David :Bienvenu]

the import bug is fixed on the 1.8.1 branch for tb 2.0

Comment 103

[John Hanson]

I am still having this problem when I am trying to transfer a local folder (imported from Eudora) to the IMAP "sent" folder.  

Initially I was getting the "message contains bare newlines" as originally reported with this bug.  Then I copied the local folder to a NEW local folder.  The error dialog disappeared but now TBird just ignores the transfer if a troublesome email is included in the bunch I'm trying to copy.

I was using 1.5.0.2 but even with 3.0 alpha still get the error (or ignoring the copy if funny email is there).

Comment 104

[Richard Blumel]

I dont know if this is part of my problem, I was running 1.5.0.2 of tb and was migrating from local boxes to imap/cyrus. I am getting bare new line errors, so I update to daily build 20060528 I think 3.0a1 and still cant move this msg. Does this fix not resolve bare newlines that are already in the the msg database.... 

Comment 105

[Richard Blumel]

Here is a hex dump from tcpdump of the mesg it is trying to send. http://pastebin.com/744441

Comment 106

[Virendra Singh]

* Work Around *

I am new to this group, so please forgive me if I sound stupid :-)

I too was facing this problem even in 1.5.0.4 (for Solaris/sparc), apparently it was being cased by "^M" characters, and here is how I got over it (of course manually):

1) Stop thunderbird client
2) Go to the Mail files under <HOME>/.thunderbird/<profile>/Mail/Local Folders
3) Perform the following steps on the file WITHOUT *.msf or *.sbd, for example on Inbox:
cat Inbox | sed "s/^From Sun/From - Sun/g" |sed "s/^From Tue/From - Tue/g" | sed "s/^From Mon/From - Mon/g" | sed "s/^From Wed/From - Wed/g" | sed "s/^From Thu/From - Thu/g" | sed "s/^From Fri/From - Fri/g" | sed "s/^From Sat/From - Sat/g" | sed "s/>From -/From -/g" | sed "s/^M//g" > Inbox.new

This command will also take care of a few other corruptions which occur when you download mail from IMAP to your Local Folders, and later when you try to put them back on IMAP account, it gives errors.

4) Now, move this Inbox.new to Inbox (of course you better take backup of Inbox before doing step 1 itself, in case things to go wrong.).
5) Start Thunderbird again, and start moving the mails, hopefully it'll get you through this time.

I hope it helps.

Comment 107

[raal]

I´m using TB ver. 3.0a2 2008070803. 
IMAP server Cyrus

When I Send message with attachment then copying message to Sent folder fail.
Error: The current command did not succeed. The mail server responded : Message contains bare newlines

Comment 108

[Ravi]

I was not facing this issue in TB2, but it started appearing when I switched to TB3 Alpha 1 and Alpha 2 (version 3.0a2 (2008072418)) and try to send/reply messages with attachments or other mime data.
IMAP server is Cyrus.

Comment 109

[Dan Mosedale (:dmosedale, :dmose)]

Reopening based on comment 108; added qawanted keyword.  Assuming this can be reproduced, it should block tb3, since it's dataloss.

Comment 110

[Mark Banner (:standard8)]

(In reply to comment #109)
> Reopening based on comment 108; added qawanted keyword.  Assuming this can be
> reproduced, it should block tb3, since it's dataloss.
> 
Normally we'd deal with these in new bugs...

Anyway think I know the regression, my mistake in bug 429891, taking.

Comment 111

[Mark Banner (:standard8)]

So now I look at it, I don't think bug 429891 is the culprit (as its change would not have caused a regression such as this, and the bug which made a change before that was checked in in 2007), but its possibly not helping.

I'm going to deal with my issue in a separate bug.

David, if you've got any ideas, please help. At the moment I can't reproduce this locally - but its interesting the two reports we have got both use cyrus IMAP.

Comment 112

[David :Bienvenu]

Cyrus is the only server I know that really complains about this. I can try with my test account on MoMo's Cyrus server.

Comment 113

[David :Bienvenu]

I tried creating a local message with barenewlines, and copying it to a cyrus imap server worked fine. So at this point, I really need a reproducible test case. I'll try to verify that our Cyrus server complains about this, by disabling the code that fixes barenewlines before appending to an imap folder.

Comment 114

[Ravi]

This seems to happen mostly only when replying (to the bare newline messages). After the I get the error popup saying copying to the IMAP Sent folder failed, if I return to the composition window and save the message in a local Drafts folder and then copy manually from local Drafts to the IMAP Sent folder, it copies fine!
So can you try to reproduce this by replying to a bare newline message instead of manual copy? 

Comment 115

[Boris Folgmann]

If necessary for any tests you can contact me to get an test account on a working cyrus-imapd-2.2.12-8.1.RHEL4 installation on one of my servers.

Comment 116

[Bill Sheppard]

I often triggered this bug when copying a bunch of messages from a local folder to an Imap folder - that might be a quick way to try to catch the bug.  Note that I haven't seen it in many months (I'm running on a Sun IMAP server).

Comment 117

[Mark Banner (:standard8)]

I think we need to fix this.

Comment 118

[Wayne Mery (:wsmwk)]

see also bug 301010

Comment 119

[David :Bienvenu]

I wonder if the remaining issue is bug 381472 ? If anyone has a local folder and imap server that shows this problem, and can share it with me, please contact me.

Comment 120

[WADA:World Anti-bad-Duping Agency]

There was a case in which Tb is culprit, Bug 460636(fixed on 2008-10-22. see also Bug 466892). 
> Bug 460636 nsMsgSaveAsListener sometimes inserts extra LF characters
If the extra LF is inserted between consecutive [CRLF]s by Tb himself, "bare newline" issue can occur.

Comment 121

[David :Bienvenu]

raal, are you seeing this with TB 3.0 b2, or nightly 3.0 trunk builds?

Comment 122

[raal]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090331 Lightning/1.0pre Shredder/3.0b3pre
This bug does not occur.

Comment 123

[David :Bienvenu]

ok, thx, I'm going to mark this WFM. If there still exists a reproducible bug in 3.0 trunk builds, let's open a new bug for it.

Comment 124

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #123)

David Bienvenu, "LF without CR" and "NUL character" related issue can still occur, if such data is passed to Tb by POP3/IMAP server, or if such mail data is imported by Tb.
>(examples)
> Original issue of this bug : "LF without CR" is imported
> Bug 301010 : MS Exchange server passes "LF without CR" to Tb
> Bug 477799 : NUL is imported from Outlook family's mailbox
So, protection from such data is still required.
(A) A way to protect Tb from contaminated mail data
    (== be tolerant with RFC violation by server or other mail client)
    e.g. Reject contaminated data, replace contaminated data
(B) A way to avoid passing contaminated data to server
    (== avoid RFC violation by Tb)
    e.g. Remove contaminated data, replace contaminated data
Note:
When NUL case(Bug 477799), Tb doesn't violate RFC fortunately, because Tb currently stops appending mail data at NUL.

Comment 125

[David :Bienvenu]

we do try to fix CRLF issues when appending to the imap server. NULL characters are a bit different, but I think much rarer.