115179 - Null character in body of mail message results in corruption of that an following message.

Status: RESOLVED WORKSFORME

(MailNews Core :: Movemail, defect)

Description

[Peter B. West]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.6+) Gecko/20011212
BuildID:    20011030

If a null character occurs in the text of a message processed by movemail, the
message is truncated at that point, and the beginning of the following message
is lost.  The remainder of the subsequent message is joined to the truncated
message.

Reproducible: Always
Steps to Reproduce:
1.Process mail including a null character with movemail
2.
3.

Actual Results:  As described.

Expected Results:  Messages viewable in full.

I have not seen this bug for some time, because I unsubscribed from the source
which I suspected was responsible, and because I was anxious to avoid mail
corruption.  That's why the build ID is an old one.  I have just re-subscribed,
using procmail to filter the suspicious messages, and I have seen the null
characters in these messages.  However, I have not tried to reproduce the
corrupt messages by processing these null-containing messages with movemail.

Comment 1

[Andrew Schultz]

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.7+) Gecko/20020111
I can confirm that Mozilla perceives a NULL byte as the end of a message.  This
was in a doctered local folder.

If I do View->Message Source, the rest of the message shows up fine.  This is a
possible workaround.

Comment 2

[Peter B. West]

The serious difficulty occurs when the message is being received.  The message
withe the null is truncated and the following message (at least - it may affect
more than just the following message) is beheaded.  The two fragments then
become one message as far as Mozilla is concerned.  The message folder contains
only the mutilated message.

Comment 3

[Andrew Schultz]

I don't use movemail, so I can't fully see what's going on, but looking at the
source code, it looks like the problem resides in xpcom/io/nsFileStream.cpp
(nsRandomAccessInputStream::readline)

It reads (n-1) bytes, NULL terminates them, and then goes looking for EOL.  If
it finds no EOL, it assumes that the line was too long for the buffer, but
having a NULL within the buffer would prevent it from finding EOL after the
NULL.  Finding no EOL, it would advance the filepointer (n-1) bytes, but only
bytes before the NULL would get back to the calling function.

This is particularly problematic when (as mentioned by Peter), the start of a
new message is within the buffer after the NULL byte.  In this case, the first
part of the next message is lost and the rest of it is appended to the first one.

This is a much more serious bug from the perspective that it gives a new meaning
to the term "mail bomb".  Anyone could trash a Mozilla-movemail user's mail by
sending mail with NULL bytes near the end.

I will attach a patch that might fix this.  I can't test this as I don't use
movemail.  It compiles and runs ok, but might not work as expected.

Comment 4

[Andrew Schultz]

Attachment: patch

Comment 5

[Keyser Sose]

Marking NEW. Andrew you should contact the correct owner on this list
http://www.mozilla.org/owners.html and get a review/sr for the patch to be
checked in. Thanks for the patch.

Comment 6

[Peter B. West]

I will try to test this offline by sending myself some of the messages which
have caused problems in the past.  However, I am reluctant to test this "live"
for obvoius reasons.

What I am doing at the moment is using procmail to redirect messages from the
troublesome source into a mozilla folder, where they continue to cause me
problems by not having their "viewed" status recognozed across a restart.  Is
there any information on how to use procmail to direct incoming into mozilla
folders?

Comment 7

[Andrew Schultz]

sorry... fell off my radar.
adding dataloss keyword

I tried to test out the patch by creating a movemail account (via NS4
migration), but I was unable to get it to receive any mail.  :(

If this is still a problem for you, this bug will probably belongs in XPCOM
(based on my patch).

Comment 8

[Andrew Schultz]

Attachment: patch v2

a patch that actually works

Comment 9

[Andrew Schultz]

ok, I can't confirm that the patch fixes the bug (although it should) because I
can't get movemail to work.  However, it compiles ok and mozilla runs fine with
the patch.

==> XPCOM

to sum up: movemail uses nsRandomAccessInputStream::readline to read a file that
might have a null in it, but that routine does not properly handle a null,
resulting in dataloss.  see comment 3 for more gory details.

the patch makes readline properly handle a null.  

Comment 10

[Andrew Schultz]

no really, ==> XPCOM

Comment 11

[Doug Turner (:dougt)]

nsRandomAccessInputStream and subclasses are probably not the ideal thing to
use.  Can you fix up the callers to use necko and the nsILineInputStream
interface for
readline().

Comment 12

[Andrew Schultz]

I had a sneaky suspicion you would say that... :)
That looks like it's a bit beyond the scope of my brain, especially since I
can't even get movemail to work to test it out.  :(

I can plug away at it, and might accomplish something before 1.1, but don't hold
your breath.

Does this bug belong back in Movemail then?

Comment 13

[Doug Turner (:dougt)]

back to movemail.  

Comment 14

[Peter B. West]

Has there been any movement on this one?

Comment 15

[Blu3]

Someone please fix up mozilla's mail client to handle emails with NULLs in them.
 The latest nightly (nov 13th/2002) is worse at handling emails with NULLs in
them.  I have to manhandle my INBOX by hand and delete the emails.  I used to be
able to select the group of emails around and including the emails with the NULL
in them and was able to delete them and then advance through my INBOX normally.

Now, I can't do anything.  If Moz tries to access an email with a NULL in it, it
messes up $MAILBOX (INBOX) completely.  I have to restart Moz to get back to my
emails prior to the NULL containing emails.

Comment 16

[Brant Gurganus]

By the definitions on <http://bugzilla.mozilla.org/bug_status.html#severity> and
<http://bugzilla.mozilla.org/enter_bug.cgi?format=guided>, crashing and dataloss
bugs are of critical or possibly higher severity.  Only changing open bugs to
minimize unnecessary spam.  Keywords to trigger this would be crash, topcrash,
topcrash+, zt4newcrash, dataloss.

Comment 17

[Adam D. Moss]

->nobody

Comment 18

[tobias]

Attachment: corrupt mail

i recently received a mail containing no NULL character but =00 - dunno if it
makes a difference for you. i attached the eml file. the msgsrc can be viewed
correctly and thus i guess its correctly stored but incorrectly displayed.

Comment 19

[David :Bienvenu]

Has anyone tried this with a recent 1.5 trunk build? I made some fixes w.r.t.
handling nulls in mail messages (not directly for movemail, but in our line
buffering code).

Comment 20

[Blu3]

Yup, works fine

thanks :)

Comment 21

[Peter B. West]

I haven't tried this one, but I haven't experienced any problems for some time
now, possibly because the source of my NULL-containing messages cleaned up their
act.  I'm now running on stable builds, not nightlies.

Comment 22

[David :Bienvenu]

marking wfm, then - this is probably a dup of one or more of a couple bugs I
fixed recently.