63162 - mail messages and attachments truncated

Status: RESOLVED INCOMPLETE

(MailNews Core :: Networking: SMTP, defect, P1)

Description

[philippe tapon]

mail messages and attachments truncated
 at line 143 (text) and line 38 (?) of HTML with embedded image

Comment 1

[Keyser Sose]

Reporter we need more information. What build are you using? OS? Platform? Is it in all mail messages? Is it POP or IMAP? Please provide more information.

Comment 2

[philippe tapon]

more info.  Build ID 2000120713; Linux on Macintosh.  2 (POP) mail messages were
affected.  I got one message in full, read it, but the next time I opened it
(from my local hard drive) it *appeared* truncated.  My reply to the sender
*also* got truncated.  (I'm using text composition only.)  NOTE: I cannot
reproduce bug (yet).

Comment 3

[philippe tapon]

should add: checked Inbox-übermessage in ~/.mozilla : text file itself is
actually truncated.

Comment 4

[Keyser Sose]

Any luck reproducing it?

Comment 5

[philippe tapon]

I got a truncation when I moved a message from the Unsent Folder back to the Drafts folder.  My impression is that it truncates when you move a message to the folder and then (?) open it before it has had a chance to copy the whole thing.  Just a guess.

Comment 6

[Keyser Sose]

Marking NEW as per comments.

Comment 7

[Sheela Ravindran]

change qa contact to myself,
Add to the cc list

Comment 8

[Sheela Ravindran]

reassign->fenella

Comment 9

[scottputterman]

cc'ing naving.  Navin, is this a dup of the corruption bug you fixed a few weeks
ago?

Comment 10

[fenella]

To Esther ..

Comment 11

[Navin Gupta]

reporter, can you try again with latest nightly builds. If it has do with 
move/copy messages it may have been recently fixed.

Comment 12

[Mack Earnhardt]

I had problems today using 20010227 on WinNT. I sent a pdf to someone who
reported they couldn't open it. I recreate the file thinking it was a settings
issue and tried to send it but could not. The SMTP connection would show
activity for a short time and then stop.  After some delay, the send would time
out. After several tries, I zipped the pdf and sent it _apparently_
successfully.  Recipient could not open it. I sent a copy to myself and found
the attachment had been liposuctioned in chunks of 27 lines each.  Attached below:

Comment 13

[Mack Earnhardt]

Attachment: mime e-mail as sent

Comment 14

[Mack Earnhardt]

Attachment: mime e-mail as received

Comment 15

[scottputterman]

reassigning to ducarroz. Esther,Fenella, or JF, could you try this out with the
attached mail and see if you can reproduce this.

Comment 16

[Christopher Blizzard (:blizzard)]

OK, I did some testing with my 0.8.1 build and it's definitely got a problem.  I
tried sending myself a gzipped file that was about a meg in size after it was
compressed.  The on disk temporary files looked fine and the sent mail was saved
successfully to my imap server 5 states away but the file comes in through in
email truncated.

Upping the priority on this, setting it as an SMTP transport problem since
everything else seems to have worked fine and giving it a 0.8.1 milestone even
though the odds of actually getting it done in that timeframe are very low. 
Corrupting attachments is really really bad.  Let's see if we can track this
down and get it on the right people's radar.

Comment 17

[Christopher Blizzard (:blizzard)]

Adding my buddy seth since he's good about getting traction on issues like this
for me.

Comment 18

[scottputterman]

Jean-Francois or Varada, 

Is this a dup of our other bugs where sending has problems with large amounts of
data, especially on Mac and Linux?  If it is, then it's not going to be able to
be done for mozilla0.8.1. We have numerous people looking into this and haven't
had much luck.  See 70928.

Comment 19

[scottputterman]

We need to fix this. If it turns out to be a dup we can mark it that way.

Comment 20

[Christopher Blizzard (:blizzard)]

*** Bug 72846 has been marked as a duplicate of this bug. ***

Comment 21

[Christopher Blizzard (:blizzard)]

OK, here's an update of what I've found.

First of all, the fact that the message was truncated was completely false.  It
wasn't truncated on the server.  But when I used view->message source it looked
truncated in imap.  I'm guessing there's a limit on how much we download or
something.  It looked just fine if I viewed it as a POP account.  This is a
different problem.

However, the attachments _are_ currupted.  Here's a line from the attachment:

TryyuFN29BPLX1IF/YStZPFWXoxlEruItfMi+kbG5fYJ5kgr0XWP0Mymo9nfpmtotvKm/uG8
MVVPszuTIDlho2sgjGDvllNS0cMNXfjzRWALtMCOzejXeHvOpsd1cLw5rhtsfLyRGZOAGS2T9NA2a2GZg+ZXyIAcsNSbC8ksJpZAEa/SUlCY3GaWH9vqRfNHosRY/rt+PMcQ1GXDjfTRRx
LNLyKq9/OC5Y9TiGJcHxpFXBcb2I47CWHk/f2C8cqww4thUmGf9WJjPVEX92NZlLFA+5Buf7

Note the middle line is longer than it is supposed to be.

Where is this translation done?  I can probably debug this if someone points me
at the right file.

Comment 22

[Christopher Blizzard (:blizzard)]

I need to verify that this isn't happening at the tranport layer to the SMTP
server.  The on disk file that it starts out as is probably a good place to start.

Comment 23

[Brendan Eich [:brendan]]

Is blizzard the only one working on this currently?

/be

Comment 24

[Christopher Blizzard (:blizzard)]

OK. This looks like a transport problem to me. The on-disk image is just fine
but there is some data lossage when sending to the SMTP server or something. If
I take the resulting base64 encoded data and strip out the headers I end up with:

-rw------- 1 blizzard blizzard 1896198 Mar 22 14:19 before
-rwx------ 1 blizzard blizzard 1845782 Mar 22 14:19 after

That's why we have some long lines in the mime encoded data - some of the
newlines have been stripped out somewhere. Digging more.

Comment 25

[Christopher Blizzard (:blizzard)]

Attachment: patch

Comment 26

[Christopher Blizzard (:blizzard)]

This patch fixes the short write problem in |nsSocketBOS::Write| with
non-blocking sockets.  This is where we were loosing data.  With this patch in I
was able to send two 1.3 meg attachments without any corruption.

"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"
"PR_Write may do a short write on a non-blocking fd"

Comment 27

[Brendan Eich [:brendan]]

writte is uselessly initialized to 0, but otherwise I like it. 
r/sr=[e-mail], reassigning to dougt.

This bug is from 12/18, but dougt's changes landed on 2/21.  Two bugs, or just
same bug in old code?

/be

Comment 28

[Brendan Eich [:brendan]]

s/writte/&n/

/be

Comment 29

[Darin Fisher]

[this is not dougt's fault... i wrote this code thinking that even blocking
 writes could be partial, conforming to POSIX's definition of write.]

Comment 30

[Brian Ryner (not reading)]

r=bryner

Comment 31

[Christopher Blizzard (:blizzard)]

This is in the 0.8.1 branch.

Comment 32

[Christopher Blizzard (:blizzard)]

Fix is in.

Comment 33

[Darin Fisher]

I THINK WE NEED TO RECONSIDER THE SOLUTION TO THIS BUG!


the statement that the thread invoking this function blocks until *all* data is
written, is wrong.  consider the following code:

while (count) {
  int n = write(fd, buf, count);
  if (n < 0) break;
  buf += n;
  count -= n;
}

the first call to write could return (n > 0 && n < count).  the second call
could return (n == -1).  then this code (taken from PR_Write) would return the
error and mask the fact that some bytes were written.  this seems completely
wrong IMO.  in my opinion this convention is a mistake.  i understand that it is
the way NSPR has always been... but, i have my doubts about whether or not it
sould be the convention for nsIOutputStream::{Write,WriteFrom,WriteSegments}.  
consider nsIOutputStream::WriteFrom, for example.  A caller might say "write 256
bytes from this input stream"... but what if the input stream only has 10 bytes
available.  should WriteFrom block until the input stream has more bytes?  what
if that was EOF on the input stream?  then should WriteFrom return an error
code?  and mask the fact that it was able to write 10 bytes?  i don't think so.
 *no* caller of WriteFrom would want it to block the calling thread until *all*
data was written.

in the case of mailnews's use of the socket transport's blocking output stream,
it should really be the one to sit in the while loop... not the implementation
of nsSocketBOS::Write. 

Comment 34

[Darin Fisher]

sorry to be such a stick in the mud about this, but we are being VERY
inconsistent.

Comment 35

[Doug Turner (:dougt)]

We have a working patch.  Please submit an alternative patch.

Comment 36

[Darin Fisher]

The current fix can/should stay for now... but, when I get a chance I'd like to
go back and work on a patch that fixes mailnews' usage of nsIOutputStream::Write

Comment 37

[Darin Fisher]

downgrading severity to minor since we have a working solution.

Comment 38

[skiingyac]

I am having the same problem with linux build 2001071108.  I have a email message in my imap inbox which I have opened before and the full message was displayed.  Now, I can only view the headers and the first line.



The message is displayed in the message list as having a size of 6KB.  If I go to view source, I can see the headers and only the first line of the body.



I can use webmail and see the entire message still.  Closing the mail windows and reopening it with CTRL-2 doesn't change anything.  I have to fully close mozilla and reopen it, then I can view the full message.  I wasn't doing anything abnormal when it happened.

Comment 39

[Charles Gillet]

Attachment: This message gets truncated inside the mime block for the image/tiff chunk

Comment 40

[Charles Gillet]

Is this bug still alive for Mozilla 1.0 RC1 ? I am having a problem with it
truncating certain attachments from our fax system that come in tiff format. 
The mail reader does not properly interpret the mime encoding and does not
display the message as having an attachment.  The view source windows displays a
truncated mime encoded block of text.  But when I do a "save as" on the message,
it saves to file with no truncation (but with annoying windoze ^M's ending each
line).

I use IMAP w/ssl (UW 2001a) on Redhat 7.2/intel.  Am I posting this to the right
place?  I will attach a problematic (intact) mail message that mozilla trashes.

Comment 41

[Jean-Francois Ducarroz]

Using a Mozilla RC1 build on Windows, this message display correctly and I can
see and open the tiff attachment. I am using POP. Let me check IMAP...

Comment 42

[Jean-Francois Ducarroz]

works too from IMAP for me

Comment 43

[Charles Gillet]

Well, to clear IMAP of any guilt, I opened the message over IMAP with Netscape
4.79-linux-intel and it sees the attachment just fine.  As the bug was filed
against Linux, does anyone else watching this bug have a Linux/intel test bed to
try that attached message on?

Comment 44

[Charles Gillet]

the posted image/tiff attachment which didn't work in RC1 works now in RC2 on my
setup.  

Comment 45

[nchorner]

I think I'm experiencing the same problem.  I have a .mbox archive of old
messages, and I'm running Mac OS X (10.1.5) with Mozilla 1.1 released last night
(8/26).  (I had the same problem in 1.0.)  Some of the messages in the mailbox
have attachments, but in the mail program they load for a few seconds and then
show up blank.  Other mail programs display the messages correctly.  I checked
the message source as Mozilla sees it against the actual text of the .mbox file,
and Mozilla is truncating the messages at around 700 lines or just under 48,500
characters (don't know if that's helpful or useless).

Comment 46

[[e-mail] (gone - use [e-mail] instead)]

The nsbeta1+ and mozilla0.8.1 marking added by putterman and brendan on
2001-03-21  and 2001-03-22, respectively, are no longer useful.

Clearing those.

Comment 47

[Mark Banner (:standard8)]

Is anyone still seeing this? With no duplicates or recent complaints I really doubt that this is a problem any more.

Comment 48

[Joshua Cranmer [:jcranmer]]

RESO INCO per lack of response to comment 47. If you feel this change was made in error, please respond to this bug with your reason why.