110067 - mail attatchment corruption when configured for offline

Status: VERIFIED FIXED

(MailNews Core :: Backend, defect)

Description

[Judson Valeski]

Using win2k, NS6.2.

problem: large mail attatchements are corrupted when viewing or saving to disk.

steps to repro:
1. receive a large mail attatchment (someone sent me a 1.5 Mb powerpoint preso)
in your inbox.
2. right click the attatchment and save it to disk.
3. open the file in an appropriate viewer (again, for me it was powerpoint).

actual results:
viewer pukes indicated that it can't open/read the file.

expected results:
viewer is able to open the file.

Comment 1

[Judson Valeski]

I just saw this w/ a 475k byte zip attatchment.

Comment 2

[Peter Trudelle]

May be branch-onlyh, this works for me using recent trunk builds on Win2K.

Comment 3

[scottputterman]

a 5MB file worked fine on the trunk as well.  Jud, were you able to send the
file to mscott?

Comment 4

[Judson Valeski]

sorry, should have posted status. I emailed the 475k attatchment to mscott a few
hours ago. I should also note that downloading and using the file using Outlook
express worked fine, ruling out the possibility that a corrupt file was sent.

Comment 5

[Marek Z. Jeziorek]

I experience it very frequently on my Win 2000 N6.2 (as well prior daily/weekly
builds). I am on nsmail-1. The only other person I know who got it also
(michaell) was on nsmail-1. The size of the attachment doesn't matter. When I
look at the temporary file it usually is very small 1k-30k. I expereinced it for
MS Word, Excel and PowerPoint. When I use the web interface to nsmail-1 (again
6.x doesn't work because of some stupid Javascript problem) I can get the
attachment.

I sent mscott several times about it, but none of the times I could reproduce it
again.

Comment 6

[Mitch Green]

I concur w/ Marek.  The size of the attachment doesn't matter.  And it seems to
happen most frequently with MS Office documents (although they also comprise the
majority of the attachments I receive).  And as Jud mentions, I'm able to
correctly download/open a attachment using another mail client, eliminating the
attachment as the actual problem.

Comment 7

[Judson Valeski]

changing summary from "mail attatchment corruption"

Comment 8

[Judson Valeski]

I also just verified that this is broken on the trunk.

some data:
Outlook Express is downloading the complete file (based on file size), whereas,
NS is only downloading the first 4096 bytes (undoubtedly one chunk of a buffer).
I bet we're fine w/ attatchements <= 4096 bytes.

Comment 9

[Scott MacGregor]

looks like mime parts on demand broke again......

Comment 10

[David :Bienvenu]

Do all of you having the problem have the folder in question configured for
offline use? I tried it with a folder not configured for offline use and can't
reproduce the problem (at least for zip and .exe files - I'll try a word file next).

Comment 11

[Susie Wyshak]

FYI I just received a 70MB zip file with no corruption. It self extracted
successfully after I saved it to disk.

Using Win2K with Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.4)
Gecko/20011019 Netscape6/6.2 

Sent by a 6.2 user on Win 98.

Comment 12

[Mitch Green]

There are two locations to select my Inbox for Offline use.  The first is in the
"Edit | Mail & News Account Settings | Offline & Disk Space " dialog.  None of
the Offline checkboxes are checked.  Specifically, "Make the messages in my
Inbox available when I'm working offline" is NOT checked.

When I choose the "Select..." button in that group though, my Inbox is selected
for offline use when I go offline. 

I haven't even attempted to work offline for months. 

Comment 13

[David :Bienvenu]

Mitch, that means your inbox is selected for offline use. I'm pretty sure Jud's
is as well.

Comment 14

[David :Bienvenu]

If I configure my Inbox for offline use, I can reproduce this on a .pps file.
This problem happens when we download the message for offline use when you read
it (not if you download the message for offline use by doing an offline sync, or
by downloading selected/flagged messages). What happens is that we're putting
the partially downloaded message in the offline store, and pretending it's the
whole message. We should not put partially downloaded messages in the offline
store. I'll look at a fix.

Comment 15

[David :Bienvenu]

Attachment: proposed fix

If we're going to potentially download this message by parts, don't put it in
the local cache. This will mean that a few messages which we would have
downloaded normally (e.g., all the parts were inline) won't end up in the
offline store, but that's better than messages we don't download completely
ending up in the offline store.

Comment 16

[Judson Valeski]

I'm config'd for offline use.

Comment 17

[David :Bienvenu]

Even with this fix, you will need to delete your offline store (or your .msf
file) in order to be able to download the attachments successfully.

Comment 18

[scottputterman]

David, does your last comment mean you'd have to do this to make the current
messages work or would we have to delete the .msf file to make messages after
your fix work as well?

Comment 19

[David :Bienvenu]

the first one. After the fix, all newly downloaded messages should work fine
whether or not you delete your .msf file.

Comment 20

[Scott MacGregor]

i should note that i'm not configured for offline use yet the attachment judson
sent me is corrupt. That's probably because when we forwarded the message we
didn't fetch the part because he is configured for offline on his inbox. So the
attachment I'm trying to open is already corrupt.

That explains why I've never had any problems opening attachments before except
for this one of Judsons. 

clarifying the summary to note that this only happens when you are configured
for offline.

Comment 21

[grylchan]

*** Bug 107314 has been marked as a duplicate of this bug. ***

Comment 22

[grylchan]

adding myself to cc. Esther if you want help on qa
part let me know. Other test cases for this bug
can be found in dupe bug 107314

Comment 23

[esther]

Gary, this is specific to Offline and you already had a bug for this, so I will
reassign the QA to you. Thanks.

Comment 24

[David :Bienvenu]

Can I get some reviews, please?

Comment 25

[Navin Gupta]

Comment on attachment 57940 [details] [diff] [review]
proposed fix

r=naving

Comment 26

[Scott MacGregor]

Comment on attachment 57940 [details] [diff] [review]
proposed fix

sr=mscott

Comment 27

[David :Bienvenu]

fix checked into trunk - will wait for it to bake on trunk before checking into
branch.

Comment 28

[Jaime Rodriguez, Jr.]

Adding PDT for tracking and 6.2 branch review purposes.

Comment 29

[grylchan]

These are my observations using 2001111603 commercial trunk
build on NT 4.0:

 Folder (not inbox) not selected for download use:
  - while online, I could open an attachment (ppt,doc,zip,txt,jpeg,pdf,rtf)
    or save it with no problem
  - while offline, obviously i could not open or save an attachemnt
 Folder (not inbox) selected for download use:
  - Online
    -as I click on a mesg w/out attach, it turns dark grey and is downloaded
    -click on a mesg w/a certain attach (jpeg,txt, rtf,vcf), it turns dark
     grey and is downloaded. I can open/save attachments with no problem.
    -click on a mesg w/attach (doc,ppt,xls,pdf,exe,htm,dll) and it doesn't
     get downloaded (envelope stays white).  I can open/save attach with
     no problem. 
      Is this the expected result? It's not a size issue (as i tried different
      sized files and i do not have the pref 'do not download mesgs > x kb'
      set.
  - Offline (did not download mesgs at the prompt)
    - can read any mesgs that have been downloaded. Mesgs that had
      attachments (rtf,jpeg,txt) i can open or save the file with
      no problems
    - mesgs that were not downloaded obviously I can not read or open,save
      the attachment
  - Offline (did say yes at download mesg prompt)
    - all mesgs were downloaded/envelope dark gray
    - able to read any mesg and open/save any attachment with no problem

 Inbox (set the pref 'make my mesgs in my inbox available when i am working
        offline)
   Online
       - all mesgs downloaded
       - able to open/save attachments with no problems
   Offline (did not download at prompt since already downloaded)
       - able to read all mesgs
       - able to open/save attachments with no problems
   
Let me know if I missed anything. But looks ok so far.

Comment 30

[Jaime Rodriguez, Jr.]

Adding ETA of 11.19. mscott, pls let us know, when you think this is ready for
checkin everywhere needed.

Comment 31

[scottputterman]

reassigning to bienvenu.

This has been landed on the trunk and tested.  

Comment 32

[Judson Valeski]

works for me using today's windows trunk build.

Comment 33

[Jaime Rodriguez, Jr.]

PDT- for the Netscape 6.2 branch. Most users will not run into this problem
because it requires changing the default state.

Comment 34

[Judson Valeski]

re-nominating for 6.2 branch.

Comment 35

[Jaime Rodriguez, Jr.]

Adding workaround from scottip's email ...

A workaround is to change the "Do not download messages larger than 50KB" on and
then this will work (of course, you won't have the message when you are offline).

Comment 36

[Jaime Rodriguez, Jr.]

PDT+, pls check this one in asap.

Comment 37

[Navin Gupta]

checked in on branch

Comment 38

[Sven Hazejager]

Looks like the bug is fixed. Great! What version of Netscape (or are we talking
Mozilla here?) should I download to get the fix?

Thanks!

Comment 39

[Niranjani Raya]

Time Inc has encountered the same problem. Where can I download the build from
which includes the fix? 

Comment 40

[scottputterman]

any nightly build (Mozilla or Netscape) will have this fix.

Comment 41

[Niranjani Raya]

Cu is a little concerned about this. 
He asked if 
- there is a way to deal with this outside of a new version?  
- they have rolled out at least half of their clients (that is about 6500).  A
new version means literally thousands of dollars in labor for them.  
- there is anything they can do to patch the existing installs?

Comment 42

[Sven Hazejager]

Where can I find Netscape nightly builds? Mozilla is at mozilla.org, but I can't
find Netscape...

Comment 43

[scottputterman]

David, correct me if I'm wrong on this.  To avoid this problem in a build
without this fix you could:

1.  Go to Account Settings
2.  Select the Offline panel for the affected account.
3.  Turn the "Do not download messages > 50 KB" on.

or don't make the folder useable for offline.

This would make it so that future messages aren't affected.  To make it so that
current messages work you would have to delete the .msf file of the affected folder.

This made it work for me.

Comment 44

[David :Bienvenu]

I think you'd have to make that pref be 30K instead of 50K and turn it on, as
you say. Or, you could turn off mime parts on demand by editing your prefs.js -
not sure what AOL/TW IS would prefer.

Comment 45

[grylchan]

Using 20011126 commercial bits on NT 4.0, Mac OS 10.1, linux 2.2
and 20011127 on Mac 9.1

Did most of the extensive testing on Nt4.0 and Mac 9.1
For Windows, I tested both a clean install and installing 
a new verison over previous version.

For windows:
 A.tested a folder with attachments (before being selected for download)
   attachments tested: doc,zip,xls,jpeg,rtf,ppt,txt
   Size of attachments varied
   -tested attachments can be read,opened,saved while online
 B.tested a folder with attachments (after being selected for download)
   attachments tested: doc,zip,xls,jpeg,rtf,ppt,txt
   Size of attachments varied
   1.tested attachments can be read,opened,saved while online
     -Files > 30kb
         -not downloaded when you select a mesg
         -could open/save the attachments with no problems
     -Files < 30kb
         - selecting mesg results in it being downloaded
         - no problems opening/saving an attachment
   2 [details] [diff] [review].tested attachments can be read,opened,saved while offline
      - no problems opening/saving an attachment (those that were downloaded)
      - obviously mesgs not downloaded, i had no access to
   3.tested attachments, while offline, after downloading all mesgs
    - no problems opening/saving an attachment
On Mac 9.1:
 Using attachments (doc,ppt,jpeg,xls,txt,pdf)
 A. tried opening/saving attachments, no problems 
 b. selected folder for offline use, then tried clicking on various mesgs while
    still online
     -Files > 30kb
         -not downloaded when you selected them
         -could open/save the attachments with no problems
     -Files < 30kb
         - selecting mesg results in it being downloaded
         - no problems opening/saving an attachment
 C. downloaded entire folder and went offline
    -all mesgs/attachments downloaded
    -able to read/save any attachment

In addition I tested various ways of downloading mesgs w/attachs
[download/sync,get flagged/selected mesgs,download mesgs at download
prompt, folder properties menu (select this folder and dwnld now button),
when i create new folders make them available for offline, make my inbox
available when I work offline]
and did not have any problems with opening/saving attachments while
on/offline.

Verified there is no attachment corruption when a folder is
selected for offline use.

Marking as verified.

Comment 46

[David :Bienvenu]

*** Bug 92993 has been marked as a duplicate of this bug. ***

Comment 47

[Boris Zbarsky [:bzbarsky]]

*** Bug 86206 has been marked as a duplicate of this bug. ***

Comment 48

[Michael Buckland]

Adding to status whiteboard AOLTW and plussing bugs definite requirements,
adding just AOLTW for possibles.

Comment 49

[Michael Buckland]

Updating AOLTW