746711 - Saving attachment produces mail with "This body part will be downloaded on demand" （when browser.cache.memory.enable=false/browser.cache.disk.enable is false and IMAP folder with offline-use=Off)

Status: RESOLVED WORKSFORME

(MailNews Core :: Networking: IMAP, defect)

Description

[Alain Knaff]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0
Build ID: 20120310194926

Steps to reproduce:

Attempted to save the attachment of a message


Actual results:

The saved file had the name of the attachment, but had as contents a mail file having the attachment, and with body text "This body part will be downloaded on demand".

Saving it several times in a row didn't help.

Moving the message to a local folder, and then saving the attachment works ok.



Expected results:

Saving the attachment should just save the attachment, not anything more.

Comment 1

[WADA:World Anti-bad-Duping Agency]

POP3? IMAP? I assume IMAP.
Offline-use=On folder? Offline-use=Off folder? (Folder Properties/Synchronization)
I assume IMAP & Offline-use=Off folder.

There is known issue in application/octet-stream handling like next.
(i) mime-parts-on-demand downloads non-inline-diplayable part data when requested. So, he puts "This body part will be downloaded on demand" in partial mail data to show message body.
(ii) application/octet-stream handling code tries to show part in inline as many as possible.
(iii) action of (ii) changes internal status of the part to "not attachment for IMAP code", even thouh the part is shown as if attachment at attachment pane.
(vi) then fetch for the part by (i) is not invoked upon save.
This case?

Can you show us message headers and structure of the mail?
(1) Save the mail as .eml file.
(2) Edit .eml file by test editor. 
(2-1) Remove many data lines for attachment data. (data itself is irrelevant)
(2-2) Remove/replace personal iformation. Message body text is also irrelevant.
(3) Attach .eml file to this bug(never paste long data, please)

Comment 2

[Alain Knaff]

Attachment: One mail with attachment showing the problem

A mail with an attachment showing the problem.
But actually, *any* attachment will exhibit this (i.e. so far, I haven't yet seen a case where it worked in this 11.0.1 version when accessed directly from the imap server)

Interestingly enough, the saved attachment seems to not only have the body replaced with "This body part will be downloaded on demand", but also all non-selected attachments (if there were multiple attachments)...

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Alain Knaff from comment #2)
> One mail with attachment showing the problem

"Content-Type: application/octet-stream of multipart/mixed" case.

Can you see problem with well-formed Content-Type: for .pps file?
  Content-Type: application/vnd.ms-powerpoint;
(1) Save the mail to .eml file
(2) Edit .eml file
(2-1) Change Subject: for ease of testing
(2-2) application/octet-stream => application/vnd.ms-powerpoint
(3) Drag&Drop the .eml file to thread pane of same IMAP folder
    (manual import of mail via Drag&Drop of .eml file)

Do you have entries in Tools/Options/Attachments?
Can you see your problem with clean mimeTypes.rdf file?
  rename mimeTypes.rdf to mimeTypes.rdf.Backup, and restart Tb

Comment 4

[Alain Knaff]

Nope, even after removing mimeTypes.rdf, the problem still persists.

Where may I find your example .eml file?

Comment 5

[WADA:World Anti-bad-Duping Agency]

FYI.
Open bugs which has "octet-stream" in bug summary.
https://bugzilla.mozilla.org/buglist.cgi?list_id=2908088;chfieldfrom=1y;short_desc_type=allwordssubstr;short_desc=octet-stream;resolution=---;chfieldto=Now;query_format=advanced;product=Core;product=Documentation;product=MailNews%20Core;product=Mozilla%20Localizations;product=Mozilla%20Messaging;product=mozilla.org;product=NSPR;product=NSS;product=SeaMonkey;product=Toolkit;product=Webtools

(In reply to Alain Knaff from comment #4)
> Where may I find your example .eml file?

What do you call by "my example .eml file"? My request is check with mail on which you saw your problem, with altering Content-Type: from application/octet-stream to application/vnd.ms-powerpoint.

Comment 6

[Alain Knaff]

Actually, the problems happens for _all_ attachments, even just attaching /etc/issue to a mail, in case I didn't make this clear yet. The example was just that: an example that I had lying around on top of my folder.

It happens on Thunderbird 11 on my machine at work on Fedora 16

It doesn't happen (on the same messages, browsed off the same imap server) on my machine at home with thunderbird 3.1.20 on Kubuntu 10.04

Comment 7

[rsx11m]

(In reply to WADA from comment #1)
> POP3? IMAP? I assume IMAP.
> Offline-use=On folder? Offline-use=Off folder? (Folder Properties/Synchronization)
> I assume IMAP & Offline-use=Off folder.

Alain, please provide information on the settings in Synchronization & Storage if the folder in question is synchronized (i.e., "Keep messages on this computer" and the respective box for the folder are checked in the "Advanced" view, where these preferences can be found in Edit > Account Settings).

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to Alain Knaff from comment #6)
> It doesn't happen (on the same messages, browsed off the same imap server)
> on my machine at home with thunderbird 3.1.20 on Kubuntu 10.04

Problem depends on "whole mail data is held locally" or not.
If offline-use=on folder and auto-sync is enabled(enabled by default), whole mail data is downloaded to offline-store file. So, as you saw on mail copied to local mail folder, problem doesn't occur unless data in offline-store file is corrupted.

Comment 9

[Alain Knaff]

The "Keep messages on this computer setting" is off (actually it doesn't say that, but the text is unpastable...). In "Advanced", everything in unchecked.

What is the name of "offline-store file"? Maybe if this is corrupted, I can "fix" it by removing that file, and cause it to be reconstructed?

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Alain Knaff from comment #9)
> What is the name of "offline-store file"?

"Offline-store file" I called is file named Inbox(not Inbox.msf) if folder named Inbox, which is used for IMAP folder of offline-use=On(at Keep this .../Advanced, folder is Checked).
So, it's irrelevant to your PC for which you reported comment #9.
How about your machine at home?

Comment 11

[Alain Knaff]

Indeed, removing all .msf files, and restarting thunderbird (at work) did not help.

On my machine at home (where saving attachments works ok), the "Keep messages on this computer" box is unchecked as well.

Comment 12

[rsx11m]

If not synchronization, maybe it's a caching issue of some sort. There have been previous problems associated with that, but those should be fixed by now.

You may need to double-check the per-folder synchronization settings that the box is still unchecked after removing the ".msf" files (but I'd assume they are reset to the default which in your case should be off).

Go into Edit > Preferences > Advanced > General and click on Config Editor. There, enter browser.cache into the search bar of the window that opens and double-click on the two entries browser.cache.disk.enable and browser.cache.memory.enable to set them "false" (which will disable caching entirely, thus Thunderbird may behave sluggish now). Restart Thunderbird and see if the problem is resolved now. If yes, we'd have to further investigate what's happening in the cache.

By any chance, do you know if the server you are connecting to is an MS Exchange server? There have been issues with downloading attachments on demand, but that should affect viewing and saving equally and affect both sites where you are accessing it from...

Comment 13

[Alain Knaff]

Both browser.cache.disk.enable and browser.cache.memory.enable are already set to false. Thunderbird has a reasonable speed, except at start, due to a bug in Lightning.

.msf files came back after the restart (and bug is still present).

No MS Exchange server involved.

Interestingly enough "Open" rather than "Save" on the attachment works... So if I may venture a guess, I'd think that the bug must exist rather late in the "pipeline", after the download happened, but just before the attachment is written to disk. Or else, Open would fail too.

Comment 14

[rsx11m]

David, any idea what may be the difference between "Open" and "Save"? The "Open" should use /tmp as download location, but that should be independent from whether or not the attachment is available for on-demand downloading.

Comment 15

[David :Bienvenu]

none of this is really making sense. A file this small (3K for the test e-mail) shouldn't trigger mime parts on demand) so we should be downloading the whole message. The pps file in the test-mail can't be read by my version of PowerPoint (it seems way too small).

When I save the attachment, I get a 274 byte file w/o the download text in it.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to rsx11m from comment #14)
> any idea what may be the difference between "Open" and "Save"?

Difference between them is a characteristics of bug 739025.

Alain Knaff, can you see your problem with clean mimeTypes.rdf on your non-home PC?
  rename mimeTypes.rdf to mimeTypes.rdf-Backup, restart Tb

Comment 17

[Alain Knaff]

It seems that the particular sentence "This body part will be downloaded on demand" only shows up with certain messages (possibly involving multipart/alternative).

However, the confusion between saving the entire message and saving just one attachment appears for _all_ messages (I've found no exception yet), even if the attachment is just a small text file.

If the attachment is a picture, then it the inline display of the picture shows a logo of "broken image" (meaning that not just "Save" is affected by this bug, but inline display as well).

"Open" is not affected: it displays attachments all right, even images that show up as broken inline.

The bug 739025 does not seem to be related to this one, as this one (746711) happens for all mime types (whether they contain a question mark or not), is related to saving attachments (and not about a missing open dialog). But it does have one point in common: it involves a computer too... :-)
Or did you mean a different bug, and there is a typo in the number?

Comment 18

[Alain Knaff]

Attachment: Message with just a small test file as attachment

This is a much simpler test case than my first example.

Comment 19

[Alain Knaff]

Attachment: imap log file

Imap log made with
export NSPR_LOG_MODULES=imap:5
export NSPR_LOG_FILE=/tmp/imap.log
thunderbird -safe-mode

Comment 20

[Alain Knaff]

The mimeTypes.rdf didn't re-appear since I first removed it (and problem still exists)

However, the .msf files did re-appear

Comment 21

[WADA:World Anti-bad-Duping Agency]

(In reply to Alain Knaff from comment #20)
> However, the .msf files did re-appear

Sorry, do next to hide folder named mimeTypes.rdf-Backup, please.
  delete mimeTypes.rdf-Backup.msf,
  rename mimeTypes.rdf-Backup to mimeTypes.rdf-Backup.msf,
  restart Tb
  (== rename mimeTypes.rdf to mimeTypes.rdf-Backup.msf, restart Tb)
Tb doesn't consider ".msf file only file set" "mail folder".

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to Alain Knaff from comment #19)
> imap log file

Two "UID fetch 88435 (UID RFC822.SIZE BODY[])" are seen in log, one for message body display, one for text/plain attachment.
It may be relevant to bug 740453.
Do you touch Memory/Disk Cache setting?
  browser.cache.memory.enable
  browser.cache.disk.enable

Comment 23

[Alain Knaff]

(In reply to WADA from comment #21)
> (In reply to Alain Knaff from comment #20)
> > However, the .msf files did re-appear
> 
> Sorry, do next to hide folder named mimeTypes.rdf-Backup, please.

As stated before, I've actually completely removed mimeTypes.rdf, and it didn't re-appear since then.

>   delete mimeTypes.rdf-Backup.msf,

I don't have that one either

>   rename mimeTypes.rdf-Backup to mimeTypes.rdf-Backup.msf,
>   restart Tb
>   (== rename mimeTypes.rdf to mimeTypes.rdf-Backup.msf, restart Tb)
> Tb doesn't consider ".msf file only file set" "mail folder".

That last sentence doesn't quite parse... English please?

Comment 24

[Alain Knaff]

(In reply to WADA from comment #22)
> (In reply to Alain Knaff from comment #19)
> > imap log file
> 
> Two "UID fetch 88435 (UID RFC822.SIZE BODY[])" are seen in log, one for
> message body display, one for text/plain attachment.
> It may be relevant to bug 740453.
> Do you touch Memory/Disk Cache setting?
>   browser.cache.memory.enable
false

>   browser.cache.disk.enable
false


And indeed, setting browser.cache.memory.enable to true fixes the issue!

However, if saving attachments with browser.cache.memory.enable=false is unsupported, maybe there should be a clearer indication of this fact (such as a popup pointing this out, rather than producing a bad file)

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to Alain Knaff from comment #23)
> That last sentence doesn't quite parse... English please?

In mail directory(for local mail folder), Tb uses ABC, ABC.msf, ABC.sbd(directory for files for subfolders) for a local mail folder named ABC. And, if local mail folder, Tb considers a set of file/directory as "local mail folder named ABC" only when file of ABC exists. If ABC.msf only, or if ABC.sbd only,  or if ABC.msf & ABC.sbd only, Tb doesn't show folder named ABC. So, keeping a file as "XYZ.msf" can be used as a temprary backup of a file named XYZ in mail directory.
However, this is applicable to "files/directories in Mail directory for local mail folder" only.
Because mimeTypes.rdf is file in root of profile directory and is not "Mail directory", above is not applicable to mimeTypes.rdf. Sorry for my confusion.

Comment 26

[rsx11m]

(In reply to Alain Knaff from comment #24)
> However, if saving attachments with browser.cache.memory.enable=false is
> unsupported, maybe there should be a clearer indication of this fact

The goal should rather be to make it work properly. If the MIME part = attachment is downloaded on demand but not cached, it would just need to be refetched from the server, which is the part that apparently doesn't work and hence the message is presented which indicates that the MIME part isn't available.

So, the questions are why an attachment possibly not even triggering download on demand prompts that placeholder to appear; or, maybe that's the indicator for some issue in the underlying logic where a MIME part may be treated as on demand but not cached where in fact it's supposed to be part of the message, but for some reason the full message isn't refetched when not found in the cache?

Comment 27

[Rob Janssen]

We have recently updated from SeaMonkey 2.10.1 to 2.13.1 in our company (we don't track each and every update), and from what I hear from the helpdesk this issue has raised its ugly head again.
We have seen the "27 byte file when saving attachment" issue before, and it other results (error messages from Word or PDF viewer when opening attachments directly from mail).
We use UW IMAPD and I have already set a locked preference to mitigate the issue:
lockPref("mail.imap.mime_parts_on_demand_threshold", 500000);

However, it still happens for larger messages and I think its frequency has increased with the latest update (when that is not a coincidence).

It looks like the problem is worsened by my efforts to avoid caching of messages on the local filesystem (which is a critical security and privacy issue):

lockPref("mail.server.default.offline_download", false);
lockPref("mail.server.default.autosync_offline_stores", false);
lockPref("offline.autoDetect", false);
lockPref("offline.download.download_messages", 0);
lockPref("offline.send.unsent_messages", 0);
lockPref("offline.startup_state", 2);

The mail program should be able to operate with an IMAP server and no local caching of messages, even when that would be so much more efficient.

Comment 28

[Wayne Mery (:wsmwk)]

Rob, Alain,

Does this still reproduce with a current version?
If if it does still fail, does it also fail using a nightly build from https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/ ?  (which uses newer cache code)

Comment 29

[Rob Janssen]

No longer using the program in that environment.

Comment 30

[:aceman]

Yes, this may have changed with the cache2 rework. The last report here is from 2012.

Comment 31

[WADA:World Anti-bad-Duping Agency]

(In reply to :aceman from comment #30)
> Yes, this may have changed with the cache2 rework. The last report here is from 2012.

This bug is issue when both Memory Cache and Disk Cache are intentionally disabled for security reason or unknown reason or other reason i can't understand.
Was code for browser.cache.memory.enable=false && browser.cache.disk.enable=false case drastically improved by the rework for transferring from cache1 to cache2?
Or cache2 supports "no data held in memory cache" and "no data held in disk cache"?

Comment 32

[Alain Knaff]

I retested now with Thunderbird 52.9.1 (version that comes with Debian), and I can confirm that the bug no longer exists, thanks.

Saving attachments now works (saved file contains the attachment, and nothing but the attachment), and it works both no matter whether browser.cache.memory.enable and browser.cache.disk.enable are set to true (default) or to false.

This has been tested both with a CSV file as an attachment, and an ODT file (Openoffice). Both work ok.

N.B. In my comment 24, from 7 years ago, I had already confirmed that everything was working fine with browser.cache.memory.enable and browser.cache.disk.enable set to true.