Closed Bug 1589649 Opened 3 years ago Closed 2 years ago

Saving an attachment gives a file of 27 bytes, opening works

Categories

(Thunderbird :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: mozilla, Unassigned)

References

Details

Attachments

(5 files)

When I attempt to save an attachment (no matter which) I get a file of just 27 bytes.

Saving entire message works ok.

Opening the attachment (rather than saving it) also works ok

It's always the same 27 bytes:

hexdump -C scan_ghu297_2019-10-18-07-42-26.pdf
00000000 4e 18 ac 6e 87 72 a5 aa ed c2 29 65 6d e7 68 c2 |N..n.r....)em.h.|
00000010 79 68 69 d7 9d a2 77 5e 99 a9 dd |yhi...w^...|
0000001b

Gene, haven't we fixed the "27 byte" problem? Under which circumstances did that happen exactly?

Reporter: The attachment is in a message in an IMAP folder and that folder is not synchronised for offline use, right?

Flags: needinfo?(gds)

It's indeed in an IMAP folder

... not synchronised for offline use, right?

Right-click (context menu), Properties, Synchronization tab.

The 27 bytes are the encoded string "attachment will be downloaded on demand" (approximately). But this should never be what is downloaded.
Reporter, you say you only see this when you download and save the pdf but not when you just open the pdf in a pdf reader? (Both should be doing basically the same thing it seems.)

When I attempt to save an attachment (no matter which) I get a file of just 27 bytes.

Are you saying you see this for any attachment on any message, or just for the pdf on one message? Just want to make sure I understand.

I don't know which recent bug, if any, where this was addressed. However you might trying using the lastest 68.1.x version and see if you have the same problem. If that doesn't help we might need an example .eml and maybe an IMAP:5 and IMAPCache:5 log that you record while accessing the attachment(s). Google "thunderbird log" for details.

Also, need to know if you are using offline store or just fetching/storing locally only the message headers -- see comment 3. (Offline storage is the default and if in use for the folder you should never have to "download on demand" since the complete message, including attachments, is downloaded and stored in the profile mbox file or maildir files.)

On a folder with no offline storage I just downloaded a pdf attachment and it worked OK with tb version 60.8.0. Could also open the pdf into a reader.

Flags: needinfo?(gds)

Reporter, you say you only see this when you download and save the pdf but not when you just open the pdf in a pdf reader?

Exactly. Open correctly launches the application with the correct content, but save only saves those 27 bytes.

Are you saying you see this for any attachment on any message, or just for the pdf on one message? Just want to make sure I understand.

I'm seeing this for some file types (.doc, .pdf), but not for others (.txt, wihout attachment). I'm not 100% sure either that it is related to the mime type, extension or something else. Only that with some mails it works, whereas for others it doesn't

If that doesn't help we might need an example .eml and maybe an IMAP:5 and IMAPCache:5 log that you record while accessing the attachment(s).

Log is attached.

Google "thunderbird log" for details.

You do know that google turns up lots of hits about how to do this on old thunderbird versions? I could just have said "I vaguely remember other people having had the same issue as me, if you google, you might find their forum posts with details" :-)
Ok, in the end, I did find instructions that worked, but please, next time post a link. You're the expert after all...

Sorry, didn't mean to make it difficult to find the logging info. I never remember the exact URL and just search "thunderbird log" on google and the first link finds it. Maybe that doesn't work for everyone everywhere. Will keep that in mind.
Anyhow, thanks for the log and the sample email. I will check them right now.

Well, the sample email saves the .doc attachment OK for me. Also, don't see anything odd in the log you attached.

  1. Were you saving attachment when the log was recorded?
  2. Still need to know if you have offline storage configured. You can right click on a folder and select properties and then look under "synchonization". Is "set this folder for offline use" checked or not checked? (You can also see the sync setting for all folder in one place by looking at account settings, synchronization & storage, advanced...)
  3. Do you have View Attachments Inline set or not. It's under the "View" menu.
  4. Need a re-do the log with the IMAPCache:5 enabled. This is mentioned here: https://wiki.mozilla.org/MailNews:Logging . Specifically you need to add it to the MOZ_LOG env var and re-record the log:
    MOZ_LOG=IMAP:5,IMAPCache:5
  5. Please state exactly what you are doing in tb while recording the log.
  6. Please also include the "capability" and "id" commands and responses in the log. This helps to identify the imap server type and its supported behavior. Editing out repetitive and personal information is OK.

Thanks!

Flags: needinfo?(mozilla)
Whiteboard: [closeme 2019-11-11]
Attached file moz_log.txt
  1. Yes I was saving the attachment while the log was recorded. That was the whole point... d'oh
  2. No, offline use is not configured
  3. I do have View Attachments Inline.
  4. I have attached a new version of log, with MOZ_LOG=IMAP:5,IMAPCache:5
  5. I launch thunderbird, scroll to my test message, open it, and save the attachment. Then I leave thunderbird again. I've filtered out the folder names out of the log, because I do indeed consider that a privacy issue (names of people, and of circumstances where I know them from). Folder names should not be logged by default... I've also filtered out my list of tags.
  6. In this new log, everything except LIST and FLAGS commands is in.
Flags: needinfo?(mozilla)

Sorry, just re-discovered that IMAPCache logging is only available on 68.* (not on your 60.9 version).

Anyhow the log looks a bit funny in that it appears that the whole message is fetched a couple times and then some individual parts are fetched. Once the whole message is fetched, it should be stored in ram cache and accessed from there without having to access individual parts.

There was some regressions regarding attachments but I thought they only occurred with 68.0. I don't think that Bug 1579341 affected 60.9 but not 100% sure. So you might try updating to the latest 68.* version if not too much trouble.

Also, the message I see fetched in the log seems to be smaller than the fetch mime on demand threshold which is default 30000 bytes (mail.imap.mime_parts_on_demand_threshold). In addition, the sample email you attached above is also below the threshold for on demand part fetches and when I access it the whole message is fetched. Have you by any chance changed this parameter?

I might also request that you attach the email with the .odt attachment that I see in the log. Maybe there is something unique about it that I will see by testing it here.

Whiteboard: [closeme 2019-11-11]

Alain, please repeat with version 68 and see Gene's comment 10.

Flags: needinfo?(mozilla)
Whiteboard: [closeme 2020-01-11]

The issue is still present in 68.2.2

Flags: needinfo?(mozilla)

Alain, Can you produce the log again with 68.2.2? That should now record the "IMAPCache" activity that wasn't logged in 60.9. Also, if possible, attach or send me via email the .eml file with the .odt attachment that is causing the problem. Re: comment 10.
Thanks.

Flags: needinfo?(mozilla)
Whiteboard: [closeme 2020-01-11] → [closeme 2020-04-01]
Version: 60 → 68

Resolved per whiteboard

Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
Whiteboard: [closeme 2020-04-01]

Still an issue on 68.8.0. Not on all attachments, but I saw it happen on a signature.asc attachment of 313 bytes. Became 44 bytes "This body part will be downloaded on demand." when saved.

Flags: needinfo?(mozilla)

Often "signature" is ambiguous as to whether it is really an attachment in some emails. Were you really wanting to save it as a file? Are you still seeing the problem with "real" attachments like pdf's, jpg's etc?
If you still want me to look at this, please submit the log as requested in comment 13 above.

FYI, there has been a recent change to increase the threshold for fetch parts on demand from 30K to 500K bytes and to disable it completely: bug 1629292. However, this is only in the daily build right now. You might consider making a similar change in the config editor to work around the problem.
Change mail.imap.mime_parts_on_demand_threshold from default 30000 to 500000, or make it even bigger if you want. You can also go all the way and disable fetch on demand by setting mail.server.default.mime_parts_on_demand to false.

See Also: → 1629292

(In reply to gene smith from comment #16)

Often "signature" is ambiguous as to whether it is really an attachment in some emails. Were you really wanting to save it as a file?

errr yes. What were you think I was trying to do with it? Print it out and use it as rolling paper or what? :-)

Are you still seeing the problem with "real" attachments like pdf's, jpg's etc?

But, point taken, and I continued with other types of attachments. And indeed, only signature.asc does this. All other attachments (pdf's, C sources, odt, ... and even a vcard) work ok (out of 22 tried).

If you still want me to look at this, please submit the log as requested in comment 13 above.

FYI, there has been a recent change to increase the threshold for fetch parts on demand from 30K to 500K bytes and to disable it completely: bug 1629292. However, this is only in the daily build right now. You might consider making a similar change in the config editor to work around the problem.
Change mail.imap.mime_parts_on_demand_threshold from default 30000 to 500000, or make it even bigger if you want. You can also go all the way and disable fetch on demand by setting mail.server.default.mime_parts_on_demand to false.

I'm not sure I get all of this, but I noticed that Thunderbird now takes 4.8GB of cache space, which is a tad excessive...

Ok, sorry, I was thinking when you wrote "signature" you meant just a simple file where you put your name and maybe a cute saying, not the pgp type, signature.asc.

You can try turning off mail.server.default.mime_parts_on_demand to false and restart tb. This will clear any emails in ram cache that were recently accessed and may allow any problem attachments to be accessed and saved.
Note: There is another item called mail.imap.mime_parts_on_demand that is used for legacy profiles. I would suggest setting it to false too before restarting tb just to be sure that the whole email is fetched and stored. Both of these are accessed using the config editor.

Nope, even with both mime_parts_on_demand set to false, I still only get a 44 byte file "This body part will be downloaded on demand." when attempting to download the PGP signature. Even after restarting thunderbird.

So now it happened again with an (important) PDF file that I was sent. I can view it all right by opening it, but there's no way to save it to disk directly from thunderbird. This is getting seriously annoying. Can't we just roll back the change that made this happen? The code paths for invoking an external application, and for saving straight to disk can't be that different, can they?

Oddly enough, if I configure Thunderbird in Edit->Preferences->Attachments to "Always ask" for PDF document, I can save the attachment by doubleclicking on it (as if opening it), and then by choosing "Save" in the "Ask" dialog box, rather than choosing an application. So what is different in both code paths (directly saving on one hand, and pretending to open, and then save on the other hand)?

I tried this on a lot of emails with pdf, doc and jpg attachments and don't see a problem. My test folder is not synchronized for offline use, just like yours. Also tried setting the attachment types to "Always ask" and other settings and don't see a difference.

I don't know of any changes that can be "rolled back" that would affect this; I think you originally reported the problem for tb 60 and I haven't seen other reports of this or a similar problem recently.

Have you always been setup so all folders are not sync'd and you store only headers? In the profile area, you may see an INBOX file and an INBOX.msf file (or whichever folder you are having the problem with). If you are wanting only to store headers and not sync emails bodies with the server, you can delete the INBOX file containing message bodies from the past. Regardless if you delete it or not, I would recommend doing an "repair" of the INBOX or whatever folder. This will re-download the headers and clean up any possible corruption in the .msf index that may be causing the issue. Right click on folder, select properties and click "repair folder" button.

The repair can take a while depending on mailbox size since all header need to be re-downloaded. So if you prefer not to do this, if you create a new folder and move the offending message to it, does it save attachments OK? (Make sure the new folder is configured to not sync too before copying to it.)

If a repair/clean-up of the offending folder doesn't help or if you don't want to do it, then I guess we'll need another IMAP:5,IMAPCache:5 log like I requested in comment 10 and comment 13. Before or while you record the log I would ask that that you display the "Order Received" column for the folder and tell me the number for the offending message. This number will appear in the log as the message's imap UID number and I can tell when looking at the log which message you are trying to save the attachment for.

The problem does indeed still occur with that message moved to a new specific folder.

The new folder has been created with default settings. On the folder properties, "Select this folder for offline use" is unchecked, if that's what you mean by "configured to not sync".

Problem still happens after Clicking on FolderProperties->RepairFolder.

As for the log, I'm feeling really uncomfortable providing it, as it includes names of all the folders created in thunderbird, which I feel uneasy sharing, and unconfident to snip as I'm afraid I'd miss something.

In case it is relevant, the offending message was a forwarded mail with an attachment (PDF). Incidentally, the PDF is a signed PDF, but should that really be relevant? Thunderbird would need to "look inside" to tell, but why would it, for a simple save operation? Size of PDF is 511335 bytes.

And, as said, opening it works alright, and even saving it from the "What should Thunderbird do with this file?" dialog.
This dialog is indeed a useful workaround for this particular case here (where PDF is an actual document, that I'd just like to get onto an USB stick), but in other situations (such as trying to analyze a potentially spammy/exploity attachments) this approach would be far too dangerous.

Ok, managed to reproduce it on a "neutral" (non-confidential) message.

I filled a file named empty.bin with all zeroes, 511335 bytes, and mailed it to myself => I could save the attachment all right.
I forwarded the first message to myself => I could still save the attachment
I renamed empty.bin to empty.pdf, and mailed it to myself => now I could no longer save the attachment. Apparently it cares about extension (and mime type), but (fortunately) not about contents.

Attached file imap-cache.log.bz2

Imap Cache log, with LIST of all my folders snipped (except INBOX and the target 000-test folder).

Offending message is the test message with zero-filled "PDF". It is the only message in folder, however its Order-Received is 2, as there used to be another message in that folder before (namely the initial message showing the bug with the signed PDF).

Setting browser.cache.memory to true does indeed solve the issue (I figure this should probably be "safe" and not fill up my disk, as it says "memory" ...)

(In reply to Alain Knaff from comment #26)

Setting browser.cache.memory to true does indeed solve the issue (I figure this should probably be "safe" and not fill up my disk, as it says "memory" ...)

I assume you must mean that you set browser.cache.memory.enable to true? Without this being true, tb will not have a place to store messages and will have to fetch each part as it is accessed and, if false, may cause problems. But this is only an issue if the folder does not have offline disk storage like you are setup. Do you know how this got set to false? It defaults to true and is not something a user would typically change.

There are some other values associated with it that determines the amount of RAM allocated for message storage/cache. Specifically, browser.cache.memory.capacity (total ram allowed for cache, default 200M) and browser.cache.memory.max_entry_size (max ram for a given message part, 25M). Make sure they are reset to default values (not bold in config editor). Or you can make them larger if you want.

I just tried your attached sample email https://bugzilla.mozilla.org/attachment.cgi?id=9150758 with my normal setup and the attachment saved OK (a large but corrupted pdf). I then set browser.cache.memory.enable to false and restarted tb (to clear memory cache). Attempted the attachment save again and I got the 27 bytes! So that does seem to be the key.

I haven't yet looked at the new log you attached.

(In reply to gene smith from comment #27)

(In reply to Alain Knaff from comment #26)

Setting browser.cache.memory to true does indeed solve the issue (I figure this should probably be "safe" and not fill up my disk, as it says "memory" ...)

I assume you must mean that you set browser.cache.memory.enable to true?

Indeed.

Without this being true, tb will not have a place to store messages and will have to fetch each part as it is accessed and, if false, may cause problems. But this is only an issue if the folder does not have offline disk storage like you are setup. Do you know how this got set to false? It defaults to true and is not something a user would typically change.

I might have set this to false at the time when that overly aggressive disk caching was introduced (which caused the crash on my computer at work, and which made the server of our Linux user group unusable for a couple of days), trying to stop all possible avenues how this caching might activate again, and not noticing that it said "...memory...".

There are some other values associated with it that determines the amount of RAM allocated for message storage/cache. Specifically, browser.cache.memory.capacity (total ram allowed for cache, default 200M) and browser.cache.memory.max_entry_size (max ram for a given message part, 25M). Make sure they are reset to default values (not bold in config editor). Or you can make them larger if you want.

Right now, all of these have default values. I'll keep my fingers crossed that it doesn't now fill all my RAM...

I just tried your attached sample email https://bugzilla.mozilla.org/attachment.cgi?id=9150758 with my normal setup and the attachment saved OK (a large but corrupted pdf).

Yeah, it's expected that it shows up as corrupted, as it is not actually a PDF. It's a zero-filled file...

I then set browser.cache.memory.enable to false and restarted tb (to clear memory cache). Attempted the attachment save again and I got the 27 bytes! So that does seem to be the key.

I haven't yet looked at the new log you attached.

Ok

Those prefs are also used in mozilla/firefox code for caching, I think. But tb and ff are completely separate programs and don't share code at runtime, AFAIK. But they do share a lot of source code.
Not sure what "agressive disk caching" you are referring to. The only disk cache tb uses is when you store messages for offline use and it create mbox files containing each folder's messages.

Anyhow, we might just close this bug as invalid but I think it has uncovered a bug. With ram caching turned off like you had it, saving an attachment should still work. I can make a one line change to the code to make it work so the save works with mem caching disabled. The change forces a full message fetch instead of a fetch by parts. However I haven't yet determined why it is fetching by parts. As you found, when you open the attachment into an application, it fetches it correctly and, in that case, fetches the whole message like it should.

Edit: The fetch by parts only fetches the message headers and mime headers and the message body (which is empty for the test message). If it fetched the actual attachment there would be no problem with fetch by parts. Also, looked at your log file and I see the exact same thing here when I record the log with your test message. Also, just to be clear, I understood that the attachment to to test message was a pdf with all A's so I open it to a text editor application and it is fetched OK only then (but not when saving).

(In reply to gene smith from comment #29)

Those prefs are also used in mozilla/firefox code for caching, I think. But tb and ff are completely separate programs and don't share code at runtime, AFAIK. But they do share a lot of source code.
Not sure what "agressive disk caching" you are referring to. The only disk cache tb uses is when you store messages for offline use and it create mbox files containing each folder's messages.

In any case, back when this was introduced, it filled my disk on my work PC so much that my entire KDE session crashed for lack of free space.

And our Linux server, which was also an IMAP server was bogged down for days on end because all users' thunderbirds downloaded their entire mailboxes full of years old mails.

It happened. And it did cause us real tangible problems. And very probably not just us. I wish developers who come up with such ideas would sometimes think about the consequences... Maybe this caching got tones down a couple of notches later on, but we didn't notice as everyone of us just disabled it...

Anyhow, we might just close this bug as invalid but I think it has uncovered a bug. With ram caching turned off like you had it, saving an attachment should still work.

Indeed...

I can make a one line change to the code to make it work so the save works with mem caching disabled. The change forces a full message fetch instead of a fetch by parts. However I haven't yet determined why it is fetching by parts. As you found, when you open the attachment into an application, it fetches it correctly and, in that case, fetches the whole message like it should.

Indeed. There must be some difference in both code paths, or else the bug would occur in both cases...

Edit: The fetch by parts only fetches the message headers and mime headers and the message body (which is empty for the test message). If it fetched the actual attachment there would be no problem with fetch by parts. Also, looked at your log file and I see the exact same thing here when I record the log with your test message. Also, just to be clear, I understood that the attachment to to test message was a pdf with all A's so I open it to a text editor application and it is fetched OK only then (but not when saving).

See Also: → 1673093

Still issue in 91.8.0
Seems to happen in the case of large attachments.
I managed finally to save attachments after setting
browser.cache.memory.capacity
and
browser.cache.memory.max_entry_size
to 40000000

Still issue in 91.8.0

Glad you were able to work around the issue. But just to be clear, this is a "won't fix" issue with the current code base.

Seeing the same issue -- attachments unopenable, always 27 bytes -- with TB 91.9.0/linux.

Virgo pointed (thx!), @ Dovecot ML, to this issue

Here, was occurring only in the case of messages+attachments relocated to any (sub)folder other than INBOX.
Attachments of messages in INBOX always open in app of choice with no issue.
Once relo'd to a different folder, on open, the attachment DLs to /tmp space, as 27 bytes; cannot be opened.
MV back to INBOX cures the problem

Occured always with pdfs, and sometimes-to-alot with other mime types (doc, jpg) ...

All TBIrd caching, including memory, was disabled.

Setting

browser.cache.memory.capacity"   40000000
browser.cache.memory.enable"   true
browser.cache.memory.max_entry_size   40000000
mail.imap.mime_parts_on_demand"   false
mail.server.default.mime_parts_on_demand   false

does the trick. Attachments behaving themselves no matter where the message is located. So far.

You need to log in before you can comment on or make changes to this bug.