Saving an attachment gives a file of 27 bytes, opening works
Categories
(Thunderbird :: General, defect)
Tracking
(Not tracked)
People
(Reporter: mozilla, Unassigned)
References
Details
Attachments
(6 files)
When I attempt to save an attachment (no matter which) I get a file of just 27 bytes.
Saving entire message works ok.
Opening the attachment (rather than saving it) also works ok
It's always the same 27 bytes:
hexdump -C scan_ghu297_2019-10-18-07-42-26.pdf
00000000 4e 18 ac 6e 87 72 a5 aa ed c2 29 65 6d e7 68 c2 |N..n.r....)em.h.|
00000010 79 68 69 d7 9d a2 77 5e 99 a9 dd |yhi...w^...|
0000001b
|
||
Comment 1•5 years ago
|
||
Gene, haven't we fixed the "27 byte" problem? Under which circumstances did that happen exactly?
Reporter: The attachment is in a message in an IMAP folder and that folder is not synchronised for offline use, right?
|
Reporter | |
Comment 2•5 years ago
|
||
It's indeed in an IMAP folder
|
|
||
Comment 3•5 years ago
|
||
... not synchronised for offline use, right?
Right-click (context menu), Properties, Synchronization tab.
|
||
Comment 4•5 years ago
|
||
The 27 bytes are the encoded string "attachment will be downloaded on demand" (approximately). But this should never be what is downloaded.
Reporter, you say you only see this when you download and save the pdf but not when you just open the pdf in a pdf reader? (Both should be doing basically the same thing it seems.)
When I attempt to save an attachment (no matter which) I get a file of just 27 bytes.
Are you saying you see this for any attachment on any message, or just for the pdf on one message? Just want to make sure I understand.
I don't know which recent bug, if any, where this was addressed. However you might trying using the lastest 68.1.x version and see if you have the same problem. If that doesn't help we might need an example .eml and maybe an IMAP:5 and IMAPCache:5 log that you record while accessing the attachment(s). Google "thunderbird log" for details.
Also, need to know if you are using offline store or just fetching/storing locally only the message headers -- see comment 3. (Offline storage is the default and if in use for the folder you should never have to "download on demand" since the complete message, including attachments, is downloaded and stored in the profile mbox file or maildir files.)
On a folder with no offline storage I just downloaded a pdf attachment and it worked OK with tb version 60.8.0. Could also open the pdf into a reader.
|
|
Reporter | |
Comment 5•5 years ago
|
||
Reporter, you say you only see this when you download and save the pdf but not when you just open the pdf in a pdf reader?
Exactly. Open correctly launches the application with the correct content, but save only saves those 27 bytes.
Are you saying you see this for any attachment on any message, or just for the pdf on one message? Just want to make sure I understand.
I'm seeing this for some file types (.doc, .pdf), but not for others (.txt, wihout attachment). I'm not 100% sure either that it is related to the mime type, extension or something else. Only that with some mails it works, whereas for others it doesn't
If that doesn't help we might need an example .eml and maybe an IMAP:5 and IMAPCache:5 log that you record while accessing the attachment(s).
Log is attached.
Google "thunderbird log" for details.
You do know that google turns up lots of hits about how to do this on old thunderbird versions? I could just have said "I vaguely remember other people having had the same issue as me, if you google, you might find their forum posts with details" :-)
Ok, in the end, I did find instructions that worked, but please, next time post a link. You're the expert after all...
|
|
Reporter | |
Comment 6•5 years ago
|
||
|
|
||
Comment 7•5 years ago
|
||
Sorry, didn't mean to make it difficult to find the logging info. I never remember the exact URL and just search "thunderbird log" on google and the first link finds it. Maybe that doesn't work for everyone everywhere. Will keep that in mind.
Anyhow, thanks for the log and the sample email. I will check them right now.
|
|
||
Comment 8•5 years ago
|
||
Well, the sample email saves the .doc attachment OK for me. Also, don't see anything odd in the log you attached.
- Were you saving attachment when the log was recorded?
- Still need to know if you have offline storage configured. You can right click on a folder and select properties and then look under "synchonization". Is "set this folder for offline use" checked or not checked? (You can also see the sync setting for all folder in one place by looking at account settings, synchronization & storage, advanced...)
- Do you have View Attachments Inline set or not. It's under the "View" menu.
- Need a re-do the log with the IMAPCache:5 enabled. This is mentioned here: https://wiki.mozilla.org/MailNews:Logging . Specifically you need to add it to the MOZ_LOG env var and re-record the log:
MOZ_LOG=IMAP:5,IMAPCache:5 - Please state exactly what you are doing in tb while recording the log.
- Please also include the "capability" and "id" commands and responses in the log. This helps to identify the imap server type and its supported behavior. Editing out repetitive and personal information is OK.
Thanks!
|
||
Updated•5 years ago
|
|
|
Reporter | |
Comment 9•5 years ago
|
||
- Yes I was saving the attachment while the log was recorded. That was the whole point... d'oh
- No, offline use is not configured
- I do have View Attachments Inline.
- I have attached a new version of log, with MOZ_LOG=IMAP:5,IMAPCache:5
- I launch thunderbird, scroll to my test message, open it, and save the attachment. Then I leave thunderbird again. I've filtered out the folder names out of the log, because I do indeed consider that a privacy issue (names of people, and of circumstances where I know them from). Folder names should not be logged by default... I've also filtered out my list of tags.
- In this new log, everything except LIST and FLAGS commands is in.
|
|
||
Comment 10•5 years ago
|
||
Sorry, just re-discovered that IMAPCache logging is only available on 68.* (not on your 60.9 version).
Anyhow the log looks a bit funny in that it appears that the whole message is fetched a couple times and then some individual parts are fetched. Once the whole message is fetched, it should be stored in ram cache and accessed from there without having to access individual parts.
There was some regressions regarding attachments but I thought they only occurred with 68.0. I don't think that Bug 1579341 affected 60.9 but not 100% sure. So you might try updating to the latest 68.* version if not too much trouble.
Also, the message I see fetched in the log seems to be smaller than the fetch mime on demand threshold which is default 30000 bytes (mail.imap.mime_parts_on_demand_threshold). In addition, the sample email you attached above is also below the threshold for on demand part fetches and when I access it the whole message is fetched. Have you by any chance changed this parameter?
I might also request that you attach the email with the .odt attachment that I see in the log. Maybe there is something unique about it that I will see by testing it here.
|
|
||
Comment 11•5 years ago
|
||
Alain, please repeat with version 68 and see Gene's comment 10.
|
|
||
Comment 13•5 years ago
|
||
Alain, Can you produce the log again with 68.2.2? That should now record the "IMAPCache" activity that wasn't logged in 60.9. Also, if possible, attach or send me via email the .eml file with the .odt attachment that is causing the problem. Re: comment 10.
Thanks.
|
||
Comment 14•5 years ago
|
||
Resolved per whiteboard
|
|
Reporter | |
Comment 15•5 years ago
|
||
Still an issue on 68.8.0. Not on all attachments, but I saw it happen on a signature.asc attachment of 313 bytes. Became 44 bytes "This body part will be downloaded on demand." when saved.
|
|
||
Comment 16•5 years ago
|
||
Often "signature" is ambiguous as to whether it is really an attachment in some emails. Were you really wanting to save it as a file? Are you still seeing the problem with "real" attachments like pdf's, jpg's etc?
If you still want me to look at this, please submit the log as requested in comment 13 above.
FYI, there has been a recent change to increase the threshold for fetch parts on demand from 30K to 500K bytes and to disable it completely: bug 1629292. However, this is only in the daily build right now. You might consider making a similar change in the config editor to work around the problem.
Change mail.imap.mime_parts_on_demand_threshold from default 30000 to 500000, or make it even bigger if you want. You can also go all the way and disable fetch on demand by setting mail.server.default.mime_parts_on_demand to false.
|
|
Reporter | |
Comment 17•5 years ago
|
||
(In reply to gene smith from comment #16)
Often "signature" is ambiguous as to whether it is really an attachment in some emails. Were you really wanting to save it as a file?
errr yes. What were you think I was trying to do with it? Print it out and use it as rolling paper or what? :-)
Are you still seeing the problem with "real" attachments like pdf's, jpg's etc?
But, point taken, and I continued with other types of attachments. And indeed, only signature.asc does this. All other attachments (pdf's, C sources, odt, ... and even a vcard) work ok (out of 22 tried).
If you still want me to look at this, please submit the log as requested in comment 13 above.
FYI, there has been a recent change to increase the threshold for fetch parts on demand from 30K to 500K bytes and to disable it completely: bug 1629292. However, this is only in the daily build right now. You might consider making a similar change in the config editor to work around the problem.
Change mail.imap.mime_parts_on_demand_threshold from default 30000 to 500000, or make it even bigger if you want. You can also go all the way and disable fetch on demand by setting mail.server.default.mime_parts_on_demand to false.
I'm not sure I get all of this, but I noticed that Thunderbird now takes 4.8GB of cache space, which is a tad excessive...
|
|
||
Comment 18•5 years ago
|
||
Ok, sorry, I was thinking when you wrote "signature" you meant just a simple file where you put your name and maybe a cute saying, not the pgp type, signature.asc.
You can try turning off mail.server.default.mime_parts_on_demand to false and restart tb. This will clear any emails in ram cache that were recently accessed and may allow any problem attachments to be accessed and saved.
Note: There is another item called mail.imap.mime_parts_on_demand that is used for legacy profiles. I would suggest setting it to false too before restarting tb just to be sure that the whole email is fetched and stored. Both of these are accessed using the config editor.
|
|
Reporter | |
Comment 19•5 years ago
|
||
Nope, even with both mime_parts_on_demand set to false, I still only get a 44 byte file "This body part will be downloaded on demand." when attempting to download the PGP signature. Even after restarting thunderbird.
|
|
Reporter | |
Comment 20•5 years ago
|
||
So now it happened again with an (important) PDF file that I was sent. I can view it all right by opening it, but there's no way to save it to disk directly from thunderbird. This is getting seriously annoying. Can't we just roll back the change that made this happen? The code paths for invoking an external application, and for saving straight to disk can't be that different, can they?
|
|
Reporter | |
Comment 21•5 years ago
|
||
Oddly enough, if I configure Thunderbird in Edit->Preferences->Attachments to "Always ask" for PDF document, I can save the attachment by doubleclicking on it (as if opening it), and then by choosing "Save" in the "Ask" dialog box, rather than choosing an application. So what is different in both code paths (directly saving on one hand, and pretending to open, and then save on the other hand)?
|
|
||
Comment 22•5 years ago
•
|
||
I tried this on a lot of emails with pdf, doc and jpg attachments and don't see a problem. My test folder is not synchronized for offline use, just like yours. Also tried setting the attachment types to "Always ask" and other settings and don't see a difference.
I don't know of any changes that can be "rolled back" that would affect this; I think you originally reported the problem for tb 60 and I haven't seen other reports of this or a similar problem recently.
Have you always been setup so all folders are not sync'd and you store only headers? In the profile area, you may see an INBOX file and an INBOX.msf file (or whichever folder you are having the problem with). If you are wanting only to store headers and not sync emails bodies with the server, you can delete the INBOX file containing message bodies from the past. Regardless if you delete it or not, I would recommend doing an "repair" of the INBOX or whatever folder. This will re-download the headers and clean up any possible corruption in the .msf index that may be causing the issue. Right click on folder, select properties and click "repair folder" button.
The repair can take a while depending on mailbox size since all header need to be re-downloaded. So if you prefer not to do this, if you create a new folder and move the offending message to it, does it save attachments OK? (Make sure the new folder is configured to not sync too before copying to it.)
If a repair/clean-up of the offending folder doesn't help or if you don't want to do it, then I guess we'll need another IMAP:5,IMAPCache:5 log like I requested in comment 10 and comment 13. Before or while you record the log I would ask that that you display the "Order Received" column for the folder and tell me the number for the offending message. This number will appear in the log as the message's imap UID number and I can tell when looking at the log which message you are trying to save the attachment for.
|
|
Reporter | |
Comment 23•5 years ago
|
||
The problem does indeed still occur with that message moved to a new specific folder.
The new folder has been created with default settings. On the folder properties, "Select this folder for offline use" is unchecked, if that's what you mean by "configured to not sync".
Problem still happens after Clicking on FolderProperties->RepairFolder.
As for the log, I'm feeling really uncomfortable providing it, as it includes names of all the folders created in thunderbird, which I feel uneasy sharing, and unconfident to snip as I'm afraid I'd miss something.
In case it is relevant, the offending message was a forwarded mail with an attachment (PDF). Incidentally, the PDF is a signed PDF, but should that really be relevant? Thunderbird would need to "look inside" to tell, but why would it, for a simple save operation? Size of PDF is 511335 bytes.
And, as said, opening it works alright, and even saving it from the "What should Thunderbird do with this file?" dialog.
This dialog is indeed a useful workaround for this particular case here (where PDF is an actual document, that I'd just like to get onto an USB stick), but in other situations (such as trying to analyze a potentially spammy/exploity attachments) this approach would be far too dangerous.
|
|
Reporter | |
Comment 24•5 years ago
|
||
Ok, managed to reproduce it on a "neutral" (non-confidential) message.
I filled a file named empty.bin with all zeroes, 511335 bytes, and mailed it to myself => I could save the attachment all right.
I forwarded the first message to myself => I could still save the attachment
I renamed empty.bin to empty.pdf, and mailed it to myself => now I could no longer save the attachment. Apparently it cares about extension (and mime type), but (fortunately) not about contents.
|
|
Reporter | |
Comment 25•5 years ago
|
||
|
|
Reporter | |
Comment 26•5 years ago
|
||
Imap Cache log, with LIST of all my folders snipped (except INBOX and the target 000-test folder).
Offending message is the test message with zero-filled "PDF". It is the only message in folder, however its Order-Received is 2, as there used to be another message in that folder before (namely the initial message showing the bug with the signed PDF).
Setting browser.cache.memory to true does indeed solve the issue (I figure this should probably be "safe" and not fill up my disk, as it says "memory" ...)
|
|
||
Comment 27•5 years ago
|
||
(In reply to Alain Knaff from comment #26)
Setting browser.cache.memory to true does indeed solve the issue (I figure this should probably be "safe" and not fill up my disk, as it says "memory" ...)
I assume you must mean that you set browser.cache.memory.enable to true? Without this being true, tb will not have a place to store messages and will have to fetch each part as it is accessed and, if false, may cause problems. But this is only an issue if the folder does not have offline disk storage like you are setup. Do you know how this got set to false? It defaults to true and is not something a user would typically change.
There are some other values associated with it that determines the amount of RAM allocated for message storage/cache. Specifically, browser.cache.memory.capacity (total ram allowed for cache, default 200M) and browser.cache.memory.max_entry_size (max ram for a given message part, 25M). Make sure they are reset to default values (not bold in config editor). Or you can make them larger if you want.
I just tried your attached sample email https://bugzilla.mozilla.org/attachment.cgi?id=9150758 with my normal setup and the attachment saved OK (a large but corrupted pdf). I then set browser.cache.memory.enable to false and restarted tb (to clear memory cache). Attempted the attachment save again and I got the 27 bytes! So that does seem to be the key.
I haven't yet looked at the new log you attached.
|
|
Reporter | |
Comment 28•5 years ago
|
||
(In reply to gene smith from comment #27)
(In reply to Alain Knaff from comment #26)
Setting browser.cache.memory to true does indeed solve the issue (I figure this should probably be "safe" and not fill up my disk, as it says "memory" ...)
I assume you must mean that you set browser.cache.memory.enable to true?
Indeed.
Without this being true, tb will not have a place to store messages and will have to fetch each part as it is accessed and, if false, may cause problems. But this is only an issue if the folder does not have offline disk storage like you are setup. Do you know how this got set to false? It defaults to true and is not something a user would typically change.
I might have set this to false at the time when that overly aggressive disk caching was introduced (which caused the crash on my computer at work, and which made the server of our Linux user group unusable for a couple of days), trying to stop all possible avenues how this caching might activate again, and not noticing that it said "...memory...".
There are some other values associated with it that determines the amount of RAM allocated for message storage/cache. Specifically, browser.cache.memory.capacity (total ram allowed for cache, default 200M) and browser.cache.memory.max_entry_size (max ram for a given message part, 25M). Make sure they are reset to default values (not bold in config editor). Or you can make them larger if you want.
Right now, all of these have default values. I'll keep my fingers crossed that it doesn't now fill all my RAM...
I just tried your attached sample email https://bugzilla.mozilla.org/attachment.cgi?id=9150758 with my normal setup and the attachment saved OK (a large but corrupted pdf).
Yeah, it's expected that it shows up as corrupted, as it is not actually a PDF. It's a zero-filled file...
I then set browser.cache.memory.enable to false and restarted tb (to clear memory cache). Attempted the attachment save again and I got the 27 bytes! So that does seem to be the key.
I haven't yet looked at the new log you attached.
Ok
|
|
||
Comment 29•5 years ago
•
|
||
Those prefs are also used in mozilla/firefox code for caching, I think. But tb and ff are completely separate programs and don't share code at runtime, AFAIK. But they do share a lot of source code.
Not sure what "agressive disk caching" you are referring to. The only disk cache tb uses is when you store messages for offline use and it create mbox files containing each folder's messages.
Anyhow, we might just close this bug as invalid but I think it has uncovered a bug. With ram caching turned off like you had it, saving an attachment should still work. I can make a one line change to the code to make it work so the save works with mem caching disabled. The change forces a full message fetch instead of a fetch by parts. However I haven't yet determined why it is fetching by parts. As you found, when you open the attachment into an application, it fetches it correctly and, in that case, fetches the whole message like it should.
Edit: The fetch by parts only fetches the message headers and mime headers and the message body (which is empty for the test message). If it fetched the actual attachment there would be no problem with fetch by parts. Also, looked at your log file and I see the exact same thing here when I record the log with your test message. Also, just to be clear, I understood that the attachment to to test message was a pdf with all A's so I open it to a text editor application and it is fetched OK only then (but not when saving).
|
|
Reporter | |
Comment 30•5 years ago
|
||
(In reply to gene smith from comment #29)
Those prefs are also used in mozilla/firefox code for caching, I think. But tb and ff are completely separate programs and don't share code at runtime, AFAIK. But they do share a lot of source code.
Not sure what "agressive disk caching" you are referring to. The only disk cache tb uses is when you store messages for offline use and it create mbox files containing each folder's messages.
In any case, back when this was introduced, it filled my disk on my work PC so much that my entire KDE session crashed for lack of free space.
And our Linux server, which was also an IMAP server was bogged down for days on end because all users' thunderbirds downloaded their entire mailboxes full of years old mails.
It happened. And it did cause us real tangible problems. And very probably not just us. I wish developers who come up with such ideas would sometimes think about the consequences... Maybe this caching got tones down a couple of notches later on, but we didn't notice as everyone of us just disabled it...
Anyhow, we might just close this bug as invalid but I think it has uncovered a bug. With ram caching turned off like you had it, saving an attachment should still work.
Indeed...
I can make a one line change to the code to make it work so the save works with mem caching disabled. The change forces a full message fetch instead of a fetch by parts. However I haven't yet determined why it is fetching by parts. As you found, when you open the attachment into an application, it fetches it correctly and, in that case, fetches the whole message like it should.
Indeed. There must be some difference in both code paths, or else the bug would occur in both cases...
Edit: The fetch by parts only fetches the message headers and mime headers and the message body (which is empty for the test message). If it fetched the actual attachment there would be no problem with fetch by parts. Also, looked at your log file and I see the exact same thing here when I record the log with your test message. Also, just to be clear, I understood that the attachment to to test message was a pdf with all A's so I open it to a text editor application and it is fetched OK only then (but not when saving).
|
||
Comment 31•3 years ago
|
||
Still issue in 91.8.0
Seems to happen in the case of large attachments.
I managed finally to save attachments after setting
browser.cache.memory.capacity
and
browser.cache.memory.max_entry_size
to 40000000
|
|
||
Comment 32•3 years ago
|
||
Still issue in 91.8.0
Glad you were able to work around the issue. But just to be clear, this is a "won't fix" issue with the current code base.
|
||
Comment 33•3 years ago
|
||
Seeing the same issue -- attachments unopenable, always 27 bytes -- with TB 91.9.0/linux.
Virgo pointed (thx!), @ Dovecot ML, to this issue
Here, was occurring only in the case of messages+attachments relocated to any (sub)folder other than INBOX.
Attachments of messages in INBOX always open in app of choice with no issue.
Once relo'd to a different folder, on open, the attachment DLs to /tmp space, as 27 bytes; cannot be opened.
MV back to INBOX cures the problem
Occured always with pdfs, and sometimes-to-alot with other mime types (doc, jpg) ...
All TBIrd caching, including memory, was disabled.
Setting
browser.cache.memory.capacity" 40000000
browser.cache.memory.enable" true
browser.cache.memory.max_entry_size 40000000
mail.imap.mime_parts_on_demand" false
mail.server.default.mime_parts_on_demand false
does the trick. Attachments behaving themselves no matter where the message is located. So far.
|
||
Comment 34•3 years ago
|
||
I've been getting this for quite a while, and even now in 103.0b5 (beta) Linux Ubuntu 22.04
|
|
||
Comment 35•3 years ago
|
||
(In reply to km from comment #34)
I've been getting this for quite a while, and even now in 103.0b5 (beta) Linux Ubuntu 22.04
Trying the settings in comment 33 are probably the only solution. It assumes you are not using offline store and you are accessing messages with very large attachments. If that's not the case, will need a lot more info.
Also, see comment 32.
|
|
||
Comment 36•3 years ago
|
||
The settings in comment 33 did work for me., though (In reply to gene smith from comment #35)
(In reply to km from comment #34)
I've been getting this for quite a while, and even now in 103.0b5 (beta) Linux Ubuntu 22.04
Trying the settings in comment 33 are probably the only solution. It assumes you are not using offline store and you are accessing messages with very large attachments. If that's not the case, will need a lot more info.
Also, see comment 32.
Those settings did seemingly eliminate the problem for me. I hope there will be some more fundamental resolution at some point.
|
||
Comment 37•2 years ago
|
||
I can confirm this problem still exists in the latest Thunderbird package on Debian 10 amd64: 1:91.12.0-1~deb10u1
We have the problem since we disabled local caching (offline mode sync).
The attachments of emails with attachments larger than about 20 MB cannot be saved or opened correctly any more!
We had to disable the offline caching as we have a large shared mailbox that took way too much space on rather small notebook SSDs.
For example, we have an email with 3 PDF files attached (22 MB + 1 MB + 1 MB). If we save the attachments to a folder on disk (either all at once or separate file by file), they are all only 45 bytes long and consist of "binary garbage"(?). Opening the files in a Hex editor shows that it's not a truncated PDF header or end. And it's the same bytes for all 3 different PDFs.
We cannot open the PDFs in Thunderbird for preview as well. It shows 0 of 0 pages.
The only solution that worked for us is the workaround with browser.cache.memory.max_entry_size set to -1 and browser.cache.memory.capacity set to 20000000.
|
|
||
Comment 38•2 years ago
|
||
sebastian, Thanks for the comment. I think the main issue that limits the message size when there is no offline store is that we use RAM cache and not DISK cache. With disk cache we could store any size message, within reason. This was proposed at one time but was never implemented due to technical issues (sync vs. async I think: Re bug 1302422 comment 3 and down).
But I think developer ISHIKAWA, Chiaki maybe has proposed a solution that he wrote about in bug 1302422 comment 7 and has mentioned recently something about a "buffered i/o" patch that he has proposed. I don't know what this involves or much about it so I'll ask him if it's relevant to this issue of ram vs. disk cache.
|
|
||
Updated•2 years ago
|
Hi,
I am trying to grok the problem and a possible resolution. I think I need to read this thread a few times.
My main usage is POP3 and I have not trusted IMAP servers very much. (Sorry about this.)
It seems to be mainly of IMAP problem, and I am not entirely sure if the buffering READ helps (YES, actually I was suggesting the repeated calls to READ) in Bug 1302422 comment 7, not buffered write to be exact).
But one thing for sure. Due to the local ISP's mail size limit, I have NOT sent or received a message file that is more than 20MB.
Actually the size limit is somewhere around 18MB, and this size limit is AFTER the mime encoding such as base64 including the attachment) and so the actual message size is capped around 10+ MB.
So there may be a problem of sending/receiving large e-mails (with attachments).
OTOH, I doubt if there was such inherent limit for POP3 e-mail fetching and download, but maybe I have overlooed obvious.
I have worked on the really low-level layer and if there are some meta-level decisions to stop downloading large e-mail message, I have not encountered the situation because I don't think I have tested largish e-mail transfer (larger than 20MB, that is).
I will check if I can send/receive 10+ MB e-mail message (with attachments) locally over this weekend.
Also I will study bug 1302422 comment 7, and re-read this whole thread.
I got bored after spending the summer break for two days since I cannot go out due to heat and the Covid-19 outbreak: I think Japan is registering the largest # of new patients right now. We come lately.
This is something I can do while watching missed TV series.
Thank you for your patience.
It may take a little longer.
Even my local mail server, I think I am using davecot for local testing, complained that my e-mail exceeds the
globally set size limit when I tried to send audacity win installation file as attachment. It is slightly above 20MB, but expands to 29.3 MB after encoding.
Stay tuned.
|
|
||
Comment 41•2 years ago
|
||
Thanks so much for your help!
Gene, thank you for the explanation and pointing to the matching expert.
Disk caching of opened attachments sounds reasonable. Relieves the pressure from a small RAM cache but also avoids having to offline sync a 50 GB mailbox entirely. (What we can't do anymore unfortunately. :( )
(In reply to ISHIKAWA, Chiaki from comment #40)
It may take a little longer.
Even my local mail server, I think I am using davecot for local testing, complained that my e-mail exceeds the
globally set size limit when I tried to send audacity win installation file as attachment. It is slightly above 20MB, but expands to 29.3 MB after encoding.
Stay tuned.
For POP3 download, there is no problem trying to send and receive a big attachment file.
TB displayed the attachment of audacity windows installation binary that lies around in my temp directory
as 21.4 MB.
Actually while I try to send it to my local mail server on the same linux PC, it expanded to 29.3 MB from 21.4 after mime/encoding
and TB actually warns if I want to send this big e-mail.
I said yes.
Then I could retrieve it without any issue.
SMTP daemon recorded the size of the e-mail as "size=30849007".
Some system tuning I had to add.
It turns out on my Debian GNU/Linux system, SMTP is handled by postfix/smtpd and it has a default message limit as 10,000,000.
I had to enlarge the limit by adding
message_size_limit=35000000
to /etc/postfix/main.cf.
If the size limit was 30000000, the sending failed because the message size became "size=30849007" with the headers.
The pop3 daemon was part of dovecot, but it did not seem to have per message limit.
Both dovecot and postfix seem to have the total size limit of mail queue, but in my single test, it did not matter.
So, again my conclusion is that the attachment download issue seems to be IMAP-specific?
I have no idea if header-only fetch is applicable to POP3.
I forgot to add.: I can re-visit, bug 1302422 comment 7, but right now not entirely sure if that is applicable to the issue here.
Let me re-read THAT message thread again. If the read is blocking, then we may have a problem indeed because
that blocking read may stop GUI to operate until it is released.
Anyway, I had a problem updating to 102.1.2, essentially losing the connection to my local ISP for a while, and I spent hours trying to fix the issue.
So much unexpected work for the weekend.
Again, it may take longer than I originally thought.
|
|
||
Comment 44•2 years ago
|
||
(In reply to ISHIKAWA, Chiaki from comment #42)
So, again my conclusion is that the attachment download issue seems to be IMAP-specific?
That's probably the case, I'd guess:
With POP3, your server is only meant as a temporary "letter box". You check it regularly and take the emails "out" of the letter box and store them on your local device. Although it's possible to keep a copy of the emails on the server, POP3 is for people that want to have their emails on their device. When you open an email fetched by POP3 or an attachment of such an email, the email always already resides on your hard disk in your mbox/Maildir before you open it.
With IMAP, you want to keep your email storage on the server. You can benefit from the advantage of having your emails and folders in sync between different client devices (notebook, workstation, mobile phone) automatically. As the server-side storage is the "single source of truth".
Additional offline copies of your emails / folders are not necessary. But it can be useful to be able to work offline (on an airplane or something like that) or to speed up access. When you open an email or its attachments in this use cases, you might not have a local copy of the email (maybe headers only). So first you have to load the email from the central server storage, either into RAM or onto a local disk. If I get Glen right, Thunderbird only uses a limited amount of RAM (but not disk space) to load emails from the server if you don't have an offline copy. And this seems to fail for attachments larger than 20 MB if you don't manipulate the config.
In our case it's like that: We have a local Linux mailserver (Postfix + Dovecot) with RAID disks, automated nightly backups, etc. For us it's a better / safer place to keep the emails in this central storage than scattered on different client notebooks that can get lost, broken, corrupted due to user / usage errors, ...
As client devices, we use older laptops with Debian Linux and small SSDs (not larger than 128 GB) or smartphones (Android). For both types of devices, we had to stop keeping a local copy of the entire IMAP storage because the mailboxes are simply too large for the small disks. :(
|
||
Comment 45•2 years ago
|
||
I am observing the same fault with IMAP that sebastian is:
"For example, we have an email with 3 PDF files attached (22 MB + 1 MB + 1 MB). If we save the attachments to a folder on disk (either all at once or separate file by file), they are all only 45 bytes long and consist of "binary garbage"(?). Opening the files in a Hex editor shows that it's not a truncated PDF header or end. And it's the same bytes for all 3 different PDFs.
We cannot open the PDFs in Thunderbird for preview as well. It shows 0 of 0 pages."
with the following difference - it does not matter how big the attachment is. It happens with relatively small PDF attachments as well (I just observed it with a 195 kB PDF).
sebastian's workaround, i.e. setting browser.cache.memory.max_entry_size to -1 and, in my case, browser.cache.memory.capacity to 40000000, seems to "fix" the problem.
I should mention that this has been a problem with Thunderbird for many years.
Thunderbird 91.12.0 (64-Bit) on Gentoo Linux, kernel uname -a Linux t470 5.4.203-gentoo #1 SMP Sat Jul 23 23:01:54 CEST 2022 x86_64 Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz GenuineIntel GNU/Linux
|
|
||
Comment 46•2 years ago
|
||
Sebastian and Chiaki, thanks for you interest and for looking into the issue. I've been able change TB's caching of imap emails from memory based to disk based while keeping the default parameters values under browser.cache. in Config Editor. However, there are two issues that prevented it from working initially.
- With the current memory cache, which can be accessed synchronously, before the entry is used the first 100 bytes are "peeked at" using a standard read() call. The first line of the cache entry is checked to make sure it looks like a email header as it should. If it should fail, the message is just fetched from the server:
https://searchfox.org/comm-central/rev/c849e112e4752d4707cec8aa887d68eb475f4375/mailnews/imap/src/nsImapProtocol.cpp#9519
There is a comment in the code that it won't work with disk cache. I haven't found a workaround for this yet so I've just commented out this block of code. In my testing I've haven't seen a cache entry that was saved and then read back that was not valid when it is opened. - Then before the "pump" is setup to pipe the cache entry to the "listener", another synchronous access is attempted to obtain the number of bytes available to be read from the cache entry:
https://searchfox.org/comm-central/rev/c849e112e4752d4707cec8aa887d68eb475f4375/mailnews/imap/src/nsImapProtocol.cpp#9544
This always returns zero bytes available. I tried, as Chiaki suggested, doing the Available() call in a loop until I saw a non-zero result. But even with 100 loops it didn't always return a non-zero value so I commented this out too.
So my modified code assumes that if the disk cache entry can be opened it can be used. So far in my testing I haven't seen a problem with very large messages (or average sized ones).
However, even with disk cache instead of memory cache there are still default limits that aren't infinite. It appears that the default upper limit on a disk cache entry is a bit more than twice the limit of a memory cache entry: 51.2MB vs. 25MB (browser.cache.disk.max_entry_size vs browser.cache.memory.max_entry_size).
Also the default total cache size is not a lot bigger: 256MB vs. 200MB (browser.cache.disk.capacity vs. browser.cache.memory.capacity). This can be increased or decreased in the user-accessible systems setting under General | Disk Space | "Override automatic cache management". However, for disk cache I'm not sure this limit is meaningful since the documentation says that browser.cache.disk.capacity is only relevant if browser.cache.disk.smart_size.enabled is false: https://searchfox.org/mozilla-central/rev/5f1fc7348a9fb1189fe1c2139d68f928f8cfe510/modules/libpref/init/StaticPrefList.yaml#957. I'll have to ask the cache2 experts at mozilla but I suspect with "smart_size" enabled lets the cache grow to a "reasonable" size without using up all your disk space.
Anyhow, I'll go ahead and make a "try" build with my changes and post a link where Sebastian (or anyone) can try my proposed fix with a patched daily build.
|
|
||
Comment 47•2 years ago
|
||
(In reply to Stephen Bosch from comment #45)
I am observing the same fault with IMAP that sebastian is:
with the following difference - it does not matter how big the attachment is. It happens with relatively small PDF attachments as well (I just observed it with a 195 kB PDF).
Can you attach, share or send me the 195kb pdf that doesn't work?
|
|
||
Comment 48•2 years ago
•
|
||
The try build is here: https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=33b3e3087465284c8de4b2d052a2b0b45725b8c0. You can get the appropriate link for installation there or use these direct links:
Linux64: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/YIL6l7npRI2YlxEvIQKT0A/runs/0/artifacts/public/build/target.tar.bz2 (Just unpack to directory thunderbird, cd to thunderbird and run from command line thunderbird.)
To test this proposed change you should set everything under browser.cache. back to default and it should be able to bring in the messages with large attachments, I think.
Note: Instead of caching messages to memory, this will now cache them to disk files. On linux (and maybe OSX, not sure) the disk cache files appear at $HOME/.cache/thunderbird/<your-tb-profile-dir-name>/cache2/entries/ . Each message, when first opened, is download and stored in its own file. There will be a few other system related files here too.
For windows, the cache2\entries\ are at the top level of where the profile is stored.
When TB shuts down, the cache files are deleted so messages will have to be re-downloaded before being re-cached. (This is the same as cache to memory did.)
Edit: Actually the disk cached messages or parts survive a restart and are used in the new session.
|
|
||
Comment 49•2 years ago
|
||
(In reply to gene smith from comment #47)
(In reply to Stephen Bosch from comment #45)
I am observing the same fault with IMAP that sebastian is:
with the following difference - it does not matter how big the attachment is. It happens with relatively small PDF attachments as well (I just observed it with a 195 kB PDF).
Can you attach, share or send me the 195kb pdf that doesn't work?
Hi Gene,
I've e-mailed you an example PDF (which triggers the same issue) instead of attaching it here because it contains personally identifying information. This one is 175 kB, according to the filesystem driver.
Sadly, Sebastian's workaround no longer seems to work. Thunderbird was updated to 91.13 on September 3rd, perhaps that's the reason, but I really don't know.
(One thing that may be relevant: the file as attached has this filename: S-2022-00528_#12752.pdf. If I open it within the webmail client (Gmail) it appears as S-2022-00528_#12752.pdf and can be opened. In the Thunderbird IMAP URL or after saving in Firefox, it is displayed as S-2022-00528_%2312752.pdf.)
|
|
||
Comment 50•2 years ago
|
||
(In reply to Stephen Bosch from comment #49)
(In reply to gene smith from comment #47)
(In reply to Stephen Bosch from comment #45)
I am observing the same fault with IMAP that sebastian is:
with the following difference - it does not matter how big the attachment is. It happens with relatively small PDF attachments as well (I just observed it with a 195 kB PDF).
Can you attach, share or send me the 195kb pdf that doesn't work?
Hi Gene,
I've e-mailed you an example PDF (which triggers the same issue) instead of attaching it here because it contains personally identifying information. This one is 175 kB, according to the filesystem driver.
Sadly, Sebastian's workaround no longer seems to work. Thunderbird was updated to 91.13 on September 3rd, perhaps that's the reason, but I really don't know.
(One thing that may be relevant: the file as attached has this filename: S-2022-00528_#12752.pdf. If I open it within the webmail client (Gmail) it appears as S-2022-00528_#12752.pdf and can be opened. In the Thunderbird IMAP URL or after saving in Firefox, it is displayed as S-2022-00528_%2312752.pdf.)
Sorry for very late reply to this. I can open the un-truncated pdf attachment in TB in the email you sent me back in Sept. Being a very small attachment, it shouldn't be a problem or require the workaround. However, if the pdf attachment is to an email and that email is an attachment to another email, you will see a problem opening the attachment, but only if you don't have offline storage.
I'm working on a solution to this.
I don't think the %23 is a problem in the filename since I think it is just the escape value for #. FWIW, the 45 bytes you see is just the encoded string that TB puts in saying something like "This attachment will be opened on demand". If your email is using html I think this will be 45 bytes and if it's just text it will be 27 bytes as in the bug summary title.
|
||
Comment 51•2 years ago
|
||
Resolved duplicate per Gene's Bug 1805186 Comment 12.
Description
•