1506800 - On Mac TB 60 is sending attachment as Apple-double causing problems with mail servers and at the recipient's end

Status: RESOLVED FIXED

(MailNews Core :: Composition, defect)

Description

[lang]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:63.0) Gecko/20100101 Firefox/63.0

Steps to reproduce:

Tried to send an email with an attached Word for Mac document.


Actual results:

Delivery failed. Received the following message from Microsoft Exchange:

Diagnostic information for administrators:

Generating server: BCLDCEX02.bcl.local

[e-mail]
BCLDCEX02.bcl.local
Remote Server returned '554 5.6.0 Corrupt message content; STOREDRV.Deliver.Exception:ConversionFailedException.MimeException; Failed to process message due to a permanent exception with message The message content has become corrupted. MimeException: MIME content error: Wrong magic number in Apple MIME attachment. [Stage: CreateReplay]'


Expected results:

Reinstalled version 52.9.1. Same message sent perfectly.

Comment 1

[Jorg K (CEST = GMT+2)]

Nice to know that Thunderbird is used by the The Victorian Bar Inc ;-)

Looking in Google for "Wrong magic number in Apple MIME attachment" finds a few hits, so the issue is not new, however, it appears to be new in TB.

Apple's glorious "two fork" approach for files (resource and data fork) makes TB send the attachment in the "Apple double" format which isn't well received at times, see bug 1503645. There the suggestion was to remove the resource fork of the document before sending. Does this happen with all message with attachments or just some?

Reporter, you you please do this for us:

Using TB 52.9.1, send a sample message and then save the sent message as a .eml file (File > Save As > File) and attach it here, see "Attach File" above. Then do the same with TB 60.3.0. Note that you can install TB 60 in parallel on the same machine without doing any damage.

I'd like to compare the two sent messages to see what has changed. I'm not aware of any changes to Apple processing between 52 and 60 (although there was bug 1417187 where we removed BinHex support (HQX) when saving to disk which is unrelated to sending).

Comment 2

[lang]

Attachment: Test Message v52.9.1.eml

Comment 3

[lang]

Attachment: Undeliverable_ Test Message v60.3.0.eml

Comment 4

[lang]

G'Day Jorg, love it that you have "Europe - Australia - América" as the language options on your homepage!

I confirm that the problem has only started as soon as I updated to v60.3.0 and only appears to occur with MS Word documents and not PDFs.

As requested, I have uploaded 2 sample messages of the same email and Word attachment sent with v52.9.1 and v60.3.0.

Please let me know if you need any further information from me.

Regards, Tony Lang

Comment 5

[Jorg K (CEST = GMT+2)]

Thanks, but that's not quite what I asked for, perhaps I wasn't clear. I asked for the *sent* message (from the Sent folder), not the received one from the inbox. But anyway, the rejection message contains the original message at the end. Maybe for completeness you can just attach the 60 version from your Sent box to be sure. No need for the 52 version.

Anyway, looking at the 52 version in attachment 9024910 [details], I see a plaintext message
  This is a test message ...
with a single attachment:

Content-Type: application/msword; name="20181113 1st draft regulations.doc"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="20181113 1st draft regulations.doc"

No Apple-double anywhere to be seen.

In the other message we have the same text part:
  This is a test message ...

but then comes Apple-double :-(

--------------adD787D06E85BBE7840989B79F
Content-Type: application/applefile
Content-Transfer-Encoding: base64

BxYFAAAAAgAAAAAAAAAAAAAAAAAAAAAABQD8MAMAAABYAAAAIgAAAAkAAAB6AAAAIAAAAAgA
...

--------------adD787D06E85BBE7840989B79F
Content-Type: application/octet-stream;
	name="20181113 1st draft regulations.doc"
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename="20181113 1st draft regulations.doc"

0M8R4KGxGuEAAAAAAAAAAAAAAAAAAAAAPgADAP7/CQAGAAAAAAAAAAAAAAABAAAAYAAAAAAA
...

So for some reason TB 60 decided to send out the attachment as Apple-double :-(((

Richard, can you help me here. If you send an attachment using TB 60, does it now get always sent as Apple-double when it wasn't before. That would be quite terrible. I'm not aware of any changes in this area. Sadly also, our "regression findind service" Alice White doesn't work on Mac.

Comment 7

[Jorg K (CEST = GMT+2)]

It could be that some M-C "content sniffing" service has changed?? We'll have to investigate where this comes from but sadly we don't have any Mac developers.

Comment 8

[Jorg K (CEST = GMT+2)]

Actually, the reporter of bug 1503645 mentioned Mojave. Has that caused the change in behaviour?

Comment 9

[Jorg K (CEST = GMT+2)]

Digging  bit through the code it appears that on Mac we either send a ZIP file on the file encoded as Apple-double:
https://searchfox.org/comm-central/rev/943ba91d2b7c867fb4d131bd59ebed03ae8eb462/mailnews/compose/src/nsMsgAttachmentHandler.cpp#759

We check for a resource fork
https://searchfox.org/comm-central/rev/943ba91d2b7c867fb4d131bd59ebed03ae8eb462/mailnews/compose/src/nsMsgAttachmentHandler.cpp#843
and then then the file extension (and don't send the resource for for TXT, JPG and some others. DOC or PDF are not in the list.

So we go and encode the file as Apple-double. That code hasn't changed in ages, so I don't understand why the behaviour has changed now. I also don't understand why TB 52 did NOT send as Apple-double.

Who can read the code better than I can? Magnus and Alfred, anything obvious I'm overlooking?

Unless HasResourceFork() has changed.

Comment 10

[Jorg K (CEST = GMT+2)]

Hmm, HasResourceFork() uses this *deprecated* Apple system call FSGetCatalogInfo():
https://developer.apple.com/documentation/coreservices/1565356-fsgetcataloginfo

I assume the DOC and PDF file (bug 1503645) have resource forks.

Comment 11

[Magnus Melin [:mkmelin]]

I'm a bit confused. Why do we ever opt to send apple-double?

Comment 12

[Jorg K (CEST = GMT+2)]

I was an Apple person once upon a time in the late 1990ies when they still had PowerPC chips. I can tell you that a Mac file consists of two parts/files/forks, data and resources. The resource fork contains the file type resource since Mac files don't need a file extension. It can also contain images/icons, etc. Transmitting only the data part won't be well received by people using Mac-to-Mac e-mail.

That brings us to the options I can think of:
1) Never use Apple-double, always only transmit the data fork.
2) Have a preference that lets you disable (or enable?) Apple-double.
3) Never send Apple-double if the file has an extension.
4) Move the "non-Apple-double" list at
   https://searchfox.org/comm-central/rev/943ba91d2b7c867fb4d131bd59ebed03ae8eb462/mailnews/compose/src/nsMsgAttachmentHandler.cpp#856
   to a pref and at the same time add things like DOC, DOCX, PDF, ODT, XLS, ODS, etc.
5) Hard-code that list.

What's more important would be to understand what has changed.

I made some changes here, can anyone see what I did wrong? Or maybe right since apparently the Apple-double encoding didn't work in TB 52 (and no one noticed) and now it does again:
https://searchfox.org/comm-central/diff/9e29ae6d3fc944b42de513a26081cb3fc78cd44c/mailnews/compose/src/nsMsgAttachmentHandler.cpp#864

So Magnus, I hope I could dissipate the confusion and you can a) look whether you can see the offending change and b) help make as decision as to how to proceed.

Comment 13

[Jorg K (CEST = GMT+2)]

Interesting reading here: https://en.wikipedia.org/wiki/Resource_fork
Quote: However, as forks can make it difficult to exchange files with other operating systems, this feature is not in common use. Even in macOS, resource forks are seldom used anymore.

So maybe option 1) "complete removal" is the way to go.

Comment 14

[Jorg K (CEST = GMT+2)]

Tony, although according to the program source code TB should send an attachment as Apple-double if a resource fork is found, my colleague Richard, the only one with a Mac, wasn't able so far to produce an e-mail with that encoding on his Mac. He used a PDF from a message I had received privately from the reporter of bug 1503645.

So my questions:
Was the document sent from the same Mac using TB 52 and TB 60 or from different machines?
Does the problem exist for all Word documents or just some?
Does the problem exist for PDFs as well?

Comment 15

[Richard Marti (:Paenglab)]

Tried with Jörg different mails with TB 60 on Mac with Mojave but wasn't able to send as apple-double. Maybe MS Office as one of the last dinosaurs does still save with resource fork and I have no MS Office. Also saving the attachments from the messages in the bugs didn't create the resource fork.

Comment 17

[Jorg K (CEST = GMT+2)]

Attachment: 1506800-remove-apple-double.patch

Option 1) - completely remove Apple-double from Compose. We can't remove it from MIME since we still need to be able to process messages encoded that way.

I don't have a Mac, so let's see whether this compiles:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=cd5b5802c2e19ce11136d7d2e38eb405756dec29

Comment 18

[Jorg K (CEST = GMT+2)]

Comment on attachment 9025066 [details] [diff] [review]
1506800-remove-apple-double.patch

You want to do this?

Comment 19

[Jorg K (CEST = GMT+2)]

Attachment: 1506800-no-AppleDouble-if-entension.patch

Or better like this?

Comment 20

[Magnus Melin [:mkmelin]]

Comment on attachment 9025188 [details] [diff] [review]
1506800-no-AppleDouble-if-entension.patch

Review of attachment 9025188 [details] [diff] [review]:
-----------------------------------------------------------------

Yes for now, but I'd add a pref to switch it on if people get upset and need it for something. When it's well received, we can just take the larger patch.

Comment 21

[lang]

I am responding to Jorg's questions as follows:

Question 1: Was the document sent from the same Mac using TB 52 and TB 60 or from different machines?
Response 1: Same machine.

Question 2: Does the problem exist for all Word documents or just some?
Response 2: Appears to exist for all Word documents.

Question 3: Does the problem exist for PDFs as well?
Response 3: As previously indicated, problem does not appear to exist for PDFs.

May I reiterate that this problem has only begun with TB 60. I have been using TB since its inception in 2004, originally in Windows and since 2012 on my Mac, and I have never had this problem before. There has obviously been some change between TB 52 and TB 52 that has created this problem.

I appreciate the work you and your colleagues have been doing to investigate this problem, and I promise I will make a donation accordingly!

Comment 22

[lang]

My last post should read: "There has obviously been some change between TB 52 and TB 60 that has created this problem."

Comment 23

[Jorg K (CEST = GMT+2)]

Yes, but hard for us to work out what changed. As I said, reading the code, we should always send the undesired AppleDouble. I have no idea why a) that didn't happen in 52 and b) it also doesn't happen for my colleague Richard in 60.

That said, a patch is on the way and you will have a "special" version within 24 hours.

Comment 24

[Jorg K (CEST = GMT+2)]

Attachment: 1506800-no-AppleDouble-with-pref.patch

Magnus, since you requested a preference, I'm moving the ugly (and incomplete, why no PNG, for example) hard-coded list out into the pref. So by default, if the file has an extension, we don't use AppleDouble, if it has one, it needs to be whitelisted. Do you have a better name for the pref?

Comment 25

[Magnus Melin [:mkmelin]]

Comment on attachment 9025332 [details] [diff] [review]
1506800-no-AppleDouble-with-pref.patch

Review of attachment 9025332 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM, r=mkmelin

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/9e43ff6048ca
Don't sent AppleDouble for files with extension unless whitelisted. r=mkmelin

Comment 27

[Jorg K (CEST = GMT+2)]

TB 60.3.2/60.4 ESR:
https://hg.mozilla.org/releases/comm-esr60/rev/5ce5e6f8fc5132ee53f0c2bbe24e6eeb8f832590

Tony, please try an unofficial TB 60.3.2 from here:
https://queue.taskcluster.net/v1/task/c9W7m4HPRNSrWBwhE83HhA/runs/0/artifacts/public/build/target.dmg

Comment 28

[lang]

Thanks Jorg, I confirm that TB 60.3.2 fixes the problem. Thanks again for all your work in resolving this issue so promptly, I appreciate it.

Comment 29

[Jorg K (CEST = GMT+2)]

60.3.2 will most likely not be released. The fix will go into TB 60.4 due out in mid-December.

Comment 30

[info]

Thank you to all of you who took the time to help out and squash this bug! Much appreciated! Look forward to TB 64!

-Colin

Comment 31

[Jorg K (CEST = GMT+2)]

You mean 60.4? You can use the program from comment #27 (last line).

Comment 32

[info]

🙄yes.... 60.4 😂

Comment 33

[[e-mail]]

thanks for the update. It fixes also my problem (s. 1507177) But now the editing of adresses in the adressbook lead to the crash of Thunderbird 60.3.2

Comment 34

[Magnus Melin [:mkmelin]]

(Please prefix bug numbers with "bug" to get them auto linked, like bug 1507177)

In case you get a reproducible crash, file a new bug, and give the crashid. You can find it under Help | Troubleshooting information.

Comment 35

[[e-mail]]

thanks for the update 60.3.2
It fixes also my problem with doc. and docx. attachments (bug 1507177) 
But now the editing of adresses in the adressbook or attempt to write a new address leads to the crash of Thunderbird 60.3.2
File of new bug 1507699

Comment 36

[Jorg K (CEST = GMT+2)]

Beta (TB 64 beta 3):
https://hg.mozilla.org/releases/comm-beta/rev/bb567fd203e4

Comment 37

[Wayne Mery (:wsmwk)]

https://support.mozilla.org/en-US/questions/1241165
https://support.mozilla.org/en-US/questions/1240161

Ssumo search is so horrible, I can't spend time seaching for more - but I don't think there is much more