446321 - Cascadable Multi-part Decapsulation API

Status: NEW

(MailNews Core :: MIME, enhancement)

Description

[Aron Rubin]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
Build Identifier: thunderbird3

Description:
  Mozilla Core has cascadable multi-part decoding (nsIStreamConverter) but appears to be lacking cascadable multi-part decapsulation. The current mail code queries only the top level MIME part in nsIMsgMessageService.openAttachment and in the nsIMsgMessageFetchPartService. Also it is entirely unclear (to me) as to where to discover which encapsulated parts exist without full decapsulation.

Use Cases:
  * typical MIME multi-part decapsulation
  * MIME encapsulated MS-TNEF multi-part decapsulation
  * MIME encapsulated compressed archive decapsulation
  * MIME encapsulated MS-TNEF encapsulated compressed archive decapsulation
  * S/MIME multi-part stream decapsulation

Required Changes:
  * hierarchical part/section addressing
  * part traversal should flatten certain encapsulations (as for attachment
    pane)
  * decapsulation should have parameterized limitations for security (i.e. max
    depth, max breadth)


Reproducible: Always

Steps to Reproduce:
1.
2.
3.

Comment 1

[David Ascher (:davida)]

Aron, thanks for the bug.

Cc'ing some relevant folks.

(Aron is the author of LookOut, an add-on which does the TNEF attachment handling to better interoperate w/ Exchange users) 

Comment 2

[Aron Rubin]

Does anyone have any thoughts on this matter. Is this something that is already covered. Are there any good design ideas that come to mind?

Comment 3

[David :Bienvenu]

It's definitely not covered. I have some questions, though. 

>where to discover which encapsulated parts exist without full decapsulation

We only know if a part potentially encapsulates sub-parts from the content type, right? And the decapsulation is different for each content type. I don't know if we can avoid having libmime know about these special content types, though I can imagine a scheme where decapsulaters register content-types they know how to handle with libmime.

There are a few gotchas - delete/detach of attachments is not going to work with decapsulated parts, unless we know how to re-encapsulate. So I suspect that the attachment info is going to need a boolean that says the part is actually a sub-part of an encapsulated part, so we can disable detach/delete. And IMAP Mime Parts on Demand will not be able to deal with decapsulated parts, only real parts.

I'm imagining a scheme where decapsulater register for content type(s). If libmime encounters a part with that content type, it asks the de-encapsulater if it wants to handle that type - if it does, it streams the part to the de-encapsulater, and the de-encapsulater then streams the decoded parts to the original emitter as if they were real, but with a boolean that says they're not real. The decapsulaters would need to have access to things like s/mime since parts can be encapsulated using s/mime.

It sounds like you were thinking of a scheme where we extent the part query mechanism of libmime urls to allow decapsulation of parts, or fetching of de-encapsulated parts (but first we need to know about those sub-parts). We'd still need a way of associating a decapsulater with that encapsulated part...

Comment 4

[Aron Rubin]

Part on demand is definitely an issue here but only in terms of performance. That is why I listed parametrization as a requirement. Traversal into non-MIME multipart sections may set to occur with or without intervention on IMAP. I decided to read RFC 3501 (IMAP v4) a little more. It refers to nested MIME parts and also the ability to request partials. That ability to request partials (byte ranges) may enable us to request TNEF headers in the same way because each TNEF chunk has a length in its header. Likewise RFC 2311 refers to nested S/MIME parts. I imagine the decapsulaters would be registered as stream converters. If certain mime types could be qualified as a encapsulation types I imagine that would suffice to flag the message service to use a decapsulation interface.

Comment 5

[Aron Rubin]

One thing to note here is the reason we are talking about IMAP is because if the full message body is available there is no issue. I don't see MIME part specific mechanisms for POP. POP has XTND XLST for headers and TOP (line from start) but that is not sufficent. Therefore I will discount POP from part-on-demand considerations.

Ok so here is flow that I am thinking about for IMAP:
C: FETCH 123456 (BODYSTRUCTURE)
S: (("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "7BIT" 1152
         23)("APPLICATION" "MS-TNEF" ("NAME" "winmail.dat")
         NIL NIL "BASE64" 4554 73) "MIXED")
C- "streamconv;1?from=application/ms-tnef&to=*/*" supports nsMultipartInput
C- tnefFetchPart.getHeader( part = 0 )
C: FETCH 123456 (BODY.PEEK[1.2]<0.20>)
S: FETCH 123456 BODY[1.2]<0> eJ8+IhAPAQaQCAAEAAAA

Comment 6

[Aron Rubin]

Are there any more thoughts here? Design or implementation issues that are expected?

Comment 7

[David :Bienvenu]

Well, beyond what I already mentioned, the implementation issues are mainly that libmime is difficult to work with. 

It's also not clear to me how you would deal with the detach/delete attachment ui w.r.t. TNEF encapsulated parts. When we detach an attachment, we recreate the message with an empty stub for the part. I would think the TNEF stuff would complicate that.

Comment 8

[Gerry Healy]

I've done a bit of hacking recently on the Lookout extension and was also thinking how tnef handling could be a little more integrated.

My C++ is not as good as it could be, but I'd be happy to help with testing (I'm running OSX 10.4.11 on a G4 PPC). I find myself with quite a bit of time on my hands these days so with a bit of assistance I could contribute quite a bit more.