2034726 - Rust crate mail_parser viablity assesment based on all 323 Thunderbird .eml test cases

Status: RESOLVED WONTFIX

(MailNews Core :: MIME, task)

Description

[Max Emig [:maxe]]

I present to you, actually using a competent enough LLM to parse .eml and unit-tests to asses the possibility that we one day wire up the pretty promising mail-parser by Stalwart, that we actually already ship for some godforsaken reason, I have not yet tried to actually identify, as the primary MIME-parser:

Results

• Total files: 323
• Pass: 295 (91.3%)
• Hard fail: 8 (2.5%) — upstream bug required
• Soft fail: 20 (6.2%) — sane post-processing possible, no upstream bug needed
• Total fail: 28 (8.7%)

---

Hard Fails — upstream bug required

1. text/html inside multipart/related misclassified as PartType::Text

text/html parts inside multipart/related are returned as PartType::Text instead of PartType::Html, causing html_body_count: 0 despite correct content being decoded.

• 25-HTML+embedded-image.eml
• 27-plaintext+(HTML+embedded-image).eml
• 29-(HTML+embedded-image)+attachment.eml
• 30-plaintext+(HTML+embedded-image)+attachment.eml

2. MIME boundary leakage from nested multipart/alternative inside multipart/related

MIME boundary markers from a nested multipart/alternative leak into the body text of the preceding part. The inner structure is not parsed; raw headers appear in body output.

• content-utf8-rel-alt.eml
• test-rel-alt.eml
• plaintext-with-key-and-windows-1252-encoded-eml-attachment.eml
• plaintext-with-windows-1252-encoded-eml-attachment.eml

---

Soft Fails — post-processing sufficient, no upstream bug needed

1. cid-referenced images classified as Binary (attachment) instead of InlineBinary (18 files)

cid-referenced image parts inside multipart/related are returned as Binary (attachment) instead of InlineBinary (inline). The Content-ID is correctly parsed and exposed, so a consumer can reclassify in post-processing by matching cid values against src="cid:..." references in the HTML body.

• 05-HTML+embedded-image.eml
• 07-plaintext+(HTML+embedded-image).eml
• 09-(HTML+embedded-image)+attachment.eml
• 10-plaintext+(HTML+embedded-image)+attachment.eml
• 15-HTML+embedded-image.eml
• 17-plaintext+(HTML+embedded-image).eml
• 19-(HTML+embedded-image)+attachment.eml
• 20-plaintext+(HTML+embedded-image)+attachment.eml
• bug1358565.eml
• clickableContent.eml
• content-utf8-alt-rel.eml
• content-utf8-alt-rel2.eml
• content-utf8-rel-only.eml
• malformed_multipart_mixed_cid.eml
• sample08.eml
• sampleContent.eml
• test-alt-rel.eml
• test-alt-rel-with-attach.eml

2. Fragmented multipart HTML bodies (2 files)

A single logical HTML body is split across multiple parts and returned as separate fragments. Mergeable in post-processing.

• partial-encrypt-for-alice-html.eml
• partial-encrypt-for-carol-html.eml

As a bonus, if we were to implement the cid-matching against html as post-processing for images, we would end up fixing 1-2 bugs we just marked as won't fix.

---

Sidenote: No IMAP protocol handler good enough for us to pull it straight in exists, as-of-right-now.

Comment 1

[Max Emig [:maxe]]

we one day wire up the pretty promising mail-parser by Stalwart, that we actually already ship for some godforsaken reason, I have not yet tried to actually identify

It's used in the EWS (Exchange Web Services) protocol handler for parsing e-mail headers: Message-ID, From, To, Cc, Bcc, timestamps, attachments.

Specifically the MessageHeaders trait is implemented on mail_parser::Message, so EWS uses mail-parser to parse raw RFC 822 messages fetched from Exchange.

---

Is this assessment correct?

Comment 2

[Brendan Abolivier [:babolivier]]

EWS currently uses mail-parser, that's correct, at least for now.

Something we want to look into at some point, which this assessment does not seem to take into account, is the memory footprint. Email messages can get very large (several GBs per message), and mail-parser's approach is to load it all into memory, whereas our existing parser works by streaming message contents from the raw RFC822 source (e.g. local storage or the server) to its destination. This avoids having to take up potentially gigabytes of memory just to load/display a single message. This is important because, as much as reasonably possible, we want Thunderbird to work on configurations with low amount of resources such as memory. For this reason, we want to be loading full messages into memory as little as technically possible.

Streaming is not supported by mail-parser, and it does not look like there is a plan to support it any time soon. For now, it's excusable in EWS because we don't yet support parsing EWS responses from a stream (we consume the entire stream then parse its contents). However, this is something we want to address in the future for the same reasons as previously stated, at which point we'll need to move away from mail-parser, and will probably move to using Thunderbird's existing parsing facilities (which would be better for consistency with other protocols anyway).

Comment 3

[Max Emig [:maxe]]

which this assessment does not seem to take into account

Correct, I built myself a fixture to compare our expectation of test cases to the output of mail_parser as guidance for some of the tougher bug 1405234 cases.

Then, I got curious and and orchestrated the comparison for all existing .eml-files.

Email messages can get very large (several GBs per message)

Do you have an example where one can get an above like 100 MB message, for my understanding? But fair point, I hope this bug will serve as guidance for future tb-planning.

Comment 4

[Brendan Abolivier [:babolivier]]

(In reply to Max Emig from comment #3)

Email messages can get very large (several GBs per message)

Do you have an example where one can get an above like 100 MB message, for my understanding? But fair point, I hope this bug will serve as guidance for future tb-planning.

Adding attachments to a message can get its size to grow quite quickly. Not all providers support messages that large (and most popular email providers don't, because storage isn't exactly cheap), but that's not a limit in protocol specs, and we obviously don't know what every provider on the planet allows or doesn't. As an aside, even 100MB can matter for old repurposed hardware.

Comment 5

[Max Emig [:maxe]]

Not something that can work with the current prerequisites.