760412 - Multipart/alternative (with Lightning): Out of 3 alternative parts, TB does not display nor offer as attachment any of (1) text/plain or (2) text/html, but only (3) text/calendar (mail with invite from google calendar)

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Gaetano Giunta]

Attachment: the email in question

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0
Build ID: 20120420145725

Steps to reproduce:

[filing under general, though it might be Lightning-related]

receive an email. Email was sent using the "send msg to guests" button in goggle calendar's event page


Actual results:

TB only showed the event info, not the plaintext part of the mail (the "message" in the google calendar form)

.ics file was correctly shown as atachment as well, and could be saved

The funny thing is: the popup-notification window when mail was received showed instead the plaintext message


Expected results:

tb should have displayed the plaintext part of the mail above the event invite

Comment 1

[Hashem Masoud]

(In reply to Gaetano Giunta from comment #0)
checked your attachment and the text part appears for me (which shows the following):

Hello,
I am so sorry, but someone had scheduled a meeting exactly at the same time  
as ours. Lets try gotowebinar:  
https://www3.gotomeeting.com/register/128585726
Thanks for your patience.
Melissa

Comment 2

[Ludovic Hirlimann [:Usul]]

Gaetano did you try in Thunderbird's safe-mode ?

Hashem do you have lightning installed ?

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to Gaetano Giunta from comment #0)
> TB only showed the event info, not the plaintext part of the mail (the
> "message" in the google calendar form)

Where can we see "ATTACHMENT part of text/calender" in the mail?

Mail structure.
  Content-Type: multipart/mixed; boundary=0016e6d976262e053a04c1649135
  --0016e6d976262e053a04c1649135
  Content-Type: multipart/alternative; boundary=0016e6d976262e053304c1649133
    --0016e6d976262e053304c1649133
    Content-Type: text/plain;
    --0016e6d976262e053304c1649133
    Content-Type: text/html;
    --0016e6d976262e053304c1649133
    Content-Type: text/calendar;
    --0016e6d976262e053304c1649133--
  --0016e6d976262e053a04c1649135
  Content-Type: application/ics; name="invite.ics"
  --0016e6d976262e053a04c1649135--

Because multipart/alternative and Tb has capability to show text/plain or text/html as message body, Tb chooses one of text/plain or text/html part in multipart/alternative, and because Tb is not a mailer who shows text/calender as message body, Tb ignores text/calander in multipart/alternative according to definition of multipart/altenative.
What's wrong?

Comment 4

[Hashem Masoud]

(In reply to Ludovic Hirlimann [:Usul] from comment #2)
> Gaetano did you try in Thunderbird's safe-mode ?
> 
> Hashem do you have lightning installed ?

No, I don't.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to Gaetano Giunta from comment #0)
> the email in question
> Sender: Google Calendar <[e-mail]>

Does Google intentionally ignore difference among RFCs for multipart/mixed, multipart/alternative, multipart/related?

As for text/calender in multipart/related which is not pointed by text/html.
(i)  Lightning shows it as attachment
(ii) Tb already has quirks for it and shows it as if attachment,
     if one of name(Content-Type:), attachment(Content-Disposition:),
     filename(Content-Disposition:) is specified.

As for text/calender which is placed in multipart/alternative.
(iii) Tb 12 already shows it as if attachment too,
      if one of name(Content-Type:), attachment(Content-Disposition:),
      filename(Content-Disposition:) is specified. 
      However, even if this part is shown as if attachment by Tb,
      following problem occurs in Tb 12.0.1.
      - The text/calander part can't be opened due to following error.  
        ! This attachment appears to be empty.
          Please check with the person who sent this.
          Often company firewalls or antivirus programs will destroy attachments.
      - If save is requested, nothing is saved. Saved file is not created.
      This is already reported phenomenon to other bug.

In any case of malformed text/calender under multipart/related and malformed text/calender under multipart/related, 
(iv) text/calender part is not shown as if attachment by Tb, if no name(Content-Type:), no attachment(Content-Disposition:), no filename(Content-Disposition:). This is also known issue.

To see/save malformed text/calender under multipart/alternative, following workaround is already available.
(1) mailnews.display.show_all_body_parts_menu = true
(2) View/Message Body As/All Body Parts.
    Because any multipart/xxx is treated as multipart/mixed in this mode,
    any part can be opened by an application and can be saved as a file.
(3) Change back to mailnews.display.show_all_body_parts_menu=false for daily use.

Comment 6

[Gaetano Giunta]

@wada

> Does Google intentionally ignore difference among RFCs for multipart/mixed, multipart/alternative, multipart/related?

No clue. Ask them! :-)

Thanks for the detailed explanation you have given so far, but I think you might have misunderstood the problem.

When opening the mail, I DO see the calendar invite, both inline in the mail body and as attachment that can be saved as .ics.

The problem is that I DO NOT see the textual part which reads "I am so sorry, but someone ...".
This text is sent as parts 1 (text) and 2 (html) of the 3 which are sent as alternatives within the 1st part of the top level "mixed" multipart

Comment 7

[Gaetano Giunta]

PS: of all recipients of this mail, I was the only one having this problem, so other mail readers act differently (I'd expect gmail's own and apple mail to have been used here)

Comment 8

[Gaetano Giunta]

Attachment: Screenshot of tb displaying the mail

Comment 9

[Gaetano Giunta]

Lat note: when enabling mailnews.display.show_all_body_parts_menu = true, I see the missing text displayed twice: once in a "part" of its own, once just above the invite box

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Gaetano Giunta from comment #6)
> The problem is that I DO NOT see the textual part which reads "I am so
> sorry, but someone ...".
> This text is sent as parts 1 (text) and 2 (html) of the 3 which are sent as
> alternatives within the 1st part of the top level "mixed" multipart

Oh, sorry for my misunderstanding.

When did your problem started to occur?
No problem with Tb 12 and started to occur after upgrade to Tb 12.0.0 or 12.0.1?

Do you see your problem with any combination of of View/Message Body As and View/Display Attachments Inline ?
(Original HTML/Simple HTML/Plain Text, Display Attachments Inline=Checked/Unchecked)

Do you see your problem with new profile, with no add-on?
(1) thunderbird.exe -ProfileManager, create a new profile, start with it.
    When account Wizard is shown, cancel it,
    and manually create a dummy News or POP3 account(via Manual Setup).
    (any non-existent server such as news.news.news or pop3.pop3.pop3).
    "Local Folders" account is created.
(2) Create a folder(call FolderX) under Local Folders.
    Drag&Drop .eml file of the mail to thread pane of FolderX.
    (import of .eml by Drag&Drop)
What is shown by Tb 12?

As invitation is shown, you probably install Lightning.
Does your problem occur with Lightning installed to the new profile?

Comment 11

[Thomas D. (:thomas8)]

(In reply to WADA from comment #10)
> (In reply to Gaetano Giunta from comment #6)
> > The problem is that I DO NOT see the textual part which reads "I am so
> > sorry, but someone ...".
> > This text is sent as parts 1 (text) and 2 (html) of the 3 which are sent as
> > alternatives within the 1st part of the top level "mixed" multipart
> When did your problem started to occur?
> No problem with Tb 12 and started to occur after upgrade to Tb 12.0.0 or
> 12.0.1?

didn't test for regression range

> Do you see your problem with any combination of of View/Message Body As and
> View/Display Attachments Inline ?
> (Original HTML/Simple HTML/Plain Text,
>  Display Attachments Inline=Checked/Unchecked)

With all of these combinations (exactly as above), result is same:
(TB12.0.1, new profile, no addons, WinXP)

a) with Lightning:
- only text/calender (alternative part 3) is shown
- text/plain and text/html (alternative part 1 and 2) are missing entirely (from display, and attachment pane)
- only .ics is shown as real attachment

b) without Lightning:
- correctly toggles between: View msg body as > Plain text vs. HTML (alternative parts 1 or 2 are shown)
- text/calendar (alternative part 3) is entirely missing (never displayed, not in attachment pane)
- .ics is always shown as real attachment

> Do you see your problem with new profile, with no add-on?
> What is shown by Tb 12?

Exactly same as above (tried new profile, no addons, with and without Lightning).

> As invitation is shown, you probably install Lightning.
> Does your problem occur with Lightning installed to the new profile?

Yes, as above.

Comment 12

[Thomas D. (:thomas8)]

This is the reduced problem in a nutshell:

msg with multipart/alternative part containing 3(!) alternative parts:
(1) text/plain
(2) text/html
(3) text/calendar

a) with Lightning:
- we never show any of (1) text/plain or (2) text/html
- we only show (3) text/calendar

b) without Lightning:
- we correctly toggle between (1) text/plain or (2) text/html
- we never show (3) text/calendar

Even if that triple alternative structure produced by google might be unfortunate or malformed, we need to handle this case better (robustness principle).

Comment 13

[WADA:World Anti-bad-Duping Agency]

"Show text/calender under multipart/alternative as if attachment when name/filename/attachment is specified" is a new functionality of recent Tb.
This may affect on Lightning's subpart selection/handling.

Do you see similar problem in Tb 12/Lightning on (a) "text/calender under multipart/related" or (b) "text/calender mail"?

(a) Edit .eml file of "the email in question" by text editor, and change multipart/alernative to multipart/related, then Drag&Drop .eml file to thread pane of a local mail folder.

(b) Mail's Content-Type: is "Content-Type: text/calender", and message body of the mail is text/calendar data.

Comment 14

[WADA:World Anti-bad-Duping Agency]

Issue in "text/calender under multipart/alternative" without Lightning is currently processed by Bug 505024. Setting dependency for ease of problem analysis, tracking, and search.

Comment 17

[José M. Muñoz]

Let's get this straight:
Bug 505024: In an multipart/alternative message, the text/calendar part isn't shown unless Lightning is installed.
Bug 760412: In an multipart/alternative message, the text/html part isn't shown if Lightning is installed. Different issue.
Since TB78 Lightning is always "installed" (integrated into TB), so bug 505024 has been resolved and bug 760412 permanently becomes an issue.

Comment 21

[José M. Muñoz]

There used to be this add-on to always show the text/calendar part:
https://addons.thunderbird.net/en-US/thunderbird/addon/sfoa/

Comment 22

[Noah]

I think this is the correct issue for us to continue on. Interestingly, I saved the .eml for two different effected emails, removed the entire text/calendar section and opened it again and they now display the text/html version as well as the attached .ics file. Seems like if there is a way to prefer the text/html OR ignore the text/calendar version, it would resolve this bug.

Comment 23

[José M. Muñoz]

Please read the RFC: https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html
The multipart/alternative type is syntactically identical to multipart/mixed, but the semantics are different. In particular, each of the parts is an "alternative" version of the same information. User agents should recognize that the content of the various parts are interchangeable. The user agent should either choose the "best" type based on the user's environment and preferences, or offer the user the available alternatives. In general, choosing the best type means displaying only the LAST part that can be displayed.

So if the message contains text/html followed by text/calendar, only text/calendar part will be displayed. If the event is also included as a .ics attachment, there is no point to have a text/calendar part with the same information. Microsoft OL clearly infringes the RFC here. That is not to say that TB shouldn't handle the case better.

Comment 24

[Noah]

To be honest, I'm not a web-person to this detail, just an avid user for many years who wants Thunderbird to keep going strongly. The emails are from Google Calendar, which is also a large user base, so I wouldn't want this to be the cause of lost users. Happy to continue to test/report back/etc. to help with this issue.

Comment 25

[José M. Muñoz]

Sorry, I re-read bug 1725690 comment #0 now, it's not Microsoft, but Google. I thought that Google didn't put any content into the HTML part, but for updated events, that's apparently not the case. The bug has been on file since 2012. You'll have to see whether the TB team prioritizes it. Sadly, only showing one MIME part is deeply buried in 20+ y/o MIME code that no one wants to touch.

Comment 26

[Noah]

Hopefully it can be addressed at some point soon. It is certainly a bother for those imporant event update emails.

Comment 27

[Zoe Martin]

We will ship this fix: https://github.com/Betterbird/thunderbird-patches/blob/main/91/bugs/760412-always-show-text-calendar.patch

Comment 28

[Noah]

Does this mean it will get into Thunderbird?

Comment 29

[Noah]

I can confirm that the fix noted for Betterbird in comment 27 works. Betterbird displays the text/HTML and then the ics context (assuming due to inline attachments).

Comment 31

[Magnus Melin [:mkmelin] (away, back June 30)]

Attachment: Bug 760412 - Always display text/calendar part along with chosen text/html part. r=darktrojan

This makes us always show the generated invite html together with the normal html part.
Invites usually have an ok display in there generated by the server or sending client, and I think we don't want to normally show both parts at the same time. This patch changes the imip invite to just show an expandable summary on top of the html from the email. By clicking the arrow the details (like we had them) will be displayed, so that it still is accessible for the cases where the html representation is not there in the email, or it's bad.

Displaying the normal html content of the email solves problems expecially for "cancelled with a note" updates where you otherwise can't see the note because it's only inside the mail content.

Based on https://github.com/Betterbird/thunderbird-patches/blob/main/91/bugs/760412-always-show-text-calendar.patch

Comment 33

[Noah]

I'm happy to test out this patch if there is an easy way to.

Comment 34

[bhakta0007]

I'd be happy to test this too - Happy to see this getting fixed [I have been waiting for this fix for years now! https://bugzilla.mozilla.org/show_bug.cgi?id=1581070]

Comment 35

[bhakta0007]

I took the latest nightly build [https://ftp.mozilla.org/pub/thunderbird/nightly/2021/08/2021-08-30-10-47-50-comm-central/thunderbird-93.0a1.en-US.mac.dmg]

I still see the issue that I reported in the original bug I filed [1581070 ].

I am not sure if the fix is supposed to be present in 30Aug nightly this build?

Comment 36

[Magnus Melin [:mkmelin] (away, back June 30)]

It's not yet fixed. I need to adjust some tests before getting it landed.

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/565fc4b596ea
Always display text/calendar part along with chosen text/html part. r=darktrojan

Comment 38

[Magnus Melin [:mkmelin] (away, back June 30)]

I think we eventually want to uplift to 91, since there's so cases where it's needed to get the information someone sent you.

Comment 39

[bhakta0007]

I confirm this is fixed in the nightly build: 93.0a1 (2021-09-01) (64-bit)

@Magnus - thank you so much for getting this resolved.

Comment 40

[Henry Wilkes (they/them) [:henry-x]]

After this change, this CSS selector is unused https://searchfox.org/comm-central/rev/88d88aa4086176def5c63925b9d727b593ec6b99/calendar/base/themes/common/imip.css#28-34 . It used to be used for the imipHtml-header, which was the table <caption class="header">, but is now a <summary>. Can you clean it up?

Comment 41

[Magnus Melin [:mkmelin] (away, back June 30)]

Attachment: Bug 760412 - followup to remove now unused css selector. r=henry

Comment 42

[Noah]

Confirming that 93.0a1 is working properly for these emails from Google Calendar this morning. Very excited for this to be pushed to the regular release. Thank you all for your hard work and attention to this long existing bug!

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4ff85c5a8693
followup to remove now unused css selector. r=henry

Comment 44

[Rob Lemley [:rjl]]

Comment on attachment 9237555 [details]
Bug 760412 - Always display text/calendar part along with chosen text/html part. r=darktrojan

[Approval Request Comment]
To show missing information from invites. Landed in 93 milestone.

Comment 45

[Wayne Mery (:wsmwk)]

Comment on attachment 9237555 [details]
Bug 760412 - Always display text/calendar part along with chosen text/html part. r=darktrojan

[Triage Comment]
Approved for esr91

(it also has tests)

Comment 46

[Rob Lemley [:rjl]]

Thunderbird 91.2.0:
https://hg.mozilla.org/releases/comm-esr91/rev/8a40fbc006d5
https://hg.mozilla.org/releases/comm-esr91/rev/0d3752ae10fc

Comment 50

[Richard Leger]

Does this patch has the following side effect/regression?
https://thunderbird.topicbox.com/groups/enterprise/T0661b5915a2d00e8-M43050dbf43c83726521bb141/expand-calendar-event-details-by-default

Comment 51

[Magnus Melin [:mkmelin] (away, back June 30)]

It's expected yes.

Comment 52

[Ben Bucksch (:BenB)]

I've run into that as well, and I agree with Richard that this UI is unfortunate. I've filed https://bugzilla.mozilla.org/show_bug.cgi?id=1742101