Closed Bug 1334937 Opened 4 years ago Closed 4 years ago

text/calendar MIME part wrongly displayed when Lightning is not installed

Categories

(MailNews Core :: MIME, defect)

Unspecified
All
defect
Not set
normal

Tracking

(thunderbird52+ fixed, thunderbird53 fixed, thunderbird54 fixed, seamonkey2.47 wontfix, seamonkey2.48 fixed, seamonkey2.49esr fixed, seamonkey2.50 fixed, seamonkey2.51 fixed)

VERIFIED FIXED
Thunderbird 54.0
Tracking Status
thunderbird52 + fixed
thunderbird53 --- fixed
thunderbird54 --- fixed
seamonkey2.47 --- wontfix
seamonkey2.48 --- fixed
seamonkey2.49esr --- fixed
seamonkey2.50 --- fixed
seamonkey2.51 --- fixed

People

(Reporter: t50, Assigned: jorgk-bmo)

References

Details

(Keywords: regression, reproducible)

User Story

Facts:
-------
a) Reproducible for Google Calendar Invitations
c) No problem in older SM because those versions did not 
   wrongly show .ics contents inline.
d) Valid regression, SM /B may not show .ics without having an
   integrated .ics viewer. 
e) Also TB Problem, with 'wessage body as full html" and Lightning 
   disabled TB Daily 53.0a1 (2017-01-09) (64-bit) also shows that
   .ics source code
b) Appeared with SM 2.46

More investigation required
---------------------------
./.

Attachments

(3 files, 1 obsolete file)

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:49.0) Gecko/20100101 Firefox/49.0 SeaMonkey/2.46
Build ID: 20161213183751

Steps to reproduce:

I always received attachments .ics(with calendar invitations) in version 2.39 and was listed (rendered & previewed correctly). After upgrade to 2.46 it is not interpreted any more (no frame of calendar meeting and invited persons - pure not parsed at all message only).


Actual results:

I can't see email messages with .ics (calendar files) properly in 2.46.
I had to rollback to v2.39 where everythink still works ok with .ics calendar preview.


Expected results:

Please repair preview and rendering of ics files in 2.46 like it was working in 2.39 and earlier.
Severity: normal → critical
Component: MailNews: General → MailNews: Message Display
Dupe of Bug 505024 ?
Not sure if it might be related or a dupe of bug 1326271. If yes should be fixed in the next version.
I don't use any plugins in seamonkey and OS (Lightning or sth.)
(In reply to Mike from comment #3)
> I don't use any plugins in seamonkey and OS (Lightning or sth.)

I really really doubt that that is true.  2.39 showed something different to a simple text listing? I doubt. 
Please forward to me by email such an email which was rendered as a Lightning like calendar invitation box.
Flags: needinfo?(t50)
I attached the screenshots which ilustrates bugged rendering of invite.ics between 2.39(or older was OK) and 2.46(which is buggy).
Screenshots was made on the same e-mail when was previewed and opened in both versions - build version also included.
Flags: needinfo?(t50)
I only ever saw such raw calandar entry mails when Lightning was not installed. There were a lot of changes in the mailnews code between 2.39 and 2.46. They might just be processed differently now. If would be good if you can attach a sample anonymized eml here.

Please refrain from further personal accusations.
More or less REPRODUCIBLE with official  German SeaMonkey 2.46 FINAL  (Windows NT 6.1; WOW64; rv:49.0)  Gecko/20100101 Firefox/49.0 Build 20161213183751  (Default Classic Theme)  on German WIN7 64bit.

Reporter's problem is not related to standard .ics attachments, which AFAIK  never were visible with SeaMonkey.

But if I create an invitation for a guest from my Google Calendar, it will be shown similar to reporter's screenshots with SM 2.39, but 2.46 only will show  source code.

I will attach a sample file for further investigation
Severity: critical → normal
User Story: (updated)
Keywords: reproducible
Summary: No preview and no reading of email attachment files .ics in ver 2.46 - in 2.39 all worked OK → No preview and no reading of email attachment files .ics from Google Calendar Invitation
Steps how to reproduce with SM 2.46:

0. Download Attachment
1. Launch SM email → Change 'View Message body' to simpe of full HTML if 
   necessary
2. Menu 'File → Open → Browse for Attachment → [open]
   » .eml will open in separate Window and show some .ics source

With 2.39 I see an invitation similar to reporter's screenshots
(In reply to Rainer Bielefeld from comment #9)
I forgot to mention in step 1: Disable Lightning, if necessary!

I think the core of the problem is that SM 2.46 shows the attached .ics (inline) without having an integrated .ics viewer.
SM must not show inline attachments which it can not render!

After some more investigation I will forward this one to MailNews CORE
User Story: (updated)
Also REPRODUCIBLE with attachment "Invitation From Google Calendar (.eml)" and  Server-Installation of  official  en-US SeaMonkey 2.50a1 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0 Build 20170118003001  (Default Classic Theme) on VirtualBox Ubuntu 14.04.1:

.ics wrongly shown inline.
So OS=All
OS: Unspecified → All
Summary: No preview and no reading of email attachment files .ics from Google Calendar Invitation → .ics attachment wrongly shown inline although no viewer available
b1): With steps from comment 9
     Already REPRODUCIBLE with  English SeaMonkey 2.46a2  (Windows NT 6.1; WOW64;
     rv:49.0)  Gecko/20100101 Firefox/49.0 Build 20160723030242  
    (Default Classic Theme)  on German WIN7 64bit
b2): Still ok with  English SeaMonkey 2.45  (Windows NT 6.1; WOW64; rv:48.0) 
    Gecko/20100101 Firefox/48.0 Build 20160710233343  
    (Default Classic Theme)  on German WIN7 64bit 
b): So probably appeared with MailNews Core 49
User Story: (updated)
Component: MailNews: Message Display → Attachments
Product: SeaMonkey → MailNews Core
Version: SeaMonkey 2.46 Branch → 49
f) No obvious DUP found with <https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=DUPs1334937&sharer_id=41036>
Status: UNCONFIRMED → NEW
Ever confirmed: true
User Story: (updated)
Thank You Rainer for Your professional analysis. 
I can't attach You genuine ics's becouse I preview "restricted to me invitations" from Google Mail System.
I could try anonymize it in source maybe, but not now.
All invitations "rendering" worked in Seamonkey versions 2.39 & earlier with coloured borders and was OK.
After direct upgrade to 2.46 rendering doesn't work.
I didn't know if I need any plugin or something to render this or something has broken.
I enetered this as a Bug becouse rollback to 2.39 helped me to recover .ics rendering functionality.
If You have any solution for me to this or safe plugin to install for viewing in 2.46 - it is welcomed.
Thank's.
This is basically bug 505024. However, the behaviour of the system changed a bit in bug 574989 which landed on TB 49. So TB 49 = SM 2.46 behaves differently to prior versions.

The problem is the MIME structure of the message. The top level is multipart/mixed so the first part is the message, and the second part the ICS attachment.

The message itself is multipart/alternative with three (!!) parts instead of two:
text/plain
text/html
text/calendar

For multipart/alternative the last (!) part should be displayed to get the richest display according to RFC 1521. Sadly, the last part is not the HTML but some funny text part.

Anyway, I'll attach a patch in a minute.
Component: Attachments → MIME
For people interested in history:
https://hg.mozilla.org/comm-central/rev/ab68d375e7e4#l4.136
Blocks: 574989
Keywords: regression
OK, here's the patch. In order to understand it, you need to understand the prioritising we have implemented.

When the display is set to HTML (TB: View > Message Body As > Original HTML), we're looking for a rich part in the alternative section, regardless of the order.

text/plain is prioritised as "plain text" and text/html, text/enriched, text/richtext and sadly also text/calendar were prioritised as "rich text". The last rich part won, so in this case text/calendar over text/html.

Obviously this is wrong, since text/calendar is "plain text" MIME type and not a rich one. So just removing that from the group fixes the problem.

Voilà.

Now the bigger problem is getting this reviewed. FRG, can you do reviews?
Assignee: nobody → jorgk
Status: NEW → ASSIGNED
Attachment #8832864 - Flags: review?(iann_bugzilla)
>> Now the bigger problem is getting this reviewed. FRG, can you do reviews?

Occasionally for small things. I think this one qualifies and IanN will not shoot me. I can't approve any uplifts down to comm-release.
Comment on attachment 8832864 [details] [diff] [review]
1334937-remove-text-calendar.patch

r=frg

Don't forget to change the patch heading to reflect this.

Thanks
Attachment #8832864 - Flags: review?(iann_bugzilla) → review+
https://hg.mozilla.org/comm-central/rev/a2e039a852d7c66b373d7020847f30bc15783e8a

Thanks for the review. I changed the commit message and even the patch itself to clarify this in a comment ;-)
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Comment on attachment 8832864 [details] [diff] [review]
1334937-remove-text-calendar.patch

Clear regression from bug 574989, and, might I add, the only one although we did quite some changes in that bug ;-)

This needs to go into all branches, especially for TB 52 ESR. I'll handle the C-A and C-B uplifts.

I forgot to say in the previous comment: My new landing policy is to trigger a build after every M-I to M-C merge. One had just happened after the last Daily build, so this push/build was necessary.
Attachment #8832864 - Flags: approval-comm-release+
Attachment #8832864 - Flags: approval-comm-beta+
Attachment #8832864 - Flags: approval-comm-aurora+
Target Milestone: --- → Thunderbird 54.0
Summary: .ics attachment wrongly shown inline although no viewer available → text/calendar MIME part wrongly prioritised over text/html in multipart/alternative
Thanks. I will wait till its in c-b and then push it to c-r.
Had a Lotus Notes message in my drafts folder which I copied there some time ago to test something. It showed the same behaviour after disabling Lighting. With the fix here a proper preview and message is shown.

User agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0 SeaMonkey/2.51a1
Build identifier: 20170202161617
Status: RESOLVED → VERIFIED
Jorg,

I just noticed that with Lightning enabled my Notes message is no longer recognized as an invitation and I am unable to add it to a calendar. This might need to be backed out.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
https://hg.mozilla.org/comm-central/rev/a2e039a852d7c66b373d7020847f30bc15783e8a (comment #20)
https://hg.mozilla.org/comm-central/rev/3cbd2f30207c05929b94af61fea9b9f4184aa8eb (bustage fix)

Oops, Linux and Mac didn't like |text/*| in a comment.

But now I see that there is a complaint in comment #24. Can you attach the message in question or send it to me in a PM.
I can see it now. The HTML part displays fine, but Lightning doesn't pick up the invitation.
Hmm, I did an experiment an removed the attachment. Lightning gets the invitation from the third mime part, the text/calendar, not the invitation.

I think the patch is right as far as the display is concerned, so not to display the text/calendar part instead of the text/html part, but then Lightning doesn't see that part. Let's take it out for now.

Backout:
https://hg.mozilla.org/comm-central/rev/dca11f96a228ccdcc9ec5a0123f82d6f8348139e
https://hg.mozilla.org/comm-central/rev/ca5f57cbe502fa561f667282abfd0914f522eb3f
Just tried to anonymize my mail and remove the company information but failed so sorry can't send you my testcase right now.

I agree. The display part is right. Looks like it just needs a patch somewhere else too.
MMD, if you have a multipart/alternative message with three MIME parts:
text/plain
text/html
text/calendar
how does Lightning grab the last part as invitation?

If I change the MIME processing to always select the html part for display, then Lightning doesn't see the invitation. So does Lightning just grab the displayed part and extract an invitation if it's text/calendar?

Also I'm wondering how this worked before bug 574989, for example in TB 45.x. With Lightning, we get the invitation and without Lightning I get the html part displayed, although it's not the last part. Hmm.
Change was this: https://hg.mozilla.org/comm-central/rev/ab68d375e7e4.

Terje, do you have an idea?
Flags: needinfo?(makemyday)
Flags: needinfo?(bugzilla)
MMD, there doesn't appear to be a test assuring that with Lightning installed see that text/calendar part as invitation. Here
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=a2e039a852d7c66b373d7020847f30bc15783e8a
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=3cbd2f30207c05929b94af61fea9b9f4184aa8eb
the patch from this bug was in place prioritising the text/html part over the text/calendar part for display, yet, no test failed.
Confirming the regression. After backing out revision ab68d375e7e4 locally the behaviour is this:
With Lightning: Invitation is displayed.
Without Lightning: HTML part is displayed.

https://hg.mozilla.org/comm-central/rev/ab68d375e7e4 isn't huge change, so it shouldn't be to hard to fix, well, it's MIME ;-(

I noticed a MIME calendar hook here:
https://dxr.mozilla.org/comm-central/rev/756fa9e390ab08f995382c9a9dfc6bf4b72421ca/mailnews/mime/src/mimei.cpp#466
in mime_find_class().

Looks like a parameter to the call of that function got changed here:
https://hg.mozilla.org/comm-central/rev/ab68d375e7e4#l4.61
https://hg.mozilla.org/comm-central/rev/ab68d375e7e4#l4.75
FRG, you can try this.
Attachment #8832864 - Attachment is obsolete: true
Flags: needinfo?(makemyday)
Attachment #8833038 - Flags: feedback?(frgrahl)
Attachment #8833038 - Flags: feedback?(bugzilla)
(Oops, I hit <enter> while the caret wasn't in the text area and the comment got submitted).

Terje, can you explain why you changed 'true' to 'false'
https://hg.mozilla.org/comm-central/rev/ab68d375e7e4#l4.61
https://hg.mozilla.org/comm-central/rev/ab68d375e7e4#l4.75
and what consequences it might have to turn it back.
The test from bug 574989 and bug 253830 still passes:
mozmake SOLO_TEST=message-window/test-view-plaintext.js mozmill-one
Comment on attachment 8833038 [details] [diff] [review]
1334937-call-mime_find_class-true.patch

Quick check looks ok to me. With Lightning disabled I get the html view. 

Enabled the invite buttons etc. work.
Attachment #8833038 - Flags: feedback?(frgrahl) → feedback+
Comment on attachment 8833038 [details] [diff] [review]
1334937-call-mime_find_class-true.patch

You can stick an r+ onto it then. Your review is as good as anyone's (no offence intended) since absolutely nobody understands this stuff, unless maybe Terje. I'll wait for his feedback.
Attachment #8833038 - Flags: review?(frgrahl)
Comment on attachment 8833038 [details] [diff] [review]
1334937-call-mime_find_class-true.patch

Ok then

r=me
Attachment #8833038 - Flags: review?(frgrahl) → review+
Attachment #8832864 - Flags: approval-comm-release+
Attachment #8832864 - Flags: approval-comm-beta+
Attachment #8832864 - Flags: approval-comm-aurora+
I did some more debugging on this.

With the current state
  mime_find_class (ct, sub_hdrs, self->options, false);
  if (clazz && clazz->displayable_inline_p(clazz, sub_hdrs)) {
all parts text/plain, text/html and text/calendar are considered displayable, regardless of whether Lightning is installed or not.
With Lightning installed, the text/calendar part is grabbed by Lightning and displayed as invitation. Without Lightning, the text/calendar part is displayed instead of the HTML part. Lowering, as I did yesterday, the priority of the text/calendar part has as a consequence that it was never prioritised and Lightning didn't find it.

With the previous and new planned state
  mime_find_class (ct, sub_hdrs, self->options, true);
  if (clazz && clazz->displayable_inline_p(clazz, sub_hdrs)) {
text/calendar is only considered displayable if Lightning is installed. So with Lightning installed, the text/calendar part passes into the prioritisation, has the same priority as HTML, since it's last, it gets selected and Lightning turns it into an invitation. Without Lightning, text/calendar is not considered displayable so the HTML part is displayed.

So going back to 'true' is the right fix.
Flags: needinfo?(bugzilla)
https://hg.mozilla.org/comm-central/rev/84718664672cc6451b3fe7d136f4788235af8107
Second time lucky ;-)
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Summary: text/calendar MIME part wrongly prioritised over text/html in multipart/alternative → text/calendar MIME part wrongly displayed when Lightning is not installed
Comment on attachment 8833038 [details] [diff] [review]
1334937-call-mime_find_class-true.patch

I'll uplift this once I've convinced myself in a Daily that it's really working this time ;-)
Attachment #8833038 - Flags: feedback?(bugzilla)
Attachment #8833038 - Flags: approval-comm-beta+
Attachment #8833038 - Flags: approval-comm-aurora+
Attachment #8833038 - Flags: approval-comm-release+
I did some tests with with attachment 2017 [details]-02-02 12:12 CET, Rainer Bielefeld and it's source and
Server-Installation of  unofficial (by FRG)  en-US SeaMonkey 2.51a1 (NT 6.1; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0 Build 20170204194856  (Default Classic Theme) on German WIN7 64bit 
and 
TB Daily 54.0a1 (2017-02-04) (64-bit):

- With Lightning active
  -- Lightning view of .ics shown inline
  -- Drag and drop of .ics from Mail Heading attachment pane to Lighning works fine
- With Lightning INactive
  -- HTML (or text) view of .ics shown

With SeaMonkey I additionally gambled around a little with calendar and .ics attachments: Everything seems to work fine.
Status: RESOLVED → VERIFIED
Thanks for verifying, but there is still some confusion.

ICS attachments are *never* shown inline. The message has three MIME parts:
text/plain
text/html
text/calendar

Without Lightning, the HTML part is displayed since the text/calendar part is classified non-displayable. With Lightning, the text/calendar part is displayed, but Lightning grabs that and displays its fancily formatted invitation.

None of this has anything to do with the ICS attachment, see attachment 389696 [details] from bug 505024 where you have those three message parts, but *without* the ICS attachment. The complaint in that bug is that without Lightning, the HTML part is displayed, but the user has no idea that there is an invitation in an non-displayable third MIME part.

BTW, without fixing this bug here, the text/calendar part *was* displayed even without Lightning. In one way that was good, since it helped bug 505024 a bit since users *did* see the invitation. In another way it wasn't good, since users lost the HTML display.

To fix bug 505024 one would have to pretend that the third non-displayable part was indeed an attachment. Since this is hard, that bug from 2009 is still not solved. As you can see, changing one 'false' to 'true' in MIME can have some surprising effects ;-(
1. Is this bug the reason I cannot add calendar invites to my google calendar any more?

2. Isn't Lightning installed and turned on by default in all Thunderbird and Seamonkey installations? Does this mean people are turning off that functionality, and then seeing that once turned off that it doesn't work?
When will You release new official build with this bug fixed on Seamonkey homepage ?(Windows installer counrty localized version)
You need to log in before you can comment on or make changes to this bug.