.ICS text/calendar (VCal) in multipart/alternative is not visible, unless Lightning is installed

NEW
Unassigned

Status

MailNews Core
MIME
--
major
9 years ago
7 months ago

People

(Reporter: Michael Baffoni, Unassigned, Mentored)

Tracking

({testcase})

unspecified
x86
Windows XP
testcase
Bug Flags:
blocking-thunderbird3 -

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [lang=c++][datalossy][patchlove])

Attachments

(2 attachments, 4 obsolete attachments)

(Reporter)

Description

9 years ago
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.1pre) Gecko/20090718 Shredder/3.0b4pre

I have received a .ICS calendar meeting attachment, but no visual evidence of the invitation exists; no attachment item, nor text display within the message.  I used to receive these as just text within the message.

Here is the MIME type and beginning of attachment (from message source):
Content-class: urn:content-classes:calendarmessage
Content-Type: text/calendar;
	name="meeting.ics";
	method=REQUEST
Content-Transfer-Encoding: quoted-printable

BEGIN:VCALENDAR
METHOD:REQUEST
PRODID:Microsoft CDO for Microsoft Exchange
VERSION:2.0
BEGIN:VEVENT
DTSTAMP:20090717T002452Z
DTSTART:20090723T163000Z
Component: Mail Window Front End → MIME
Product: Thunderbird → MailNews Core
QA Contact: front-end → mime
Version: Trunk → 1.9.1 Branch
Component: MIME → Lightning Only
Product: MailNews Core → Calendar
QA Contact: mime → lightning
Version: 1.9.1 Branch → unspecified
(Reporter)

Comment 1

9 years ago
Is the change to component "Lightning" correct?  I don't have lighting installed, I'm using MeetingMaker to handle my calendar info.  Also, this issue is evident in Tbird safe mode so I'm sure it is not being affected by add-ins.
Hmm maybe I read that wrong then.
Component: Lightning Only → MIME
Product: Calendar → MailNews Core
QA Contact: lightning → mime

Comment 3

9 years ago
Do you have View | Display Attachments inline checked?
Can you attach a sample .eml?
(Reporter)

Comment 4

9 years ago
Created attachment 389696 [details]
.eml of message with issue.

Calendar attachment does not appear even when display attachments inline is turned off.
(Reporter)

Comment 5

9 years ago
BTW, I tried to do some cleanup of the data within the calendar invite; if it doesn't work, let me know of an individual working on this issue and I can send the original to them so I don't post potentially sensitive information to the ticket.  However, I did open the email with the altered data and it still appears the same - no attachment and no text, with or without display attachments turned on using Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.1pre) Gecko/20090720 Shredder/3.0b4pre.

Comment 6

9 years ago
The sample shows just fine for me. (As an invitation text without lightning, an invitation with lightning.)
(Reporter)

Comment 7

9 years ago
Interesting.  I have two different machines (one XP 64bit, one Xp 32bit) and both don't show any attachment, even with attachment not displayed inline (they have completely different profiles, one is not a copy of the other).  

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2pre) Gecko/20090727 Shredder/3.0b4pre.  

I don't know if it matters, but I've never had lightning installed in my profiles, but they both are upgrades from 2.0 profiles.

Comment 8

9 years ago
You tried it as .eml file?
(Reporter)

Comment 9

9 years ago
Correct, I saved the file from the bug and double-clicked on it to replicate your experience.

Comment 10

9 years ago
I just experienced this issue with TB 3.0b2 and then b4 (tried the upgrade to see if it'd help). I'm on Mac OS X 10.4. The profile used to be in use on TB2... so it looks like it is related to that. I don't recall having lightning installed - it certainly is not installed now. Are there any steps I could take to narrow down what the issue is? It seems like something one would want fixed before 3.0 is released --> requesting blocking3.0 for that reason -- attachments are just invisible, and it'd be better if it was at least an attachment or even visible as plaintext inside the email...
Flags: blocking-thunderbird3?
The steps I'd start with would be:

* reduce the testcase to something more obvious, like just the text "I'm the text/plain part" "I'm the text/html part" "I'm the attachment description" to be sure that everyone will know what they are seeing

* make sure it's a regression - when I open the .eml in 2.0, I see the same thing I see in 3.0 - and if it is find the one-day window, which will do more good than anything else possibly could

* make sure whether it's associated with having used a profile in 2.0, or with having updated from 2.0, or both, or neither: does a new profile on 3.0 do the same? does a fresh install of 3.0b4 do the same? does a new profile created by the fresh install do the same? does it actually require having installed Lightning (in 2.0, or in 3.0? a particular version of Lightning?)?

* see whether it's Tb-only, or the same in a comparable build of SeaMonkey
Given reproducibility issues, we can't block on this bug as it stands today.  Please follow the suggestions in comment 11, and, once those issues are sorted out, if it still seems like it should be a blocker, please renominate.  Thanks!
Flags: blocking-thunderbird3? → blocking-thunderbird3-
Keywords: qawanted

Comment 13

9 years ago
Possibly related but perhaps not.

Even in the sample email that was submitted the message can be opened viewed and all the invite information is present.  It's not "lost".  Question I have is more that the invite correctly interpreted and displayed in the message offers no interface (that I see) where you could save or run the invite against an external calendar.

For example here on OSX I expected to see a .ics or calendar attachment that I could save to my desktop.  But that option doesn't exist instead if I wanted to enter this information into iCal I would have to manually create the invite.

This seen via TB3B4, fresh install no addons or anything.  I do see the same behavior in TB2 so (OSX again) I'm wondering if this is by design?  Is this a ''bug'' or an interface design decision?  Should I be able to save/view/ download the Calendar information from a message?

Assumption is that the same logic that interprets 'text and html' and doesn't provide and interface for "saving the Text Version" of a the original email and doesn't display a multipart message with text and html versions as having attachments is what 'hides' you from being able to save a calendar invite?

(mind you if this is seen entirely off topic from this posted bug let me know and I'll take my thoughts elsewhere)

Comment 14

9 years ago
(In reply to comment #13)
> (mind you if this is seen entirely off topic from this posted bug let me know
> and I'll take my thoughts elsewhere)

I believe your problem is described in bug 242937 - this bug is about not seeing anything (so not even seeing the ICS data in the email text itself).
(Reporter)

Comment 15

9 years ago
Created attachment 405560 [details]
Reduced test case of email containing VCAL

Attaching reduced testcase.

Per comment 11, confirmed that this behavior is not a regression, the same behavior exists in Seamonkey, Tbird 2, and Tbird3; it exists in clean profiles or upgraded profiles.  

Lightning is not installed, nor has it been installed in prior versions either.

I played around with the .eml quite a bit, and essentially the VCAL doesn't show as text as long as the attachment type is text/calendar.  Messing with that at least allows the text to show, but clobbers the actual message text.
(Reporter)

Comment 16

9 years ago
Removing regression.  Can we mark this dataloss since there is no practical way in the standard UI interface to retrieve the data in the message short of saving the email as a file and cutting out the VCAL information manually (or is that considered a reasonable workaround)?  Half the time, I don't know the VCAL is there to go retrieving it from the message source....

I'm surprised that if Tbird doesn't understand the application type it doesn't at least just display it as an (unknown) attachment.  It would be reasonable to at least just save the meeting.ics as a standard text file....
Keywords: regression
Keywords: testcase

Comment 17

8 years ago
First, I'm glad I'm not the only one seeing this bug.

Without looking at the code (I'll be doing that shortly), my hypothesis is that Thunderbird only displays one text/* multipart MIME part, and Outlook invitations are sent with text/plain, text/html and text/calendar.  The code needs to be changed to "teach" Outlook that text/calendar should be presented as a file attachment, not "ignored" when displaying the text/html or text/plain.

Now, to see if I can come up with a fix ...

Comment 18

8 years ago
Just thinking out loud ... it looks like src/mailnews/mime/src/mimemalt.cpp:MimeMultipartAlternative_display_part_p might be a starting point for the fix for this bug.

As an aside, won't line 255 return in a memory leak as *ct isn't PR_FREEIF'ed?

Updated

7 years ago
Duplicate of this bug: 646779

Comment 20

7 years ago
I just wanted to mention that I'm having the same issue with Thunderbird 9.1 on Mac OS X 10.7.

Comment 21

7 years ago
Correction Thunderbird 9.0.1.
All of attached mails is multipart/alternative and text/plain, text/html, text/calender part are contained in it. Because of multipart/alternative, any part in it is ALTERNATIVE each other. Where can we see ATTACHMENT in multipart/alternative?

To see this special part in malformed mail as if attachment, two ways are currently available.
(i)  Lightning extension shows this text/calender as if attachment.
     (broken by Tb 8, but will be fixed by Tb 10. see bug 713380)
(ii) View/Message Body As/All Body Parts,
     with mailnews.display.show_all_body_parts_menu = true
     (implemented by Bug 602718. available since Tb 8)
     In order to see any part, any multipart/xxx is treated as multipart/mixed,
     and any part is shown as if attachment at attachment pane.    

Similar malformation is seen in multipart/related which was perhaps originally born by MS. Bug 674473 is for such malformed multipart/related case.
By that bug, "non-referred part and non-displayable part in malformed multipart/related" will be shown as if attachment.

Similar enhancement will be needed in case of wrong use of multipart/alternative by some mailers and some mail systems of some companies who don't respect mail RFC.
  - Limit application of "alternative" to mime part which Tb knows only.
    Apply to text/plain, text/html, and some predefined text/xxx only.
  - Show any other part as if attachment at attachment pane.
I believe this is natural enhancement if mail like next.
  multipart/alternative
    text/plain, text/html, application/pdf, audio/wav, video/x-mpeg
  In this case, any part can be actual/valid ALTERNATIVE, because PDF version
  of mail, voice version of mail, video version of mail is possible.
  Even if Tb can render text/plain or text/html only, I believe audio/wav etc.
  is better shown in attachment pane for user's convenience.
And, "doesn't choose single part only" and "doesn't ignore other parts than part of mail sender's highest preference order" is never RFC violation by Tb. 

This enhancement can do nothing for "reversed order in multipart/alternative" case. I think reversed order case is far rare than "text/calender in multipart/alternative" case and is relieved by View/Message Body As/All Body Parts. However, enhancement like next, partially ignore "mail sender's preference order", is better for multipart/alternative.
  - When View/Message Body As/Original/Simple HTML, search text/html part only.
    If only text/plain is contained or if text/html part is null/blank,
    use text/plain part.
  - When View/Message Body As/Plain Text, search text/plain part only.
    If only text/html is contained, or if text/plain part is null/blank,
    use text/html part and convert it to Text.
Severity: normal → enhancement
Summary: .ICS text/calendar (VCal) attachment is not visible → .ICS text/calendar (VCal) in multipart/alternative is not visible, unless Lightning is installed

Comment 23

6 years ago
Taking. I have a fix in the works.
Assignee: nobody → squibblyflabbetydoo
Status: NEW → ASSIGNED

Comment 24

6 years ago
Created attachment 613852 [details] [diff] [review]
WIP patch

This should fix the issue. Bienvenu, what do you think of this solution? Among other things, it special-cases text/calendar parts and provides a name for them so that they don't get skipped. As an alternative, we could add a displayableInline property to nsMsgAttachmentData and show all non-displayable-inline parts in the attachment list.
Attachment #613852 - Flags: feedback?(dbienvenu)

Comment 25

6 years ago
(In reply to Jim Porter (:squib) from comment #24)
> As an alternative, we could add a displayableInline property to nsMsgAttachmentData
> and show all non-displayable-inline parts in the attachment list.

Hmm, maybe we should do this, since apparently some MUAs actually send attachments as multipart/alternative subparts: bug 636253.

Updated

6 years ago
Duplicate of this bug: 259480

Updated

6 years ago
Duplicate of this bug: 301441

Comment 28

6 years ago
(In reply to Jim Porter (:squib) from comment #25)
> (In reply to Jim Porter (:squib) from comment #24)
> > As an alternative, we could add a displayableInline property to nsMsgAttachmentData
> > and show all non-displayable-inline parts in the attachment list.
> 
> Hmm, maybe we should do this, since apparently some MUAs actually send
> attachments as multipart/alternative subparts: bug 636253.

This actually works with the current patch.

One limitation, however, is that text/whatever parts will get shown inline after the "proper" body (so you'd have, e.g. text/html and text/whatever shown inline). This is because we try to find a handler for the *exact* MIME type when determining if we're at the last inlinable part, but then when we try to show it, we fudge things, and treat unknown text/* parts as text/plain. Maybe we should add an option to mime_create that forwards exact_match_p onto mime_find_class? This would also let me remove the "text/calendar" handling in mime_find_class. Ideas?

Comment 29

6 years ago
Comment on attachment 613852 [details] [diff] [review]
WIP patch

I haven't tried the patch, but it seems like the pragmatic thing to do.
Attachment #613852 - Flags: feedback?(dbienvenu) → feedback+

Comment 30

6 years ago
Hey Jim,

Thanks for taking this on.  I was the opener of Bug 301441 (I think), and while I don't have time to do a full build, if you have an image for MacOSX built, I'll be happy to test.  Just drop me a pointer (mozilla-bugs "the round A symbol" ofcourseimright.com).

Eliot

Comment 31

6 years ago
I presume Lightning still picks things up correctly with this patch applied?

Comment 32

6 years ago
(In reply to Ian Neal from comment #31)
> I presume Lightning still picks things up correctly with this patch applied?

Well, that depends a lot on how strict you're being with "correctly". Lightning will almost certainly require a patch to work with this code, specifically to hide .ics attachments when it's handling them through a notification bar.
(Reporter)

Comment 33

6 years ago
I hope it doesn't.  I use a different meeting calendar, but I really appreciate the HTML markup view within Tbird that lightning does on the .ics, and still allow me to associate a different helper app to .ics instead of lighting "gobbling" it up.
(In reply to Jim Porter (:squib) from comment #32)
> (In reply to Ian Neal from comment #31)
> > I presume Lightning still picks things up correctly with this patch applied?
> 
> Well, that depends a lot on how strict you're being with "correctly".
> Lightning will almost certainly require a patch to work with this code,
> specifically to hide .ics attachments when it's handling them through a
> notification bar.

Could you elaborate wat kind of code changes are needed? I don't plan to remove the HTML representation of the ics attachment, I think this would go in the wrong direction. If the email message has its own html representation, then I could imagine providing a way to show alternative representations, but I would prefer showing the lightning HTML template first.

Comment 35

6 years ago
This issue has really annoyed me for a while now.  I used to use mutt to handle the multipart/alternative calendar attachment until I came across this thunderbird addon:

https://addons.mozilla.org/en-US/thunderbird/addon/show-all-body-parts

It does the trick but requires an extra mouse/menu/click when viewing a meeting invite email.

FWIW, I really like how Claws Mail handles these multipart/alternative parts. It has a couple action icons on the right of the message window (one icon per multipart and others for real attachments).

Updated

6 years ago
Duplicate of this bug: 779909

Comment 37

6 years ago
What is the current status on this problem?

With SeaMonkey 2.10 a multipart/alternative message with text, html and test/calendar parts (as produced by Outlook and Google Calendar) shows as the text part in the mail pane and a "meeting.ics" file in the attachment pane which cannot be opened or saved.   When the "View -> Message Body as -> All body parts" trick is used, all parts are shown in the attachment pane and the meeting.ics can be opened or saved.
While cumbersome, it is usable as a workaround to access the meeting.ics (which we can import into the calendar app we use - not lightning)

However, in SeaMonkey 2.11 and 2.12 the attachment pane no longer shows up when a message like this is opened.   I'm sure I have read a bug (for Thunderbird) that discussed this change, but now I can no longer find it.

Unfortunately this means that our users no longer see the trigger to try the "all body parts" workaround when they receive a message (the nonfunctional meeting.ics attachment).  So we have been unable to upgrade past 2.10.
Today I installed 2.13beta and unfortunately it has not been fixed.

We need either the old behaviour which we can use with the workaround, or better the solution depicted above (which I understand will mean that we see the text part in the message pane and the text/calendar part as an attachment in the attachement pane where it can be saved without using "View -> Message Body as -> All body parts")

Any idea in what version we can expect this fix?  (e.g. the one in comment #25)

Comment 38

6 years ago
(In reply to Rob Janssen from comment #37)
> What is the current status on this problem?

My patch fails for all but the simplest test cases, and I haven't had time to fix it up. I'm unassigning myself from this for now, since I'm unlikely to work on any MIME parsing bugs until the new Javascript MIME parser is in place and turned on. If anyone else would like to take this on, I'll be happy to help provide guidance, though.
Assignee: squibblyflabbetydoo → nobody
Status: ASSIGNED → NEW

Updated

6 years ago
Whiteboard: [lang=c++][mentor=squib]

Updated

6 years ago
Duplicate of this bug: 792434
Comment hidden (me-too)

Comment 41

6 years ago
I'd just like to put in a vote for this bug: I have messages that come from colleagues using WebEx and they are all multipart/alternative with the last alternative being text/calendar. Stupid. The icing on the cake is that the last part is base64 encoded, so even looking at the message source (which I'm totally willing to do) won't reveal the source of the message invite. Did I mention that WebEx doesn't put the date and time of the meeting into the text portion of the message? So I get these meeting invites all the time that say "come to my meeting" and I have absolutely no idea when the meeting is going to be. Stupid. Not tb's fault - to be sure - but insanely frustrating nonetheless.

I read the long and nasty saga of bug 674473 and I certainly wouldn't want to repeat that here. Jim, thanks for taking on the responsibility for both bug 674473 and for taking a swing at this one.

Jim: if your fix only works 5% of the time, would you feel comfortable releasing it so at least 5% of the time I can read these messages? Or is the other 95% such a horrible failure that it's worse than the current behavior?

I'm happy to provide raw material for test case email messages if that would be helpful.

Thanks to everyone on the tb team for years of great work: I've been using tb since before it had a name and have always been very happy with it. Keep up the great work!

Comment 42

6 years ago
(In reply to Christopher Schultz from comment #41)
> Jim: if your fix only works 5% of the time, would you feel comfortable
> releasing it so at least 5% of the time I can read these messages? Or is the
> other 95% such a horrible failure that it's worse than the current behavior?

The problem is that it breaks most messages that use multipart/alternative and emits a bunch of bogus "attachments".

Comment 43

6 years ago
(In reply to Jim Porter (:squib) from comment #42)
> The problem is that it breaks most messages that use multipart/alternative
> and emits a bunch of bogus "attachments".

Fantastic. Thanks for the attempt.

I went ahead and enabled mailnews.display.show_all_body_parts_menu=true so now at least i can get tb to show me the data for those messages (decoded from base64, which is an enormous help as my base64 is a little rusty). I just need to be smart enough now to turn it on when I see a meeting invite with no meeting data evident in it.
Comment hidden (me-too)

Updated

5 years ago
See Also: → bug 783603

Updated

5 years ago
Duplicate of this bug: 879288
Comment hidden (advocacy)

Comment 47

5 years ago
Following Wayne,  i posted an example message over at https://bugzilla.mozilla.org/show_bug.cgi?id=783603

Comment 48

4 years ago
I just got bitten by this -- I got an email invite that I didn't even realize was an invite, and therefore I didn't respond to it in a timely fashion.

One can legitimately argue whether Outlook should be mixing up text/plain and text/html parts with a text/calendar part that has info in it that isn't displayed in the other two, but it is what it is, and Thunderbird really needs to cope with it better than it is.
Comment hidden (me-too)
(Assignee)

Updated

4 years ago
Mentor: squibblyflabbetydoo
Whiteboard: [lang=c++][mentor=squib] → [lang=c++]
Comment hidden (me-too)
Comment hidden (me-too)

Comment 52

4 years ago
i pinpointed the bug to a single line

Content-Type: multipart/mixed;
vs.
Content-Type: multipart/alternative;

the email is sent from a python code, and the attachment is visible both cases on gmail webclient.


Content-Type: multipart/alternative;      --> not showing attachment
vs.
Content-Type: multipart/mixed;         --> does show attachment

example eml file:
not showing attachment:   http://pastebin.com/33cvj9Pi
showing attachment:    http://pastebin.com/yjbgVK6w
Comment hidden (me-too)

Updated

4 years ago
Keywords: qawanted

Updated

4 years ago
Duplicate of this bug: 1088094
Comment hidden (me-too)

Updated

4 years ago
Severity: enhancement → major
Comment hidden (me-too)
Comment hidden (advocacy)

Comment 58

4 years ago
Ideally we need an attachment so we can open it with the calendar. A renderer would be a nice addition but you still need to be able to see the ics file. It's worth pointing out that the Lightning code already has a renderer.

Comment 59

4 years ago
Sorry, when I say "the calendar" I mean "the calendar of our choice". That may or may not be lightning.
Comment hidden (me-too)
Comment hidden (advocacy)
Comment hidden (me-too)
Comment hidden (me-too)

Comment 64

4 years ago
Can the renderer simply be copied from Lightning?
Comment hidden (me-too)
Comment hidden (me-too)
Comment hidden (me-too)

Comment 68

3 years ago
I've just marked a *lot* of comments as me-too and/or advocacy. Please don't comment with "I'm seeing this too" or "when are you fixing it" or "This is a terrible bug and it is shameful it's not fixed yet".

We know all of that. Telling us again won't help - it just makes the bug far harder to scan and understand if people *are* actually interested in fixing it.



Now... Andrew/Mike, is the JS MIME parser thing something that ever happened? I can't find it from a quick search in bugzilla and MXR, but maybe I missed it (Andrew, poking you because of bug 447842 which looked related).

Irrespective of that, AIUI the easiest way to avoid the lack of clarity / visibility of the ics attachment is to ignore and/or replace-with-multipart/mixed any use of multipart/alternative if and only if there is a part that with mimetype text/calendar . Is that right, and if not, can you suggest an alternative (minimal) way to address this

If that sounds acceptable, where does that need to be addressed? Still mailnews/mime/src/mimemalt.cpp and friends, or elsewhere?
Flags: needinfo?(mconley)
Flags: needinfo?(bugmail)
Yes, I believe the JSMIME code is now in the tree. jcranmer, can you confirm?
Flags: needinfo?(mconley)
Flags: needinfo?(bugmail)
Flags: needinfo?(Pidgeot18)
(In reply to :Gijs Kruitbosch from comment #68)
> Now... Andrew/Mike, is the JS MIME parser thing something that ever
> happened? I can't find it from a quick search in bugzilla and MXR, but maybe
> I missed it (Andrew, poking you because of bug 447842 which looked related).

JSMime has, in effect partially landed. It's already in place for most header parsing and emission (as of TB 32 or so).  It's *not* in place for parsing the body and message display--libmime is *very* featureful, so there is a very good deal of work that needs to get done to get it even in a state where it could be enabled by preference.

> Irrespective of that, AIUI the easiest way to avoid the lack of clarity /
> visibility of the ics attachment is to ignore and/or
> replace-with-multipart/mixed any use of multipart/alternative if and only if
> there is a part that with mimetype text/calendar . Is that right, and if
> not, can you suggest an alternative (minimal) way to address this

If you have Lightning installed, you probably want to treat text/calendar properly in a multipart/alternative. Otherwise, you'd want to treat it as a basic attachment.

> If that sounds acceptable, where does that need to be addressed? Still
> mailnews/mime/src/mimemalt.cpp and friends, or elsewhere?

Quite frankly, the multipart/alternative handling of libmime is a giant black hole, and the attachment handling is near as bad.
Flags: needinfo?(Pidgeot18)
I think Joshua addressed the JSMime stuff.  The API added for bug 447842 is something that could be used to work-around the problem (it gets to hear about all the MIME parts, not just what libmime emits for specific display), but it would require a second streaming of the message and hooking the result up to the message reader.

Pre-JSMime-conversion, the simplest 95% fix is probably to always treat non-displayable multipart/alternative leaf node subparts as attachments rather than trying to re-interpret multipart/alternative as multipart/mixed.  (Basically, if MimeMultipartAlternative_display_part_p returns false and it's not multipart/*, emit that it's an attachment).  It seems like this would avoid screwing up lightning, since lightning would indicate that the part is displayable.

Patch-size-wise, the fix is probably fairly small, but it's still probably a bit of effort to come up to speed on the libmime bits.  (So, not impossible, but a bit of a slog.  Although one might get really lucky with just cargo culting some of the attachment emitting bits.)

(I do need to very explicitly disclaim that I'm not able to help with writing the patch or reviewing because I'm swamped with the Gaia email app and think it's most productive for me to work on that instead.)

Comment 72

3 years ago
Just confirming that it still exists and that its a real pain.  Enough to abandon thunderbird (I have been using it what feels like over 10 years).

I have tried all the related add-ons on the site (msg/eml/ics/winmail.dat) and others besides.  I have lightning installed and I have tried external applications.  Nothing works.  The problem is that most of the time I dont even know that someone has sent me an invite because it is not obvious that something is missing.

I do not want to import it into lightning, I just need the info visiible inside the email.  How hard can it be.,  Even Gamil works fine.  I can send sample emials if required

Comment 73

3 years ago
I'm confirming what Rohit says that not only it exists, but the Add-Ons (like "Show All Body Parts") don't solve the problem.  Mac's email, and iPhone's have no problem handling these attachments IF you know they are there - usually the symptom is a blank message from someone, but i missed a meeting earlier this week because the message had a considerable amount of text, so I didn't notice something was strange. 

This is a Data-Loss-Bug and should be treated with the urgency that implies, but its been there five years now.

Comment 74

3 years ago
Implementation suggestions:
Step 1: Let "text/calendar" be decoded as text/plain. That should be a 1-line change in libmime.
Step 2: Make a display feature similar to vcard. vcard is a libmime plugin (!). See mailnews/mime/cthandlers/vcard/mimevcrd.cpp/h for the code. Make sure that it still also shows as attachment that I can save as file.

Comment 75

3 years ago
mimei.cpp line 479 has specific code for text/calendar. Apparently it's that what messes us up here. (I was already wondering, because unknown attachment types should normally show as file attachment. text/calendar leaves no trace whatsoever, which is the bug here. Apparently, it's caused by this code.)

Comment 76

3 years ago
-> Neil. See the last 2 comments.
Assignee: nobody → neil

Comment 77

3 years ago
Created attachment 8600315 [details] [diff] [review]
Hack to display it as plain text

The last WIP patch doesn't work for me, I still don't see the ICS attachment.
Also, reading the code, it also seems to do something that is wrong or at least debatable according to RFC 2046 5.1.4. <https://www.ietf.org/rfc/rfc2046.txt>:
"What is most critical, however, is that the user not automatically be shown multiple versions of the same data."

At least not normally, text/calendar is an exception due to a bug in Exchange. Exchange uses multipart/alternative, and does not put all information in the plaintext and HTML parts, it only puts the description there and not the date/time and place, so the plaintext part is useless.
As WADA already diagnosed in comment 22, this is a clear violation of Internet standards, which explicitly state that the "content of the various parts are interchangeable."

So, here is a HACK that implements step 1 above: display text/calendar as plain text. It's really ugly, because it's a data format for machines in Field: value form, not for humans. So, I don't consider it very useful for end users, but at least much better than status quo, because it does contain and display all data.

Comment 78

3 years ago
Neil, please take a look whether you can migrate the Lightning text/calendar libmime plugin from Lightning to mailnews, so that text/calendar will be displayed even when Lightning is not enabled (addon disabled).

Updated

3 years ago
Attachment #613852 - Flags: feedback-

Comment 79

3 years ago
Comment on attachment 8600315 [details] [diff] [review]
Hack to display it as plain text

This is a band-aid, but still better than before. Unpretty, but at least no information loss.
Tested without Lightning. Still needs testing with Lightning enabled. It should not change anything in this case, because plugins have priority over built-in types, so the Lightning plugin should still be used even with this patch.
Attachment #8600315 - Flags: review?(neil)
Created attachment 8602813 [details] [diff] [review]
Alternative approach

When I was looking into this last week (i.e. before the last few comments above) I noticed that we simply display the last alternative that we can, and ignore subsequent alternatives, even though strictly speaking they should be a better format. I therefore came up with this approach which shows those alternatives as attachments (and they show up in the attachment pane and also in the message list as having attachments). In this case this then allows you to launch the invitation using an external calendaring application.
So, there are at least three cases:
1. Invitation sent as a text/calendar message
   In this case you see the invitation as plain text.
2. Invitation sent as multipart/mixed
   In this case you see the invitation as an attachment.
3. Invitation sent as multipart/alternate
   In this case the invitation is ignored.

Also, try toggling mailnews.display.show_all_body_parts_menu in config editor and then using View - Message Body As - All Body Parts. (Caution! This will start marking all your multipart messages as having attachments. The marking will go away when you switch display option and reload the message.)
Created attachment 8603340 [details] [diff] [review]
Yet another idea

For completeness, this is a variant of attachment 8602813 [details] [diff] [review] whereby the additional multiparts are only attached if they have a file name (which applies in the attached vcal example).
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 84

3 years ago
Far too much time has passed with this and other calendar bugs. It's simply unusable in the current release state. I'm afraid we have now decided to drop Thunderbird and move our department onto eMClient on Windows and Apple Mail on Mac. So long and thanks for all the delays.

Comment 85

3 years ago
Agreed Ian - six years, and a major data loss bug - still marked "New", it just shows that noone really cares about TB. 

Just missed another meeting this week because of a hidden calendar invite that I didn't even know was there. 

As calendar invites become really common, and TB is the only significant mail client that just ignores them I can see how it becomes untenable to keep using them, I'm about to jump ship myself.

Comment 86

3 years ago
While I share frustration over the length of time this bug is open, as I myself am not in a position to fix it, I have refrained from complaining.  And so to those who do, I would ask if they would volunteer some time to come up with a fix.  Several attempts have been made, and I would imagine since none of them were taken, that the problem is tricky and involved.  Anyway, thanks to those who ahve tried.
Neil, could you explain why this important bug is currently stalled?
What is needed/missing to get it moving again?

This has been elevated to "major" by rkent in 2014, 7 duplicates, 24 votes, and obviously biting people badly when they miss their invitations because Thunderbird doesn't show them (regardless of the fact the respective messages might be malformed by other widespread mailers, we should safeguard against that).
Flags: needinfo?(neil)

Comment 88

3 years ago
(In reply to Eliot Lear from comment #86)
> While I share frustration over the length of time this bug is open, as I
> myself am not in a position to fix it, I have refrained from complaining. 
> And so to those who do, I would ask if they would volunteer some time to
> come up with a fix.

Now would be a good time to remind folks: I'm happy to mentor anyone who'd like to take this bug on. Unfortunately, I don't have time to work on it myself, nor do I expect I will in the near future, but I should have time to at least point people in the right direction.

Comment 89

3 years ago
I am also seeing this problem on OSX Yosemite.

Comment 90

3 years ago
I can confirm that adding Lightning lets you view the invite in TB. Lightning itself unfortunately has plenty of bugs, so you can't download the invite, export it to iCal or do any of the other things which you would expect to do with a calender invite.  (See Bug #1217237 )

Comment 91

2 years ago
V38.2 and still a problem.  I have users sending emails back to clients saying "your email is empty. Can you tell me the meeting date please?" and effectively making us look stupid because we couldnt see their .ICS invitation that they had actually sent.  And adding Lightning just to see these isnt the correct solution (its an unnecessary workround, and an unrequired addon for our business).

Comment 92

2 years ago
After nearly 7 years of eating ICS attachments, I've given up on TB and have moved to Postbox. Losing ICS attachments was one of the main reasons (lightning is basically unusable).

Comment 93

2 years ago
Note that in Lightning bug #1243407 the Lightning team confirm that there is no way to export single events in Lightning so I'm not sure why the Thunderbird team suggest that Lightning is the reason not to solve this 7 year old data-losing bug.

Comment 94

2 years ago
Neil won't be replying in the next several months. So to move this foward we have proposals in Neil's comment 80, 81 and comment 82.  Ben, what is your opinion of Neil's proposals?
Flags: needinfo?(neil) → needinfo?(ben.bucksch)

Comment 95

2 years ago
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #94)
> Neil won't be replying in the next several months. So to move this foward we
> have proposals in Neil's comment 80, 81 and comment 82.  Ben, what is your
> opinion of Neil's proposals?

I know it wasn't addressed to me, but I think it sounds like a good proposal.

Comment 96

2 years ago
I would prefer a variation on comment 77, but instead of displaying as plain text, display the keyword-value pairs as an HTML table.

I found using mailnews.display.show_all_body_parts_menu (comment 81) to be totally unacceptable. I started using lightning as the workaround just to get the displays right.

But the solution expressed in comment 80 - comment 82 is certainly preferable to nothing being done.

Updated

2 years ago
Whiteboard: [lang=c++] → [lang=c++][datalossy][patchlove]

Updated

2 years ago
Duplicate of this bug: 760412

Comment 98

2 years ago
ITs all very well closing other bugs as duplicates of this bug, but if this bug is being progressed then they all might as well be closed....including this one.  Otherwise effectively its saying "Save wasting your time.  Yeah, we know about this problem.  We are not going to deal with it and dont need you telling us about it loads of times."

Comment 80 and 81 dated 7th May 2015 (1 year ago) and no progress since.  And the whole bug report approaching its SEVENTH birthday!

Yes, Im frustrated.  Sorry.

Comment 99

2 years ago
...if this bug ISNT being progressed*

Comment 100

2 years ago
In case somebody needs a simple workaround just to deal with the very occasional Outlook user, here's what I did:

1. View message source; scroll down to "Content-Type: text/calendar" section
2. Copy the block of base64 text and decode it. (On Linux, I use base64 -d; there are also online decoders like https://www.base64decode.org/ ).
3. Save the resulting text (should start with "BEGIN:VCALENDAR" and end with "END:VCALENDAR") into a file. E.g., event.vcs.
4. Open up Google Calendar. In the left margin, click on the "Other calendars" menu and pick "Import calendar."
5. Select your file and click "Import". It should say, "Processed one event. Successfully imported one event."

After that, you should have the event on your calendar. This definitely isn't optimal, but it seems better than having to email back a client and ask them to send you details in plain text.

Comment 101

2 years ago
"After that, you should have the event on your calendar. This definitely isn't optimal, but it seems better than having to email back a client and ask them to send you details in plain text."

Is it??!

Comment 102

2 years ago
As I thought was obvious, I mean that it seems better to me. If it isn't better for you, I will gladly offer you a full refund on your purchase price.

Comment 103

2 years ago
Im concerned that offers of 'workrounds' (however short or long-winded) being posted on this bug report page only serves to further delay any potential real fix to the problem that MIGHT just be in the offing.  (Sort of a "oh look, it doesnt stop the show because there is something they can do without it so we will lower the priority even more" sort of thing). I know it shouldnt, but given we are approaching 7 years old with no solution on the horizon, it is easy to understand why such a concern should be had.  (Even proposals - comment 79 and onwards - just have "yes, I agree"'s yet no one willing to commit or progress).

(BTW, Im afraid your offer of the *simple* workround or even the virtual refund doesnt apply to me - Ive had to find a different solution due to lack of this functionality.  I would have really struggled to try to explain to 'Mary, in Accounts' and all the other users about "content-type's" and "base64 decoding") :-)

Comment 104

2 years ago
I'm exactly none of the developers, but I find your sense of entitlement offputting here. If you'd like things to worry about, you might try worrying that being demanding toward people who you are not paying and who are often volunteers might delay them working on this. In favor of, say, something where they get to deal with pleasant people, or just something they find personally rewarding.

On my part, all I did was try to help, and I've gotten two grumbly emails from some internet rando. So you've certainly convinced me that trying to help here is a bad idea.
As a developer who's worked on this area of the code (but unfortunately has no time to delve back into libmime at the moment), I can assure you that workarounds posted here have no bearing on the schedule for fixing this.

I may look into this when we're further along the process of migrating from libmime to jsmime, but probably not before then. Any significant improvements to libmime would be lost once we switch to jsmime for everything, and I'd prefer to spend my limited time on improvements that will last.

Comment 106

2 years ago
Hi Jim Porter.

I have recently done some other bug-fixes to libmime and the handling of multipart/alternative massages, and I was pondering if I should try to fix this bug.

You said in an earlier comment that you could be willing to mentor and point in the right direction.
So, please, any pointers would be welcome. Where should I start to look in order to fix this bug?

Terje B.
Flags: needinfo?(squibblyflabbetydoo)
If I were you, I'd take a look at the existing patches, (especially the last two by Neil) and see how they work. They might already do what we need, but some tests would be nice.
Flags: needinfo?(squibblyflabbetydoo)

Updated

2 years ago
Duplicate of this bug: 1295965

Updated

2 years ago
Duplicate of this bug: 687063

Updated

2 years ago
Duplicate of this bug: 582251

Updated

2 years ago
Duplicate of this bug: 342443

Updated

2 years ago
Duplicate of this bug: 1003241

Updated

2 years ago
Duplicate of this bug: 242937

Comment 114

2 years ago
Terje, are you still interested, we could fix this ancient old bug. Without looking I'd say that Neil's patches don't apply any more since you've made changes in mimemalt.cpp.

BTW, I got here via Neil's review queue.
Flags: needinfo?(ben.bucksch) → needinfo?(bugzilla)

Comment 115

2 years ago
Comment on attachment 8600315 [details] [diff] [review]
Hack to display it as plain text

I don't think you're going to get this review :-(
Attachment #8600315 - Flags: review?(neil)

Comment 116

2 years ago
In *HTML* view in a TB 53 Daily attachment 405560 [details] looks very nice, the last part is displayed: Content-Type: text/calendar. For attachment 389696 [details] also the last part Content-Type: text/calendar is displayed.

In both case one clearly sees that there is an attachment. This was most likely already fixed by Terje in bug 574989. I remember he added text/calendar here:
https://hg.mozilla.org/comm-central/rev/ab68d375e7e4#l4.140

In plaintext view there is no attachment shown.

Terje, can we improve this further?
Assignee: neil → nobody

Updated

2 years ago
Attachment #8603340 - Attachment is obsolete: true

Updated

2 years ago
Attachment #8602813 - Attachment is obsolete: true

Updated

2 years ago
Attachment #8600315 - Attachment is obsolete: true

Updated

2 years ago
Attachment #613852 - Attachment is obsolete: true

Comment 117

2 years ago
Hi Jorg.

I am sorry, but I am busy with work related stuff now.
It might be some time before I will look at this again.

I highly recommend that some one else looks at it, because I might never get around to it.

Kind regards

Terje B.
Flags: needinfo?(bugzilla)

Comment 118

a year ago
I'm coming back to this bug after fixing bug 1334937.

Looking at attachment 389696 [details] and attachment 405560 [details].
With Lightning installed, both show the invitation. 
With Lightning disabled, both show the HTML part.

Looking at the messages they have three parts:
text/plain
text/html
text/calendar.

With Lightning not installed, the text/calendar part is not displayable, see bug 1334937 comment #39.

So I guess you can't have it both ways: Bug 1334937 wanted to see the HTML part and this bug here wants so see an attachment where there isn't one.

The test message from bug 1334937, attachment 8832838 [details], is different. It has those three parts *and* it has a .ics attachment, and that shows, with or without Lightning.

Comment 119

a year ago
The previous comments show that Lightning is distrusted.

Comment 120

a year ago
There are a few issues here.  The first is that when using mulitpart/alternative, one uses the preference order of the sender in the order that the receiver understands.  Thus, in your example, you act on text/calendar if you understand text/calendar, and otherwise move up the list to text/html.

The challenge here is that I don't believe the standard matches the user's expectations, particularly when lightning isn't installed.  In particular, the user may wish to view the appointment *and* "double click" on it to hand it off to a handler for Outlook or iCal.

Comment 121

a year ago
Right.

Bug 1334937 comment #46 explains that when you have three parts
text/plain
text/html
text/calendar
and *no* ICS attachment, the difficulty is to display the third usually hidden part additionally or as attachment. RFC 1521 says:
  Receiving user agents should pick and display the last format
  they are capable of displaying.  In the case where one of the
  alternatives is itself of type "multipart" and contains unrecognized
  sub-parts, the user agent may choose either to show that alternative,
  an earlier alternative, or both.

Comment 122

a year ago
This seems really a bug in Microsoft Exchange, not a bug in Thunderbird. However, since so many people send out event invitations via Exchange, it seems other MUAs have added special-case algorithms to deliberately violate the MIME specification if the sender was MS-Exchange and show multiple multipart/alternative parts simultaneously if one of them is an ICS calendar and the others are text/plain or text/html.

The problem is that Exchange sends out calendar invitations as

Example A:

- multipart/alternative
  * text/plain
  * text/html
  * text/calendar

even in cases where the text/plain and text/html parts are empty, or otherwise are no legitimate representation of the same information as the calendar file (e.g., just a human written cover message).

In an ideal world, where Microsoft respected Internet standards, Exchange would instead have sent out

Example B:

- multipart/mixed
  * multipart/alternative
    o text/plain
    o text/html
  * text/calendar

This way, the client would always display either the plain-text or HTML cover letter (which presumably are just alternative renderings of exactly the same information), plus show below that the ICS file as a proper attachment.

But since Exchange sends out so many inappropriate Example A messages, it may be worthwhile to add circumvention code. Thunderbird could detect the structure of Example A and convert it into the structure of Example B before displaying it. That transform can be made conditional to the presence of header fields (such as
regular expression /^x-ms-exchange-/i), such no messages sent my non-Microsoft products are affected.

I can forward example messages.

Comment 123

a year ago
Regarding Comment 122, one of the use cases here is really quite simple: we want to be able to link a MIME handler to a text/calenar message.  That is what Bug 301441 was requesting lo those many year ago (marked a duplicate of this one).  And so it's not just a matter of whatever exchange bug might exist.  At the time, and I don't know if it has changed, it was not possible to register a MIME handler in the UI.

Comment 124

10 months ago
Could it be that this bug is also related o that https://bugzilla.mozilla.org/show_bug.cgi?id=1396754 is a duplicate of this but with groupwise instead of exchange?.
You need to log in before you can comment on or make changes to this bug.