482131 - UTF-8 encoded invitation subject not displayed correctly

Status: RESOLVED FIXED

(Calendar :: E-mail based Scheduling (iTIP/iMIP), defect)

Description

[Oliver Böttcher]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7
Build Identifier: Version 2.0.0.19 (20081209)

Already stated here: https://bugzilla.mozilla.org/show_bug.cgi?id=476444 (Comment #1)

This problem occurs if umlauts and/or german Eszett (ß) should appear in subject line, resulting in ommitting the subject line completely.

[...]
X-Provags-ID: V01U2FsdGVkX18R9uKeYMYp3GELX2dSwy8jnvVfhanf2WAy4/+
 d8FwrA8wjpUPUjc9wRyMvRTVC3oxCwHlhvwpjeMXqRtcdES+Nt
 CEfYOM4UUqdfPtXng50AnHURF6cVR1tsKUpuaWMwwrYcGqfp5o
 cEQ==
Envelope-To: xxx

=?UTF-8?B?U3ViamVjdDogQW50d29ydCBUZXJtaW5laW5sYWR1bmc6IEZyYW5rLiA=?=
 =?UTF-8?B?R2Vyb2xkLiBCZXJhdGVuIG1pdCBCU0UgSGVycm4gU2Nod2VpdHplciB6dW0gQmU=?=
 =?UTF-8?B?Z2lubiBkZXIgUGxhbnVuZyBTdGHDn2Z1cnQgdW5kIERhcm5zdGVkdA==?=
Content-class: urn:content-classes:calendarmessage
Content-type: text/calendar; method=REPLY; charset=UTF-8
Content-transfer-encoding: 8BIT


BEGIN:VCALENDAR
[...]

I didn't figured out any scheme when this happens. See posting here: http://www.sunbird-kalender.de/forum/viewtopic.php?f=13&t=2358&p=11288#p11288
(German)

Reproducible: Sometimes

Steps to Reproduce:
1. Write Invitation or get invitation with Umlauts or ß
2.
3.
Actual Results:  
Malformed Invitation

Comment 1

[Stefan Sitter [:ssitter]]

Does this issue still exists using Thunderbird 3.0b2 or 3.0b3pre together with recent Lightning 1.0pre nightly test build?

Comment 2

[Oliver Böttcher]

Thanks for your reply and tip.

Invitation reply is still displayed as plain text. I tried the following subject: "Test Test Test Test Test Test äöü ß"

The Invitation itself is displayed as an invitation but without subject.

Thunderbird 3.0b2 and Lightning 1.0pre nightly build downloadad today

Comment 4

[Oliver Böttcher]

On comment #2 I made a mistake. The invitation itself is displayed with a subject while the invitation reply isn't.

Just to be correct.

Comment 7

[Oliver Kurz]

Bug also occurs with Thunderbird 2.0.0.19 and Lightning 0.9.

The problem does not necessarily show up, if umlauts or eszett are used, only if the subject line is long enough and containing umlauts at the end.

For example the subject
"äöüß"
should work, while
"This title will be damaged, as it is long and contains umlauts like äöüß at the end"
does not.

source of working invitation with subject "äöüß": 

------------------------
[...]
To: [...]
Subject: Termineinladung: =?UTF-8?B?w6TDtsO8w58=?=
Content-type: multipart/mixed; boundary="Boundary_(ID_qyG4ZdjoAsiZ+Jo19dCbWQ)"
Message-Id: [...]
Date: Tue, 17 Mar 2009 14:52:57 +0100 (CET)


--Boundary_(ID_qyG4ZdjoAsiZ+Jo19dCbWQ)
Content-type: multipart/alternative;
 boundary="Boundary_(ID_ryU4ZdJoASiZ+Jo21dCbwA)"


--Boundary_(ID_ryU4ZdJoASiZ+Jo21dCbwA)
Content-type: text/plain; charset=UTF-8
Content-transfer-encoding: 8BIT

[...]


--Boundary_(ID_ryU4ZdJoASiZ+Jo21dCbwA)
Content-type: text/calendar; method=REQUEST; charset=UTF-8
Content-transfer-encoding: 8BIT

BEGIN:VCALENDAR
[...]
------------------------

source of broken invitation with subject "This title will be damaged, as it is long and contains umlauts like äöüß at the end" (observe the missing "Subject:" in the second example:

------------------------
[...]
To: [...]
Message-Id: [...]
Date: Tue, 17 Mar 2009 14:53:16 +0100 (CET)

=?UTF-8?B?U3ViamVjdDogVGVybWluZWlubGFkdW5nOiBUaGlzIHRpdGxlIHdpbGw=?=
 =?UTF-8?B?IGJlIGRhbWFnZWQsIGFzIGl0IGlzIGxvbmcgYW5kIGNvbnRhaW5zIHVtbGF1dHM=?=
 =?UTF-8?B?IGxpa2Ugw6TDtsO8w58gYXQgdGhlIGVuZA==?=
Content-type: multipart/mixed; boundary="Boundary_(ID_qyG4ZdjoAsiZ+Jo19dCbWQ)"


--Boundary_(ID_qyG4ZdjoAsiZ+Jo19dCbWQ)
Content-type: multipart/alternative;
 boundary="Boundary_(ID_ryU4ZdJoASiZ+Jo21dCbwA)"


--Boundary_(ID_ryU4ZdJoASiZ+Jo21dCbwA)
Content-type: text/plain; charset=UTF-8
Content-transfer-encoding: 8BIT

[...]

--Boundary_(ID_ryU4ZdJoASiZ+Jo21dCbwA)
Content-type: text/calendar; method=REQUEST; charset=UTF-8
Content-transfer-encoding: 8BIT

BEGIN:VCALENDAR
[...]
------------------------

Comment 8

[Daniel Boelzle [:dbo]]

Hmm, I am trying to reproduce this issue, but it works for me using the following event title: "test test test test test (tester äöü".
(I've been using TB 3.0b3pre, 20090328, same lightning)

Reporter, could you please check a recent TB+lightning nightly?

Comment 9

[David von Oheimb]

I tried several recent lightning nightly builds, e.g. of today, with TB 2.0.0.18, and neither of them works at all! E.g. Go->Calendar is not available, and the lightning preferences tab has some problems, too.

Comment 10

[David von Oheimb]

P.S. I should have mentioned that my platform is Ubuntu Linux. 
With lightning 0.9 (build 2008091718) I can reproduce the bug wrt. UTF-8 chars.

Comment 11

[Stefan Sitter [:ssitter]]

David, ensure that you installed libstdc++5 before installing Lightning 0.9 <http://www.mozilla.org/projects/calendar/lightning/system-requirements.html#linux>. If not: install it and reinstall Lightning.

There exists no Lightning nightly builds for Thunderbird 2. Current nightly builds require Thunderbird 3 Beta 2 or newer. They will not work in Thunderbird 2.

Comment 12

[David von Oheimb]

Thanks Stefan for your hints. libstdc++5 is already installed on my system.

Comment 13

[Oliver Böttcher]

(In reply to comment #8)

> Reporter, could you please check a recent TB+lightning nightly?

Same again. Invitation works, Reply doesn't using the same subject like u've tested.
TB nightly 20090408 + same lightning.

Can anyone confirm that it isn't a problem only with win32 builds? I'll try to figure this out in the meantime.

Comment 14

[Stefan Sitter [:ssitter]]

There is an function encodeMimeHeader() that encodes the subject to UTF-8:
<http://mxr.mozilla.org/comm-central/ident?i=encodeMimeHeader>

In this function is passes the substring after "Subject: " to encodeMimePartIIStr_UTF8() and specifies the parameter fieldnamelen with the length of "Subject: ".

Please correct me if I wrong but I read the IDL like the following: <http://mxr.mozilla.org/comm-central/source/mailnews/mime/public/nsIMimeConverter.idl#82>

    * @param header UTF-8 header to encode.
    * @param fieldnamelen Header field name lengt (ex: "From: " = 6)

So either pass the complete string including "Subject: " or fieldnamelen should be 0 like in <http://mxr.mozilla.org/comm-central/source/mailnews/mime/test/unit/test_EncodeMimePartIIStr_UTF8.js#57>?

Comment 15

[Daniel Boelzle [:dbo]]

Attachment: patch

I am not really sure how the interface is to be used, i.e. whether the field name is to be excluded or not. It seems to work equally good for me (though looking at the sent message source the first encoded line is not fieldNameLen chars shorter than necessary).
However, since I couldn't reproduce the initially reported problem, could Oliver please try this patch whether it helps?

Comment 16

[David von Oheimb]

Here is a simpler, more direct fix: replace
  encodeMimeHeader("Subject: " + aSubject) + "\r\n");
by
  "Subject: " + encodeMimeHeader(aSubject) + "\r\n");

If you like it, please incorporate it.

Comment 17

[Mark Banner (:standard8)]

Comment on attachment 380428 [details] [diff] [review]
patch

Sorry for the delay in reviewing this.

I couldn't reproduce either as it turns out.

However, I've taken a look, and it is a bit unclear what fieldNameLen is doing, but from my tests what you are doing in the path is the right thing afaict whether or not it will fix the bug I'm not sure. I've a feeling it is something to do with the strange characters being near the 72 char limit.

>+                return mimeConverter.encodeMimePartIIStr_UTF8(header, false, "UTF-8", fieldNameLen, 72);

nit: you can replace 72 by nsIMimeConverter::MIME_ENCODED_WORD_SIZE

r=Standard8 with that fixed.

Comment 18

[Daniel Boelzle [:dbo]]

Changed that and pushed to comm-central <http://hg.mozilla.org/comm-central/rev/856938cc9dbe>

-> FIXED

Comment 20

[Andrej]

In file components\calItipEmailTransport.js (vers. 0.9) on line 352 i replaced 
encodeMimeHeader("Subject: " + aSubject) + "\r\n");
with
"Subject: " +encodeMimeHeader( aSubject) + "\r\n");
and looks like all is working fine now for me.
will use this until real fix.

Comment 21

[David von Oheimb]

Attachment: Second time proposing fix

In both versions 1.0b2pre and in 1.1a1pre (which I recently pulled from the repository), this bug still exists.  The fix apparently done on 2009-09-02 08:15:57 PD does *not* solve the problem. Yet moving the "Subject: " out of the encodeMimeHeader function call, as indicated in the patch I provide here again, solves the problem. Please, incorporate this pretty obvious fix.

Comment 22

[Mark Banner (:standard8)]

Comment on attachment 423323 [details] [diff] [review]
Second time proposing fix

Sorry, I'm just not getting the time to review this. Passing to Daniel as he did the original patch.

Comment 23

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 423323 [details] [diff] [review]
Second time proposing fix

r=philipp

Sorry for the long delay

Comment 24

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Pushed to comm-central <http://hg.mozilla.org/comm-central/rev/8002e7e57005>
-> FIXED