167800 - iCal calendars with international characters imported or subscribed incorrectly

Status: RESOLVED WORKSFORME

(Calendar :: General, defect)

Description

[ajbu]

Steps to reproduce:
1) Subcribe or import webcal://ical.mac.com/ical/Japanese32Holidays.ics
2) Look at the titles

Data doesn't seem to be UTF-8, I am not sure what the format / problem is.

Opening http://ical.mac.com/ical/Japanese32Holidays.ics also displays 
incorrectly.
Maybe the content type is incorrect, or the saving..

Comment 1

[Mike Potter]

Can we vome up with what the files *should* look like.
I've been talking to the guy at Apple, maybe we can help them figure this one
out. (It doesn't sound like a problem with the calendar)

Comment 2

[gekacheka]

If I subscribe to http://ical.mac.com/ical/Japanese32Holidays.ics with Tools |
subscribe, the characters are corrupted.
If I download http://ical.mac.com/ical/Japanese32Holidays.ics and then open it
locally, the characters look fine (to check if mozilla can find your japanese
fonts, try opening a japanese site like yahoo.co.jp).

So something in the subscribe path is mangling the characters --- is it using a
7bit character stream or something?

(I am using 20031416-cal on moz1.3b)

Comment 3

[Simon Woods]

Sounds like this is this a duplicate or at least another manifestation of 181557?

http://bugzilla.mozilla.org/show_bug.cgi?id=181557

Comment 4

[Dennis Haney]

*** Bug 190786 has been marked as a duplicate of this bug. ***

Comment 5

[Eric Belhaire]

*** Bug 181557 has been marked as a duplicate of this bug. ***

Comment 6

[Eric Belhaire]

Attachment: suppression of UTF-8 conversion in calendarManager.js.

This patch is based on Filip Jirsak proposition in the newsgroup at :
news://news.mozilla.org:119/b9506s$jeo$[e-mail]
> Filip Jirsák <[e-mail]> posted on 05.05.2003 at 08:27
>============================================================================================

>Hi,
>I found problem in file calendarManager.js in calendar/content directory.
There is following line:
>saveDataToFile( ThisCalendarObject.getAttribute(
"http://home.netscape.com/NC-rdf#path" ), CalendarData, "UTF-8" );
>When I replace it with this:
>saveDataToFile( ThisCalendarObject.getAttribute(
"http://home.netscape.com/NC-rdf#path" ), CalendarData, null );

>(only "UTF-8" -> null), it works fine for me. But I think the first form is
correct - because it calls convertFromUnicode("UTF-8", aDataStream); and this
can IMHO do nothing. But it does. So my solution is only workaround, I must
download sources of Mozilla and look what exactly
@mozilla.org/intl/scriptableunicodeconverter does.
>regards
>Filip JIrsák

I tested it for a fews days and it seems to work. The only problem is that the
Holidays file in the server (i.e.
http://www.mozilla.org/projects/calendar/caldata/FrenchHolidays.ics) seems to
take the bug into account and are coded in iso-8859-1 instead of UTF-8.

Comment 7

[Mike Potter]

Checked into cvs.

Comment 8

[Eric Belhaire]

*** Bug 211939 has been marked as a duplicate of this bug. ***

Comment 9

[Eric Belhaire]

*** Bug 181555 has been marked as a duplicate of this bug. ***

Comment 10

[Gergely Papp]

Unfortunately this workaround while solving the problem of character encoding
creates more problems.
Without the patch I can create entities (both tasks and events) with title,
description, etc. using Hungarian character set (ISO-8859-2) and publish them on
the server. When refreshing the entities are displayed correctly, except for the
Unicode replacement of the characters.
With the patch I keep loosing titles and descritpions. I haven't got the time to
find the pattern for loosing data, but find it very annoying. I guess it has
something to do with C style null terminated strings.

(I'm using Mozilla 1.4 with Calendar 20030612)

Comment 11

[Eric Belhaire]

reopen based on comment

Comment 12

[Mike Potter]

New contact from [e-mail] to [e-mail]
Filter on string OttawaMBA to get rid of these messages. 
Sorry for the spam.

Comment 13

[Simon Paquet [:sipaq]]

Comment on attachment 127152 [details] [diff] [review]
suppression of UTF-8 conversion in calendarManager.js.

Patch was checked in

Comment 14

[Matthew (lilmatt) Willis]

*** Bug 214276 has been marked as a duplicate of this bug. ***

Comment 15

[Tilmann Kuhn]

I subscribe to an online calendar that is delivered as text/html using
iso-8859-1. Some of the task's summaries contain international characters such
as "הצ��" for example. In that case calendar refuses to import the task title at
all and shows "Untitled" instead in the task list.

I used Calendar 2004121718-cal

Comment 16

[db]

The problem is still there in the nightly build I downloaded yesterday (but the
about box says Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a5)
Gecko/20050111 Mozilla Sunbird/0.2RC2))

Comment 17

[Joey Minta]

(In reply to comment #16)
> The problem is still there in the nightly build I downloaded yesterday (but the
> about box says Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8a5)
> Gecko/20050111 Mozilla Sunbird/0.2RC2))

Then you didn't download the correct nightly.  Please use this link:
http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/nightly/latest-trunk/

Comment 18

[Greg]

I have a similar problem with the file at :
http://www.mozilla.org/projects/calendar/caldata/FrenchHolidays.ics

Can you delete (in this file) the events at :
2003/10/21
2003/06/09
2004/04/12
2004/05/31

They contain bogus chars (not UTF-8), which appear badly in Sunbird.

Comment 19

[Greg]

> I have a similar problem with the file at :
> http://www.mozilla.org/projects/calendar/caldata/FrenchHolidays.ics
>
> Can you delete (in this file) the events at :
> 2003/10/21
> 2003/06/09
> 2004/04/12
> 2004/05/31

Sorry for the inconvenience : it's not 2003/10/21, it's 2003/04/21. The others dates are ok.

Comment 20

[Joey Minta]

The attached testcase worksforme in Sunbird 0.3a1+.  Resolving this bug accordingly.

Greg,
   Please create a new version of the FrenchHolidays file without the non-UTF8 characters, and file a new bug with the new holiday-file.  Simon Paquet will take care of updating the file on the mozilla server.

Comment 21

[Greg]

> Greg,
>    Please create a new version of the FrenchHolidays file without the non-UTF8
> characters, and file a new bug with the new holiday-file.  Simon Paquet will
> take care of updating the file on the mozilla server.
> 

Done. It has number 316680. Thanks for your answer.

Comment 22

[Reed Loden [:reed]]

The bugspam monkeys have been set free and are feeding on Calendar :: General. Be afraid for your sanity!