523307 - utf-8 decoding botched somehow

Status: RESOLVED WORKSFORME

(MailNews Core :: Internationalization, defect)

Description

[Eyal Rozenberg]

For the last several months, certain UTF-8 messages, which are reported as windows-1255 or iso-8859-8-i and have both plain text and HTML parts (these may be sufficient rather than necessary criteria), are no longer properly decoded when forcing the charset to UTF-8. 

Moreover, sometimes one even gets parts of the HTML itself, e.g. the doctype string, in the decoded message.

This happens, of course, with and without my BiDi Mail UI extension.

This started happening between 2009-07-09 and 2009-07-12.

Comment 1

[Eyal Rozenberg]

There are no Windows builds between 2009-07-09 and 2009-07-12 so I can't pinpoint the exact day.

Another note: The windows-1255 decoding result also differs before and after the regression, it's not just the post-coersion decoding.

Comment 2

[Eyal Rozenberg]

Attachment: triggerring message

Comment 3

[Eyal Rozenberg]

Attachment: triggerring message

Simplified testcase.

It turns out you don't need the text MIME part. But if you only have text instead of HTML with the same issue (UTF-8 reported as windows-1255 or 8859-8-i), the problem does not manifest.

Comment 4

[Eyal Rozenberg]

Bug does _not_ manifest with a clean profile using the same mail folders as the dirty one. But it _does_ manifest with a clean profile into which I've copied my prefs.js, addressbook, bookmarks and cookies, and using the same mail folders.

Comment 5

[Wayne Mery (:wsmwk)]

bad, sounds like.

this is the only open intl bug, with regression keyword + created since fall 2006.

Comment 6

[Eyal Rozenberg]

Manifestation seems to also be affected by the choice of View Message Body as Simple HTML/Original HTML. With some messages I see manifestation with Original and Simple HTML but not with text; with other messages, it's just Simple HTML.

Comment 7

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 8

[:aceman]

Jorg, can you please check this?

Comment 9

[Jorg K (CEST = GMT+2)]

The message shows as mojibake since it's shown as per
  Content-Type: text/html; charset=iso-8859-8-i
in Hebrew although the HTML is in UTF-8.

Comment 10

[Jorg K (CEST = GMT+2)]

We have to decode based on the mail header, and that's wrong, so the bug is invalid.

Comment 11

[Eyal Rozenberg]

@JorgK: 
With due respect - I was talking about what happens when you _force_ UTF-8 decoding, not with the default we have to apply based on the header.

However, I don't see the bug right now, so changing the resolution to WFM. I will reopen if I encounter it again though.

Comment 12

[Jorg K (CEST = GMT+2)]

Sorry, I read too many bugs every day and some details escape me.