11763 - Regression: Quoting in Reply/Forward of Japanese msgs is broken in Msg window body

Status: VERIFIED DUPLICATE of bug 11094

(MailNews Core :: Composition, defect, P1)

Description

[Katsuhiko Momoi]

** Observed with 8/12/99 Win32 M9 build **

This was working until a few builds ago and so this is
a regression.

1. Select a JPN msg (HTML or plain)containing a Japanese header & body text
   with no attachment
2 [details] [diff] [review]. Now reply or forward (w/ auto auto) either under HTML or plain text mail
   send.
3. You see that the Message window body seems to be in raw JIS rather than
   in Japanese.

Note: header quote is not working -- this is a known bug.
      But the body text quote should show in Japanese.

Comment 1

[Katsuhiko Momoi]

Setting the priority and sedverity.
This should be fixed for M9.

Comment 2

[Katsuhiko Momoi]

Assigning myself as QA.

Comment 3

[nhottanscp]

I think the reply bug 11094 has not been fixed.
Did you try us-ascii or latin1, are those working?

Comment 4

[Katsuhiko Momoi]

I'm not sure if Rich has taken out the Meta charset listening
in Bug 11094. It's not clearly indicated there. I assumed that he did not
take the action and in the meantime, the crash stopped occuring.

Rich, please tell us -- is non-ASCII quoting supposed to work
with today's build?

Comment 5

[rhp (gone)]

I didn't take out the emitting of a meta tag so this should be there.

Also, plain text has been messed up the past few days...quoting was very
broken.

- rhp

Comment 6

[Katsuhiko Momoi]

This problem of raw JIS display in reply-forward/quote happens
with both plain text and HTML mail. So this seems like
a real regression and not related to the other bug, then.

Comment 7

[Katsuhiko Momoi]

BTW, Latin 1 high-bit characters are quoted OK.

Comment 8

[Jean-Francois Ducarroz]

Akkana did some modification few days ago to fix a conversion problem with
japanese mail. Maybe this problem is a side effect of her fix.

I reassign the bug to Rich as it's a composition backend issue.

Comment 9

[rhp (gone)]

This is still that problem of the editor dealing with HTML that has a META tag
that specifies a charset. I'll attach an example of the data that comes out of
the quoting operation and just trying to load this into viewer or apprunner
seem to give me lots of problems.

- rhp

Comment 10

[rhp (gone)]

Attachment: Example of mail message quoting

Comment 11

[Katsuhiko Momoi]

Since what's broken is "quoting Japanese text", I'll
modify the sumamry to fit the problem better.

Comment 12

[nhottanscp]

I can see Rich's attachment using 4.5 Win32 or today's Win32 build. I think
that's a correct UTF-8 html.
When I saved that as a local file and tried to open it by the editor then I got
problems (either not redraw or redraw with many gargabe). I am seeing the
similar problem by opening Netscape JA page which has also a META tag with
charset.

Comment 13

[Akkana Peck]

Saved the attachment as a URL to make it easier to test in the editor.
OnEndDocumentLoad is called once (correct), so this isn't the double-load
problem, but something else.

Comment 14

[nhottanscp]

Attachment: I18n mail smoke test

Comment 15

[nhottanscp]

Using the smoke test data, I saw different results between local and imap.
Local file, I saw any replied message got redraw problem (could be a dup of
11094). While imap, I got the same problem momoi mentioned for the second
(ISO-2022-JP) and forth message (UTF-8).
There may be other problem for imap. Reassign to Rich for further investigation.

Comment 16

[rhp (gone)]

There seems to be a problem with IMAP operations and getting completion calls
on running URL operations. Does this work with local mail, but not IMAP. If so,
this is a duplicate of a bug that jefft has on his plate.

- rhp

Comment 17

[nhottanscp]

>Does this work with local mail, but not IMAP.
Not so sure because of the redraw problem, but I can see at least correct
Japanese (at least no raw JIS with incorrect draw/redraw) in case of local mail.

Comment 18

[Katsuhiko Momoi]

This problem actually occcurs under both IMAP and local mail.
Once the garbage is cleared out, you'll see no-Japanese strings
under local mail.

Comment 19

[Katsuhiko Momoi]

Also there seems to be a difference of the following type:

Under IMAP; you see raw JIS.
Under local mail: you see 8-bit characters (but not Japanese)

This result is obtained for reply/quoting exactly the same msg.

Comment 20

[rhp (gone)]

Ok, here is the deal.

The quoting we do with IMAP is a hack that is guaranteed not to work for
Japanese or any other non us-ascii text...and it didn't ever work. I am waiting
on a stream routine from mscott that won't be in the product until M10.

No, on POP, seems like we have 2 problems:

 1 - redraw problem
 2 - UTF-8 rendering problem

I have no idea what the redraw problem is....we just tell the editor to load a
URL and we are done. Second, the data we are telling the editor to load is the
"Example of mail message quoting" data that I've attached to this bug which
looks valid to me.

For both these problems, I'm really not sure what I can do to fix these. I can
try to trace it and debug it, but I need to get multipart related fixed and
about 10 other bugs fixed before Friday when I will be leaving for sabbatical.

I will try to look at it later today.

- rhp

Comment 21

[Katsuhiko Momoi]

Well, IMAP reply/quote for Japanese text at least displays OK
on the last pre-necko build, 7/28/99. Are you saying that this was
a fluke?

Comment 22

[rhp (gone)]

Big fluke...consider yourself lucky.

- rhp

Comment 23

[rhp (gone)]

Ok, I've traced this broke on the line before we call to LoadURL on the editor.
I have attached this quoted HTML file to this bug. If you try to load this in
viewer, you get weird behavior. When you try to load the first time, you get
the repaint problem. If you just force a reload, it comes up fine.

I have a feeling that there is data there after the first load, and hitting
some keys in the editor gives you the repaint, but the character rendering is
wrong.

Frank, if this doesn't belong to you, please reassign, but it appears to be a
problem loading this HTML that specifies UTF-8 in the META tag since my tests
had nothing to do with mail or the editor.

- rhp

Comment 24

[rhp (gone)]

Attachment: Here is the HTML quoted material - if you load twice, it works fine

Comment 25

[Akkana Peck]

There's a style sheet issue here.  If you load this file in the browser, it
works.  If you load it in the editor, the editor starts but you see nothing in
the window.  But if you apply the style sheet "Editor Styles 1" (from
Format->Apply Style Sheet), everything looks fine on Linux, even the Japanese
text; on Mac, the Japanese text doesn't show up but everything else does.

Comment 26

[Katsuhiko Momoi]

I think there is another bug lurking in this, too.
I norticed that quoting a 1-line Latin 1 msg under HTML send option
actually dumps the original headers and body without any quoting
marker like ">" or "|" and sends it out. The body is still Latin 1
as it arrives but the headers if they contain 8-bit Latin characters
seem to be in raw UTF-8.

Comment 27

[Katsuhiko Momoi]

OK. The header bug seems to have been also reported in Bug 5493.

Comment 28

[bobj]

I saved Rich's attachment
  Created an attachment (id=1222)

as a local HTML file and opened it with today's apprunner from Sweetlou and it
displays fine for me (as Naoki also confirmed in an earlier comment.)

Comment 29

[Frank Tang]

rhp: Don't get confused. After Necko land, a lot of thing are broken in viewer
but not apprunner.
> If you try to load this in
> viewer, you get weird behavior. When you try to load the first time, you get
> the repaint problem. If you just force a reload, it comes up fine.
I just talk to haishd ~3:00 about this issue. He try to trace down a problem
only happen in the viewer with Meta tag page display as blank untill hit reload.
(It seems exactuly the samething you described) We trace down the problem and
find out the mSHist in nsWebShell::ReloadDocument is null so it cannot find the
URL to reload. Radha recently change the history cod and it probably is broken
in viewer. Howerver, this is probably not the cause of the bug. You should trace
this bug by checking whether the nsWebShell::ReloadDocuemnt call RefreshURL with
a valid URL.
BTW, is that true the current behavior looks like the same as 11094. If so, it
depend on Necko to fix a notification issue. See 11094 for details.

Comment 30

[rhp (gone)]

I did load it in apprunner and saw the same bad behavior. I was just trying to
isolate the behavior away from mail/news.

- rhp

*** This bug has been marked as a duplicate of 11094 ***

Comment 31

[Katsuhiko Momoi]

Naoki, Rich and I just discussed teh current state of this bug
and came to an agreement that this bug should have been really 2 bugs.

One -- the IMAP non-ASCII quote problem which has dependency on mscott's
       work. This goes to rhp.
Two -- The Ender problem which cannot display the mail message containing
       Japanese correctly though the same page can be displayed
       OK with Browser. This goes to akkana.

Bug 11094 started out as a crashing bug and it is confusing as to
what the current status is. And so it would be best to file a new
bug on Two.

Comment 32

[Katsuhiko Momoi]

I'll be filing 2 new bugs.

Comment 33

[Katsuhiko Momoi]

The 2 issues here are dealt with elsewhere.
Verifying as dupliacte.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ef8d645d18ec
[fenix] For https://github.com/mozilla-mobile/fenix/issues/11763: Shows confirmation dialog when deleting last tab from collection
https://hg.mozilla.org/integration/autoland/rev/211ff6145b81
[fenix] For https://github.com/mozilla-mobile/fenix/issues/11763: Fixes unit tests