1737055 - "Repair text encoding" greyed out, when text encoding is wrong.

Status: RESOLVED DUPLICATE of bug 1701828

(Core :: Internationalization, defect)

Description

[Mary Arnold]

Attachment: examples.png

User Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.7113.93 Safari/537.36

Steps to reproduce:

Go to website:
https://the-japan-news.com/news/article/0007901286
https://the-japan-news.com/news/article/0007901240
Expected text encoding to be correct.

Actual results:

Text encoding is wrong.

Expected results:

Text encoding should have been correct, and if wrong, should have the option to repair, with the additional option to choose alternative encodings.

See also Bug 1731482 and Bug 1736248 (Thunderbird). I do not have the expertise to say what the encoding should be, or shouldn't be, but I do know that I used to go and manually change the encoding when I encountered stuff like this, and was able to fix it all up. Call me grumpy if you like. That's 'cause I am.

Comment 1

[Mary Arnold]

Attachment: viewsource.png

<meta name="keywords" content="The Japan News, The Daily Yomiuri, Yomiuri Shimbun, デイリーヨミウリ, 読売新èž, 英字新èž, 英語 教育,日本 ニュース" />

Comment 2

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::Layout: Text and Fonts' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 3

[Jonathan Kew [:jfkthame]]

I don't think this is a Layout bug, it's about the (non-)detection of the encoding. Once there's misinterpreted content in the DOM, there's nothing Layout can do about it. I thought we had an Encodings component somewhere (under DOM?) but can't seem to find it now, so moving this to Internationalization for further consideration. Henri will know more about how this works, I think.

One factor may be that the <meta charset=utf8> tag is too far into the document; according to https://www.w3.org/International/questions/qa-html-encoding-declarations#quickanswer, it's supposed to be within the first 1K bytes, but in this case it's considerably further down than that (around 1.7K).

Comment 4

[Henri Sivonen (:hsivonen)]

(In reply to Jonathan Kew (:jfkthame) from comment #3)

One factor may be that the <meta charset=utf8> tag is too far into the document; according to https://www.w3.org/International/questions/qa-html-encoding-declarations#quickanswer, it's supposed to be within the first 1K bytes, but in this case it's considerably further down than that (around 1.7K).

Indeed this seems to be the root cause. The patch for bug 1701828 fixes it.

It's not immediately obvious to me why the menu item got disabled here. The page is weird by being a POST response (the GET request uses JS to generate POST, and the actual content is in response to the POST).

Comment 5

[Henri Sivonen (:hsivonen)]

Pretty sure the menu effect arises from the POST via https://searchfox.org/mozilla-central/rev/4c184ca81b28f1ccffbfd08f465709b95bcb4aa1/parser/html/nsHtml5StreamParser.cpp#519

Comment 6

[Henri Sivonen (:hsivonen)]

Considering that the patch for bug 1701828 fixes this site, I think it's not worthwhile to put much time into figuring out what the menu interation was. However, my best guess is that it's an interaction between POST not being idempotently reloadable and the first 1024 bytes being ASCII. In theory, the menu behavior could be fixed. In practice, it's probably not worthwhile unless this shows up again even after bug 1701828 has landed.

Comment 7

[Mary Arnold]

Thank you. I can't check if it's been fixed until whatever version of Firefox turns up (95? 96?). If I find any other examples of the repair text menu option not being available, especially if they aren't fixed in the version that bug 1701828 lands in, I'll post 'em here.