1602213 - X-Content-Type-Options: nosniff turns off the encoding detector

Status: RESOLVED FIXED

(Core :: DOM: HTML Parser, defect)

Description

[Dimitar]

Attachment: On the left is the a page from the website in Chrome and on the right is the same page in Firefox.

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0

Steps to reproduce:

1. Open Firefox
2. Enter http://dolnocerovene.tk in the address bar

Actual results:

The text in the navigation bar is garbled, but the content isn't. The page is encoded in Windows-1251 and the text is in Bulgarian(Cyrillic alphabet).

Expected results:

All of the text should be in Bulgarian and not garbled.

Comment 1

[Gingerbread Man]

Please close all tabs from that site, clear the cache, then try again. Does Chrome still show the correct text then?

I see the same garbled text in Vivaldi 2.9. Furthermore,

1. Opened this page in Vivaldi.
2. Right-clicked the site link and saved the file as index.html
3. Opened the file in Notepad++ and chose Encoding → Character sets → Cyrillic → Windows-1251.
   The text was still garbled in the same spots. It really looks like it was improperly saved and the browser is just displaying what the site serves.

Comment 2

[Dimitar]

When I cleared the cache in Chrome, the site's navigation bar became garbled. Refreshing it multiple times did nothing, until I pressed Ctrl + F5. Then the text was shown correctly, but sometimes it gets garbled again.

In Firefox when I enabled the Disable Cache(Developer Tools > Network tab) the text was either garbled or not after every F5 press(it was switching between the two), but when I disabled that option, the text switched a few times and fixed itself.

Comment 3

[Alphan Chen [:alchen]]

Is there a solid STR for this bug?
I cannot reproduce the symptom.

Comment 4

[Henri Sivonen (:hsivonen)]

I don't see the problem that the screenshot shows. Reporter, do you still see it in the latest Nightly?

However, I do see http://dolnocerovene.tk/site/novini.php decoded as windows-1252. The page's declaration is typoed as window-1251 (without the s in windows). I need to debug why the detector isn't doing its thing instead then.

Comment 5

[Henri Sivonen (:hsivonen)]

(In reply to Henri Sivonen (:hsivonen) from comment #4)

I don't see the problem that the screenshot shows. Reporter, do you still see it in the latest Nightly?

However, I do see http://dolnocerovene.tk/site/novini.php decoded as windows-1252. The page's declaration is typoed as window-1251 (without the s in windows). I need to debug why the detector isn't doing its thing instead then.

The page has X-Content-Type-Options: nosniff, so we don't do content-based detection. Apparently Chromium does. :-(

Comment 6

[Henri Sivonen (:hsivonen)]

(In reply to Henri Sivonen (:hsivonen) from comment #5)

The page has X-Content-Type-Options: nosniff, so we don't do content-based detection. Apparently Chromium does. :-(

Did we get cross-browser consensus that X-Content-Type-Options: nosniff should turn off character encoding sniffing? Apparently, Chromium sniffs and, arguably, the right way to turn off character encoding sniffing is to put charset in the Content-Type header.

Comment 7

[Henri Sivonen (:hsivonen)]

Attachment: Bug 1602213 - Do not let X-Content-Type-Options: nosniff affect encoding detection, because Chrome runs its encoding detector.

Comment 9

[Henri Sivonen (:hsivonen)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1625c039b9ca7cc632fb5dce12ca38965a0e1cc

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bf9d222c0f0b
Do not let X-Content-Type-Options: nosniff affect encoding detection, because Chrome runs its encoding detector. r=sstreich

Comment 11

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/20805 for changes under testing/web-platform/tests

Comment 12

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

Comment 13

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/bf9d222c0f0b

Comment 14

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Is this a patch we should consider uplifting to Beta for Fx72 or can this fix ride Fx73 to release?

Comment 16

[Henri Sivonen (:hsivonen)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #15)

Is this a patch we should consider uplifting to Beta for Fx72 or can this fix ride Fx73 to release?

This bug impacts fewer locales on 72 than on 73, but I guess it would be prudent to uplift, yes. (The test case change won't apply cleanly, but I think we don't need to uplift testing.)

Comment 17

[Henri Sivonen (:hsivonen)]

Comment on attachment 9116120 [details]
Bug 1602213 - Do not let X-Content-Type-Options: nosniff affect encoding detection, because Chrome runs its encoding detector.

Beta/Release Uplift Approval Request

• User impact if declined: Users of the Japanese, Russian, and Ukrainian localizations as well as users who've manually enabled a Cyrillic detector and users who are browsing content on the .jp domain with any localization can experience some long-tail pages as completely unreadable when they would be readable with this patch.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The risk is low, because this removes a recently-added bit of code whose impact is (now) well understood. I marked this as covered by automated tests, because it is covered on trunk. However, the test part doesn't uplift meaningfully. Please just drop the test case changes when uplifting.
• String changes made/needed: None

Comment 18

[Henri Sivonen (:hsivonen)]

(Note: While the uplift would be useful, I don't expect it to fix the specific case reported here when applied to 72, so the steps to reproduce here can't be used for verifying the uplift.)

Comment 19

[Julien Cristau [:jcristau]]

Comment on attachment 9116120 [details]
Bug 1602213 - Do not let X-Content-Type-Options: nosniff affect encoding detection, because Chrome runs its encoding detector.

detect text encoding in more cases, low risk, approved for 72.0b9

Comment 20

[Dorel Luca [:dluca]]

https://hg.mozilla.org/releases/mozilla-beta/rev/51983bf763d0