1469580 - Character encoding detection doesn't follow the W3C specification

Status: RESOLVED FIXED

(Core :: DOM: HTML Parser, defect, P3)

Description

[Ophir LOJKINE]

Attachment: test-encode.html

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Build ID: 20180611123338

Steps to reproduce:

1. open the attached html file (test-encode.html)
2. open the web console


Actual results:

'windows-1258' is shown in the console logs


Expected results:

'windows-1251' should be shown in the console logs according to the W3C specification on character encoding scanning [1] :

> If it is an unmatched U+0022 QUOTATION MARK character (") [...] Return nothing.

Chrome returns the expected character encoding.

[1] https://html.spec.whatwg.org/multipage/urls-and-fetching.html#algorithm-for-extracting-a-character-encoding-from-a-meta-element

Comment 1

[Ophir LOJKINE]

Note: the file should be opened from the local file system or from a web server that doesn't specify an encoding in it's headers. Bugzilla automatically serves html files with a 'Content-Type: text/html; charset=UTF-8' header.

Comment 2

[Gingerbread Man]

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0
20180619102337

Comment 3

[Henri Sivonen (:hsivonen)]

The bug is right about at:
https://hg.mozilla.org/projects/htmlparser/file/2620240e520b/src/nu/validator/htmlparser/impl/TreeBuilder.java#l3304

Comment 4

[Henri Sivonen (:hsivonen)]

Attachment: Bug 1469580 - Ignore charset= in <meta content> if followed by unmatched quote.

Review commit: https://reviewboard.mozilla.org/r/252704/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/252704/

Comment 5

[Henri Sivonen (:hsivonen)]

Timing out on importing the html5lib tests properly into WPT and just fixing the actual bug.

Comment 6

[Henri Sivonen (:hsivonen)]

Attachment: Bug 1469580 - Ignore charset= in <meta content> if followed by unmatched quote.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a145a897c94c
Ignore charset= in <meta content> if followed by unmatched quote. r=mrbkap

Comment 8

[Cristina Coroiu [:ccoroiu]]

Backed out changeset a145a897c94c (bug 1469580) for lint failure at /builds/worker/checkouts/gecko/tools/lint/wpt.yml 

Backout: https://hg.mozilla.org/integration/autoland/rev/fb1b1c8b13ba1d7f691898b1e59b4a468489a344

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a145a897c94c14d74e1642d7335f1ca0efb44d8a

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=216978235&repo=autoland&lineNumber=290

[task 2018-12-14T10:12:30.226Z] executing ['bash', '-cx', 'cd $GECKO_PATH && ./mach lint -l wpt -f treeherder']
[task 2018-12-14T10:12:30.237Z] + cd /builds/worker/checkouts/gecko
[task 2018-12-14T10:12:30.237Z] + ./mach lint -l wpt -f treeherder
[task 2018-12-14T10:12:31.055Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2018-12-14T10:12:31.055Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2018-12-14T10:12:32.850Z] Installing setuptools, pip, wheel...done.
[task 2018-12-14T10:12:33.999Z] running build_ext
[task 2018-12-14T10:12:33.999Z] building 'psutil._psutil_linux' extension
[task 2018-12-14T10:12:33.999Z] creating build
[task 2018-12-14T10:12:33.999Z] creating build/temp.linux-x86_64-2.7
[task 2018-12-14T10:12:33.999Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2018-12-14T10:12:33.999Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-12-14T10:12:33.999Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-12-14T10:12:33.999Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2018-12-14T10:12:33.999Z] creating build/lib.linux-x86_64-2.7
[task 2018-12-14T10:12:33.999Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2018-12-14T10:12:33.999Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2018-12-14T10:12:33.999Z] building 'psutil._psutil_posix' extension
[task 2018-12-14T10:12:33.999Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2018-12-14T10:12:33.999Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2018-12-14T10:12:33.999Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2018-12-14T10:12:33.999Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2018-12-14T10:12:33.999Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2018-12-14T10:12:33.999Z] 
[task 2018-12-14T10:12:33.999Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2018-12-14T10:12:34.067Z] No specific files specified, running the full wpt lint (this is slow)
[task 2018-12-14T10:16:17.469Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/tools/lint/wpt.yml:0 | Lint process exited with return code 1 (wpt)
[task 2018-12-14T10:16:17.469Z] TEST-UNEXPECTED-ERROR | testing/web-platform/tests/html/syntax/parsing/quotes-in-meta.html:10 | Tabs used for indentation (INDENT TABS)
[taskcluster 2018-12-14 10:16:18.028Z] === Task Finished ===
[taskcluster 2018-12-14 10:16:18.028Z] Unsuccessful task run with exit code: 1 completed in 553.769 seconds

Comment 9

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14528 for changes under testing/web-platform/tests

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d02d14a3dd6e
Ignore charset= in <meta content> if followed by unmatched quote. r=mrbkap

Comment 11

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/d02d14a3dd6e

Comment 12

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/14528
* continuous-integration/travis-ci/pr (https://travis-ci.org/web-platform-tests/wpt/builds/468077177?utm_source=github_status&utm_medium=notification)

Comment 13

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 14

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 15

[Henri Sivonen (:hsivonen)]

https://hg.mozilla.org/projects/htmlparser/rev/bbeeee84648bdd84ba9ff280f55d60b0bf7b1e96