1836973 - IOUtils.readUTF8 and IOUtils.readJSON should strip the UTF-8 BOM if any

Status: RESOLVED FIXED

(Toolkit :: IOUtils and PathUtils, enhancement, P3)

Description

[Rob Wu [:robwu]]

Bug 1835790 is a regression caused by the replacement of OS.File.read with IOUtils.readUTF8. This regression happened because an external (JSON) file was read and failed to be parsed because there was an UTF-8 BOM in front of it.

IOUtils.readUTF8 and IOUtils.readJSON should strip the UTF-8 BOM if any, possibly as an option if you want it to be used exclusively for externally-provided files. This stripping can be implemented at https://searchfox.org/mozilla-central/rev/27e4816536c891d85d63695025f2549fd7976392/dom/system/IOUtils.cpp#1285

Comment 1

[Beth Rennie [:beth] (she/her)]

Attachment: Bug 1836973 - Skip BOM in IOUtils.read{UTF8,JSON} by default r?nika

Comment 2

[Rob Wu [:robwu]]

Attachment: Bug 1836973 - Avoid double stripping of BOM

Now that IOUtils.readUTF8 automatically drops the BOM, the consumer in
NativeManifests.sys.mjs does not need to do that manually any more.

Depends on D180136

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2eabe81260f7
Skip BOM in IOUtils.read{UTF8,JSON} by default r=nika
https://hg.mozilla.org/integration/autoland/rev/2d06f6a33a6a
Avoid double stripping of BOM r=barret

Comment 4

[Norisz Fay [:noriszfay]]

Backed out for causing mochitest failures on test_ioutils_read_write_utf8.html

Backout link

Push with failures

Failure log

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7c5101ddab4e
Skip BOM in IOUtils.read{UTF8,JSON} by default r=nika
https://hg.mozilla.org/integration/autoland/rev/32aa10fc8f27
Avoid double stripping of BOM r=barret

Comment 6

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/7c5101ddab4e
https://hg.mozilla.org/mozilla-central/rev/32aa10fc8f27