Closed Bug 1739798 Opened 3 years ago Closed 2 years ago

[wpt-sync] Sync PR 31537 - Fix TextCodecUTF8's error handling in EOF and across buffer boundaries

Categories

(Core :: Internationalization, task, P4)

Product:
Core
Component:
Internationalization
Type:
task

Tracking

()

Status:
RESOLVED FIXED
Milestone:
96 Branch
Tracking Flags:
Tracking Status
firefox96 --- fixed

People

(Reporter: mozilla.org, Unassigned)

References

(
URL
)

Details

(Whiteboard: [wptsync downstream])

Sync web-platform-tests PR 31537 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/31537
Details from upstream follow.

Andreu Botella <andreu@andreubotella.com> wrote:

Fix TextCodecUTF8's error handling in EOF and across buffer boundaries

When TextCodecUTF8 found a truncated sequence at EOF, it used to emit
one replacement character per byte in the sequence, even when it was a
prefix of a valid sequence. Additionally, in streaming mode, if it found
a lead byte for which a valid sequence would span longer than the
current available bytes, any processing of that sequence was deferred
until all such bytes were available, even if errors could be detected
earlier. Both issues are solved by always checking the validity of
partial sequences.

The approach used in this patch uses DecodeNonASCIISequence to find
the length of the maximal subpart of a partial sequence, and if the
length is equal to the partial sequence size and we're not at EOF, we
don't emit the error. However, this does not work when a byte in the
0x80 to 0xC1 range is found in a lead position, since
NonASCIISequenceLength wrongly returns 2 and DecodeNonASCIISequence
isn't enough to determine whether the partial sequence is invalid. This
is fixed by having NonASCIISequenceLength to return 0 in those cases.

Another issue with this approach is that, since the outer do-while loops
in the Decode method take do_flush && partial_sequence_size as a
condition, if a non-ASCII lead byte is found whose valid sequences would
span longer than the bytes we have, those bytes would not be processed
until the next call to Decode if do_flush is false. But as it turns
out, the do_flush condition is not in fact needed, and removing it
fixes this issue.

Fixed: 796697
Fixed: 978522
Change-Id: Ic5a78e4eca356fdc2ad4038eba9ffe455fddf3ee
Reviewed-on: https://chromium-review.googlesource.com/3263938
WPT-Export-Revision: 2dc3caabebc8c24b97e3920d5f138a70be851472

Component: web-platform-tests → Internationalization
Product: Testing → Core
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]

CI Results

Ran 11 Firefox configurations based on mozilla-central, and Firefox, Chrome, and Safari on GitHub CI

Total 6 tests and 32 subtests

Status Summary

Firefox

OK : 6
PASS : 68
FAIL : 64

Chrome

OK : 6
PASS : 60
FAIL : 72

Safari

OK : 5
PASS : 45
FAIL : 55
ERROR: 1

Links

Gecko CI (Treeherder)
GitHub PR Head
GitHub PR Base

Details

New Tests That Don't Pass

/encoding/textdecoder-streaming.any.serviceworker.html
Streaming decode: utf-8, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: UTF-8 chunk tests (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
/encoding/textdecoder-streaming.any.html
Streaming decode: utf-8, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: UTF-8 chunk tests (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
/encoding/textdecoder-streaming.any.sharedworker.html
Streaming decode: utf-8, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-8, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-8, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-8, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-8, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16le, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16le, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16le, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16le, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16le, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16be, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16be, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16be, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16be, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: utf-16be, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL)
Streaming decode: UTF-8 chunk tests (SharedArrayBuffer): FAIL (Chrome: FAIL)
/encoding/textdecoder-streaming.any.worker.html
Streaming decode: utf-8, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-8, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16le, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 1 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 2 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 3 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 4 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: utf-16be, 5 byte window (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)
Streaming decode: UTF-8 chunk tests (SharedArrayBuffer): FAIL (Chrome: FAIL, Safari: FAIL)

Pushed by wptsync@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a41356b852d2
[wpt PR 31537] - Fix TextCodecUTF8's error handling in EOF and across buffer boundaries, a=testonly
https://hg.mozilla.org/integration/autoland/rev/239bee52e127
[wpt PR 31537] - Update wpt metadata, a=testonly

https://hg.mozilla.org/mozilla-central/rev/a41356b852d2
https://hg.mozilla.org/mozilla-central/rev/239bee52e127

Status: NEW → RESOLVED
Closed: 2 years ago
status-firefox96: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch
You need to log in before you can comment on or make changes to this bug.