Closed Bug 672472 Opened 12 years ago Closed 12 years ago

hyphenator does not handle non-ASCII characters correctly

Categories

(Core :: Internationalization, defect)

Product:
Core
Component:
Internationalization
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla8

People

(Reporter: jfkthame, Assigned: jfkthame)

References

Details

Attachments

(1 file)

convert hyphenation-point offsets correctly to utf16 offsets
3.93 KB, patch
smontagu
: review+
Details | Diff | Splinter Review 
Because I misinterpreted the libhyphen API in bug 253317, hyphenation positions are not returned correctly when non-ASCII characters are present. (This doesn't affect the en-US patterns, but showed up once I started testing with more languages for bug 672320.)

The issue is that although the hnj_hyphen_hyphenate2() function takes the text as an 8-bit string, with a length in bytes, the hyphens array that it returns (when using UTF-8 dictionaries) is indexed by Unicode character count, not (as I assumed) by UTF-8 code unit positions in the input string.

This means that the conversion of hyphenation-point offsets to our UTF-16 text representation is incorrect.
Attachment #546742 - Flags: review?(smontagu)
Attachment #546742 - Flags: review?(smontagu) → review+
Pushed to mozilla-inbound:
http://hg.mozilla.org/integration/mozilla-inbound/rev/faa1737443dd
Whiteboard: [inbound]
http://hg.mozilla.org/mozilla-central/rev/faa1737443dd
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Whiteboard: [inbound]
Target Milestone: --- → mozilla8
You need to log in before you can comment on or make changes to this bug.