Fix astral plane hack to work with UCS2 python

RESOLVED FIXED

Status

Tree Management
Treeherder: Data Ingestion
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: jgraham, Assigned: jgraham)

Tracking

Details

Attachments

(1 attachment)

(Assignee)

Description

a year ago
The hack in bug 1275425 doesn't work when python is compiled with UCS2 (which happens to be the case on Heroku). To fix this a separate codepath is needed to cover cases where non-BMP characters have a length of > 1.

Comment 1

a year ago
Created attachment 8758786 [details] [review]
[treeherder] mozilla:astral_hack_ucs2 > mozilla:master

Updated

a year ago
Blocks: 1176484

Updated

a year ago
Attachment #8758786 - Flags: review+

Updated

a year ago
Assignee: nobody → james

Updated

a year ago
Blocks: 1277304

Comment 2

a year ago
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/5162728d63b8a6e36b0602d727768d8822b64d35
Bug 1277300 - Fix astral plan character hack to work with UCS2 python

Python can either be compiled in UCS2 mode or UCS4 mode, which differ in
how non-BMP characters are represented. In the former they are
represented by a (user-exposed) surrogate pair so that a single astral
codepoint is encoded in a string of length 2. In the latter all
characters are represented by a single 4 byte value and all characters
have a string length of 1 e.g. len(u"\U0010FFFF") is 2 in UCS2 python
and 1 in UCS4 python.

Several functions don't work identically between the two variants
e.g. ord() will only accept a single python character and
ord(u"\U0010FFFF") will either succeed or fail depending on the compile
time options. Similarly, regexps can't directly work with characters
outside the BMP in UCS2 Python, and one must instead match the
individual parts of the surrogate pair.

This is obviously a terrible design mistake, but it's important to us
because although most environments use the forgiving/sane UCS4
configuration, Heroku currently uses edge-case-happy UCS2
configuration. Therefore where we are dealing with non-BMP characters we
must add in individual codepaths for each variant.

There is probably a moral in this story about why "just make it an
option" is generally a terrible idea.

https://github.com/mozilla/treeherder/commit/46224c12b38873dca776ba06610208b6f611454a
Merge pull request #1547 from mozilla/astral_hack_ucs2

Bug 1277300 - Fix astral plan character hack to work with UCS2 python

Updated

a year ago
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED

Updated

a year ago
Blocks: 1292720
You need to log in before you can comment on or make changes to this bug.