2022456 - Port bug 1937541 - Migrate collation to ICU4X

Status: RESOLVED FIXED

(Thunderbird :: Upstream Synchronization, task)

Description

[Magnus Melin [:mkmelin]]

Port bug 1937541. Builds are failing.

Comment 1

[Henri Sivonen (:hsivonen)]

Previously: bug 1971367.

The new APIs relevant to the comm-central build failures are mozilla::intl::AppCollator::CompareBase and mozilla::intl::AppCollator::Compare. The former is provided to retain old and likely wrong semantics. The latter is the one that makes more sense to use.

With CompareBase, the relative order of case and accent differences depends on factors other than the string content, in the case of stable sort what order the input was in before sorting and in case of other kinds of sorting algorithms also on the specifics of the sorting algorithm.

My theory is that pre-existing use of the "base" sensitivity comes either from the semantics of an ancient Windows API or from a misunderstanding somewhere along the way.

Comment 2

[Henri Sivonen (:hsivonen)]

See bug 2022459.

Comment 3

[Henri Sivonen (:hsivonen)]

Filed bug 2022463 about sorting at "base" sensitivity via JS in comm-central.

Comment 4

[Geoff Lankow (:darktrojan)]

Attachment: Bug 2022456 - Port bug 1937541 - Migrate collation to ICU4X. r=#thunderbird-reviewers

I think it's easiest to just stop using collation keys (which Collator no longer has functions for)
and just compare string values.

Comment 5

[Henri Sivonen (:hsivonen)]

// FIXME: This doesn't do a numeric comparison, but it should.

It would make sense to add a collator configuration to AppCollator that has "variant" sensitivity and numeric mode turned on. (Perhaps that should even be the default for Compare and there should be a non-numeric config for sqlite to avoid changing behavior there.)

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/0a9b4084679a
Port bug 1937541 - Migrate collation to ICU4X. r=mkmelin

Comment 7

[Henri Sivonen (:hsivonen)]

(In reply to Henri Sivonen (:hsivonen) from comment #5)

// FIXME: This doesn't do a numeric comparison, but it should.

It would make sense to add a collator configuration to AppCollator that has "variant" sensitivity and numeric mode turned on. (Perhaps that should even be the default for Compare and there should be a non-numeric config for sqlite to avoid changing behavior there.)

Bug 2022559