1640408 - macOS: Cannot lookup words in the dictionary when they are enclosed in typographic quotation marks

Status: VERIFIED FIXED

(Core :: Internationalization, defect)

Description

[Ansgar Wohl]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:76.0) Gecko/20100101 Firefox/76.0

Steps to reproduce:

1. Go to a web page that uses typographic quotation marks, e.g.:

https://www.nytimes.com/2020/05/23/us/coronavirus-government-trust.html

or try it with this bug report where I'm going to include typographic quotation marks as well.

2. Three-finger-click or force-click (or whatever gesture you are using for dictionary lookup) on a word that has a typographic quotation mark in front of it (or after it). On the example page, the first such word is "Every". Probably, you can also try it here on this page: “Every time”.

Actual results:

A dictionary lookup for »“Every« (including the “) is performed and of course, there are no results.

Expected results:

The quotation mark should have been stripped from the word and a search for "Every" (without the quotation marks) should have been performed. This does work for other punctuation marks like standard non-typographic quotes.

When fixing this, please test it for different languages as well, e.g.

„German typographic quotation marks“
»alternative German typographic quotation marks«
«French typographic quotation marks»

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Ansgar Wohl]

Attachment: Punctuation patch v1

This patch that adds the punctuation marks from the Unicode pages "Latin-1" and "General Punctuation" to the word breaker.
I have tested the patch and it works, but I don't have time to find out how to properly submit the patch. I hope it might still be useful.

Comment 3

[Stephen A Pohl [:spohl]]

Masayuki-san, could you take a look or redirect as you see appropriate? Thank you!

Comment 4

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Well, Jonathan Kew or Makoto-san might be better for word breaker.

(If nobody wouldn't take this, I'd like to check it, but probably, I don't have spare time in a couple of months.)

Comment 5

[Jonathan Kew [:jfkthame]]

This is a legitimate issue, caused by the intl/lwbrk/WordBreaker code treating lots of Unicode punctuation characters as if they were alphabetic. I think a better fix than just hard-coding a couple of ranges, though, would be to check the General Category property; this will also catch various punctuation characters in other blocks, without needing an exhaustive list or constant maintenance. I'll put up a patch.

Comment 6

[Jonathan Kew [:jfkthame]]

Attachment: Bug 1640408 - Check Unicode general category to identify punctuation marks in word-breaker. r=m_kato

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/430ea4182cef
Check Unicode general category to identify punctuation marks in word-breaker. r=m_kato

Comment 9

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/430ea4182cef

Comment 10

[Cristian Fogel [:cfogel]]

Confirmed issues with 76.0.
Verified with 79.0b4 on macOS 10.15.5 & 10.13.6.