1346180 - Differantiate between Turkish dotted and dotless "i"

Status: RESOLVED FIXED

(Webtools Graveyard :: Pontoon, enhancement, P2)

Description

[Selim Şumlu]

The Turkish alphabet has two different "i" characters: "i" (uppercase: İ)" and "ı" (uppercase: I). Pontoon fails to differantiate these in search.

For example, when searching for "yer imleri", results with "Yer İmleri" aren't returned.

https://pontoon.mozilla.org/tr/firefox-aurora/all-resources/?search=Yer+İmleri
https://pontoon.mozilla.org/tr/firefox-aurora/all-resources/?search=yer+imleri

More info on Wikipedia: https://en.wikipedia.org/wiki/Dotted_and_dotless_I

Comment 1

[Axel Hecht [:Pike]]

This might be related to postgres behind the scenes, there are a few hits on google on that, but old ones.

Some indicate that icontains isn't as good as full-text search, so maybe https://docs.djangoproject.com/en/1.10/ref/contrib/postgres/search/ is a good answer.

Comment 2

[Jarek Śmiejczak [:jotes]]

I can confirm that Pontoon use a simple __contains which maps to the 'LIKE' in postgresql. Currently We're working on improving the speed of the search queries, I can look if any of my work will resolve your issue.

Comment 3

[Jarek Śmiejczak [:jotes]]

Reference PR: https://github.com/mozilla/pontoon/pull/554/commits

Comment 4

[Jarek Śmiejczak [:jotes]]

After some digging I found two following links:
https://www.postgresql.org/message-id/[e-mail]
http://stackoverflow.com/questions/13029824/postgres-upper-function-on-turkish-character-does-not-return-expected-result

I looked at the code to verify it and Pontoon uses icontains which translates into 'LIKE UPPER('%query%')' and that may be the reason. I'll try to look into that to estimate if it's a quick fix.

Comment 5

[Matjaz Horvat [:mathjazz]]

Selim, could you have a look at the proposed fix and see if it works OK?
https://aaaaaaaasaaaaaa.herokuapp.com/tr/pontoon-intro/all-resources/

It might take a minute to load the page for the first time.

To add translation, you can log in:
username: [e-mail]
password: supersecretpassword

Comment 6

[Selim Şumlu]

(In reply to Matjaz Horvat [:mathjazz] from comment #5)
> Selim, could you have a look at the proposed fix and see if it works OK?
> https://aaaaaaaasaaaaaa.herokuapp.com/tr/pontoon-intro/all-resources/

Yes, I can confirm that it works properly.

Comment 7

[Matjaz Horvat [:mathjazz]]

Thank you!

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla/pontoon

https://github.com/mozilla/pontoon/commit/e0008021536eae30df8eb3ed54ab671fdbdc58e5
Fix bug 1346180. Set database collation for search queries. (#588)