1022341 - Implement Approximate Multilingual Contextual Search

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, defect)

Description

[Anivar Aravind]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0 (Beta/Release)
Build ID: 20140428193838

Steps to reproduce:

Bug type : Wishlist Feature

On Multilingual environments like Indian Market , People save names in Contacts in Various Local languages or english , But we may be using a different language keyboard for searching details like contacts . We need to locate expected results even if we are using a different language keyboard. 




Actual results:

As of now firefox OS does not support Approximate Multilingual Search . It need to use same language search to get expected results


Expected results:

We already have the algorithm for this as a part of SILPA Project (Swathanthra Indian Language processing applications ) . https://github.com/Project-SILPA/inexactsearch

You can find the working demo of this algorithm at at http://silpa.org.in/ApproxSearch

This is a Fuzzy string search application[1]. This application illustrates the combined use of Edit distance[2] and Indic Soundex algorithm[3][4].

By mixing both written like(edit distance) and sounds like(soundex), we achieve an efficient aproximate string searching. This application is capable of cross language string search too. That means, you can search Hindi words in Malayalam text. If there is any Malayalam word, which is approximate transliteration of hindi word, or sounds alike the hindi words, it will be returned as an approximate match. The "written like" algorithm used here is a bigram average algorithm. The ratio of common bigrams in two strings and average number of bigrams will give a factor which is greater than zero and less than 1. Similarly the soundex algorithm also gives a weight. By selecting words which has comparison weight more than the threshold weight(which 0.6), we get the search results.


[1] http://en.wikipedia.org/wiki/Fuzzy_string_searching
[2] http://en.wikipedia.org/wiki/Levenshtein_distance
[3] http://silpa.org.in/Soundex, https://github.com/Project-SILPA/soundex
[4] http://thottingal.in/blog/2009/07/26/indicsoundex/

Implementing this algorithm will help multilingual users to find desired search results easily via contexual search . 

This is an essential feature needed for Indian Market

Comment 1

[Axel Hecht [:Pike]]

This sounds interesting.

I don't know much about how to technically implement your proposal. I'd like to call out a licensing concern, though. The code I found on silpa is under AGPL, while our code is APL, which vary in intent quite a bit. So it's not just going to be a straightforward port.

Comment 2

[Anivar Aravind]

There is no patents on these algorithms. :-) 
The code is now in Python 
Anyway the idea needs to be ported to javascript for FFOS integration . So AFAIK whoever doing that is free to license it on APL.

Comment 3

[Anivar Aravind]

Looks like Mediawiki have a jquey way of addressing some similar concerns

http://www.mediawiki.org/wiki/Universal_Language_Selector/Technical_Design#Language_Search_API

Comment 4

[BMO Automation]

Firefox OS is not being worked on