984671 - Searching for "Metro Santiago" doesn't find https://marketplace.firefox.com/app/metro-mobile-metro-de-santiago/

Status: VERIFIED FIXED

(Marketplace Graveyard :: Search, defect, P4)

Description

[krupa raj[:krupa]]

steps to reproduce:
1. Search for "Metro Santiago" or load https://marketplace.firefox.com/search?q=Metro+Santiago

observed behavior:
https://marketplace.firefox.com/app/metro-mobile-metro-de-santiago/ is one of the search results

actual behavior:
Searching for "Metro Santiago" doesn't find https://marketplace.firefox.com/app/metro-mobile-metro-de-santiago/

Comment 1

[krupa raj[:krupa]]

Note that searching for just "metro" and just "santiago" finds the app. Only when searching for "Metro Santiago" fails to find the app.

Comment 2

[Thomas Elin [:relaas]]

This is a requested fix from TEF as Metro Santiago is an app developed by one of their partners. Is there an estimated time for completion?

Comment 3

[Rob Hudson [:robhudson]]

I'll state this first so it's not buried down in the wall of text below: User's whose default locale is Spanish will find this app.

To search in Spanish:
1. Go to https://marketplace.firefox.com/?lang=es
2. Search for "metro santiago"
3. The app should be the first result.

The reason is that the name field is provided only in Spanish and we match user's language to localized name fields stored in the index.

Deeper analysis by search index field:

App slug "metro-mobile-metro-de-santiago".
* We do 2 different kinds of matches against the slug:
1. Prefix matching. If the app starts with the term being queried we can find it. In this case "metro" would work but "metro santiago" doesn't.
2. Phrase matching. When multiple terms are provided (as is this case), the terms are matched as a phrase. But since "metro" and "santiago" are separated by "de" this phrase match fails to find it. A "slop" can be specified so the phrase match query can find adjacent words with words in between and indeed, if I add a slop=1 it finds this slug.

Name "Metro Mobile Metro de Santiago S.A."
* The name is provided only in Spanish.
* The index stores localized names in different fields (field: "name_$locale").
* We have one field that combines all the localized names into a single field for searching (field: "name").
* The name field is queried with 2 different kinds of matches:
  1. A prefix query. Similar to the app slug, "metro santiago" isn't a prefix of the name.
  2. A phrase query. Similar to the app slug, the words "metro" and "santiago" aren't adjacent so a phrase query doesn't find this. Adding a slop=1 does find the name correctly.
* The "name_spanish" field is queried using a traditional match query and finds the app by searching "metro santiago". This only happens when the user's language is set to "es". So English users searching for "metro santiago" do not find it because of the above reasons.

The description and author fields are similar.


Seeing as Spanish users find the app fine, do we want to fix this for users of other languages?

Potential solutions:
* Add localized names to locales where this app is relevant. This would be on the developer to adjust their manifest (for name) or app description (within the Marketplace) and provide localized versions.

* Adjust our queries to find this app no matter what language the user is using. My fear would be that we may return irrelevant results to some users.
  1. We can add some slop to the phrase queries so multi-word queries aren't required to be strictly adjacent. This may help other searches in general.
  2. Add a standard match query to the "name" field (not just prefix or phrase).

Comment 4

[krupa raj[:krupa]]

I think we should update the slop to allow searching for words which aren't adjacent.

People who don't have their locale set to Spanish might still want to be able to search for and install this app. We need to account for people who can speak multiple languages :)

Also, since the developer wants the app to show across all regions, the app should show up in search results.

Comment 5

[Wil Clouser [:clouserw]]

Until we have an enormous catalog of apps, searching across all locales doesn't seem like a terrible idea to me.  (I have no numbers/tests to back up my guess).  productwanted for dbialer to give us the answers.

Comment 6

[Christopher Van Wiemeersch [:cvan]]

I know this is about searching the app's name, but BTW, https://marketplace.firefox.com/search?q=%3Alanguage%3Des shows all the apps that support Spanish in case some people were unaware.

FWIW, in the offline version of the Marketplace, we get every translation of the app's name/description but we add to the search index only the translated version for that user's locale. If we're not doing something like this on the backend, we should.

Comment 7

[Rob Hudson [:robhudson]]

(In reply to Wil Clouser [:clouserw] from comment #5)
> Until we have an enormous catalog of apps, searching across all locales
> doesn't seem like a terrible idea to me.  (I have no numbers/tests to back
> up my guess).  productwanted for dbialer to give us the answers.

Just to clarify, we do search across all locales (for both name and description), we just don't do it in a way that would have found this app. We only do prefix and phrase queries. This can be adjusted to do a normal match query (with a lower boosting).

(In reply to Christopher Van Wiemeersch [:cvan] from comment #6)
> FWIW, in the offline version of the Marketplace, we get every translation of
> the app's name/description but we add to the search index only the
> translated version for that user's locale. If we're not doing something like
> this on the backend, we should.

I'm not clear what you think we should do on the backend? We don't have a single user (and single locale) to only store a single translation on the backend. We store them all and query against the field that matches the querying user's locale. But we also concatenate all translations into a single field for a catch-all query so matches for queries in other languages will find apps if they match. Each can have varying boosts.

Comment 8

[Rob Hudson [:robhudson]]

PR for adding slop:
https://github.com/mozilla/zamboni/pull/1939

Comment 9

[Rob Hudson [:robhudson]]

https://github.com/mozilla/zamboni/commit/d8c9f45 

Also filed bug 998070 to improve searches in other ways.

Comment 10

[krupa raj[:krupa]]

The app is listed in the search lists for all these search terms:

https://marketplace-dev.allizom.org/search?q=metro+santiago
https://marketplace-dev.allizom.org/search?q=Metro+de
https://marketplace-dev.allizom.org/search?q=Me 
https://marketplace-dev.mozflare.net/api/v1/fireplace/search/?lang=en-US&q=S.A.&limit=25&offset=25&region=us&cache=1&vary=0

However searching for "de", doesn't find the app

https://marketplace-dev.allizom.org/search?q=de returns https://marketplace-dev.mozflare.net/api/v1/fireplace/search/?cache=1&lang=en-US&q=de&region=us&vary=0 which doesn't find 
https://marketplace-dev.allizom.org/app/metro-mobile-metro-de-santiago?src=search

Rob, is there a reason for that?

Comment 11

[Rob Hudson [:robhudson]]

(In reply to krupa raj[:krupa] from comment #10)
> However searching for "de", doesn't find the app
> 
> https://marketplace-dev.allizom.org/search?q=de returns
> https://marketplace-dev.mozflare.net/api/v1/fireplace/search/
> ?cache=1&lang=en-US&q=de&region=us&vary=0 which doesn't find 
> https://marketplace-dev.allizom.org/app/metro-mobile-metro-de-
> santiago?src=search
> 
> Rob, is there a reason for that?

"de" is more of a stopword, like "the" or "an" and likely doesn't carry much weight as a search term on its own.

Comment 12

[krupa raj[:krupa]]

(In reply to Rob Hudson [:robhudson] from comment #11)
> (In reply to krupa raj[:krupa] from comment #10)
> > However searching for "de", doesn't find the app
> > 
> > https://marketplace-dev.allizom.org/search?q=de returns
> > https://marketplace-dev.mozflare.net/api/v1/fireplace/search/
> > ?cache=1&lang=en-US&q=de&region=us&vary=0 which doesn't find 
> > https://marketplace-dev.allizom.org/app/metro-mobile-metro-de-
> > santiago?src=search
> > 
> > Rob, is there a reason for that?
> 
> "de" is more of a stopword, like "the" or "an" and likely doesn't carry much
> weight as a search term on its own.

That's what I thought but it finds other apps with "de" in it but not Metro santiago.

Anyways, the case in comment 0 is addressed. Thanks!