875455 - Reviews have mixed languages

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: Consumer Pages, defect, P2)

Description

[Maria Sandberg [:mushi]]

Attachment: .png - mixed languages in reviews

Discovered on FxOS by participant in usability testing. App details page show reviews in different languages. Region is set to "United States", user is not signed in. 

Observed behavior: 
When viewing reviews on an app details page (Twitter), there are reviews in English, Spanish, and French. 

Expected behavior: 
The Marketplace only shows me reviews in my language.

Comment 1

[Christopher Van Wiemeersch [:cvan]]

The thing is how do we know which language the review was written in? I might have my language set as French, but I'm going to write a review as English because the app is an English app. We would have to do some magic like Google Translate does to detect which language the text is most likely in.

Comment 2

[Maria Sandberg [:mushi]]

Do we do anything now, like guess that someone who set Marketplace to French will only see reviews from others with the same setting? This might need to be related to stores, so Twitter reviews in the French store are different than Twitter reviews in the US store?

Comment 3

[Christopher Van Wiemeersch [:cvan]]

So the thing is French language != French region.

The only thing we do is if you are in *France* (and that region is a "mature region") we show reviews only from users writing those reviews in France. (See bug 763699.)

Comment 4

[Wil Clouser [:clouserw]]

Our current solution in comment 3 is to show reviews only added in that region for mature regions.  Many regions are multi-lingual and I don't think that's a problem for now.  At some point you raise questions of what language "ur gr8" is in, I suppose.

Anyway, if you have UX recommendations, we're happy to hear them. :) In the meantime, I'm going to wontfix.

Comment 5

[Wil Clouser [:clouserw]]

It sounds like we store language with reviews now.  There will be the occasional broken case here (eg. I'm viewing the pages in French but writing in English) but I think the percentage of reviews falling into that scenario will be minimal.  This is worth doing, even if we only get it 90% right, imo.

Comment 6

[krupa raj[:krupa]]

Showing reviews only in your language surface a bunch of issues

a) The app may have 100 reviews in English but only 2 in the user's locale. What review count should we show?

b) Even though locale is more important, region may be important too. Yelp may have good content and thus higher reviews in the US, but the reviews for Spain may be poor. What do we show in that case?

Comment 7

[David Bialer [:dbialer]]

I think we should show all reviews, but prioritize the order.
If we can tell region for the review, I would suggest the following prioritized order:

1. Same Region->same locale of user->latest date first 
2. Other region->locale of user->latest date first (all regions that match the locale in date order)
3. Region->other locales->latest date first (other reviews in the region but in a different language than the user)
4. Other region->other locales->latest date first (non matching regions and non-matching languages in date order)

Comment 8

[Rob Hudson [:robhudson]]

We don't store region with reviews. There's bug 1004680 for that.

Comment 9

[David Bialer [:dbialer]]

As Wil said in Comment #5, we can just do languages for now.

Comment 10

[Tony Santos [:tsmuse]]

The solution to this seems to be to always show the total number of reviews, but to filter by default to only show the reviews in the users language. Because regions are mostly about laws and languages are about people, filtering by regions doesn't make a lot of sense to solve this problem. People can be in a country where they don't speak the language, those geo-political borders don't play a part in this problem. 

UI: We can give them an option (a "all reviews" "only reviews in my language" segmented control, see: http://pwalm.github.io/marketplace-style-guides/buttons.html#controls for Marketplace Style segmented control) to filter results and this control could live below the current "Write a review" button.

Comment 11

[Christopher Van Wiemeersch [:cvan]]

I think this is something we really ought to do. With most reviews in non-English languages, the reviews aren't quite useful for English speakers. (And, I'm sure the same happens for other languages.)

Can product set priority here please?

Comment 12

[David Bialer [:dbialer]]

I agree that this should be high priority and is a p2.  I think Tony's suggestion in Comment 10 makes sense.  I think there are 2 areas we need to consider, both in a signed in or not signed in condition.

1. In the app details page, below the description area, we have the first few reviews.  This may not be a good place for a dropdown - so need ui help here. 
2. On the ratings page.

One point of clarification:  I would think that if the person selects something (like 'all reviews') it should persist for the session.  In that way, moving from the app details page to the ratings page would persist the current preference until changed.

I think this remains a P2 - but can be put on repoman possibly.

Comment 15

[Mark Striemer [:mstriemer]]

Closed PRs since this got hung up on desktop UI changes.

https://github.com/mozilla/zamboni/pull/2818
https://github.com/mozilla/fireplace/pull/878

Comment 16

[Allen Short [:ashort]]

I ran the language classifier https://github.com/saffsd/langid.py over our existing reviews.

Main things I learned:
* The shorter a review is, the harder time it has guessing the language correctly. Adding some key words to check short reviews for will probably improve that a good bit ("buena", "malo", "good", "bad", "very", "muy", "exelente"). Or perhaps we decide that 1-3 word reviews are less important. Here's a scatter plot of review size vs language-id confidence: http://people.mozilla.org/~ashort/875455/scatter.png

* Half of reviews are pretty short, under 10 words. http://people.mozilla.org/~ashort/875455/wordcount.png

* As-is, accuracy is fairly good. 53% of reviews are above the 0.99 confidence level, and 83% of reviews longer than 3 words are above the 0.99 confidence level: http://people.mozilla.org/~ashort/875455/confidence.png 
Eyeballing the results didn't reveal any at that level that were misclassified, so I feel pretty good about trusting its scoring.

langid also supports restricting the languages to check for, which will probably improve accuracy.

I think using langid to classify comments, plus simple common-word filtering for low-confidence cases, is a worthwhile idea.

Comment 17

[Allen Short [:ashort]]

Added language guessing for comments, and API for filtering reviews by language:
https://github.com/mozilla/zamboni/commit/b51e9d5d05243f46f35a8145a65dd79769c01d4b

Comment 18

[David Durst [:ddurst]]

This is close to ready in case the consumer pages wanted to use this information somehow.