Closed Bug 757979 Opened 12 years ago Closed 12 years ago

Use Google Japan searchplugin for Mobile

Categories

(Mozilla Localizations :: ja / Japanese, defect)

Product:
Mozilla Localizations
Component:
ja / Japanese
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox15-)

Status:
RESOLVED WONTFIX
Tracking Flags:
Tracking Status
firefox15 - ---

People

(Reporter: bugzilla, Unassigned)

Details

(Keywords: jp-critical)

Attachments

(2 files)

Google Japan search plugin
3.38 KB, text/xml
Pike
: review-
Details
screen shot of side by side compare
503.11 KB, image/png
Details 
Japanese Firefox use Google Japan for the default search plugin.
But current Fennec use Google in English (Global one).

We should use Google Japan for Fennec too since:
 - We should provide same experience for both Firefox/Fennec users
 - Search result is quite different between English/Japanese Google

Google English will show many english only page especially when we search with english words.
Google Japanese will show Japanese pages properly even when we search with english words.

For example, when we seach with "firefox", users cannot get any Japanese informations from Google in English (current Fennec):
https://www.google.com/search?hl=en&sky=ee&ie=UTF-8&q=firefox
With Google Japanese, uses can find Japanese sites:
https://www.google.co.jp/search?hl=ja&sky=ee&ie=UTF-8&q=firefox
Google search plugin with google.co.jp, not google.com

diff from following Global one is s/google.com/google.co.jp/ only
http://hg.mozilla.org/mozilla-central/file/320b16daa7c0/mobile/locales/en-US/searchplugins/google.xml
Attachment #626576 - Flags: review?(stas)
Attachment #626576 - Flags: review?(stas) → review?(l10n)
We want to include this update for NativeUI release.
Could you review this or add CC to appropriate person if you cannot decide about this change.
Keywords: jp-critical
OS: Mac OS X → Android
Hardware: x86 → ARM
I don't see how we're ending up with the search url in comment 0, sorry.

AFAICT, the searches are https://www.google.com/m?q=firefox&ie=utf-8&oe=utf-8&aq=t and a dependent rls param. That compares to https://www.google.co.jp/m?q=firefox&ie=utf-8&oe=utf-8&aq=t, which give a very similar UX for me, if I insert a 'ja' as first language in accept-lang. No idea how close those results are within the same geolocation.

Historically, we added a specific google plugin for co.jp because that had a distinct corpus for search suggest. Is that still the case? I was actually thinking about having a discussion on wether we can get the desktop plugins integrated back again, mostly because the maintainance effort.
Axel- are we shipping customized search for any other locales? (I'm not familiar with how many localizations we ship on Fennec and how that might be different from Desktop Firefox.) If we are shipping customized search for other locales, then this request should be honored.
We're shipping two, Japanese, and Kurdish. The Kurdish one hard-codes English because the auto-detect is giving Kurdish in Arabic script instead of the Latin script that our localization uses. The input there was that users of the Latin script can't necessarily read the arabic script.
(In reply to Axel Hecht [:Pike] from comment #5)
> We're shipping two, Japanese, and Kurdish. 

If this is the case, then we need to make the change requested by Dynamis as search in the Japanese Fennec is basically useless for Japanese right now.
I see minor differences in the results so far, nothing close to useless.

To get around local settings, here's the links I compare:

https://www.google.com/m?q=firefox&ie=utf-8&oe=utf-8&aq=t&hl=ja
vs
https://www.google.co.jp/m?q=firefox&ie=utf-8&oe=utf-8&aq=t&hl=ja

I see nuances, sure, but useless?
Axel, the first of the two results is largely English. Most Japanese do read some English but they expect the search results basically all in Japanese.

What is the concern here? If we are shipping a separate build for a specific localization, why can't we ship the appropriate search plugin?
The concern is that we're failing to maintain these plugins in an orderly fashion, and that if we start adding them for minor nits, all hell breaks loose with 50 randomly diverging plugins.

Also, I would appreciate if the arguments would be adequatly describing the behaviour.

The search results I see from Berlin are largely Japanese.

One notable exception is mozilla.org, which is probably worth filing a bug about. That's annoying, but given that the top sponsored link is mozilla.jp, which is likely where you'd want folks to go anyway, I have a hard time to see the issue.
Axel, with due respect to your concerns, if we don't get search right on Fennec this time, we won't have a next time. 

Search needs to be localized to the market, especially in Japan, which is a P1 market for NativeUI Fennec. If we are shipping a separate build, please ship a separate and market-appropriate search plugin. There's nothing to debate here.

Being concerned about future issues is something we can plan for. It's an internal issue. Please don't conflate a real market need with an internal concern which is not yet even a real material issue.

Getting this right for the Japan market is something we need to fix now before launch.

Dynamis' explanation in Comment 1 says everything that needs to be said. It adequately describes the search results as unusable in the Japan market.

Jaclyn- can you please weigh in on this bug wrt the Japan market as a P1 market and the need for us to make the Fennec experience as native as possible (for the market)?
Adding dslater from PMM as well as Jaclyn.
Hi all,

Agreed that because Japan is a P1 priority, we should do everything we can to make the experience for our Japanese users as awesome as possible.

If search is indeed unusable (and searching is one of the most important use cases of a mobile browser) then I agree that something should be done to fix this in time for the GA release. 

Cc'ing Irina for further inbound Fennec marketing insights
OK, Seriously, if all you say is based on search queries that hardcode the language to English, then we should WONTFIX this one outright. I fed up with getting this "oh, but this one language is soooo super special that I don't need to provide the right data, nor humbly and honestly talk about the seen results"
(In reply to Axel Hecht [:Pike] from comment #13)
> OK, Seriously, if all you say is based on search queries that hardcode the
> language to English, then we should WONTFIX this one outright. I fed up with
> getting this "oh, but this one language is soooo super special that I don't
> need to provide the right data, nor humbly and honestly talk about the seen
> results"

Where do you see anyone saying what you have quoted? No one is saying what you claim in Comment 13.

Do you not trust the claims in Comment 1? 

What information do you require that is not already provided?
I don't think we need to argue about JA being a P1, or that we want to provide the best search option possible for that market.

What we need are some comparisons that we can all objectively look at, and some technical information on how and why the two different URLs for google JA search are different.   Here is what I see here in California.  Both contain a lot of english text.

Do we have guidance from google that there are actually differences in the processing of these search results given the the different host names, and if there are, are those differences expected to continue?
(In reply to chris hofmann from comment #15)
> Created attachment 632317 [details]
> screen shot of side by side compare
> 
> I don't think we need to argue about JA being a P1, or that we want to
> provide the best search option possible for that market.
> 
> What we need are some comparisons that we can all objectively look at, and
> some technical information on how and why the two different URLs for google
> JA search are different.   Here is what I see here in California.  Both
> contain a lot of english text.
> 
> Do we have guidance from google that there are actually differences in the
> processing of these search results given the the different host names, and
> if there are, are those differences expected to continue?

Chris, looking at your attachment, the .com results are all English. The .co.jp results are almost all Japanese results. I see very different results.

Google certainly does send different results to different host names. Any user in Japan, even those using en-US Firefox, are served the .co.jp results.
Can we:

- get the right search query by testing directly with the latest Japanese version of Fennec, on a device 
- take screenshots of the results
- compare that with screenshots taken with the Japan search plugin installed on the device
- add screenshots to the bug 
- reach a conclusion

How does this sound? Does anybody already have a Japanese version of Fennec installed?
I'm still surprised that google sends different results for .com/?hl=ja in different geos, in particular, that its behaviour is worse from inside Japan than it is from California or Germany.

Anywho:

For desktop, google.com globally redirects to .de or .fr or whatnot, giving local results.

Apparently the same doesn't hold true for google.com/m?

Can someone verify with google that that's intended behaviour?

If so, we need to find a solution that does that at least for all or 14 locales in multi, and don't special-case Japanese. Give previous experiences, we should come up with a new setup, I'd suggest to add the localized plugins to the en-US repo. That way, folks can fix things as needed by contracts in a timely fashion, and once, and not some on central, and then a bunch on aurora and all that mess.
For Google search queries, Google has requested we use Google.com, and allow their preference/locale detection routines do the right thing. If they're not doing the right thing, we should be working with them to fix it server-side, especially if it's a Tier 1 locale like Japan. Hard-coding results in the query may have some side-effects we don't want (like not honoring preferences the user sets Google-side).

Can we provide IP blocks from where we're seeing English results, as well as a capture from something like livehttpheaders to see what we're sending for things like accept-language and the like? Happy to get on this with Google as a priority, and I'd like to investigate fixing it at the source rather than in the client if we can.
There seems to be some confusion about urls, quality, redirection etc.
Let me explain more details.

(In reply to Axel Hecht [:Pike] from comment #3)
> I don't see how we're ending up with the search url in comment 0, sorry.

Fennec default search plugin will show google.com/m results like these:
when we select Japanese for Android OS and use Fennec in ja locale:
https://www.google.com/m?q=firefox&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:ja:official
when we select English for Android OS and use Fennec in en-US locale:
https://www.google.com/m?q=firefox&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official (English)

As you see, both are google.com/m (no redirection) and they are not useful to show the difference with google.com/m and google.co.jp/m. So I wrote simplest url by hand to show the difference. Sorry if this make you confuse.

> AFAICT, the searches are
> https://www.google.com/m?q=firefox&ie=utf-8&oe=utf-8&aq=t and a dependent
> rls param. That compares to
> https://www.google.co.jp/m?q=firefox&ie=utf-8&oe=utf-8&aq=t, which give a
> very similar UX for me, if I insert a 'ja' as first language in accept-lang.
> No idea how close those results are within the same geolocation.

As Gen also explained, UI will be shown in Japanese but search result items will not for Japanese even with ja accept-lang and from Japan geolocation.
# Note: redirection by geolocation is not work for google.com/m.

> Historically, we added a specific google plugin for co.jp because that had a
> distinct corpus for search suggest. Is that still the case? I was actually
> thinking about having a discussion on wether we can get the desktop plugins
> integrated back again, mostly because the maintainance effort.

I think user experience is important than maintainance cost. But I don't know enough about other locale and how Google search work in other lang/geolocation.
Let me focus on ja locale case here.

When I talked with Google Japan staff who is in charge of Search in 2009, Google Japan require us to keep using google.co.jp in Firefox. The reason was not for search suggest.

The redirection is not enough accurate and dependable. It depends on the IP of the client and if the gateway server of user's organization is not in Japan, the redirection will not work correctly.
# For example, redirection will not work in Google Japan office he explained.

(In reply to Axel Hecht [:Pike] from comment #18)
> For desktop, google.com globally redirects to .de or .fr or whatnot, giving
> local results.
> 
> Apparently the same doesn't hold true for google.com/m?

No unfortunately. google.com and google.com/m are different.
 - google.com/m will not redirect to google.co.jp/m
 - google.com can show search results for Japanese but google.com/m will not
 - google.com/m can show Japanese UI but search result is not for Japanese.
(In reply to Kev [:kev] Needham from comment #19)
> For Google search queries, Google has requested we use Google.com, and allow
> their preference/locale detection routines do the right thing. If they're
> not doing the right thing, we should be working with them to fix it
> server-side, especially if it's a Tier 1 locale like Japan. Hard-coding
> results in the query may have some side-effects we don't want (like not
> honoring preferences the user sets Google-side).

If Google can solve this problem server-side it's happy but as I wrote, current google.com/m don't.
 - google.com/m does not redirect to google.co.jp/m
 - google.com/m does not show search results for Japanese (only provide ja UI)
And it cannot solve completely on the server-side as Google Japan staff said
 - google cannot determine user geolocation accurately from IPs

To provide good user experience for users Google do at least
 - support redirect from google.com/m to google.co.jp/m
 
Kev, have you confirmed with Google Japan staff?
I guess that Google request to use google.com as a general rule but as for Japanese, Google Japan request to use google.co.jp.

I'll try to confirm with Google Japan staff again.
(In reply to dynamis (Tomoya ASAI) from comment #21)
> To provide good user experience for users Google do at least
>  - support redirect from google.com/m to google.co.jp/m

Sorry, I submitted during the edit. I intended to write:
To provide good user experience for users Google need do at least
 - determine proper country from IP accrately
   - but it cannot, according to Google Japan
 - support redirect from google.com/m to google.co.jp/m
   - since only google.co.jp/m provide appropriate results for ja now

I'm requesting to use google.co.jp/m not because ja is special P1 locale.
We just should provide reasonable result for users. I believe Google want same.
Comment on attachment 626576 [details]
Google Japan search plugin

Kev, any updates from you here?

Technically, I don't think we should continue down the road of putting google plugins in l10n. It's bitten us before. Thus marking this r-.
Attachment #626576 - Flags: review?(l10n) → review-
(In reply to Axel Hecht [:Pike] from comment #23)
> Comment on attachment 626576 [details]
> Google Japan search plugin
<...>
> 
> Technically, I don't think we should continue down the road of putting
> google plugins in l10n. It's bitten us before. Thus marking this r-.

CCing gavin for some technical input. Even though this is mobile, any decision here may bleed into desktop, and you're likely the best guy to ask about this.

I'd prefer to have in particular the google plugin in en-US, even if we need special cases for a locale. I could see us doing more ifdefs in the en-US one, if needed, or just put plugins not used in en-US next to the ones that are.

I'd really like to get to a scheme where we can make fixes like the https one in a well confined space, and have it magically trickle down the trains as en-US migrates, and locales pick stuff up.

What do you think?
Putting locale-specific search plugins in a single place rather than having to manage them across multiple l10n repos makes sense to me. We can devise some mechanism in the search service to select the appropriate plugin based on locale, or figure out some build-time solution (I don't know how easy that'd be with repacking).
Note, the standard google plugin changed in bug 783373, apparently that change is going to be in the final beta of fx15, can you check its behaviour again?
Bug 783373 is mobile only, just to be clear. Testing would need to happen in Firefox 15 for Android beta 6.
tracking-firefox15: --- → ?
Ugh, sorry, didn't realize we were tracking both Fennec and Desktop here. I'll talk to Google and see if we can get mobile addressed if it's an issue.
We aren't tracking any Desktop changes here, AFAICT.
Summary: Use Google Japan searchplugin → Use Google Japan searchplugin for Mobile
I found some report about 404 with old default Google search (with beta5)  but I confirmed new default Google search solve it (with beta6).

At the same time, locale detection also works fine with new one!
Thanks.
(In reply to dynamis (Tomoya ASAI) from comment #30)
> At the same time, locale detection also works fine with new one!

No need to use locale specific searchplugin now.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
Great, thanks for the patience. I'm happy the en-US google patch changed things for the better.

Can you un-land the mobile/searchplugins/google-jp.xml plugin still? Thanks.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: