1177298 - Write UA overrides for top Japanese Sites

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Mike Taylor [:miketaylr]]

I'll double check, but I think this is the current list of candidates.

#2.1-2.7, *.yahoo.com.jp - Spoof as Chrome
#46, mixi.jp - "AppleWebKit" in UA will get SP site
#60, lohaco.jp - Would have to spoof as Chrome
#68, uniqlo.com/jp/ -"Mobile Safari" in UA to get SP site
#7,  www.rakuten.co.jp - Would have to spoof as Chrome
#92, www.nhk.or.jp - Needs "webkit" in UA string

Comment 1

[:shell escalante]

needinfo to Mike or Karl to double check additional sites I thought were fixed by UA strings. Not sure if i read the comments in the webcompat spreadsheet right... but worth asking (more eyes don't hurt). =]

#009 livedoor.jp  needs an android version 4 or 5 ua token (shell: not sure if UA token same as string or different?)
#54 * www.aeon.co.jp Fixed by Android version number in UA
#73.0 navitime.co.jp Needs Android version # + "AppleWebKit" to get to touch site
#73.1 * www.navitime.co.jp mobile at http://touch.navitime.co.jp. "AppleWebKit" in UA is minimal requirement to get 302 to touch site.

Comment 2

[Karl Dubost💡 :karlcow]

Shell,

To note the *deployed* status on UA override, see this file.
https://hg.mozilla.org/mozilla-central/raw-file/tip/mobile/android/app/ua-update.json.in

:Sebastian recently identified an issue for clean install of the device.
See Bug 1178760 and Bug 1179300
and Mike's email https://groups.google.com/forum/#!topic/mozilla.compatibility/RA0-_sRSWHY

The comment in the spreadsheet is what it needs to work for each of these sites. 

Hopefully everything will be fixed for next nightly and if not we can still probably UA override for demo on the client side.

Does it answer partly your questions? Do not hesitate to ask more.

Comment 3

[Karl Dubost💡 :karlcow]

Shell,

ok double checked today.
* Livedoor. no need for UA override. Just unprefixing.
* aeon. no need for UA override. Just unprefixing.
* navitime. NEED UA override + unprefixing

Does it help? ^_^

Comment 4

[Mike Taylor [:miketaylr]]

Yes, thanks Shell. I had missed Navitime--the rest you showed are covered by us adding Android version to the UA string by default. I'll write a status update on overrides in my report this afternoon.

Comment 5

[:shell escalante]

yes, super helpful. thanks guys!

Comment 6

[Mike Taylor [:miketaylr]]

I just noticed spoofing as Chrome for www.rakuten.co.jp, we get the mobile layout--but it needs to be whitelisted for the unprefixing service. I'll spin out a bug to handle whitelisting and ua spoofing at the same time.

Comment 7

[Mike Taylor [:miketaylr]]

Also to note: the redirection for www.navitime.co.jp -> touch.navitime.co.jp has been fixed (https://webcompat.com/issues/944), so no need for an override.

Comment 8

[Mike Taylor [:miketaylr]]

Attachment: 1177298.-Add-UA-overrides-for-top-Japan-sites.-r.patch

Karl, we have to add overrides for both www.yahoo.co.jp and m.yahoo.co.jp to avoid an infinite redirect loop. Given that we're also sending overrides for 7 other yahoo.jp domains... I wonder if we should just override all subdomains? i.e., "yahoo.co.jp": "chrome"? WDYT?

Comment 9

[Mike Taylor [:miketaylr]]

Note to self: let's not land this patch until 1184320 has landed. Might need to clean up bitrot, but will be simple.

Comment 10

[Mike Taylor [:miketaylr]]

Attachment: 1177298.-Add-UA-overrides-for-top-Japan-sites.-r.patch

On second thought, being more verbose is probably the right thing to do here--we haven't tested all yahoo.co.jp domains so who knows what the consequences might be.

Yahoo! is in the process of considering Fennec support, so these overrides are temporary anyways.

Note, this patch was rebased against Bug 1184320, so that has to land first.

Comment 11

[Mike Taylor [:miketaylr]]

1184320 is on fx-team now, this is safe to land now.

Sheriffs, just adding some UA overrides. Builds fine and works as expected--No try run needed.

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/a4d672467b9b

Comment 13

[Tim Taubert [:ttaubert] (inactive)]

https://hg.mozilla.org/mozilla-central/rev/a4d672467b9b

Comment 14

[rikingcoding]

Have we considered that maybe the Chrome version should be Chrome/43.0.0000.00 instead of Chrome/43.0.2357.93 to more clearly identify as a spoofed user-agent?

Comment 15

[Mike Taylor [:miketaylr]]

(In reply to rikingcoding from comment #14)
> Have we considered that maybe the Chrome version should be
> Chrome/43.0.0000.00 instead of Chrome/43.0.2357.93 to more clearly identify
> as a spoofed user-agent?

What are the benefits to identifying as a spoofed UA, transparent or otherwise?

Comment 16

[Clément Lefèvre]

(In reply to Mike Taylor [:miketaylr] from comment #15)
> (In reply to rikingcoding from comment #14)
> > Have we considered that maybe the Chrome version should be
> > Chrome/43.0.0000.00 instead of Chrome/43.0.2357.93 to more clearly identify
> > as a spoofed user-agent?
> 
> What are the benefits to identifying as a spoofed UA, transparent or
> otherwise?

Probably to let know website owners and developers that you had no choice other than spoofing your User Agent to get the website to work. Because this is not the way it should be done.
This can maybe lead some of them to consider having a mobile version which is not just seeking for a User Agent with Webkit, as spoofing your UA should just be considered as a hack and a fix due to the website behavior with standard mobile platforms.

Comment 17

[Karl Dubost💡 :karlcow]

Clément,

Mike and I are working on Web Compatibility issues, but to give a bit more context.
https://webcompat.com/ and Tech Evangelism Mobile/Desktop components on Bugzilla.

(In reply to Clément Lefèvre from comment #16)
> Probably to let know website owners and developers that you had no choice
> other than spoofing your User Agent to get the website to work. Because this
> is not the way it should be done.

This is the positive part of the story. What you are advocating for here is education and that's cool. 
And it's the work we do in Web Compat. You are welcome to help on analyzing, contacting, etc. There's more on our plate than we can handle. See https://webcompat.com/contributors


The daily story is a bit darker. Basically when doing UA spoofing, we are trying to lie to the Web site for different reasons:

1. The Web developer/site owner just didn't realize and is eager to fix it. Best case scenario. We contact them through the work of Web Compatibility issues (when we can find a contact)

2. The Web developer knows the issue and is not willing to fix it.
   - too many things to do, not a priority
   - project manager, boss is not interested in fixing it (business priority, not billable)
   - client is not interested in fixing it (business priority, no budget for requesting a fix)
   - the issue is in a library the developer has no control on
   - the site is legacy and not maintained anymore

Firefox Android has very very little market to no market shares in different geographical areas. 

Order of priorities:
User > Webdev > Browser

UA spoofing is not about warning the webdev/site owners/client that the site is not well coded, it's about giving a good experience to users. So the Web Compatibility in this scenario tries to find out what is the best scenario for the users. We are not happy about these choices usually. Sad panda face. It's just that we have to do it in a way which deviates the minimum from what is expected by the scripts on the server. We are not talking to Webdevs but to machines through the UA. Note that Web developers will not even notice a UA which is slightly different in their logs.

Comment 18

[Clément Lefèvre]

Karl, what I was meaning to say is more that it can give some signals.
From what I can see from work, sometimes companies gather some stats about the platform of their clients, and for this purpose, they're looking at User Agent. So, having a meaningful part of faked UA to get the website to work for users on some platforms could raise some alerts and give them some informations.

About the little market share of Firefox for Android, I guess the problem can be exactly the same on Firefox OS, so that it can be valuable for there too.

Because that still to be a very sad point that some companies or people just ignore part of the user and just care about Android (by Chrome for Android, and no other browsers) and iOS for mobile, and sometimes only Webkit for desktop.

Comment 19

[Karl Dubost💡 :karlcow]

(In reply to Clément Lefèvre from comment #18)
> Karl, what I was meaning to say is more that it can give some signals.

Understood, and we want to avoid to send this signal for two reasons. Any new hook, you provide to external services will be used for filtering:

1. for working around browser bugs.
2. for blocking the browser because not supported by the service.

> From what I can see from work, sometimes companies gather some stats about
> the platform of their clients, and for this purpose, they're looking at User
> Agent.

Yes. It's why when possible, we fake for example a Firefox OS UA to be Firefox Android. It's not perfect but it makes it a bit better, but sometimes no luck for Firefox, we need to be Chrome. 

btw, we are slightly off-topic, feel free to contact me directly or to send an email on compatibility @ lists.mozilla.org

Comment 20

[Karl Dubost💡 :karlcow]

All these sites send the desktop version to Firefox Android. A simple Chrome UA sends the right version.
Only a couple of them requires -webkit- prefixes released in 49.

bylines.news.yahoo.co.jp
finance.yahoo.co.jp
blogs.yahoo.co.jp
address.yahoo.co.jp
carview.yahoo.co.jp
crowdsourcing.yahoo.co.jp
dir.yahoo.co.jp
fortune.yahoo.co.jp
gyao.yahoo.co.jp
job.yahoo.co.jp
logistics.yahoo.co.jp
lohaco.jp
movies.yahoo.co.jp
netallica.yahoo.co.jp
omiai.yahoo.co.jp
search.yahoo.co.jp
tickets.yahoo.co.jp
toku.yahoo.co.jp
toto.yahoo.co.jp
tv.yahoo.co.jp
wifi.yahoo.co.jp
xbrand.yahoo.co.jp
konkatsu.yahoo.co.jp
partner.yahoo.co.jp
secure.mobile.yahoo.co.jp/p/ycal/top
weather.yahoo.co.jp
sports.yahoo.co.jp

Comment 21

[Mike Taylor [:miketaylr]]

Karl, is Yahoo! JP not willing to fix their mobile detection? If not, do you recommend we send overrides for the list in Comment #20? (I recall you have/had a meeting with them soon)

Comment 22

[Karl Dubost💡 :karlcow]

(In reply to Mike Taylor [:miketaylr] from comment #21)
> Karl, is Yahoo! JP not willing to fix their mobile detection? If not, do you
> recommend we send overrides for the list in Comment #20? (I recall you
> have/had a meeting with them soon)

The comment is not a request to add the UA override. Just a status, because I tested ~100 Yahoo! Japan domains these last couple of days.

I do have a meeting on Monday for Yahoo! Japan www.yahoo.co.jp but all these sites are not managed by the same teams and departments. And they do not have the same logic for UA detection or building the site. 
AT the meeting on Monday, my plan is to ask them help for outreach to the rest of the company and then figure out what is the best strategy for UA override. :)

Comment 23

[Mike Taylor [:miketaylr]]

Ah OK. I misunderstood -- sounds good!