Closed Bug 1177298 Opened 5 years ago Closed 5 years ago

Write UA overrides for top Japanese Sites

Categories

(Firefox for Android :: General, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
Firefox 42
Tracking Status
p11 + ---
firefox42 --- fixed
fennec 42+ ---

People

(Reporter: miketaylr, Assigned: miketaylr)

References

()

Details

(Whiteboard: [compat][sitepatch-built])

Attachments

(1 file, 1 obsolete file)

I'll double check, but I think this is the current list of candidates.

#2.1-2.7, *.yahoo.com.jp - Spoof as Chrome
#46, mixi.jp - "AppleWebKit" in UA will get SP site
#60, lohaco.jp - Would have to spoof as Chrome
#68, uniqlo.com/jp/ -"Mobile Safari" in UA to get SP site
#7,  www.rakuten.co.jp - Would have to spoof as Chrome
#92, www.nhk.or.jp - Needs "webkit" in UA string
Assignee: nobody → miket
tracking-fennec: --- → ?
tracking-p11: --- → +
Whiteboard: [compat]
Depends on: 1178760
needinfo to Mike or Karl to double check additional sites I thought were fixed by UA strings. Not sure if i read the comments in the webcompat spreadsheet right... but worth asking (more eyes don't hurt). =]

#009 livedoor.jp  needs an android version 4 or 5 ua token (shell: not sure if UA token same as string or different?)
#54 * www.aeon.co.jp Fixed by Android version number in UA
#73.0 navitime.co.jp Needs Android version # + "AppleWebKit" to get to touch site
#73.1 * www.navitime.co.jp mobile at http://touch.navitime.co.jp. "AppleWebKit" in UA is minimal requirement to get 302 to touch site.
Flags: needinfo?(miket)
Flags: needinfo?(kdubost)
Shell,

To note the *deployed* status on UA override, see this file.
https://hg.mozilla.org/mozilla-central/raw-file/tip/mobile/android/app/ua-update.json.in

:Sebastian recently identified an issue for clean install of the device.
See Bug 1178760 and Bug 1179300
and Mike's email https://groups.google.com/forum/#!topic/mozilla.compatibility/RA0-_sRSWHY

The comment in the spreadsheet is what it needs to work for each of these sites. 

Hopefully everything will be fixed for next nightly and if not we can still probably UA override for demo on the client side.

Does it answer partly your questions? Do not hesitate to ask more.
Flags: needinfo?(kdubost)
Shell,

ok double checked today.
* Livedoor. no need for UA override. Just unprefixing.
* aeon. no need for UA override. Just unprefixing.
* navitime. NEED UA override + unprefixing

Does it help? ^_^
Flags: needinfo?(sescalante)
Yes, thanks Shell. I had missed Navitime--the rest you showed are covered by us adding Android version to the UA string by default. I'll write a status update on overrides in my report this afternoon.
yes, super helpful. thanks guys!
Flags: needinfo?(sescalante)
tracking-fennec: ? → 42+
I just noticed spoofing as Chrome for www.rakuten.co.jp, we get the mobile layout--but it needs to be whitelisted for the unprefixing service. I'll spin out a bug to handle whitelisting and ua spoofing at the same time.
Also to note: the redirection for www.navitime.co.jp -> touch.navitime.co.jp has been fixed (https://webcompat.com/issues/944), so no need for an override.
Karl, we have to add overrides for both www.yahoo.co.jp and m.yahoo.co.jp to avoid an infinite redirect loop. Given that we're also sending overrides for 7 other yahoo.jp domains... I wonder if we should just override all subdomains? i.e., "yahoo.co.jp": "chrome"? WDYT?
Attachment #8634868 - Flags: feedback?(kdubost)
Note to self: let's not land this patch until 1184320 has landed. Might need to clean up bitrot, but will be simple.
On second thought, being more verbose is probably the right thing to do here--we haven't tested all yahoo.co.jp domains so who knows what the consequences might be.

Yahoo! is in the process of considering Fennec support, so these overrides are temporary anyways.

Note, this patch was rebased against Bug 1184320, so that has to land first.
Attachment #8634868 - Attachment is obsolete: true
Attachment #8634868 - Flags: feedback?(kdubost)
Attachment #8635411 - Flags: review?(s.kaspari)
Attachment #8635411 - Flags: review?(s.kaspari) → review+
1184320 is on fx-team now, this is safe to land now.

Sheriffs, just adding some UA overrides. Builds fine and works as expected--No try run needed.
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/a4d672467b9b
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 42
Have we considered that maybe the Chrome version should be Chrome/43.0.0000.00 instead of Chrome/43.0.2357.93 to more clearly identify as a spoofed user-agent?
(In reply to rikingcoding from comment #14)
> Have we considered that maybe the Chrome version should be
> Chrome/43.0.0000.00 instead of Chrome/43.0.2357.93 to more clearly identify
> as a spoofed user-agent?

What are the benefits to identifying as a spoofed UA, transparent or otherwise?
(In reply to Mike Taylor [:miketaylr] from comment #15)
> (In reply to rikingcoding from comment #14)
> > Have we considered that maybe the Chrome version should be
> > Chrome/43.0.0000.00 instead of Chrome/43.0.2357.93 to more clearly identify
> > as a spoofed user-agent?
> 
> What are the benefits to identifying as a spoofed UA, transparent or
> otherwise?

Probably to let know website owners and developers that you had no choice other than spoofing your User Agent to get the website to work. Because this is not the way it should be done.
This can maybe lead some of them to consider having a mobile version which is not just seeking for a User Agent with Webkit, as spoofing your UA should just be considered as a hack and a fix due to the website behavior with standard mobile platforms.
Clément,

Mike and I are working on Web Compatibility issues, but to give a bit more context.
https://webcompat.com/ and Tech Evangelism Mobile/Desktop components on Bugzilla.

(In reply to Clément Lefèvre from comment #16)
> Probably to let know website owners and developers that you had no choice
> other than spoofing your User Agent to get the website to work. Because this
> is not the way it should be done.

This is the positive part of the story. What you are advocating for here is education and that's cool. 
And it's the work we do in Web Compat. You are welcome to help on analyzing, contacting, etc. There's more on our plate than we can handle. See https://webcompat.com/contributors


The daily story is a bit darker. Basically when doing UA spoofing, we are trying to lie to the Web site for different reasons:

1. The Web developer/site owner just didn't realize and is eager to fix it. Best case scenario. We contact them through the work of Web Compatibility issues (when we can find a contact)

2. The Web developer knows the issue and is not willing to fix it.
   - too many things to do, not a priority
   - project manager, boss is not interested in fixing it (business priority, not billable)
   - client is not interested in fixing it (business priority, no budget for requesting a fix)
   - the issue is in a library the developer has no control on
   - the site is legacy and not maintained anymore

Firefox Android has very very little market to no market shares in different geographical areas. 

Order of priorities:
User > Webdev > Browser

UA spoofing is not about warning the webdev/site owners/client that the site is not well coded, it's about giving a good experience to users. So the Web Compatibility in this scenario tries to find out what is the best scenario for the users. We are not happy about these choices usually. Sad panda face. It's just that we have to do it in a way which deviates the minimum from what is expected by the scripts on the server. We are not talking to Webdevs but to machines through the UA. Note that Web developers will not even notice a UA which is slightly different in their logs.
Karl, what I was meaning to say is more that it can give some signals.
From what I can see from work, sometimes companies gather some stats about the platform of their clients, and for this purpose, they're looking at User Agent. So, having a meaningful part of faked UA to get the website to work for users on some platforms could raise some alerts and give them some informations.

About the little market share of Firefox for Android, I guess the problem can be exactly the same on Firefox OS, so that it can be valuable for there too.

Because that still to be a very sad point that some companies or people just ignore part of the user and just care about Android (by Chrome for Android, and no other browsers) and iOS for mobile, and sometimes only Webkit for desktop.
(In reply to Clément Lefèvre from comment #18)
> Karl, what I was meaning to say is more that it can give some signals.

Understood, and we want to avoid to send this signal for two reasons. Any new hook, you provide to external services will be used for filtering:

1. for working around browser bugs.
2. for blocking the browser because not supported by the service.

> From what I can see from work, sometimes companies gather some stats about
> the platform of their clients, and for this purpose, they're looking at User
> Agent.

Yes. It's why when possible, we fake for example a Firefox OS UA to be Firefox Android. It's not perfect but it makes it a bit better, but sometimes no luck for Firefox, we need to be Chrome. 

btw, we are slightly off-topic, feel free to contact me directly or to send an email on compatibility @ lists.mozilla.org
Blocks: 1301619
All these sites send the desktop version to Firefox Android. A simple Chrome UA sends the right version.
Only a couple of them requires -webkit- prefixes released in 49.

bylines.news.yahoo.co.jp
finance.yahoo.co.jp
blogs.yahoo.co.jp
address.yahoo.co.jp
carview.yahoo.co.jp
crowdsourcing.yahoo.co.jp
dir.yahoo.co.jp
fortune.yahoo.co.jp
gyao.yahoo.co.jp
job.yahoo.co.jp
logistics.yahoo.co.jp
lohaco.jp
movies.yahoo.co.jp
netallica.yahoo.co.jp
omiai.yahoo.co.jp
search.yahoo.co.jp
tickets.yahoo.co.jp
toku.yahoo.co.jp
toto.yahoo.co.jp
tv.yahoo.co.jp
wifi.yahoo.co.jp
xbrand.yahoo.co.jp
konkatsu.yahoo.co.jp
partner.yahoo.co.jp
secure.mobile.yahoo.co.jp/p/ycal/top
weather.yahoo.co.jp
sports.yahoo.co.jp
Karl, is Yahoo! JP not willing to fix their mobile detection? If not, do you recommend we send overrides for the list in Comment #20? (I recall you have/had a meeting with them soon)
Flags: needinfo?(kdubost)
(In reply to Mike Taylor [:miketaylr] from comment #21)
> Karl, is Yahoo! JP not willing to fix their mobile detection? If not, do you
> recommend we send overrides for the list in Comment #20? (I recall you
> have/had a meeting with them soon)

The comment is not a request to add the UA override. Just a status, because I tested ~100 Yahoo! Japan domains these last couple of days.

I do have a meeting on Monday for Yahoo! Japan www.yahoo.co.jp but all these sites are not managed by the same teams and departments. And they do not have the same logic for UA detection or building the site. 
AT the meeting on Monday, my plan is to ask them help for outreach to the rest of the company and then figure out what is the best strategy for UA override. :)
Flags: needinfo?(kdubost)
Ah OK. I misunderstood -- sounds good!
Blocks: 1314214
See Also: → 1505722
Whiteboard: [compat] → [compat][sitepatch-built]
You need to log in before you can comment on or make changes to this bug.