1015045 - [tarako] Add UA overrides for top Indian sites

Status: RESOLVED INVALID

(Web Compatibility :: Site Reports, defect)

Description

[Peipei Cheng (needinfo if you need my action)]

Attachment: Indian Top 10 Sites.txt

We need to override UA for top India sites regarding the release of tarako. Our goal is to create a recommended list of Indian sites for the UA whitelist.

There are 3 actions:

1. Smoke test each of these sites, preferably on B2G, with a spoofed UA to ensure that with the UA override the site appearance is improved and the site is mostly functional. Non primary function, like a vertically scrollable list of posters/images or function buried deep within the site, should not block a smoke test from passing.

2. Add the UA overrides to Gaia.

3. Open an evangelism bug for each site that is added to the UA override list in order to track the work required to remove a site from the list.

Legend for the attachment:
(Android UA) = This site requires the Android stock UA to get mobile content
(better content with Android UA) = This site receives mobile content with the Fennec UA but better mobile content with the Android stock UA
(nsfw) = not safe for work (this is an adult site)

Comment 1

[Peipei Cheng (needinfo if you need my action)]

We can do the same thing as following bugs for Tarako.
# Bug 823364 - Add UA overrides for top Colombia, Poland, Spain, and Venezuela sites
# Bug 819210 - Add UA overrides for top Brazil sites

Comment 2

[James Zhang (Spreadtrum)]

Can we add this code to sp6821a_gonk/dev-pref.js?

pref("general.useragent.override", "Mozilla/5.0 (Android; Mobile; rv:29.0) Gecko/29.0 FireFox/29.0")

Comment 3

[Peipei Cheng (needinfo if you need my action)]

(In reply to James Zhang from comment #2)
> Can we add this code to sp6821a_gonk/dev-pref.js?
> 
> pref("general.useragent.override", "Mozilla/5.0 (Android; Mobile; rv:29.0)
> Gecko/29.0 FireFox/29.0")

I think in this way we will override UA for the whole device. (correct me if Im wrong) This is too aggressive. And we could not collect tarako's information when user feedback or there is a crash report.

It would be better if we just override UA for top sites.

Comment 4

[James Zhang (Spreadtrum)]

We have verified TOP 10 websites in India, there're 4 websites have some difference between the behavior of FFOS and Android 6821.

S.No.

Websites

Results

Comments

1

http://www.google.co.in

Pass


2

https://m.facebook.com

fail

    Firefox show English(US) at bottom of page where as 6821A show English(India) Using Facebook. 
    Difference is very little but this can show there is some difference  , English(US) is shown at bottom of page where as 6821A show English(India) Using Facebook.

3

https://mobile.twitter.com

Pass


4

http://m.youtube.com

Pass


5

http://timesofindia.indiatimes.com

fail

    Firefox open Timesofindia web page where as 6821A open m.timesofindia.com when we enter "timesofindia.com". 
    Firefox is opening WEB page but Android opens WAP page there is a difference.

6

http://www.flipkart.com

Pass


7

https://accounts.google.com

Pass


8

https://rediff.com

fail

    Firefox open rediff.com web page where as 6821A open m.rediff.com. 
    Firefox is opening WEB page but Android opens WAP page there is a difference.  

9

https://m.yahoo.com

fail

    Firefox open m.yahoo.com page where as 6821A open in.yahoo.com. 
    When we enter "yahoo.com" in FFOS it opens m.yahoo.com page where as 6821A open in.yahoo.com.

10

https://touch.www.linkedin.com

Pass

Comment 5

[Steven Yang [:styang]]

Hi Lawrence, Could you help to check on this? thanks.

Comment 6

[Lawrence Mandel [:lmandel] (use needinfo)]

Please file a separate bug for each broken site and we will review each one in turn. FYI, our general process is to diagnose the issue and attempt to work with the site to fix the issue before adding an UA override. FYI, UA overrides are now updated via server side push and therefore no longer device dependent.

Comment 7

[Karl Dubost💡 :karlcow]

(In reply to Steven Yang [:styang] from comment #5)
> Hi Lawrence, Could you help to check on this? thanks.

As Lawrence, a separate bug for each site. 
Note also that some of your sites above are already part of sites which have been tested in the past.
You can check 

http://arewecompatibleyet.com/

and/or
https://wiki.mozilla.org/Compatibility/Countries

Comment 8

[Steven Yang [:styang]]

OK, as comment 2, our partner would like to override it to pass the test cases in the short term.

Hi Peipei, please create separate bugs for the sites reported by our partner if there isn't anyone addressing them. thanks.

Comment 9

[James Zhang (Spreadtrum)]

(In reply to Steven Yang [:styang] from comment #8)
> OK, as comment 2, our partner would like to override it to pass the test
> cases in the short term.
> 
> Hi Peipei, please create separate bugs for the sites reported by our partner
> if there isn't anyone addressing them. thanks.

Thanks Steven, I'll override useragent to pass the test cases in the short term and I'll remove it after Indian top 10 websites bug fixed.

Comment 10

[Abdul Rauf [:haseeb]]

FYI, rediff.com is looking into the issue and they will be fixing it soon, i can push little bit for timesofindia, will probably phone call them (mostly Tuesday)

Comment 11

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Steven Yang [:styang] from comment #8)
> OK, as comment 2, our partner would like to override it to pass the test
> cases in the short term.

I take it this means that the partner will override the UA locally. If this is the case, that's fine. If the request is to add overrides for these domains to the Mozilla UA override list, we should follow our defined process of diagnosing the each site's issues and contacting the site for a fix before adding an override.

Comment 12

[Steven Yang [:styang]]

(In reply to Lawrence Mandel [:lmandel] from comment #11)
> (In reply to Steven Yang [:styang] from comment #8)
> > OK, as comment 2, our partner would like to override it to pass the test
> > cases in the short term.
> 
> I take it this means that the partner will override the UA locally. If this
> is the case, that's fine. If the request is to add overrides for these
> domains to the Mozilla UA override list, we should follow our defined
> process of diagnosing the each site's issues and contacting the site for a
> fix before adding an override.

Yes, totally agreed. We should follow the defined process and let our partner override the UA locally. But will this modification, pref("general.useragent.override", "Mozilla/5.0 (Android; Mobile; rv:29.0) Gecko/29.0 FireFox/29.0"), override UA for ALL sites? I'm not sure what the impact will be.

Comment 13

[Karl Dubost💡 :karlcow]

(In reply to Steven Yang [:styang] from comment #12)
> Yes, totally agreed. We should follow the defined process and let our
> partner override the UA locally. But will this modification,
> pref("general.useragent.override", "Mozilla/5.0 (Android; Mobile; rv:29.0)
> Gecko/29.0 FireFox/29.0"), override UA for ALL sites? I'm not sure what the
> impact will be.


Steven,

Note that our project is to remove as much as possible the UA override and to limit the number of additions. UA override have a nasty effect at many levels:

1. They kill our marketshare stats (aka shooting in our own feet). One of the main arguments from Web sites to not fix their Web site for user agent sniffing is to pull out the "not enough marketshare" argument. By doing UA override, we identify the device as something else and indeed we do not appear in the marketshare stats.

2. Being served the wrong assets, libraries, content. For example, when we identify ourselves as Android, we are served the user experience for Mobile Android. It can work, but it also can be an additional failure. As the sites detecting Android often expect a WebKit rendering engine, so we get in return a site which is broken and where sometimes the desktop version was **more** usable than the mobile version.

See for background materials
https://wiki.mozilla.org/UA/override
https://wiki.mozilla.org/Evangelism/UA_Override_List_Policy
https://wiki.mozilla.org/Compatibility/Mobile/WipeOutUAOverides

The current list of domains with UA override is maintained at
https://hg.mozilla.org/mozilla-central/file/tip/b2g/app/ua-update.json.in
(You will see what is the format for the UA override).


I usually try to update the list at a regular pace. :)
see https://hg.mozilla.org/mozilla-central/log/e86a0d92d174/b2g/app/ua-update.json.in

Comment 14

[Karl Dubost💡 :karlcow]

**if** we do UA override for some of these sites, we will need to open separate bugs. A bug dedicated to the UA override for each site. The reason is that the current bugs are made for the issue of wrong user agent sniffing and specific Web compatibility issues. 

This bug also can be used to track the list of Web sites but not really for adding the UA override itself. Because when we negotiate with Web sites, we want to be able to individually close the UA override bugs.


I still need to understand the business context.  Were there efforts made by the local operator to contact each of these sites? If not, we can help on providing the right documentation for what they need to ask to the Web sites. I'm mentioning this because when the device was released for the Spanish market with Telefonica, Telefonica sent letters to big sites to help and encourage them to change their UA detection. It's all natural in the sense, that they have a better connection with the local market.

Comment 15

[Steven Yang [:styang]]

(In reply to Karl Dubost :karlcow from comment #14)
> **if** we do UA override for some of these sites, we will need to open
> separate bugs. A bug dedicated to the UA override for each site. The reason
> is that the current bugs are made for the issue of wrong user agent sniffing
> and specific Web compatibility issues. 
> 
> This bug also can be used to track the list of Web sites but not really for
> adding the UA override itself. Because when we negotiate with Web sites, we
> want to be able to individually close the UA override bugs.
> 
> 
> I still need to understand the business context.  Were there efforts made by
> the local operator to contact each of these sites? If not, we can help on
> providing the right documentation for what they need to ask to the Web
> sites. I'm mentioning this because when the device was released for the
> Spanish market with Telefonica, Telefonica sent letters to big sites to help
> and encourage them to change their UA detection. It's all natural in the
> sense, that they have a better connection with the local market.

Tarako is a project aiming at retail markets, not the case as TEF's. In the short term, we may need to handle it by ourselves.

Comment 16

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Steven Yang [:styang] from comment #15)
> Tarako is a project aiming at retail markets, not the case as TEF's. In the
> short term, we may need to handle it by ourselves.

Perhaps. Our first action is to try to resolve the issue with the site author.

Comment 17

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Steven Yang [:styang] from comment #12)
> Yes, totally agreed. We should follow the defined process and let our
> partner override the UA locally. But will this modification,
> pref("general.useragent.override", "Mozilla/5.0 (Android; Mobile; rv:29.0)
> Gecko/29.0 FireFox/29.0"), override UA for ALL sites? I'm not sure what the
> impact will be.

This will override the UA for all sites. However, this pref can be reset between tests if necessary.

Comment 18

[[:fabrice] Fabrice Desré]

Doing a global override is not a solution, as this may hide issues with other sites. Resetting the pref during field testing looks very impractical too.

Comment 19

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Fabrice Desré [:fabrice] from comment #18)
> Doing a global override is not a solution, as this may hide issues with
> other sites. Resetting the pref during field testing looks very impractical
> too.

Are you saying that the testing to be performed is manual testing? If so, what is the value in testing sites that are already known to have issues? Can you clarify what is being tested?

Comment 20

[Lawrence Mandel [:lmandel] (use needinfo)]

Karl - IIRC, we still have the ability to set domain specific UA overrides on the device. I seem to recall you including this capability in one of your scripts. Do you have a reference for how this is done?

Comment 21

[Karl Dubost💡 :karlcow]

Lawrence,

(In reply to Lawrence Mandel [:lmandel] from comment #20)
> Karl - IIRC, we still have the ability to set domain specific UA overrides
> on the device. I seem to recall you including this capability in one of your
> scripts. Do you have a reference for how this is done?

Last December, Jim Chen explained that:
https://groups.google.com/forum/#!msg/mozilla.compatibility/MrQQtOc7_x4/fUNzQvrgVXcJ

> Note that "general.useragent.updates.enabled" only applies to
> overrides through *updates*, overrides through old-style prefs are
> independent and still controlled by
> "general.useragent.site_specific_overrides". 

So we can still do the old way for testing.
The script for managing UA override (with the server side mechanism) is 
https://github.com/karlcow/webcompat/blob/master/moz/mozua2.sh

BUT I haven't yet implemented the site specific feature. It's not hard to do though. I was waiting for the moment we would really need it.  Created an issue.
People are welcome to create Pull Request before I do it
https://github.com/karlcow/webcompat/issues/3

Comment 22

[bhavana bajaj [:bajaj]]

we will provide input on individual bugs that are filed separately for each site, this looks like a meta bug, so clearing the nom on this.NI :james

Comment 23

[Karl Dubost💡 :karlcow]

(In reply to bhavana bajaj [:bajaj] from comment #22)
> we will provide input on individual bugs that are filed separately for each
> site, this looks like a meta bug, so clearing the nom on this.NI :james

Thanks a lot Bhavana! :)

Comment 24

[James Zhang (Spreadtrum)]

(In reply to bhavana bajaj [:bajaj] from comment #22)
> we will provide input on individual bugs that are filed separately for each
> site, this looks like a meta bug, so clearing the nom on this.NI :james

OK

Comment 25

[Karl Dubost💡 :karlcow]

I guess we can close this bug. It had no movement for the last 7 months.

Comment 26

[Hallvord R. M. Steen [:hallvors]]

I reviewed the blockers. The only one where spoofing might be an option is rediff.com, if we want to do that we should reopen bug 1016259 to discuss it there.