1129637 - Change everything.me endpoint for 2.0 and above

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::Search, defect)

Description

[Peter Dolanjski [:pdol]]

Currently, 2.0 and above devices point directly to an Everything.me endpoint.  As we transition their services over to Mozilla, changing that endpoint for 2.0 and above to a Mozilla controlled one would mean we can avoid relying on a traffic redirection solution from Everything.me in the future.

The endpoint should be updated from api.everything.me to appsearch.services.mozilla.com.

Until Mozilla hosts the new service, the server will actually redirect to api.everything.me.  

NOTE: Before this is landed, it needs to be tested to make sure the endpoint redirection is working like expected.

Comment 1

[Kevin Grandon :kgrandon]

I'll submit a pull request for each branch (2.0, 2.1, 2.2, and 3.0), and we can test each one if desired before landing.

Comment 2

[Kevin Grandon :kgrandon]

Attachment: Github pull request to 3.0

Comment 3

[Kevin Grandon :kgrandon]

Attachment: Github pull request to 2.2

Comment 4

[Kevin Grandon :kgrandon]

Attachment: Github pull request to 2.1

Comment 5

[Kevin Grandon :kgrandon]

Attachment: Github pull request to 2.0

Comment 6

[Peter Dolanjski [:pdol]]

I'm marking this as blocking because of the risk to the product of not making the change.  It is understood that it won't get to all users, but some is better than none.
Bhavana and I had an offline conversation about blocking.

Comment 7

[Kevin Grandon :kgrandon]

I've attached a few pull requests here which update this endpoint. The endpoint seems to work in the browser, but I'm getting some weird responses when interacting with the API on a device. I'm seeing the following response: {"errorCode":-9,"processingTime":0.71,"errorString":"Security Authentication denied"} 

Peter - My assumption is that this is coming back from the snapshot server. I think I probably need to reach out to some of the E.me guys. Are they working with us on bugzilla or should I talk to them offline?

Comment 8

[Kevin Grandon :kgrandon]

I tried looking for this error code within the dump of the snapshot server I have from Jan 6th, but could not find the string. I'm assuming that I either don't have the most up-to-date snapshot codebase, or that e.me is running something different.

Comment 9

[Peter Dolanjski [:pdol]]

(In reply to Kevin Grandon :kgrandon from comment #7)
> Peter - My assumption is that this is coming back from the snapshot server.
> I think I probably need to reach out to some of the E.me guys. Are they
> working with us on bugzilla or should I talk to them offline?

+NI on Ran, we can definitely communicate through the bug

Comment 10

[Ran Ben Aharon [:ranbena] EverythingMe]

It looks like it's being redirected to api.everything.me (live server) instead of fxos-stg.everything.me (snapshot server).

Comment 11

[Kevin Grandon :kgrandon]

Peter - it seems like the API redirection is incorrect. Who can investigate this for us?

Comment 12

[Peter Dolanjski [:pdol]]

Sorry for the confusion guys.  This SHOULD be redirecting to the live server for now since once verified we'll ask partners to take these patches.  By the time they apply them, I expect that api.everything.me will actually be redirecting to the snapshot server.  If they change it before that, we want it going against the live server.  Does that make sense?

Comment 13

[Kevin Grandon :kgrandon]

Peter - this works, but it also means that we can't land the patches due to the authentication failures. We need to wait until the redirection is done to land it. I think this is generally ok, but it means that if some devices ship before we uplift this, then they might not get the patch. I'm not sure what kind of release schedule we're looking at coming up though.

If this can wait to land until the redirection is done, then we should be fine here.

Comment 14

[Peter Dolanjski [:pdol]]

Why are there authentication errors on the live server but not the snapshot?
Ran, I don't know how you'll be redirecting traffic once the snapshot is live.  Will we hit this same issue at that point?

Comment 15

[Ran Ben Aharon [:ranbena] EverythingMe]

The live server (for 2.x) has only one auth requirement - apiKey parameter. The snapshot server has none.

I tested your patch and found that your redirection omits the original query string parameters from the http request, leading to missing apiKey param, and therefore an auth error.

Comment 16

[Kevin Grandon :kgrandon]

Seems like we need to update our redirect to include the apiKey, or wait until the transition is done.

Comment 17

[Ran Ben Aharon [:ranbena] EverythingMe]

(All querystring params are omitted by the redirection and all are crucial)

Comment 18

[Peter Dolanjski [:pdol]]

(In reply to Kevin Grandon :kgrandon from comment #16)
> Seems like we need to update our redirect to include the apiKey, or wait
> until the transition is done.

If we can do this before the transition that would be ideal since there are devices that are launching regularly, so the fewer that need to go through the redirection back to Mozilla (after the transition) the better.

Comment 19

[Kevin Grandon :kgrandon]

(In reply to Peter Dolanjski [:pdol] from comment #18)
> If we can do this before the transition that would be ideal since there are
> devices that are launching regularly, so the fewer that need to go through
> the redirection back to Mozilla (after the transition) the better.

We certainly can, but I guess whoever wrote the redirect needs to do that. I'm not sure who that is - peter can you follow-up with them?

Comment 20

[Peter Dolanjski [:pdol]]

Travis, can your team look into why the redirect is omitting the original query string parameters from the http request?

Comment 21

[Kevin Grandon :kgrandon]

We may want to create a new bug for the querystring issue (or just throw this bug back at me once it's addressed). Thanks!

Comment 22

[Justin Lazaro [:jlaz] (use needinfo)]

The redirect should now preserve the original http request's query string parameters.  Kevin, please feel free to test this and let me know if you have any issues

Comment 23

[Kevin Grandon :kgrandon]

I seem to be seeing some kind of connection reset error when trying to access the redirect. It's also not working in a browser. The URL I'm trying to reach:

https://appsearch.services.mozilla.com/partners/1.0/Apps/search/?limit=24&categoryId=207&iconFormat=20&apiKey=79011a035b40ef3d7baeabc8f85b862f&deviceId=fxos-app://collection.gaiamobile.org/60dfd603-5658-46ce-bb15-1b8a7e3370e3&ctx={"lc":"en-US","tz":"-5","v":"3.0.0.0-prerelease","dn":"flame","ct":"wifi","sr":"480x853.5"}& 

Justin - any ideas?

Comment 24

[Kevin Grandon :kgrandon]

Seems the redirect is back up now. Unfortunately we are still seeing the security error when redirecting to the live server :(

{"errorCode":-9,"processingTime":0.8,"errorString":"Security Authentication denied"}

Comment 25

[Ran Ben Aharon [:ranbena] EverythingMe]

I assume it's because the params are actually sent via POST not GET in the client :/

Comment 26

[Kevin Grandon :kgrandon]

Thanks for the input Ran, that sounds very likely. It sounds like we need to wait to get a staging server set up somewhere that we can use if we can't wait to land this until the cutover date.

Comment 27

[Justin Lazaro [:jlaz] (use needinfo)]

Greetings Kevin,

Can we try to test again against appsearch.services.mozilla.com?  We've configured a nginx proxy_pass backend to handle the redirect while hopefully preserving the POST data.  Thanks!

Comment 28

[Kevin Grandon :kgrandon]

Woohoo! This is working nicely after the changes, thank you!

Peter - do you think we're good to land this? Do we need any QA signoff?

Comment 29

[Kevin Grandon :kgrandon]

Comment on attachment 8559373 [details] [review]
Github pull request to 3.0

Dale - could you give this a review when you have a minute? Thanks!

Comment 30

[Peter Dolanjski [:pdol]]

(In reply to Kevin Grandon :kgrandon from comment #28)
> Woohoo! This is working nicely after the changes, thank you!
> 
> Peter - do you think we're good to land this? Do we need any QA signoff?

Please don't land it until QA verifies it.
Naoki, can you verify this next week?

Comment 31

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

If e.me does the switch over to the stage server; wouldn't it be a redirect of a redirect?  I haven't tested that scenario, nor can I until next week.

I'll have to add this portion as part of my testing on monday as e.me makes the switch over.  I assume this is the reason why you suggested that I test next week instead of today.  Is that correct?

Comment 32

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Kevin, I did a search on everything.me on the repo:
https://github.com/mozilla-b2g/gaia/search?utf8=✓&q=everything.me&type=Code
https://github.com/mozilla-b2g/gaia/search?p=3&q=everythingme&type=Code&utf8=✓

I saw that
apps/homescreen/everything.me/config/config.js  has a few refs to the server, specifically the images.
Is that ok?

Comment 33

[Kevin Grandon :kgrandon]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #32)
> I saw that
> apps/homescreen/everything.me/config/config.js  has a few refs to the
> server, specifically the images.
> Is that ok?

In v2.0 and above we do not use the legacy home screen, so it should not be needed. This code would probably get updated in bug 1134250, which could land on master if we wanted.

Comment 34

[Kevin Grandon :kgrandon]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #31)
> If e.me does the switch over to the stage server; wouldn't it be a redirect
> of a redirect?  I haven't tested that scenario, nor can I until next week.
> 
> I'll have to add this portion as part of my testing on monday as e.me makes
> the switch over.  I assume this is the reason why you suggested that I test
> next week instead of today.  Is that correct?

Yes, but generally a double redirect would be fine. If redirects A -> C and B -> C are working, it's pretty unlikely that A -> B -> C would not work.

Comment 35

[Dale Harvey (:daleharvey)]

Comment on attachment 8559373 [details] [review]
Github pull request to 3.0

Yup looks good cheers

Comment 36

[Peter Dolanjski [:pdol]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #31)
> If e.me does the switch over to the stage server; wouldn't it be a redirect
> of a redirect?  I haven't tested that scenario, nor can I until next week.
> 
> I'll have to add this portion as part of my testing on monday as e.me makes
> the switch over.  I assume this is the reason why you suggested that I test
> next week instead of today.  Is that correct?

Exactly.  Now that the snapshot server is live, this test should give you the same results as without the patch.

Comment 37

[Kevin Grandon :kgrandon]

Comment on attachment 8559373 [details] [review]
Github pull request to 3.0

Naoki - flagging you for qa-approval here. Please set the flag whenever you think this is ready. Thanks!

Comment 38

[Kevin Grandon :kgrandon]

Comment on attachment 8559374 [details] [review]
Github pull request to 2.2

These are just uplifts so carrying the review.

Comment 39

[Kevin Grandon :kgrandon]

Ok, going to go ahead and get this into master as we've changed the implementation in master to remove web results from the search app and collections appear to be working. We can wait for the qa verification if needed before the uplift.

In master: https://github.com/mozilla-b2g/gaia/commit/7894b929f1b0394f3c997f72a6482bc7813e758d

Comment 40

[Kevin Grandon :kgrandon]

Comment on attachment 8559376 [details] [review]
Github pull request to 2.0

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): Everything.me search migration.
[User impact] if declined: Maybe the thing will stop working?
[Testing completed]: Manual testing (and existing tests passing)
[Risk to taking this patch] (and alternatives if risky): Low risk, just changing the endpoint.
[String changes made]: None.

Comment 41

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

The only thing I see is a slight performance degredation, which I'm not sure if it's the rendering of the new style or the redirects.

The other thing I noticed is that the search is based on language of the device versus the geolocation, which I guess is ok and doesn't have to do with this patch.

The other thing I also noticed is that Yahoo is not showing the Japanese version of the website.  Duckduckgo, google, and Bing show the Japanese version.  This also doesn't have to do with this patch.

Comment 42

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: 2015-02-26-16-45-22.png

I spoke too soon; I found that all the icons are generic for the games collection.

Comment 43

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

* Testing w/ SIM data connection only, the search is slow.  It takes at least 7 seconds to show results.

* Collection only showed WW English rather than US icons.

Comment 44

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Comment on attachment 8559373 [details] [review]
Github pull request to 3.0

I approve of the change for 3.0.  There's some performance issues with the redirect/redirect that I would like Peter to take a look at before ok'ing any other version.  If he's ok w/ it then we can proceed.

The initial Geolocation finding will also cause a slowness in the intial look at collections, showing blank icons at first.

Comment 45

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Please see comment 44 and image in comment 42.
Is the performance issue without a SIM card acceptable?

I think the redirect/redirect causes some performance degradation.

Comment 46

[bhavana bajaj [:bajaj]]

We need to verify this once it lands on 2.0/2.1, hence the Ni on Naoki. Kevin does the same pull request for 2.0 apply to 2.1 as well ?

Comment 47

[Kevin Grandon :kgrandon]

(In reply to bhavana bajaj [:bajaj] from comment #46)
> Kevin does the same pull request for 2.0 apply to 2.1 as well ?

Yes, I'll try to uplift.

v2.0: https://github.com/mozilla-b2g/gaia/commit/8eaa7ab64bc2c6fa882d6e16766f27ed89c86f34
v2.1: https://github.com/mozilla-b2g/gaia/commit/89db88a76ae8363cd7f8e7a07d35ae24058aec36

Looks like there is a conflict in 2.2 which I'll resolve.

Comment 48

[Kevin Grandon :kgrandon]

Rebased the v2.2 patch and will land once try results come in.

Comment 49

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I looked at the code change and there's low risk in terms of any regressions.  My main concern stems from the double redirect.

Having said that, I'll take the time to verify today.

Comment 50

[Kevin Grandon :kgrandon]

In v2.2: https://github.com/mozilla-b2g/gaia/commit/89af288bad6751248ff84504fa898206fee127fe

Comment 51

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

For some reason the collections won't launch when I built my own gaia build v2,2 on the mac.  Not sure why. investigating.

Comment 52

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Side note it's landed in all 3 branches:

landed in 2.2 :
https://github.com/mozilla-b2g/gaia/commit/89af288bad6751248ff84504fa898206fee127fe

landed in 2.1 : 
https://github.com/mozilla-b2g/gaia/commit/89db88a76ae8363cd7f8e7a07d35ae24058aec36

landed in 2.0:
https://github.com/mozilla-b2g/gaia/commit/8eaa7ab64bc2c6fa882d6e16766f27ed89c86f34

Comment 53

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

logcat shows:
d 'SIGNAL_POLL'
D/wpa_supplicant(  978): nl80211: survey data missing!
V/WLAN_PSA(  210): NL MSG, len[048], NL type[0x11] WNI type[0x5050] len[028]
E/HWComposer(  206): Non-uniform vsync interval: 7265406980
E/HWComposer(  206): Non-uniform vsync interval: 6556927
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.050]  Handling mozbrowserasyncscroll event...
I/GeckoDump(  206): [system] [HierarchyManager][78262.092] mozChromeEvent
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.147]  Handling mozbrowsermetachange event...
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.149] publishing internal event: themecolorchange
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.151] publishing external event: themecolorchange
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.168]  Handling mozbrowsermetachange event...
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.170] publishing internal event: themecolorchange
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.170] publishing external event: themecolorchange
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.178] publishing internal event: titlestatechanged
I/GeckoDump(  206): [system] [HomescreenWindow][Homescreen][homescreen][78262.179] publishing external event: titlestatechanged

Nothing changes on the device though.

Comment 54

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Reverting the patch caused the collection to open on my device.  I spoke with kevin and he says it works fine on his device.  Not sure the difference.  We both had flashed a v2.2 build and then made gaia on top.  

Still investigating v2.1 and v2.0 as I don't have those branches on my machine yet.

Comment 55

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

So I checked all three versions from the builds that we have on pvt.  The latest builds should have the gaia changes and they open up the collections just fine.  I'm not sure what the difference is.  I guess it has something to do with my repo.  I'm going to troubleshoot that on the side.

I did notice performance issues when clicking on links where it will appear all white screened for several seconds ( up to 10 seconds ) before switching to the website on a wifi connection on all three versions.

Comment 56

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I just realized using the Japan VPN, I'm getting WW again for the redirect.  The geolocation is done via IP.  Is the IP getting passed through?  103.17.199.77

The ll in the logcat is showing Tokyo for the URL passed:
E/GeckoConsole( 3295): Content JS LOG at app://collection.gaiamobile.org/gaia_build_defer_create.js:295 in log: evme API request: https://appsearch.services.mozilla.com/partners/1.0/Search/bgimage/?width=127&height=127&categoryId=349&_checksum=09177edd1dd0bcd126cea02a181bbf9a&apiKey=79011a035b40ef3d7baeabc8f85b862f&deviceId=fxos-app://collection.gaiamobile.org/bf681f71-ba4d-437c-ad8e-ca5985a8187f&ctx={"lc":"en-US","tz":"-5","v":"2.1.0.0-prerelease","dn":"flame","ct":"wifi","sr":"480x853.5","ll":"35.685,139.7514"}&

Comment 57

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: 2.1_logcat.txt

Comment 58

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

To note, currently : 
appsearch.services.mozilla.com	canonical name = appsearch.prod.mozaws.net.
Name:	appsearch.prod.mozaws.net
Address: 54.213.1.224
Name:	appsearch.prod.mozaws.net
Address: 54.213.18.52

Comment 59

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: 2.0_logcat.txt

WWE for 2.0 for a different reason, can't get geolocation address.  See logcat

Comment 60

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: 2.2_logcat.txt

2.2 also has broken geolocation.

Comment 61

[Kevin Grandon :kgrandon]

I thought we had disabled geolocation from 2.2 already.

Comment 62

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Doesn't the server side still need the ip to get geolocation for the requestor?  That's what I'm referring to in regards to geolocation.  Otherwise they get the default WWE content instead of the country content.
I didn't mean the client side geolocation.

Comment 63

[Justin Lazaro [:jlaz] (use needinfo)]

Greetings Naoki,

We've redeployed the nginx reverse proxy to us-east, as we discovered that everything.me's endpoint is actually hosted on AWS's us-east region.  Instead of the traffic having to travel from the user, to the current proxy (us-west), to its destination (us-east), then back to our proxy, the latency should go down.  

I haven't performed the DNS cutover yet so we can collect more performance data, but let me know when you are able to test via IRC

Comment 64

[Justin Lazaro [:jlaz] (use needinfo)]

I've switched the appsearch endpoint to our US-East production stack as there is now a much more improved response time.  Naoki, please feel free to retest the endpoint to see if there are still performance issues

Comment 65

[Josh Cheng [:josh]]

Hi Vincent, Hi Steven,

This is the bug which replacing e.Me and already land on 2.0M Woodduck.
Could you please help to land this on 2.1S Dolphine? Thanks!

Comment 66

[Jacob Shen]

Hi Steve, as we discussed today, could you kindly verify or plan for landing this change in 2.1S?

Comment 67

[Steven Yang [:styang]]

(In reply to jshen from comment #66)
> Hi Steve, as we discussed today, could you kindly verify or plan for landing
> this change in 2.1S?

the solution has been landed in 2.1S. ni? mike for the verification on 2.1S. thanks

Comment 68

[Mike Lien[:mlien]]

verified and fixed with v2.1S
Build ID               20150419161204
Gaia Revision          1f2df8c408f9cc23863ad0991459e1e47a877e57
Gaia Date              2015-04-14 21:52:01
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/rev/6e48d0ec7755
Gecko Version          34.0
Device Name            scx15_sp7715ea
Firmware(Release)      4.4.2

Below is the part of log:
E/GeckoConsole( 1253): Content JS LOG at app://collection.gaiamobile.org/gaia_build_defer_view.js:414 in log: evme API request: https://appsearch.services.mozilla.com/partners/1.0/Apps/search/?limit=24&categoryId=207&iconFormat=20&apiKey=79011a035b40ef3d7baeabc8f85b862f&deviceId=fxos-app://collection.gaiamobile.org/cddffa30-a8bd-4f57-a5aa-76f5778168fb&ctx={"lc":"en-US","tz":"8","v":"2.1.0.0-prerelease","dn":"SP7715A","ct":"wifi","sr":"480x853.5"}&