1060254 - Test failure 'Geolocation position is: unavailable' in /testGeolocation/testShareLocation.js

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect, P1)

Description

[Andrei Eftimie]

Module:    testVerifyDisplayGeolocationNotification
Test:      /testGeolocation/testShareLocation.js    
Failure:   Geolocation position is: unavailable
Branches:  mozilla-aurora
Platforms: Win

I can reproduce this on a CI Win machine, relevant log part:
> *** WIFI GEO: startup called.
> *** WIFI GEO: wifi error: 2147746065
> *** WIFI GEO: Use request cache:false reason:No cached data
> *** WIFI GEO: Sending request
> *** WIFI GEO: sending {}
> *** WIFI GEO: gls returned status: 404 --> {"error":{"errors":[{"domain":"geolocation","reason":"notFound","message":"Not Found"}],"code":404,"message":"Not Found"}}
> *** WIFI GEO: Use request cache:false reason:No cached data
> *** WIFI GEO: Sending request
> *** WIFI GEO: sending {}
> *** WIFI GEO: gls returned status: 404 --> {"error":{"errors":[{"domain":"geolocation","reason":"notFound","message":"Not Found"}],"code":404,"message":"Not Found"}}
> *** WIFI GEO: shutdown called
> ERROR | Test Failure | {
>   "exception": {
>     "message": "Geolocation position is: unavailable",
>     "lineNumber": 79,
>     "name": "Error",
>     "fileName": "file:///c:/Users/mozauto/Desktop/mozilla-aurora_functional/mozmill-tests/firefox/tests/functional/testGeolocation/testShareLocation.js"
>   }
> }

Comment 1

[Andrei Eftimie]

This only fails on CI machines.
Locally it passes.

This is the GeoLocation log with the same build as above, in a local Win VM:
> *** WIFI GEO: startup called.
> *** WIFI GEO: wifi error: 2147746065
> *** WIFI GEO: Use request cache:false reason:No cached data
> *** WIFI GEO: Sending request
> *** WIFI GEO: sending {}
> *** WIFI GEO: gls returned status: 200 --> {"location":{"lat":46.777248,"lng":23.59989},"accuracy":18000}

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Doug, any idea what this means?

> > *** WIFI GEO: Use request cache:false reason:No cached data
> > *** WIFI GEO: Sending request
> > *** WIFI GEO: sending {}
> > *** WIFI GEO: gls returned status: 404 --> {"error":{"errors":[{"domain":"geolocation","reason":"notFound","message":"Not Found"}],"code":404,"message":"Not Found"}}

Comment 3

[Andrei Eftimie]

This also works on CI OSX machines.
> *** WIFI GEO: startup called.
> *** WIFI GEO: Use request cache:false reason:No cached data
> *** WIFI GEO: Sending request
> *** WIFI GEO: sending {"wifiAccessPoints":[{"macAddress":"84-1b-5e-43-9d-9d","signalStrength":-45},{"macAddress":"b4-14-89-fe-0a-62","signalStrength":-62},{"macAddress":"b4-14-89-fe-0a-6e","signalStrength":-71},{"macAddress":"b4-14-89-fe-0a-6d","signalStrength":-71},{"macAddress":"b4-14-89-fe-0a-6f","signalStrength":-71},{"macAddress":"b4-14-89-d0-c6-5d","signalStrength":-84},{"macAddress":"b4-14-89-d0-c6-5e","signalStrength":-84},{"macAddress":"b4-14-89-d0-c6-5f","signalStrength":-85}]}
> *** WIFI GEO: gls returned status: 200 --> {"location":{"lat":37.3734338,"lng":-121.97302440000001},"accuracy":20}

So this issue seems to be restricted to zh-CN on Windows in our CI cluster.
(I'm checking other locales on Win ATM to be sure)

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #3)
> So this issue seems to be restricted to zh-CN on Windows in our CI cluster.

Mind re-running this test on another windows machine? Or are all affected?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

In the report url I see now that all Windows versions are affected.

Comment 6

[Andrei Eftimie]

This is not a localization issue.
All builds are affected (34, 33, 32).

Seems to be a network/proxy issue.

Comment 7

[Andrei Eftimie]

Attachment: skip.patch

Disabled patch for Windows only.

Comment 8

[Andrei Eftimie]

Applies ok to all branches except esr24.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So given that all branches are affected, we think there could be a network issue. Andrei, can you please attach a HTTP log for this failure?

Comment 10

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8481156 [details] [diff] [review]
skip.patch

Review of attachment 8481156 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.

Comment 11

[Andrei Eftimie]

Attachment: skip2.patch

Lets disable across platforms, since we've now seen this fail a couple of time on OSX as well.

http://mozmill-daily.blargon7.com/#/functional/report/2f56cb3a3728c2c47cd1b44f77e6847c

Comment 12

[Andrei Eftimie]

Disabled:

https://hg.mozilla.org/qa/mozmill-tests/rev/feb1f359b418 (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/981c2a2a1998 (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/ff6596fabdc3 (mozilla-beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/b78af808400a (mozilla-release)
https://hg.mozilla.org/qa/mozmill-tests/rev/77877695e92a (mozilla-esr31)

Comment 13

[Andrei Eftimie]

Attachment: http_log.txt.zip

Managed to get a log.

Note that this is pretty big (6MB compressed, 27MB deflated).
Search for "POST /geolocation/v1/geolocate" the second instance has the 404 a few hundred lines after.

I'm not sure what to look for.
We are using SPDY for this, but I can't make anything of the SPDY stream/log.  

I do remember we've had some issues last month in regards to SPDY (and scl proxy!?), but I can't find the issue for that.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #14)
> Some tldr from the log:
> > http request [
> >   POST /geolocation/v1/geolocate?key=XYZ HTTP/1.1
> >   Host: www.googleapis.com
> >   User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:35.0) Gecko/20100101 Firefox/35.0
> >   Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
> >   Accept-Language: en-US,en;q=0.5
> >   Accept-Encoding: gzip, deflate
> >   Content-Type: application/json; charset=UTF-8
> >   Content-Length: 2
> >   Connection: keep-alive
> >   Pragma: no-cache
> >   Cache-Control: no-cache
> > ]
> > 
> > [..]
> > 
> > http response [
> >   HTTP/1.1 404 Not Found
> >   Alternate-Protocol: 443:quic
> >   Cache-Control: private, max-age=0
> >   Content-Encoding: gzip
> >   Content-Length: 124
> >   Content-Type: application/json; charset=UTF-8
> >   Date: Fri, 05 Sep 2014 11:32:58 GMT
> >   Expires: Fri, 05 Sep 2014 11:32:58 GMT
> >   Server: GSE
> >   x-content-type-options: nosniff
> >   X-Frame-Options: SAMEORIGIN
> >   X-XSS-Protection: 1; mode=block
> >   X-Firefox-Spdy: 3.1
> > ]
> 
> Again, this only happens from SCL3, I can't reproduce this from another
> location.

Andrei, please try to not include the API key in any kind of comment. We shouldn't make it for others too easy to grab it. I will mark your former comment as private.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Doug, we really would like to get some feedback from you here! Or is there someone else we could ask regarding problems with geolocation? Anthony and Lawrence, has one of you an idea?

Comment 17

[Andrei Eftimie]

Just retested this on a couple CI Windows machines, and the issue appear to have been fixed.
We still don't know what caused us to receive 404 on geolocation requests.

I will reenable the test on Nightly shortly.

Comment 18

[Andrei Eftimie]

Unskipped:
https://hg.mozilla.org/qa/mozmill-tests/rev/3403585bb229 (default)

I will monitor the results, and if all is good by tomorrow, will unskip the rest of the branches.

Comment 19

[Andrei Eftimie]

Reenabled across branches (transplanted the backout):
https://hg.mozilla.org/qa/mozmill-tests/rev/18d4a531af5f (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/19fc9a19276a (mozilla-beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/1ae848c9ab8d (mozilla-release)
https://hg.mozilla.org/qa/mozmill-tests/rev/98f17de921b7 (mozilla-esr31)

This is not mozmill-tests related, but a 3rd party. Have not seen a single failure in the last 4 days since it was active on Nightly.

Lets hope the service is stable now, we still don't know what caused the outage.

I will mark this as WFM.

Comment 20

[Andrei Eftimie]

This failed over the weekend again... we have 177 failures in the past couple of days:
http://mozmill-daily.blargon7.com/#/functional/failure?app=Firefox&branch=All&platform=All&from=2014-09-27&test=%2FtestGeolocation%2FtestShareLocation.js&func=testVerifyDisplayGeolocationNotification

I will test if this still fails and probably re-disable the test.

Comment 21

[Andrei Eftimie]

A couple things to note:

1) I don't see we log the geolocation status anymore. I'm not aware we change anything in relation to this. This would previously display the 404 message. Without this information I am not sure if the current failure is the same as the one we had last week.

2) I can reproduce this from inside scl3, does not reproduce from another network.

I've disabled the test again (backed out previous unskip patch):
https://hg.mozilla.org/qa/mozmill-tests/rev/48f736c24d46 (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/f0ca1586d961 (mozilla-aurora)
https://hg.mozilla.org/qa/mozmill-tests/rev/4e6852f92331 (mozilla-beta)
https://hg.mozilla.org/qa/mozmill-tests/rev/7b61d996a5f7 (mozilla-release)
https://hg.mozilla.org/qa/mozmill-tests/rev/b11bebb332f6 (mozilla-esr31)

Doug you haven't replied with the initial issue which started 1 month ago.
It would really help to have some more information here.


===

On a sidenote I'm wondering if we can spoof the location and still have a relevant testcase?
We would still be testing the UI of the Share Location mechanism, but instead of relying on actual geolocation data, we would provide our own, similar to other geolocation tests we have.

For this we'd have to establish if this is still a relevant testcase for us, and if it is technically feasible.

Comment 22

[Andrei Eftimie]

So it's the same error we've seen before, we get a 404 response from the location service:
> *** WIFI GEO: startup called.
> *** WIFI GEO: wifi error: 2147746065
> *** WIFI GEO: Use request cache:false reason:No cached data
> *** WIFI GEO: Sending request
> *** WIFI GEO: sending {}
> *** WIFI GEO: gls returned status: 404 --> {"error":{"errors":[{"domain":"geolocation","reason":"notFound","message":"Not Found"}],"code":404,"message":"Not Found"}}
> *** WIFI GEO: Use request cache:false reason:No cached data
> *** WIFI GEO: Sending request
> *** WIFI GEO: sending {}
> *** WIFI GEO: gls returned status: 404 --> {"error":{"errors":[{"domain":"geolocation","reason":"notFound","message":"Not Found"}],"code":404,"message":"Not Found"}}

I've checked, and we can make use of the "geo.wifi.uri" to deliver the response locally as we've done in the `testAlwaysShareLocation.js` test.

I'm looking into patching all geoLocation tests to use this location thus averting any service or network disruption.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

When we patch this test do we still have a geolocation test, which tests a real geolocation service? That's something we totally need.

Also we should really get this problem investigated! Andrei, given that you can reproduce it on SCL3 hosts, please do a full HTTP log.

Comment 24

[Andrei Eftimie]

Attachment: fix.patch

I've applied the same logic we use in the sibling test `/remote/testGeolocation/testAlwaysShareLocation.js`

Here is a full run from inside SCL3:
http://mozmill-crowd.blargon7.com/#/functional/report/f11964362bbc85ebfbc5b36de7c5a2fb

====
> When we patch this test do we still have a geolocation test, which tests a real > geolocation service? That's something we totally need.
TL;DR: Not one that we really use.

Long response, we actually have 4 geolocation tests:
> functional/testGeolocation/testNotNowShareLocation.js
> functional/testGeolocation/testShareLocation.js
> remote/testGeolocation/testAlwaysShareLocation.js
> remote/testGeolocation/testNeverShareLocation.js

For the negative testcases we never actually check the location, because we are actively blocking the request by selecting "Not Now" or "Never" to the share options.

For the 2 positive tests "Share" and "Always Share" we are checking the location. "Always Share" is using a local JSON file in combination with `geo.wifi.uri`, and with the current patch we would also change the "Share" test to use the same local file.

We're still testing how Firefox behaves, but we're not actively testing any 3rd party service.

----
I see the benefit of the service being tested, but I think that's outside mozmill's scope. More so that AFAIK this is (currently) a google service, and the failure's we've recently been seeing seem to be network / location related.

As for the http log, we still have attachment 8484920 [details]

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #24)
> I see the benefit of the service being tested, but I think that's outside
> mozmill's scope. More so that AFAIK this is (currently) a google service,

What? Why is that outside of our scope!? That's exactly a test where Mozmill and our tests are getting created for! I totally don't want that people have to test this manually across all locales.

Comment 26

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #25)
> (In reply to Andrei Eftimie from comment #24)
> > I see the benefit of the service being tested, but I think that's outside
> > mozmill's scope. More so that AFAIK this is (currently) a google service,
> 
> What? Why is that outside of our scope!? That's exactly a test where Mozmill
> and our tests are getting created for! I totally don't want that people have
> to test this manually across all locales.

That is not what I meant.
I said that "I think mozmill's scope is not to test 3rd party web services."

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are not testing a 3rd party service, but our own integration here.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So I have taken a look and this test is part of the functional tests! With the external dependency it shouldn't have been added in this suite of tests. So a locally served position sounds fine to me, but then please check again that we really have a test, which retrieves the position remotely from the real service.

Comment 29

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #28)
> So I have taken a look and this test is part of the functional tests! With
> the external dependency it shouldn't have been added in this suite of tests.
> So a locally served position sounds fine to me, but then please check again
> that we really have a test, which retrieves the position remotely from the
> real service.

I've talked with Florin and there's are still some manual exploratory tests being run against the geolocation feature.

Maybe we could add and run a remote daily test against Nightly which scope is deep testing the actual service... but looking at this test, we would actually only duplicate it.

So better maybe just change this test to check the default geolocation service only on Nightly?
Not sure how pretty this would be...

Florin, how much value do we get in testing the actual response from the provider on Beta & Release builds? We would still be testing how Firefox responds to geolocation data, but the provider is actually a hardcoded local file.

Comment 30

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Andrei Eftimie from comment #29)
> I've talked with Florin and there's are still some manual exploratory tests
> being run against the geolocation feature.

Indeed, we have a test that implies using various web pages that make use of the geolocation services. We perform this testing on Beta pretty much every time we have at least 3-4 fixes that went in. We've included this in almost every one of the past 4-5 cycles.

> Maybe we could add and run a remote daily test against Nightly which scope
> is deep testing the actual service... but looking at this test, we would
> actually only duplicate it.
> 
> So better maybe just change this test to check the default geolocation
> service only on Nightly?
> Not sure how pretty this would be...
> 
> Florin, how much value do we get in testing the actual response from the
> provider on Beta & Release builds? We would still be testing how Firefox
> responds to geolocation data, but the provider is actually a hardcoded local
> file.

I would say it's good to keep at least one test to check for the response from a real provider. Having it run periodically on Nightly sounds fine to me. This way, we can still get a warning about any potential fallout and manually double check. 

With regards to using hardcoded data for the existing tests, I think that's fine as long as we don't shortcut the Firefox logic. In fact I'd say it's a good idea to have the tests not depend on any 3rd party, which may cause unwanted failures.

So to sum up, this sounds like a perfectly fine plan to me: existing tests using local data + 1 real life geolocation test (run regularly... e.g. daily on Nightly).

Comment 31

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8496772 [details] [diff] [review]
fix.patch

Review of attachment 8496772 [details] [diff] [review]:
-----------------------------------------------------------------

::: firefox/tests/functional/testGeolocation/testShareLocation.js
@@ +12,5 @@
>  var utils = require("../../../../lib/utils");
>  
>  const BASE_URL = collector.addHttpResource("../../../../data/");
> +const TEST_DATA = {
> +  "geoRequest_url" : BASE_URL + "geolocation/position.html",

I would simply call it url. We don't have any other one, which could raise confusion.

Comment 32

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8496772 [details] [diff] [review]
fix.patch

Review of attachment 8496772 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good to me with that url naming change and an update for the commit message (which test/location).

Comment 33

[Doug Turner (:dougt)]

garvan can help!

Comment 34

[:garvan]

I can try, not sure there are outstanding questions for the geo team, the latest patch uses local data, which makes sense if the CI machines can't reliably make network requests to google geolocation service. I am not clear on this bug thread as to why only this external request would fail, and not others (assuming that this isn't the only test that makes a request from an external server).

Very good to have an external test to make sure the google location service API key is valid and working. That could break mysteriously at any point.

Comment 35

[Andrei Eftimie]

Attachment: fix2.patch

To summarize:
- both functional geolocation tests are now local
- all previous geo tests are using local data (those 2 from the remote testrun still need to be remote as we need 2 separate domains, we might be able to make them run 100% local with mozmill 2.1 if we can spoof different local domains)
- added a copy of the testShareLocation (with geo logging enabled) that runs only on Nightly, and which still does a full 3rd party service testing. This is run on a daily basis.

Some reports:
http://mozmill-crowd.blargon7.com/#/remote/report/ad8dad905f3281cb521a23ddf311585b
http://mozmill-crowd.blargon7.com/#/functional/report/ad8dad905f3281cb521a23ddf3113bbd

Comment 36

[Andrei Eftimie]

Attachment: fix2.patch

*sigh* uploaded the wrong patch

Comment 37

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #35)
> - both functional geolocation tests are now local
> - all previous geo tests are using local data (those 2 from the remote
> testrun still need to be remote as we need 2 separate domains, we might be
> able to make them run 100% local with mozmill 2.1 if we can spoof different
> local domains)
> - added a copy of the testShareLocation (with geo logging enabled) that runs
> only on Nightly, and which still does a full 3rd party service testing. This
> is run on a daily basis.

Not sure if I missed something but where do we check for the official from Google now? Is there a remote test remaining? If not, has a bug been filed which covers such a check?

Comment 38

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #37)
> Not sure if I missed something but where do we check for the official from
> Google now? Is there a remote test remaining? If not, has a bug been filed
> which covers such a check?

attachment 8506034 [details] [diff] [review] adds a new test as:
firefox/tests/remote/testGeolocation/testShareLocation.js 
(which is basically the original functional ShareLocation test, with a check to only run it against Nightly and with Geolocation logging turned on).

This would be the only test that does a full/deep coverage of google's service.

Comment 39

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 8506034 [details] [diff] [review]
fix2.patch

Review of attachment 8506034 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, but this fails for me on OSX as in the last test I see the e10s popup which we close and we leave the geolocation popup unhandled.  On Linux it passes though, both with 2.0.8. I'd say we wait for the pref to be disabled in mozmill 2.0.9.

Comment 40

[Andrei Eftimie]

This could have been landed for a while now.

Pushed:
https://hg.mozilla.org/qa/mozmill-tests/rev/01c0dff583d8 (default)

Geolocation tests are still disabled under bug 1082590 on 36.
Here's a report: http://mozmill-crowd.blargon7.com/#/remote/report/43b0604030afed1afa485902876caf19

Comment 41

[Andrei Eftimie]

All green on 35 as well, transplanted:
https://hg.mozilla.org/qa/mozmill-tests/rev/07b75e93fa1d (mozilla-aurora)

http://mozmill-crowd.blargon7.com/#/remote/report/43b0604030afed1afa4859028788f9fa 

I'll check further backports the following days.

Comment 42

[Andrei Eftimie]

All ran fine on Aurora.

Beta is all green for me:
http://mozmill-crowd.blargon7.com/#/functional/report/43b0604030afed1afa48590287d320c5
http://mozmill-crowd.blargon7.com/#/remote/report/43b0604030afed1afa48590287d39f38

Pushed:
https://hg.mozilla.org/qa/mozmill-tests/rev/8b96a6f5929e (mozilla-beta)

Comment 43

[Andrei Eftimie]

All green on release:
http://mozmill-crowd.blargon7.com/#/functional/report/43b0604030afed1afa48590287d5c0d5
http://mozmill-crowd.blargon7.com/#/remote/report/43b0604030afed1afa48590287d622d3

Transplanted to Release as well:
https://hg.mozilla.org/qa/mozmill-tests/rev/9bfa4c5996b3

Comment 44

[Andrei Eftimie]

And transplanted to ESR31 as well:
https://hg.mozilla.org/qa/mozmill-tests/rev/99ba85df489a (mozilla-esr31)