836588 - QA Review Geolocated Snippets

Status: RESOLVED FIXED

(Testing Graveyard :: WebQA, defect)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

== Description ==

Firefox OS will land on devices in specific countries at different times. This project will allow us to use the snippets area of about:home to notify users only in those countries about the availability of Firefox OS devices. 

https://wiki.mozilla.org/Websites/Snippets/Geolocated-Snippets

== Source Code ==

https://github.com/mozilla/snippets -- the code will live somewhere in that repo.

== Dev, Stage & Prod == 

TBD... wiki will include links.

== Bugzilla Info ==

Product / Component: Service / Snippets
CC: cmore, wbowden

== APIs & External Services ==

This project entails building a new service hosted by Mozilla that will serve geolocation requests.

== Mobile Support / Responsive Design? ==

No.

== SSL Support Required? ==

Yes.

== Traffic Expecations == 

66 million users / month 

== Support Level == 

QA for launch, but not ongoing.

== Timeline ==

Project is beginning now. QA begins 3/18/2013. Hard launch 4/2/2013.

Comment 1

[Justin Crawford [:hoosteeno] [:jcrawford]]

We had a quick meeting with :stephend to introduce the project. The following tests are proposed:


** service up
--> IT has bug 848499 for this.

** stress test (what load can we handle before it falls down)
*** anticipated peak: appx. 70m requests/day
--> :stephend will work with Mark Mayo's team on this

** no errors on about:home client (FF 4+) under various scenarios...
*** waiting on geo request
*** no snippets for my geo
*** geo server down
--> :stephend will connect a QA engineer with :mkelly to model and test these

Comment 2

[Osmose [:osmose, :mkelly]]

Here's a list of the main test cases we want to cover:

- Non-geolocated snippets should be shown to the user in all cases.

- If there is a geolocated snippet that matches the user's browser but does not match their country, it should never be shown.

- If there is a geolocated snippet that matches the user's browser and matches their country, it should be added to the set of snippets they may see.

- If the geolocation service is down and the user has no country value or an out-of-date country value in localStorage, they should only see non-geolocated snippets.

- If the geolocation service is down and the user has a valid country value in localStorage, they should see geolocated snippets if available.

- If the geolocation service is responding slowly and the user has no country value in localStorage, they should be shown a non-geolocated snippet immediately, and subsequent views on about:home should show geolocated snippets (if the user stayed on about:home long enough to get a response from the geolocation service).

- If the geolocation service sends an invalid or malformed response, the user should only see non-geolocated snippets without any issue.

I'll work with the QA assigned to this to figure out how to set up these conditions for testing.

Comment 3

[Justin Crawford [:hoosteeno] [:jcrawford]]

The new client-side code for Geolocated Snippets is now online. This means there is a working end-to-end system ready for review. I added instructions for doing so on the wiki:

https://wiki.mozilla.org/Websites/Snippets/Geolocated-Snippets#Testing_on_Staging

Comment 4

[Osmose [:osmose, :mkelly]]

I've added the test data we need to the staging server. 

During the steps outlined in Justin's linked directions above, you should alter the %NAME% value in the URL to geo, and the %VERSION% value to prod, resulting in:

https://snippets.stage.mozilla.com/%STARTPAGE_VERSION%/geo/prod/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/

This will send you the JS that is going to production, plus three snippets: one that is not geolocated, one geolocated to the US, and one geolocated to France. You should be able to use the Inspector in the devtools to see all 3 of these snippets on about:home, although only one will show at a time (and FR should never show if you're testing in the US).

To simulate the server being down, slow, or broken, you can change the %VERSION% string to one of the following:

- prod: Normal behavior
- serverdown: Simulates the server being down completely.
- serverslow: Simulates the server taking 3 seconds to load.
- badresponse: Simulates the server sending invalid data to the client.

So, to test cases when the service is down, you'd use the following URL:

https://snippets.stage.mozilla.com/%STARTPAGE_VERSION%/geo/serverdown/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/

Comment 5

[raymond [:retornam] (inactive)]

Thanks for info. I'll test and file any bugs I find with both of you cc'ed.

Comment 6

[raymond [:retornam] (inactive)]

Below are the results from my tests on stage:


1) https://snippets.stage.mozilla.com/%STARTPAGE_VERSION%/geo/prod/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/
 
* Using a France based proxy,  the FR snippet and the non-gelocated snippets were shown. The US one was not
* Using a US based IP,  the US sniper and the non-geolocated snippets were shown. The FR one was not
* The value for localStorage.geoCountry were correct for FR and US




2) https://snippets.stage.mozilla.com/%STARTPAGE_VERSION%/geo/serverdown/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/

* I saw a timed out request to http://doesnotexist.mozilla.org/country.js 
* I was shown the the non-geolocated snippet by default using  both US and FR IP addresses. Neither the US or FR snippets were shown after refreshing and resetting using the web-console
* The value for localStorage.geoCountry was blank for both FR and US


3)https://snippets.stage.mozilla.com/%STARTPAGE_VERSION%/geo/serverslow/%APPBUILDID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/

* I saw a request to http://geo.mozilla.org/country.js
* I was shown the non-geolocated snippet by default using both US and FR IP addresses. Neither the US for FR snippets were shown after refreshing and resetting using the web-console.
* The values for localStorage.geoCountry  were  correct for FR and US

Comment 7

[Osmose [:osmose, :mkelly]]

(In reply to raymond [:retornam] from comment #6)
> * I was shown the non-geolocated snippet by default using both US and FR IP
> addresses. Neither the US for FR snippets were shown after refreshing and
> resetting using the web-console.

Can you double-check this again? I can't replicate this with the serverslow version, and it SHOULD be showing you one of the geolocated snippets after you refresh if you wait for the country.js request to finish.

Also, what versions of Firefox are you testing on? We'd like to also make sure that snippets in general work (not necessarily geolocation, but that would be nice if it did) on Firefox 4 and maybe one or two versions in between that and the current version.

Comment 8

[raymond [:retornam] (inactive)]

(In reply to Michael Kelly [:mkelly] from comment #7)
> (In reply to raymond [:retornam] from comment #6)
> > * I was shown the non-geolocated snippet by default using both US and FR IP
> > addresses. Neither the US for FR snippets were shown after refreshing and
> > resetting using the web-console.
> 
> Can you double-check this again? I can't replicate this with the serverslow
> version, and it SHOULD be showing you one of the geolocated snippets after
> you refresh if you wait for the country.js request to finish.
> 
> Also, what versions of Firefox are you testing on? We'd like to also make
> sure that snippets in general work (not necessarily geolocation, but that
> would be nice if it did) on Firefox 4 and maybe one or two versions in
> between that and the current version.

I tried on Firefox 19.0.2 the first time.

I have tested this again on Firefox 19.0.2 and I now see the geo-located snippets. It however takes several refreshes, about 6-7 before I am presented  the geo-located snippet.


I'll test on older versions and update the bug after I am done.

Comment 9

[raymond [:retornam] (inactive)]

All my tests show that this is working as expected.

Comment 10

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to raymond [:retornam] from comment #9)
> All my tests show that this is working as expected.

Marking as fixed per this comment, since it appears QA has finished testing.  Thx, Raymond.