1082207 - QA experiment for es-ES/es-AR translation trial on beta

Status: VERIFIED FIXED

(Firefox :: Translations, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

+++ This bug was initially created as a clone of Bug #1041598 +++

Bug 1072741 added two new locales for the translation trial on beta: es-ES and es-AR (Spanish for Spain and Argentina). We should do the same QA for them as was done in bug 1041598 and then push then to production.

The only difference from the other experiments is that the two new ones have a minVersion set to FF33, and no minBuildID.

Comment 1

[:Felipe Gomes (needinfo for replies!)]

Kamil, are you interested in working on this one? :)

Comment 2

[Kamil Jozwiak [:kjozwiak]]

Sure Felipe, I'll start going through this sometime this week once we finish up signing off 31.2.0esr

Comment 3

[Kamil Jozwiak [:kjozwiak]]

minVersion testing: [PASSED]
===================

I used the following builds and ensured that the translation experiment was being installed as the minVersion conditions are met:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/33.0b9/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/34.0b1/

- 1413559672092 Browser.Experiments.Experiments DEBUG Experiments #0::evaluateExperiments() - activating experiment [e-mail]
- 1413560761084 Browser.Experiments.Experiments DEBUG Experiments #0::evaluateExperiments() - activating experiment [e-mail]

I used the following builds and ensured that the translation experiment was NOT being installed as the minVersion conditions are not met:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/32.0b9/

- 1413566859738 Browser.Experiments.Experiments DEBUG ExperimentEntry #1::isApplicable() - id=[e-mail] - test 'minVersion' failed
- 1413567230333 Browser.Experiments.Experiments DEBUG ExperimentEntry #0::isApplicable() - id=[e-mail] - test 'minVersion' failed

channel testing: [PASSED]
================

I used the following build and ensured that both experiments are not being installed due to being on the incorrect channel:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/

- 1413569947170 Browser.Experiments.Experiments DEBUG ExperimentEntry #1::isApplicable() - id=[e-mail] - test 'channel' failed
- 1413570414424 Browser.Experiments.Experiments DEBUG ExperimentEntry #0::isApplicable() - id=[e-mail] - test 'channel' failed

locale testing: [PASSED]
===============

I used the following build using several different locales and ensured that both translation experiments were not being installed:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/34.0b1

- 1413571035900 Browser.Experiments.Experiments DEBUG ExperimentEntry #1::isApplicable() - id=[e-mail] - test 'locale' failed
- 1413571229416 Browser.Experiments.Experiments DEBUG ExperimentEntry #0::isApplicable() - id=[e-mail] - test 'locale' failed

**** es-ES testing: ****

Went through the test cases listed below using the following build:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/34.0b1/mac/es-ES/

Websites used:
- http://en.wikipedia.org/wiki/Main_Page
- http://www.espnfc.us/

- ensured that the experiment stays active/enabled after restarting fx several times
- ensured extensions.bootstrappedAddons;{}
- ensured experiments.translation-beta33-es-ES.isTestGroup was created and set to either true/false
- ensured browser.translation.detectLanguage;true was created
- ensured browser.translation.ui.show matched the value of experiments.translation-beta32-pl.isTestGroup
- ensured that when experiments.translation-beta33-es-ES.isTestGroup;false, visiting a es-ES website doesn't launch the translation toolbar
- ensured that when experiments.translation-beta33-es-ES.isTestGroup;true, visiting a es-ES website show's the translation toolbar
- ensured that the translation toolbar worked without any issues (translated English into Spanish)
- ensured that the translation icon is available in the URL (selecting it slides in the translation toolbar)
- ensured that the translation icon appears grey when the translation is not being used on the website
- ensured that the translation icon appears blue when the translation is being used on the website
- ensured that you can go back between the original and translated text via the translation toolbar without any issues
- ensured that the experiment appears disabled under about:addons once it's removed/expired
- ensured that the experiment appears as disabled under about:support once it's removed/expired
- ensured that experiments.translation-beta33-es-ES.isTestGroup doesn't change values once the experiment has been removed/expired
- ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired
- ensured browser.translation.ui.show;false once the experiment has been removed/expired
- ensured that the experiment can't be installed the second time around once it's been removed/expired
- ensured that the experiment stays disabled when restarting fx

**** es-AR testing: ****

Went through the test cases listed below using the following build:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/34.0b1/mac/es-AR/

- ensured that the experiment stays active/enabled after restarting fx several times
- ensured extensions.bootstrappedAddons;{}
- ensured experiments.translation-beta33-es-AR.isTestGroup was created and set to either true/false
- ensured browser.translation.detectLanguage;true was created
- ensured browser.translation.ui.show matched the value of experiments.translation-beta32-pl.isTestGroup
- ensured that when experiments.translation-beta33-es-AR.isTestGroup;false, visiting a es-ES website doesn't launch the translation toolbar
- ensured that when experiments.translation-beta33-es-AR.isTestGroup;true, visiting a es-ES website show's the translation toolbar
- ensured that the translation toolbar worked without any issues (translated English into Spanish)
- ensured that the translation icon is available in the URL (selecting it slides in the translation toolbar)
- ensured that the translation icon appears grey when the translation is not being used on the website
- ensured that the translation icon appears blue when the translation is being used on the website
- ensured that you can go back between the original and translated text via the translation toolbar without any issues
- ensured that the experiment appears disabled under about:addons once it's removed/expired
- ensured that the experiment appears as disabled under about:support once it's removed/expired
- ensured that experiments.translation-beta33-es-AR.isTestGroup doesn't change values once the experiment has been removed/expired
- ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired
- ensured browser.translation.ui.show;false once the experiment has been removed/expired
- ensured that the experiment can't be installed the second time around once it's been removed/expired
- ensured that the experiment stays disabled when restarting fx

Possible issues:

Issue #1:

The following pref is being left behind once the experiment has been removed/expired: (just making sure this isn't a real issue)

- browser.translation.ui.welcomeMessageShown;true

Felipe, other than the possible issue above, everything looks good! Will there be a new bug for this being pushed into production? If so, I'll mark this as verified and do the spot check under production once it's pushed.

Comment 4

[:Felipe Gomes (needinfo for replies!)]

Thanks a lot Kamil! The pref being left is expected, as it's not part of the experiment itself but rather it's part of the shipped code.

There will be another bug to release to prod. I haven't filed it yet because we're still waiting for a final OK to release, but once we have it I'll file and CC you.

Comment 5

[Kamil Jozwiak [:kjozwiak]]

This has been pushed into production via bug # 1087039. I'll go through all of the test cases in comment #3 once more and ensure everything is working correctly under production. I'll update bug # 1087039 with the results once completed. Marking this as verified.