QA experiment for Vi/Tr/Po translation trial on beta

VERIFIED FIXED

Status

()

defect
VERIFIED FIXED
5 years ago
5 years ago

People

(Reporter: Felipe, Assigned: kjozwiak)

Tracking

32 Branch
Points:
---
Dependency tree / graph
Bug Flags:
firefox-backlog +

Firefox Tracking Flags

(firefox32 verified)

Details

Attachments

(1 attachment)

When bug 1035333 gets landed, we should QA the new translation experiment on the staging server and sign it off for a push to production.

We should ensure that the three locales correctly receive the experiment and that translation work on them.
OS: Mac OS X → All
Hardware: x86 → All
Version: Trunk → 32 Branch
I'll take this on as I did the QA work for the German translation experiment in bug # 1022411, this should be similar.
QA Contact: kamiljoz
QA Whiteboard: [qa+]
Flags: firefox-backlog+
Because the minimum version is set to fx 32.0 in the manifest which is still currently on Aurora, there's no way of testing each locale experiment unitl fx beta becomes fx32. In the mean time, I went through the following test cases:

Used the following staging server for the test cases below:
- https://telemetry-experiment-dev.allizom.org/firefox-manifest.json

Minimum BuildID Test Cases:

For all of the locales, I used the following build and ensured that none of the translation experiments are being installed:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b5/win32/pl/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b5/win32/tr/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b5/win32/vi/

- 1406042627426	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #2::isApplicable() - id=fx-translation-pl-beta32@mozilla.org - test 'minBuildID' failed
- 1406043217984	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #3::isApplicable() - id=fx-translation-tr-beta32@mozilla.org - test 'minBuildID' failed
- 1406043777541	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #4::isApplicable() - id=fx-translation-vi-beta32@mozilla.org - test 'minBuildID' failed

Channel Test Cases:

For all of the locales, I used the following build and ensured that none of the translation experiments are being installed:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora-l10n/firefox-33.0a2.pl.win32.installer.exe
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-07-22-03-02-01-mozilla-central-l10n/firefox-34.0a1.tr.win32.installer.exe
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-07-22-03-02-01-mozilla-central-l10n/firefox-34.0a1.vi.win32.installer.exe

- 1406044827446	Browser.Experiments.Experiments	TRACE	Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","fx-translation-pl-beta32@mozilla.org","channel"]
- 1406044827447	Browser.Experiments.Experiments	TRACE	Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","fx-translation-tr-beta32@mozilla.org","channel"]
- 1406045284487	Browser.Experiments.Experiments	TRACE	Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","fx-translation-vi-beta32@mozilla.org","channel"]

Locale Test Cases:

For all of the locales, I used the following build and ensured that none of the translation experiments are being installed:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/be/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/zh-TW/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/ro/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/kk/

- 1406047391222	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #2::isApplicable() - id=fx-translation-pl-beta32@mozilla.org - test 'locale' failed
- 1406047690068	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #3::isApplicable() - id=fx-translation-tr-beta32@mozilla.org - test 'locale' failed
- 1406047952019	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #4::isApplicable() - id=fx-translation-vi-beta32@mozilla.org - test 'locale' failed

Minimum Version Test Cases:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/pl/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/tr/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b9/win32/vi/

- 1406049104829	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #2::isApplicable() - id=fx-translation-pl-beta32@mozilla.org - test 'minVersion' failed
- 1406050147629	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #3::isApplicable() - id=fx-translation-tr-beta32@mozilla.org - test 'minVersion' failed
- 1406049935704	Browser.Experiments.Experiments	DEBUG	ExperimentEntry #4::isApplicable() - id=fx-translation-vi-beta32@mozilla.org - test 'minVersion' failed

For all the build test cases above, I also went through the following:

- ensured that the translation experiment wasn't appearing under about:addons
- ensured that the translation experiment wasn't appearing under about:support
- ensured that the translation experiment appeared as experiments.activeExperiment;false under about:config
- ensured that extensions.bootstrappedAddons;{}
- ensured that browser.translation.ui.show wasn't appearing or is set as browser.translation.ui.show;false
- ensured that the translation experiments are still not installed after fx is restarted

As mentioned above, the text portion of testing will be based on each experiment once beta merges into fx 32 (currently beta => fx 31)
Blocks: 1042184
Used the following as the staging server:
- https://telemetry-experiment-dev.allizom.org/firefox-manifest.json

**** pl locale testing: ****

Went through the test cases listed below using the following build:
- ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/32.0b1-candidates/build1/win32/pl/

Websites used:
- http://www.thenews.pl/
- http://www.warsawvoice.pl/WVpage/pages/index.php

- ensured that the experiment stays active/enabled after restarting fx several times
- ensured extensions.bootstrappedAddons;{}
- ensured experiments.translation-beta32-pl.isTestGroup was created and set to either true/false
- ensured browser.translation.detectLanguage;true was created
- ensured browser.translation.ui.show matched the value of experiments.translation-beta32-pl.isTestGroup
- ensured that when experiments.translation-beta32-pl.isTestGroup;false, visiting a pl website doesn't launch the translation toolbar
- ensured that when experiments.translation-beta32-pl.isTestGroup;true, visiting a pl website show's the translation toolbar
- ensured that the translation toolbar worked without any issues (translated English into Polish)
- ensured that the experiment appears disabled under about:addons once it's removed/expired
- ensured that the experiment appears as disabled under about:support once it's removed/expired
- ensured that experiments.translation-beta32-pl.isTestGroup doesn't change values once the experiment has been removed/expired
- ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired
- ensured browser.translation.ui.show;false once the experiment has been removed/expired
- ensured that the experiment can't be installed the second time around once it's been removed/expired
- ensured that the experiment stays disabled when restarting fx

**** tr locale testing: ****

Went through the test cases listed below using the following build:
- ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/32.0b1-candidates/build1/win32/tr/

Websites used:
- https://www.google.ca/search?q=Microsoft+Translate+tr&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:tr:official&client=firefox-beta&channel=sb&gfe_rd=cr&ei=qEnPU6D-J4SN8Qe42IGAAg

- ensured that the experiment stays active/enabled after restarting fx several times
- ensured extensions.bootstrappedAddons;{}
- ensured experiments.translation-beta32-tr.isTestGroup was created and set to either true/false
- ensured browser.translation.detectLanguage;true was created
- ensured browser.translation.ui.show matched the value of experiments.translation-beta32-tr.isTestGroup
- ensured that when experiments.translation-beta32-tr.isTestGroup;false, visiting a tr website doesn't launch the translation toolbar
- ensured that when experiments.translation-beta32-tr.isTestGroup;true, visiting a tr website show's the translation toolbar
- ensured that the translation toolbar worked without any issues (translated English into Turkish)
- ensured that the experiment appears disabled under about:addons once it's removed/expired
- ensured that the experiment appears as disabled under about:support once it's removed/expired
- ensured that experiments.translation-beta32-tr.isTestGroup doesn't change values once the experiment has been removed/expired
- ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired
- ensured browser.translation.ui.show;false once the experiment has been removed/expired
- ensured that the experiment can't be installed the second time around once it's been removed/expired
- ensured that the experiment stays disabled when restarting fx

**** vi locale testing: ****

Went through the test cases listed below using the following build:
- ftp://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/32.0b1-candidates/build1/win32/vi/

Websites used:
- http://www.vide.vi

- ensured that the experiment stays active/enabled after restarting fx several times
- ensured extensions.bootstrappedAddons;{}
- ensured experiments.translation-beta32-vi.isTestGroup was created and set to either true/false
- ensured browser.translation.detectLanguage;true was created
- ensured browser.translation.ui.show matched the value of experiments.translation-beta32-vi.isTestGroup
- ensured that when experiments.translation-beta32-vi.isTestGroup;false, visiting a vi website doesn't launch the translation toolbar
- ensured that when experiments.translation-beta32-vi.isTestGroup;true, visiting a vi website show's the translation toolbar
- ensured that the translation toolbar worked without any issues (translated English to Vietnamese)
- ensured that the experiment appears disabled under about:addons once it's removed/expired
- ensured that the experiment appears as disabled under about:support once it's removed/expired
- ensured that experiments.translation-beta32-vi.isTestGroup doesn't change values once the experiment has been removed/expired
- ensured browser.translation.detectLanguage;flase once the experiment has been removed/expired
- ensured browser.translation.ui.show;false once the experiment has been removed/expired
- ensured that the experiment can't be installed the second time around once it's been removed/expired
- ensured that the experiment stays disabled when restarting fx
Potential Issues:

Issue #1:

The translation toolbar won't appear when the experiment is initially installed, you need to completely restart fx, example (this affects pl, tr, vi):

- translation experiment has been installed and experiments.translation-beta32-pl.isTestGroup;true
- visit http://www.thenews.pl (translation toolbar will not appear no matter how many times you visit the website)
- restart fx
- visit the same website again and the translation toolbar will start appearing without any issues

I've reproduced this on all three locales using different websites. I'm not sure if this is expected behavior but I would personally expect the translation toolbar to work as soon as the experiment was installed without having to restart fx.

Issue #2: (attached image)

The "Experiment" title for the container under about:addons is not being translated under the vi locale. It's being translated in the other two locales (pl, tr). I'm not sure if this is because "Experiment" is difficult to translate in Vietnamese or perhaps it was just missed?
Flags: needinfo?(felipc)
Issue #1: upon installation, the translation toolbar should start appearing on every new tab created after installation. It's expected that it won't appear for existing tabs, even if you navigate to a different website on that tab. Can you verify that that's what you were seeing?

Issue #2: Yeah it's just missing strings, shouldn't be a problem for us since it's secondary UI. They did a sprint to translate all translation-related string but those ones were probably missed as they are not specific to translation and are part from the broader experiments system.
Flags: needinfo?(felipc)
(In reply to :Felipe Gomes (away Jul 23 - Aug 13) from comment #5)
> Issue #1: upon installation, the translation toolbar should start appearing
> on every new tab created after installation. It's expected that it won't
> appear for existing tabs, even if you navigate to a different website on
> that tab. Can you verify that that's what you were seeing?

Indeed that's what was happening. I went through the three locales once again and can confirm that the translation toolbar correctly slides into view when opening the target websites on new tabs/windows.

> Issue #2: Yeah it's just missing strings, shouldn't be a problem for us
> since it's secondary UI. They did a sprint to translate all
> translation-related string but those ones were probably missed as they are
> not specific to translation and are part from the broader experiments system.

Created bug #1042728 to address the missing translations in vi.

Felipe, I guess that's pretty much it! I'll start working on bug #1018200. Can you mark this as Resolved so I can close this as verified?
Flags: needinfo?(felipc)
Sure thing
Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(felipc)
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
QA Whiteboard: [qa+] → [qa!]
Assignee: nobody → kamiljoz
Iteration: --- → 34.1
Blocks: 1082207
You need to log in before you can comment on or make changes to this bug.