562107 - Provide mechanism to run l10n mozmill tests from m-c without a build

Status: RESOLVED INCOMPLETE

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[cmtalbert]

Ideally, a localizer will perform the following steps:
1. Clone m-c
2. Run a script which runs mozmill with l10n tests.

This bug is to track the creation of that script that gets run in step 2. 

Note this depends heavily on bug 516984 and bug 562106,

Comment 1

[Axel Hecht [:Pike]]

This sounds close to a project that Adrian is working on.

Comment 2

[Adrian Kalla [:adriank]]

(In reply to comment #1)
> This sounds close to a project that Adrian is working on.

The part about L10n tests itself - yes... But the part about running the tests without a build... I'm not so sure about, yet.


Clint: I see a flaw in that steps: the localizer needs to clone his L10n repository too, e.g.: http://hg.mozilla.org/l10n-central/pl

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Clint, can you please give more details about why we need that feature on hg? Eventually I have missed something.

Comment 4

[cmtalbert]

This came out of the crazy number of comments on the mozmill+buildbot bug.  It might not be an issue, it might be a dupe, I dunno.  

I think the goal here is to enable localizers to run localized mozmill tests without having to build firefox.

I think we can probably support this with Adrian's changes to Mozmill to improve our testing for L10N and with the test packaging work that is occurring in the mozmill + buildbot bug.

The resulting process would be something like:
1. <create lang pack for your localization>
or 
1a. clone m-c
1b. clone localization tree
1c. generate localized build or lang pack (I'm not sure how this works today)

2. Apply lang pack, or build your localized product

-- From here you have a localized product to test --
3. download the packaged mozmill harness and tests from the standard ftp site for your branch i.e. something like: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-1.9.2/firefox-3.6.5pre.en-US.win32.tests.zip
4. Unpack those tests to directory foo
5. cd foo/mozmill
6. Run the mozmill script with the localization tests (we can package a script that hides all the necessary command line fu for this)

Or, if you have a built tree:
3. cd into the objectdir
4. make mozmill-tests

We need the tests in hg because this makes it much much easier to do the packaging of the tests and the integration with "make".

Localizer does:
1. Clone 
I think really want we want here is a way to run packaged mozmill tests.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Adrian will not make any changes which will allow to run l10n test-runs. He is working on l10n tests only. But I can create a l10n test-run script which can be part of our cloud testing extension. Given by the above comments I strongly believe that all the steps will be completely covered by our extension.

Clint and Axel, please check the documentation I have put up here:
https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Cloud_Testing

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, and it will be much, much, much easier to run tests on localized builds and not only l10n tests but also normal/restart tests, update tests, and addon tests.

Comment 7

[cmtalbert]

(In reply to comment #5)
> Adrian will not make any changes which will allow to run l10n test-runs. He is
> working on l10n tests only. But I can create a l10n test-run script which can
> be part of our cloud testing extension. Given by the above comments I strongly
> believe that all the steps will be completely covered by our extension.
> 
> Clint and Axel, please check the documentation I have put up here:
> https://wiki.mozilla.org/QA/Mozmill_Test_Automation/Cloud_Testing

The idea is fine.  I have some serious concerns about the implementation details.  Where do I send feedback?  The QA Automation alias? the newsgroup? this bug? or the wiki page? :)

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Clint, you can use the mozmill-dev list. So far I haven't checked all the details for that project. I will work on the specs this and next week. So if you already have helpful feedback, it's really welcome!

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Clint and I talked about the cloud testing extension, means we were able to figure out those serious concerns. Beside that extension a test-run will be created in our mozmill-automation repository especially for l10n tests. This work is covered by bug 565196.

Clint, would it make that bug obsolete? If not we should find a way to port our automation scripts over to the packaged tests.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Move of Mozmill Test related project bugs to newly created components. You can
filter out those emails by using "Mozmill-Tests-to-MozillaQA" as criteria.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I still don't fully understand this bug. As what we have right now, any localizer can download a build from the FTP server and run the l10n tests against it. There is no need to build Firefox on your own. With one of the upcoming scripts I'm working on we will even be able to automate the download process. So you only have to specify the version, platform, and locale of the build and everything is automatically performed.

I will close this bug for now because I also never got feedback on my questions.