995404 - Automated testing of Marketplace Catalog

Status: RESOLVED FIXED

(Tracking :: User Story, defect, P1)

Description

[Thomas Elin [:relaas]]

**Tracker bug for automated testing efforts of Marketplace Catalog.

    -       [Harald] Pull Marketplace Report w/ manifest URLS (this should also include App type and file size)
    -       [Jason] QA to provide a sample Marionette script as template
    -       [Tony] QA to provide links to Engineering builds for 1.3T
    -       [Harald-Jonathan?] Adjust Script and incorporate Marketplace manifest URL’s
    -       [Harald/Nicole] Begin testing on 2 Tarako devices
        o       Capture screenshot
        o       Capture ADB logcat
        o       Install, launch, screenshot, uninstall, repeat. On to next app.
        o       Export data using Marionette
    -       [QA TBD] Analyze results and attach PASS/FAIL for each app in XLS file

Timeline
    -       Marionette script testing to begin no later than 4/16
    -       Script testing to conclude no later than 4/23
    -       Results analyzed and delivered to Thomas Elin no later than 4/25

Comment 1

[:Harald Kirschner :digitarald]

Partner Engineering doesn’t have any prior experience in writing gaia UI tests or marionette scripts. I imagine the ATeam could do it in their sleep.

The basic pseudo code would look something like:

appManifests = loadJSONFile('appManifests.json');

while app in appManifests
	// Install either a hosted or packaged app from manifest
	app.install()
	// Wait for installation to finish
	waitOrTimeout()
	// Launch app
	app.launch()
	// Wait until app is launched or timeout
	waitOrTimeout()
	// Take and store screenshot
	screenshot()
	// Close app if needed
	app.close()
	// Might be a good idea to repeat launching and screenshot, to have better data
	// Cleanup: Uninstall app
	app.uninstall()

The end result is a set of screenshots stored per app and any logs we can pull off device associated with the test. QA mentioned that it would be good to reset gaia between tests; but I am not sure if that really helps and if it wouldn't complicate testing. Anything we can do to make the tests recover from app failures to run without intervention would help.

Comment 2

[Thomas Elin [:relaas]]

+1 on priority here. Automated testing is a crucial step in completing our Marketplace Catalog assessment for Tarako and subsequent device launch End of May. 

The timeline is already taxed so we really need to have this script completed by EOD Monday (latest Tuesday 4/15) so we can ensure the testing is complete and results analyzed by 4/25.

Comment 3

[Jonathan Griffin (:jgriffin)]

Malini will be working on this script Monday.

Comment 4

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Harald Kirschner :digitarald from comment #1)
> Partner Engineering doesn’t have any prior experience in writing gaia UI
> tests or marionette scripts. I imagine the ATeam could do it in their sleep.
> 
> The basic pseudo code would look something like:
> 
> appManifests = loadJSONFile('appManifests.json');

What exactly will the 'appManifests.json' file contain? Will it be a json file mapping app name to its webapp manifest? Like this example:

{"myApp1": {
  "name": "myApp1",
  "type": "hosted",
  "description": "Some app",
  "launch_path": "some/path.html",
  "developer": {
    "name": "Dev",
    "url": ""
  },
 "myApp2": ...
}

This may work for packaged app, but what about hosted apps? The links from https://developer.mozilla.org/en-US/Marketplace/Publishing/Publish_options#Hosted_apps are busted, so I don't konw what the manifest files look like or what a hosted app's files look like

Comment 5

[Malini Das [:mdas] - Away, not checking bugmail]

Forgot to n-i? you for the question above!

Comment 6

[Andrew Williamson [:eviljeff]]

(In reply to Malini Das [:mdas] from comment #4)
> The links from
> https://developer.mozilla.org/en-US/Marketplace/Publishing/
> Publish_options#Hosted_apps are busted, 

which links?

Comment 7

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Andrew Williamson [:eviljeff] from comment #6)
> (In reply to Malini Das [:mdas] from comment #4)
> > The links from
> > https://developer.mozilla.org/en-US/Marketplace/Publishing/
> > Publish_options#Hosted_apps are busted, 
> 
> which links?

I brought it up in #mdn and jms fixed the links. It's now fixed and points to https://developer.mozilla.org/en-US/Apps/Build/Manifest. I will be testing out hosted and packaged app deployment now.

Comment 8

[Malini Das [:mdas] - Away, not checking bugmail]

I need the following information about the apps and the test environment.

* Are we going to test a phone against the live marketplace?
* If not, can we assume that all the apps we're testing against can be served on a machine that's on the same wifi network as the phone? Does the test script have to start up this machine's webserver, or is there going to be one running?
* Are we testing both packaged and hosted apps, or just one or the other?

Lastly, for the manifest, I'd prefer a structure of {"appname1":"app manifest path/app zip", "appname2":...}, since I'll need the final name of the app to launch (it using our tools) and the path for installation purposes.

Comment 9

[Malini Das [:mdas] - Away, not checking bugmail]

As an update, I have the code for both cases: 

* https://github.com/malini/marketplace_tests/tree/remote_server holds the code for the remote hosting server (it's using my personal server as a server). This works right now, and it generates the screenshots and logcat output.

* https://github.com/malini/marketplace_tests holds the code for the locally hosted apps, and this doesn't work yet because I have yet to make the server respond with the correct headers, so we can't install the apps until I fix that. As soon as it is fixed, we should be ready.

I also have to test the packaged apps, right now I've only tested hosted apps.

Comment 10

[Malini Das [:mdas] - Away, not checking bugmail]

(In reply to Malini Das [:mdas] from comment #9)
> * https://github.com/malini/marketplace_tests holds the code for the locally
> hosted apps, and this doesn't work yet because I have yet to make the server
> respond with the correct headers, so we can't install the apps until I fix
> that. As soon as it is fixed, we should be ready.

Fixed it. I'd test a packaged app, but according to https://marketplace.firefox.com/developers/validator, I can't seem to get my app manifest UTF-8 encoded. If someone can pass me a valid packaged app, I can test with it, or at least debug what's wrong.

Comment 11

[Malini Das [:mdas] - Away, not checking bugmail]

From the meeting, we're going forward with using the live marketplace. I've updated https://github.com/malini/marketplace_tests to use the remote server code, and it now has instructions on how to run the tests.

Comment 12

[:Harald Kirschner :digitarald]

From the meeting we agreed that I provide the list of apps:

I extracted a JSON file of 3265 apps:
  https://gist.githubusercontent.com/digitarald/10759223/raw/715f94ec825142c1e6232b3b620a259c493707db/1000-apps.json
using a simple node script that queried the Marketplace API:
  https://gist.github.com/digitarald/10759223#file-fetch-js

Comment 13

[:Harald Kirschner :digitarald]

Attachment: apps-by-100.zip

App list for Tarako testing, split in groups of 100.

Comment 14

[Thomas Elin [:relaas]]

Closing thread as Automated testing has concluded.

Comment 15

[Caitlin Galimidi [:c8o]]

Nice work y'all!!! 
It will be great to leverage the automated testing capability for Tarako and going forward.