1159153 - Raptor tests for hosted apps

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Fernando Jiménez Moreno [:ferjm]]

The current Raptor implementation doesn't allow to run hosted app tests. Because we are moving from a packaged to a hosted apps model, we need to support this.

Comment 1

[:Eli Perelman]

I am working to get Marionette testing enabled within Raptor, and I think that would resolve being able to do this.

Comment 2

[:Eli Perelman]

With bug 1169775 resolved, you should be able to use the @mozilla/raptor package [1] to install and run tests against a hosted app that is installed on the device:

raptor test coldlaunch --app hostedapp.domain.com 

[1] https://developer.mozilla.org/en-US/Firefox_OS/Automated_testing/Raptor

Comment 3

[Wilson Page [:wilsonpage]]

Is it possible for Raptor to target packaged apps that don't live in Gaia repo?

Comment 4

[Oleg Zasypkin [:azasypkin]]

Ni?ing Robert as well, in case Eli doesn't monitor bugzilla while he's on PTO :)

Comment 5

[Robert Wood [:rwood]]

(In reply to Wilson Page [:wilsonpage] from comment #3)
> Is it possible for Raptor to target packaged apps that don't live in Gaia
> repo?

Yes, now that Raptor has marionette support, in theory this should work (although I'm not sure anyone has actually tried it yet). This of course is assuming that the app has the same performance markers as used in the gaia apps.

Comment 6

[Wilson Page [:wilsonpage]]

(In reply to Robert Wood [:rwood] from comment #5)
> (In reply to Wilson Page [:wilsonpage] from comment #3)
> > Is it possible for Raptor to target packaged apps that don't live in Gaia
> > repo?
> 
> Yes, now that Raptor has marionette support, in theory this should work
> (although I'm not sure anyone has actually tried it yet). This of course is
> assuming that the app has the same performance markers as used in the gaia
> apps.

Nice! How? Can you show us an example Raptor command to coldlaunch test a packaged app.

Comment 7

[Robert Wood [:rwood]]

So I just gave it a quick try. I installed an app from marketplace called "bubbleshoot". I looked at the app's manifest in the source code on github to get the appOrigin:

https://github.com/qliavi/bubble-shoot/blob/master/webapp-manifest/index.php

Then I was able to have the Raptor coldlaunch test launch the app using this command line:

raptor test coldlaunch --app bubble-shoot.qliavi.com

The installed app was tapped and launched by Raptor successfully. Then after the app launch, the test timed out as expected since the app doesn't contain the gaia performance markers.

Comment 8

[Wilson Page [:wilsonpage]]

I'm confused why the manifest.developer.url [1] would correspond to a hook for Raptor to launch an app on the device. There is also nothing in that manifest that contains the string 'bubble-shoot', so again, how can Raptor make the connection between `--app bubble-shoot.qliavi.com` and the correct installed app.

[1] https://github.com/qliavi/bubble-shoot/blob/master/webapp-manifest/index.php#L18

Comment 9

[Robert Wood [:rwood]]

Raptor uses Marionette to launch the app. The app origin/name and entrypoint (if provided) is included in the selector when marionette looks for the icon element on the homescreen, in order to get the app icon coordinates, so it can be tapped:

https://github.com/mozilla-b2g/raptor/blob/master/lib/phases/cold-launch.js#L158

Comment 10

[:Eli Perelman]

Rob is correct, and we use the app's domain for a few purposes:

- In order to lookup its location on the homescreen for fast launching performance
- To correspond performance entries with the target application
- Correspond processes with target applications

If you have suggestions on how this could be made smarter or simpler, feel free to send us your ideas. :)