1169788 - [Meta] Raptor architecture refactor

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

Lots of things that we want to be able to do, and need some changes made to Raptor in order to support them:

* Ability to write higher-level performance tests. We will utilize Marionette to make this happen.
* Performance test third-party apps. Should be doable with this refactor, but specifically we need to stop allowing multiple application runs in a single test (remove APPS), and make the app determination more generic (APP=directory instead of other identifier).
* Allow phases to specify test start and end using markers.
* Do away with device configurations, custom homescreens, and make targets (death to config.json, homescreens.json, and make raptor). We should be able to handle pretty much everything we need with Marionette.
* Memory capturing changes

Comment 1

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/eliperelman/mozilla-raptor/pull/1

Forgive me.

Comment 2

[Robert Wood [:rwood]]

I'm seeing a 'phase not defined' error when attempting the reboot test (on aries or flame)

Gist: https://gist.github.com/rwood-moz/5d4703f069c3b3ea99d2

Comment 3

[:Eli Perelman]

Rawr, my mistake. I'll have a fix for you in just a sec.

Comment 4

[Robert Wood [:rwood]]

Thanks! Adding a note re: not all marks showing up in the console output for the restart-b2g test:

https://gist.github.com/rwood-moz/bb74183873d31378f152

Comment 5

[Robert Wood [:rwood]]

If you attempt to run the coldlaunch test but don't provide a --app argument, raptor attempts to run the test but the times out at app priming:

https://gist.github.com/rwood-moz/728050e02f0f8d75e173

Suggestion: Have --app be a required argument when coldlaunch is chosen, tell user it's needed and don't try to run the test without it.

Comment 6

[:Eli Perelman]

(In reply to Robert Wood [:rwood] from comment #5)
> If you attempt to run the coldlaunch test but don't provide a --app
> argument, raptor attempts to run the test but the times out at app priming:
> 
> https://gist.github.com/rwood-moz/728050e02f0f8d75e173
> 
> Suggestion: Have --app be a required argument when coldlaunch is chosen,
> tell user it's needed and don't try to run the test without it.

Sounds good, I'll add that.

Comment 7

[Robert Wood [:rwood]]

(In reply to :Eli Perelman from comment #3)
> Rawr, my mistake. I'll have a fix for you in just a sec.

Confirmed reboot test runs now with your latest PR update, thanks

Comment 8

[Robert Wood [:rwood]]

Suite name should be 'restart', not 'reboot' when running the restart-b2g test:

https://gist.github.com/rwood-moz/4ebbec55e142126fe550

Comment 9

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #8)
> Suite name should be 'restart', not 'reboot' when running the restart-b2g
> test:
> 
> https://gist.github.com/rwood-moz/4ebbec55e142126fe550

Fixed with latest update, thx

Comment 10

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #4)
> Thanks! Adding a note re: not all marks showing up in the console output for
> the restart-b2g test:
> 
> https://gist.github.com/rwood-moz/bb74183873d31378f152

All marks now showing up in console output for restart-b2g

Comment 11

[Robert Wood [:rwood]]

(In reply to :Eli Perelman from comment #6)
> (In reply to Robert Wood [:rwood] from comment #5)
> > If you attempt to run the coldlaunch test but don't provide a --app
> > argument, raptor attempts to run the test but the times out at app priming:
> > 
> > https://gist.github.com/rwood-moz/728050e02f0f8d75e173
> > 
> > Suggestion: Have --app be a required argument when coldlaunch is chosen,
> > tell user it's needed and don't try to run the test without it.
> 
> Sounds good, I'll add that.

Works great, now reports error that --app is required with coldlaunch

Comment 12

[Robert Wood [:rwood]]

Cannot do multiple runs of the reboot test:

https://gist.github.com/rwood-moz/6f3ccd8ee7fafcc58d16

Comment 13

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #12)
> Cannot do multiple runs of the reboot test:
> 
> https://gist.github.com/rwood-moz/6f3ccd8ee7fafcc58d16

My mistake, I must have chose the wrong option on the device connection prompt on my mac after the aries reboot the first time; so my mac grabbed the usb connection away from my vm. Works fine now that I set the default usb connection to go to my linux vm.

Comment 14

[Robert Wood [:rwood]]

Here's an interesting one.

Run the reboot test, works fine. Then run the coldlaunch test, and it times out waiting for system/homescreen start - it sends the command to do the stop & start b2g however b2g is never actually restarted so the marks aren't received and raptor times out.

https://gist.github.com/rwood-moz/90dfb2a98d3847d07c68

Comment 15

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #14)
> Here's an interesting one.
> 
> Run the reboot test, works fine. Then run the coldlaunch test, and it times
> out waiting for system/homescreen start - it sends the command to do the
> stop & start b2g however b2g is never actually restarted so the marks aren't
> received and raptor times out.
> 
> https://gist.github.com/rwood-moz/90dfb2a98d3847d07c68

Interesting, ok I had to do an 'adb root' after the reboot test, and before running the coldlaunch test, and then running the coldlaunch test worked fine.

Comment 16

[Robert Wood [:rwood]]

(In reply to Robert Wood [:rwood] from comment #15)
> (In reply to Robert Wood [:rwood] from comment #14)
> > Here's an interesting one.
> > 
> > Run the reboot test, works fine. Then run the coldlaunch test, and it times
> > out waiting for system/homescreen start - it sends the command to do the
> > stop & start b2g however b2g is never actually restarted so the marks aren't
> > received and raptor times out.
> > 
> > https://gist.github.com/rwood-moz/90dfb2a98d3847d07c68
> 
> Interesting, ok I had to do an 'adb root' after the reboot test, and before
> running the coldlaunch test, and then running the coldlaunch test worked
> fine.

Solved by moving to an engineering build, now 'adb root' is not needed after the reboot test, thanks

Comment 17

[Robert Wood [:rwood]]

The --log option doesn't seem to work:

https://gist.github.com/rwood-moz/083d38a55790edfa3365

Comment 18

[Robert Wood [:rwood]]

Comment on attachment 8632274 [details] [review]
Link to Github pull-request: https://github.com/eliperelman/mozilla-raptor/pull/1

Ship it! Code looks great. Exercised it well with alot of tests (too many to list here) that included:

- coldlaunch and restart-b2g tests on aries, flame, and emulator-kk
- reboot test on aries and flame
- single and multiple iterations of each
- environment vars and command line options
- intentionally failing, retries
- targeting specific device when multiple attached
- posting to visualization

Comment 19

[:Eli Perelman]

Part 1 in master: https://github.com/eliperelman/mozilla-raptor/commit/3d6a772f7ea4b5950dd4c8ac3e5be8832967f554

Comment 20

[:Eli Perelman]

All other parts have landed. Yay!