1121187 - [raptor] develop b2g-restart test

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Robert Wood [:rwood]]

The existing raptor reboot test (boot_test.js) doesn't work on the b2g-emulator, because 'adb reboot' is not supported on the emulator. It is not possible to do a simulation of a hard reboot on the emulator.

I was considering on emulator doing a soft-reset (b2g restart) instead of a reboot. Not quite the same but I believe measuring the b2g process restart time would still provide significant value for an FXOS perf test (and just using it to establish a baseline to detect regressions; not as actual real reboot performance numbers).

Eli, I was thinking I would add a new method in /lib/device/index.js called "rebootEmulator", which would do the same as "reboot" with regards to markers etc. however insead of an 'adb reboot' it would just do a b2g restart (adb stop b2g && adb start b2g) and would do a 'waitForHomescreen" (instead of "adb wait-for-device since when just restarting b2g the emulator is always running).

Then in the /lib/suite/reboot.js itself, if options.emulator is set then invoke 'rebootEmulator' instead. 

What do you think?  Or should I create a /lib/emulator/index.js (although lots will be duplicate)? Or should I create a new perf test called 'reset_test', which would work on both device and emulator, and then we just won't run the existing 'boot_test' on the emulator? Thanks in advance for your thoughts!

Comment 1

[:Eli Perelman]

I think adding a new suite phase for only restarting B2G is in order. It would function similar to the Reboot suite, but rather than doing the device reboot it would do the B2G restart. Most of the code for this to happen already exists spread through Reboot and ColdLaunch, so it may also be in order to push some of that into the base Phase for re-use. Then we can create a new test separate from reboot_test (e.g. b2g_restart_test) which would consume the new phase and be run by the simulator.

Does that sound reasonable?

Comment 2

[Robert Wood [:rwood]]

Yes that sounds great. I will put a patch together for this, but will wait until the MozDevice work (Bug 1115134) has landed.

Comment 3

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/raptor/pull/13

Add restart-b2g phase to raptor.

Comment 4

[GitHub Autolander Bot]

Attachment: [PullReq] rwood-moz:bug1121187 to mozilla-b2g:master

Comment 5

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/gaia/pull/27969

Add restart-b2g raptor test.

Comment 6

[:Eli Perelman]

Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

My main concern is just getting the time from the device rather than overriding with the time from the Node process. Mozdevice will need to be modified to fetch the device time during the B2G restart process.

Other than that, just a couple nits to address.

Comment 7

[Robert Wood [:rwood]]

Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

Thanks for the review Eli. PR updated, I'll squash the commits after review.

Comment 8

[Robert Wood [:rwood]]

Attachment: https://github.com/eliperelman/node-mozdevice/pull/5

Corresponding change to mozdevice, to return device time before b2g restart.

Comment 9

[:Eli Perelman]

Comment on attachment 8560539 [details] [review]
https://github.com/eliperelman/node-mozdevice/pull/5

Before merging this, could you also include a bump to the package.json to 1.2.2?

Comment 10

[Robert Wood [:rwood]]

(In reply to :Eli Perelman from comment #9)
> Comment on attachment 8560539 [details] [review]
> https://github.com/eliperelman/node-mozdevice/pull/5
> 
> Before merging this, could you also include a bump to the package.json to
> 1.2.2?

Right, I had that change locally but forgot to add it to my commit, d'oh, anyway thanks, done & merged:

https://github.com/eliperelman/node-mozdevice/commit/7eb661b20d6b14ccc71e939df02c82360eb0225a

Comment 11

[Robert Wood [:rwood]]

Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

PR updated as per our conversation on IRC (and your github gist for re-organizing phase.format).

Comment 12

[:Eli Perelman]

Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

Awesome job on this. R+ with nits fixed.

Comment 13

[Robert Wood [:rwood]]

Thanks Eli. Rebased, nits fixed, squashed and merged:

https://github.com/mozilla-b2g/raptor/commit/ec63a747ae6d272caa4c174c143b25cbd8143a6b

Comment 14

[GitHub Autolander Bot]

Attachment: [PullReq] rwood-moz:raptor_update to mozilla-b2g:master

Comment 15

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/gaia-node-modules/pull/91

Update gaia-node-modules.

Comment 16

[Kevin Grandon :kgrandon]

Comment on attachment 8561454 [details] [review]
https://github.com/mozilla-b2g/gaia-node-modules/pull/91

Sounds good. Feel free to update the gaia revision as well.

Comment 17

[Robert Wood [:rwood]]

Thanks Kevin. Merged:

https://github.com/mozilla-b2g/gaia-node-modules/commit/25756ef8ae9cb1e9a0c9bc893612bb8334af3ccf

Comment 18

[GitHub Autolander Bot]

Attachment: [PullReq] rwood-moz:bug1121187 to mozilla-b2g:master

Comment 19

[Robert Wood [:rwood]]

(In reply to Autolander from comment #18)
> Created attachment 8561590 [details] [review]
> [PullReq] rwood-moz:bug1121187 to mozilla-b2g:master

Green try run. Carrying fw the R+ from Eli, and have R+ from Kevin. Merged:

https://github.com/mozilla-b2g/gaia/commit/0dd0a8c8d96969398e9e04da11327d54d2569935