[raptor] develop b2g-restart test

RESOLVED FIXED

Status

RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: rwood, Assigned: rwood)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(6 attachments, 1 obsolete attachment)

(Assignee)

Description

4 years ago
The existing raptor reboot test (boot_test.js) doesn't work on the b2g-emulator, because 'adb reboot' is not supported on the emulator. It is not possible to do a simulation of a hard reboot on the emulator.

I was considering on emulator doing a soft-reset (b2g restart) instead of a reboot. Not quite the same but I believe measuring the b2g process restart time would still provide significant value for an FXOS perf test (and just using it to establish a baseline to detect regressions; not as actual real reboot performance numbers).

Eli, I was thinking I would add a new method in /lib/device/index.js called "rebootEmulator", which would do the same as "reboot" with regards to markers etc. however insead of an 'adb reboot' it would just do a b2g restart (adb stop b2g && adb start b2g) and would do a 'waitForHomescreen" (instead of "adb wait-for-device since when just restarting b2g the emulator is always running).

Then in the /lib/suite/reboot.js itself, if options.emulator is set then invoke 'rebootEmulator' instead. 

What do you think?  Or should I create a /lib/emulator/index.js (although lots will be duplicate)? Or should I create a new perf test called 'reset_test', which would work on both device and emulator, and then we just won't run the existing 'boot_test' on the emulator? Thanks in advance for your thoughts!
Flags: needinfo?(eperelman)

Comment 1

4 years ago
I think adding a new suite phase for only restarting B2G is in order. It would function similar to the Reboot suite, but rather than doing the device reboot it would do the B2G restart. Most of the code for this to happen already exists spread through Reboot and ColdLaunch, so it may also be in order to push some of that into the base Phase for re-use. Then we can create a new test separate from reboot_test (e.g. b2g_restart_test) which would consume the new phase and be run by the simulator.

Does that sound reasonable?
Flags: needinfo?(eperelman)
(Assignee)

Comment 2

4 years ago
Yes that sounds great. I will put a patch together for this, but will wait until the MozDevice work (Bug 1115134) has landed.
(Assignee)

Updated

4 years ago
Summary: [raptor] reboot test doesn't work on b2g-emulator → [raptor] develop b2g-restart test
(Assignee)

Updated

4 years ago
Blocks: 1129096
(Assignee)

Comment 3

4 years ago
Created attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

Add restart-b2g phase to raptor.
Attachment #8559811 - Flags: review?(eperelman)
Created attachment 8559812 [details] [review]
[PullReq] rwood-moz:bug1121187 to mozilla-b2g:master
(Assignee)

Comment 5

4 years ago
Created attachment 8559813 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/27969

Add restart-b2g raptor test.
Attachment #8559813 - Flags: review?(eperelman)

Comment 6

4 years ago
Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

My main concern is just getting the time from the device rather than overriding with the time from the Node process. Mozdevice will need to be modified to fetch the device time during the B2G restart process.

Other than that, just a couple nits to address.
Attachment #8559811 - Flags: review?(eperelman) → review-

Updated

4 years ago
Attachment #8559812 - Attachment is obsolete: true

Updated

4 years ago
Attachment #8559813 - Flags: review?(eperelman) → review+
(Assignee)

Comment 7

4 years ago
Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

Thanks for the review Eli. PR updated, I'll squash the commits after review.
Attachment #8559811 - Flags: review- → review?(eperelman)
(Assignee)

Comment 8

4 years ago
Created attachment 8560539 [details] [review]
https://github.com/eliperelman/node-mozdevice/pull/5

Corresponding change to mozdevice, to return device time before b2g restart.
Attachment #8560539 - Flags: review?(eperelman)

Comment 9

4 years ago
Comment on attachment 8560539 [details] [review]
https://github.com/eliperelman/node-mozdevice/pull/5

Before merging this, could you also include a bump to the package.json to 1.2.2?
Attachment #8560539 - Flags: review?(eperelman) → review+
(Assignee)

Comment 10

4 years ago
(In reply to :Eli Perelman from comment #9)
> Comment on attachment 8560539 [details] [review]
> https://github.com/eliperelman/node-mozdevice/pull/5
> 
> Before merging this, could you also include a bump to the package.json to
> 1.2.2?

Right, I had that change locally but forgot to add it to my commit, d'oh, anyway thanks, done & merged:

https://github.com/eliperelman/node-mozdevice/commit/7eb661b20d6b14ccc71e939df02c82360eb0225a
(Assignee)

Comment 11

4 years ago
Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

PR updated as per our conversation on IRC (and your github gist for re-organizing phase.format).

Comment 12

4 years ago
Comment on attachment 8559811 [details] [review]
https://github.com/mozilla-b2g/raptor/pull/13

Awesome job on this. R+ with nits fixed.
Attachment #8559811 - Flags: review?(eperelman) → review+
(Assignee)

Comment 13

4 years ago
Thanks Eli. Rebased, nits fixed, squashed and merged:

https://github.com/mozilla-b2g/raptor/commit/ec63a747ae6d272caa4c174c143b25cbd8143a6b
Created attachment 8561452 [details] [review]
[PullReq] rwood-moz:raptor_update to mozilla-b2g:master
(Assignee)

Comment 15

4 years ago
Created attachment 8561454 [details] [review]
https://github.com/mozilla-b2g/gaia-node-modules/pull/91

Update gaia-node-modules.
Attachment #8561454 - Flags: review?(kgrandon)
Comment on attachment 8561454 [details] [review]
https://github.com/mozilla-b2g/gaia-node-modules/pull/91

Sounds good. Feel free to update the gaia revision as well.
Attachment #8561454 - Flags: review?(kgrandon) → review+
Created attachment 8561590 [details] [review]
[PullReq] rwood-moz:bug1121187 to mozilla-b2g:master
(Assignee)

Comment 19

4 years ago
(In reply to Autolander from comment #18)
> Created attachment 8561590 [details] [review]
> [PullReq] rwood-moz:bug1121187 to mozilla-b2g:master

Green try run. Carrying fw the R+ from Eli, and have R+ from Kevin. Merged:

https://github.com/mozilla-b2g/gaia/commit/0dd0a8c8d96969398e9e04da11327d54d2569935
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.