1167683 - [Raptor] Failing tests cause the process to exit before reporting can finish

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

Raptor does a process.exit(1) if there are any errors in the test. This is done on the current turn of the event loop, before the reporting of any events can be completed. :(

Comment 1

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/40

Comment 2

[Robert Wood [:rwood]]

(In reply to :Eli Perelman from comment #1)
> Created attachment 8609539 [details] [review]
> Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/40

Hmmm... it just hangs after all the tests are done (and after the results table is displayed), and never exits. Tried launch test on flame-kk.

Comment 3

[Robert Wood [:rwood]]

Thanks for the update. Ok, so if I deliberately cause the launch test to fail (i.e. I manually open a different app before app priming, so priming will fail), this is the result:

https://gist.github.com/rwood-moz/7c50a43d0cc38e3130b8

Comment 4

[Robert Wood [:rwood]]

Comment on attachment 8609539 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/40

Latest update works. Tested with launch test on flame-kk.

- ran passing launch test
- ran launch test but manually launched an app so priming failed, raptor failed gracefully i.e. "suite aborted due to error"... then the error summary
- ran launch test and let the first run pass, then failed the 2nd by manually launching an app; raptor failed gracefully but wrote the raptor.log first with the values from the successfully completed run
- same test and verified raptor reported the first run to the DB/visualization

Comment 5

[GitHub Autolander Bot]

Attachment: https://github.com/mozilla-b2g/gaia-node-modules/pull/141

Comment 6

[:Eli Perelman]

Comment on attachment 8609608 [details] [review]
https://github.com/mozilla-b2g/gaia-node-modules/pull/141

Carrying over r+ for package bump.

Comment 7

[GitHub Autolander Bot]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30206

Comment 8

[:Eli Perelman]

Comment on attachment 8609609 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/30206

Carrying over r+ for package bump.

Comment 9

[:Eli Perelman]

In master: https://github.com/mozilla-b2g/gaia/commit/cbaaa1aa1c4521041ab37c28aef7c4648b8d6ae7
In npm: gaia-raptor@1.6.1