1081970 - Gracefully handle interrupts when running test suite

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Andreas Tolfsen ❲:ato❳]

When an interrupt signal (SIGINT) is sent whilst running the tests, it appears the Marionette test harness is being forcefully killed.  We should handle interrupts gracefully so that we can clean up and send test statistics to mach for output.

Current output:

  % ./mach marionette-test
  …
  ^?mach interrupted by signal or user action. Stopping.

Expected output:

  % ./mach marionette-test
  …
  SUMMARY
  -------
   0:16.93 LOG: MainThread INFO passed: 12
   0:16.93 LOG: MainThread INFO failed: 0
   0:16.93 LOG: MainThread INFO todo: 0
   0:16.94 SUITE_END: MainThread 
  Summary
  =======

  Ran 12 tests
  Expected results: 12
  Unexpected results: 0

  OK

I'm informed this needs to be handled on a per-harness level.

Comment 1

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

When debugging marionette tests, I often interrupt whole test suite by hitting Ctrl-C once a couple of tests failed. I usually got the following socket error if I re-run the test in a short period of time.

$ ./mach marionette-test layout/base/tests/marionette/test_touchcaret.py
Error running mach:

    ['marionette-test', 'layout/base/tests/marionette/test_touchcaret.py']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

MarionetteException: MarionetteException: localhost:2828 is unavailable.


  File "/Users/tlin/Projects/gecko-dev/testing/marionette/mach_commands.py", line 143, in run_marionette_test
    return run_marionette(tests, binary=binary, topsrcdir=self.topsrcdir, **kwargs)
  File "/Users/tlin/Projects/gecko-dev/testing/marionette/mach_commands.py", line 72, in run_marionette
    runner = startTestRunner(MarionetteTestRunner, options, tests)
  File "/Users/tlin/Projects/gecko-dev/testing/marionette/client/marionette/runtests.py", line 22, in startTestRunner
    runner.run_tests(tests)
  File "/Users/tlin/Projects/gecko-dev/testing/marionette/client/marionette/runner/base.py", line 754, in run_tests
    self.start_marionette()
  File "/Users/tlin/Projects/gecko-dev/testing/marionette/client/marionette/runner/base.py", line 699, in start_marionette
    self.marionette = Marionette(**self._build_kwargs())
  File "/Users/tlin/Projects/gecko-dev/testing/marionette/driver/marionette_driver/marionette.py", line 579, in __init__
    raise errors.MarionetteException(message=ex_msg)

Comment 3

[Julien Pagès (:parkouss)]

Attachment: 1081970.patch

It seems that this bug addresses two distinct things:

- stopping marionette with CTR-C should print the current tests status
- unable to restart the marionette immadiately after an interrupted run (also interrupted by CTRL-C) due to a socket error.

This patch only address the second issue. Maybe two distinct bugs should be created, as I don't think they are really related.

Comment 4

[Julien Pagès (:parkouss)]

Pushed to try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cfc2f9f8e9e

Comment 5

[Julien Pagès (:parkouss)]

So, not sure how/why but my previous try push is blocked somewhere (at least from a treeherder point of view).

I started a new one (exactly the same!) and this time it started as usual:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=db8eaa769e54

Comment 6

[Julien Pagès (:parkouss)]

All green now. :)

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/bae616b77f82

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/bae616b77f82

Comment 9

[Julien Pagès (:parkouss)]

Well I reopen this bug since we only handled one issue here - we still may want to fix what is explained in comment 0. (next time I should use the leave-open bug keyword instead of manual reopening!)

Comment 10

[Julien Pagès (:parkouss)]

Attachment: 1081970_2.patch

With this patch, running "./mach marionette-test" and hitting CTRL-C after some seconds prints:

> ...
>  0:08.16 TEST_START: MainThread test_import_script_reuse_window.py TestImportScriptContent.test_importing_script_then_reusing_it
> ^C 0:08.48 LOG: MainThread INFO 
> SUMMARY
> -------
>  0:08.48 LOG: MainThread INFO passed: 42
>  0:08.48 LOG: MainThread INFO failed: 0
>  0:08.48 LOG: MainThread INFO todo: 10
>  0:08.71 SUITE_END: MainThread 
> Summary
> =======
> 
> Ran 53 tests
> Expected results: 43
> Unexpected results: 0
> Skipped: 9
> 
> OK
> mach interrupted by signal or user action. Stopping.

Also return code is still 1, which is fine since it was interrupted.

Comment 11

[Julien Pagès (:parkouss)]

pushed to try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=73a96d8e09f0

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/acd406195e7c

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/acd406195e7c