1141724 - mozharness retry logic doesn't display failures as they come in

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Michael Shal [:mshal]]

When using mozharness/base/script.py:retry(), if the function fails with an exception, that exception isn't displayed until all retries fail. This can be frustrating when the only output you get is this:

11:35:05     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027DC770>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #1
11:35:05     INFO - retry: Failed, sleeping 60 seconds before retrying
11:36:05     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027DC770>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #2
11:36:05     INFO - retry: Failed, sleeping 120 seconds before retrying
11:38:05     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027DC770>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #3
11:38:05     INFO - retry: Failed, sleeping 240 seconds before retrying

And you're stuck waiting for a long time waiting for an actual error message to tell you what's wrong. I think we can just log the exceptions as INFO to at least get some output.

Comment 1

[Michael Shal [:mshal]]

Attachment: retry-logging

With this patch the output looks like:

12:10:46     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027B0550>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #1
12:10:46     INFO - retry: attempt #1 caught exception: global name 'sys' is not defined
12:10:46     INFO - retry: Failed, sleeping 60 seconds before retrying

Being able to tell that I forgot 'import sys' is very handy :)

Comment 2

[Jordan Lund (:jlund)]

Comment on attachment 8575526 [details] [diff] [review]
retry-logging

Review of attachment 8575526 [details] [diff] [review]:
-----------------------------------------------------------------

handy indeed!

Comment 3

[Michael Shal [:mshal]]

Comment on attachment 8575526 [details] [diff] [review]
retry-logging

https://hg.mozilla.org/build/mozharness/rev/2452460c06fc

Comment 4

[Chris Cooper [:coop] (he/him)]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/d35bfd399151