mozharness retry logic doesn't display failures as they come in

RESOLVED FIXED

Status

Release Engineering
Applications: MozharnessCore
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: mshal, Assigned: mshal)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
When using mozharness/base/script.py:retry(), if the function fails with an exception, that exception isn't displayed until all retries fail. This can be frustrating when the only output you get is this:

11:35:05     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027DC770>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #1
11:35:05     INFO - retry: Failed, sleeping 60 seconds before retrying
11:36:05     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027DC770>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #2
11:36:05     INFO - retry: Failed, sleeping 120 seconds before retrying
11:38:05     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027DC770>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #3
11:38:05     INFO - retry: Failed, sleeping 240 seconds before retrying

And you're stuck waiting for a long time waiting for an actual error message to tell you what's wrong. I think we can just log the exceptions as INFO to at least get some output.
(Assignee)

Comment 1

3 years ago
Created attachment 8575526 [details] [diff] [review]
retry-logging

With this patch the output looks like:

12:10:46     INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.gittool.GittoolVCS object at 0x027B0550>, 'c:\\builds\\moz2_slave\\m-cen-w32_g-000000000000000000\\build\\src\\gaia'), kwargs: {}, attempt #1
12:10:46     INFO - retry: attempt #1 caught exception: global name 'sys' is not defined
12:10:46     INFO - retry: Failed, sleeping 60 seconds before retrying

Being able to tell that I forgot 'import sys' is very handy :)
Assignee: nobody → mshal
Attachment #8575526 - Flags: review?(jlund)

Comment 2

3 years ago
Comment on attachment 8575526 [details] [diff] [review]
retry-logging

Review of attachment 8575526 [details] [diff] [review]:
-----------------------------------------------------------------

handy indeed!
Attachment #8575526 - Flags: review?(jlund) → review+
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.