838325 - Intermittent clone mozharness failed with "command timed out: 1320 seconds without output, attempting to kill"

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Phil Ringnalda (:philor)]

+++ This bug was initially created as a clone of Bug #817513 +++

Brought to you by bug 817513 comment 82, and the need to have cloning mozharness do the same thing that cloning tools does, set RETRY on any failure.

https://tbpl.mozilla.org/php/getParsedLog.php?id=19456211&tree=Mozilla-Inbound
Rev4 MacOSX Snow Leopard 10.6 mozilla-inbound debug test marionette on
2013-02-05 08:37:41
slave: talos-r4-snow-067

command timed out: 1320 seconds without output, attempting to kill

Comment 1

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=19453941&tree=Cedar

Comment 2

[Aki Sasaki (not active)]

This can't retry in the standard sense, since there's no tools repo to use retry.py.

We are setting retry, but only if we match one of these regexes: http://hg.mozilla.org/build/buildbotcustom/file/21b0f0b5e41c/status/errors.py#l12

I think we need to increase the timeout.

Comment 3

[Aki Sasaki (not active)]

Attachment: increase ScriptFactory clone timeout

Comment 4

[Phil Ringnalda (:philor)]

It can't retry, but it can RETRY, exactly like bug 817513 does for cloning tools, with log_eval_func=rc_eval_func({0: SUCCESS, None: RETRY}). Sucks to be hg.m.o when we come RETRYing, but if it doesn't like it, it shouldn't time us out.

Comment 5

[Ed Morley [:emorley]]

Comment on attachment 710543 [details] [diff] [review]
increase ScriptFactory clone timeout

I really don't think a 1 hr timeout is the right thing to do here.

Comment 6

[Armen [:armenzg]]

Comment on attachment 710543 [details] [diff] [review]
increase ScriptFactory clone timeout

I agree with edmorley on this one. If it hasn't checked out in 22 minutes... In fact, I would make the timeout smaller and RETRY sooner.

We should RETRY the job (not the step) the way that philor says.

If would be great if we could use hgtool.py support (which can recover better) but that would probably be more work (tech debt).

Comment 7

[Aki Sasaki (not active)]

Attachment: retry

Comment 8

[Armen [:armenzg]]

Comment on attachment 710818 [details] [diff] [review]
retry

Review of attachment 710818 [details] [diff] [review]:
-----------------------------------------------------------------

This should do. Thanks.

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 710818 [details] [diff] [review]
retry

http://hg.mozilla.org/build/buildbotcustom/rev/13b67f5569f3

Comment 10

[Legacy TBPL/Treeherder Robot]

mikeratcliffe
https://tbpl.mozilla.org/php/getParsedLog.php?id=19500451&tree=Fx-Team
Rev4 MacOSX Snow Leopard 10.6 fx-team debug test mochitest-browser-chrome on 2013-02-06 10:30:13
slave: talos-r4-snow-040

command timed out: 1320 seconds without output, attempting to kill

Comment 11

[Phil Ringnalda (:philor)]

Comment 10 is just trying to mess with your head - it's actually a build/tools/ clone, and one that retried.

Comment 12

[Aki Sasaki (not active)]

In production.

Comment 13

[Legacy TBPL/Treeherder Robot]

reuben
https://tbpl.mozilla.org/php/getParsedLog.php?id=28540324&tree=Mozilla-Inbound
Ubuntu HW 12.04 mozilla-inbound talos other on 2013-09-29 18:58:55
revision: ac541c7e88b7
slave: talos-linux32-ix-049

command timed out: 1320 seconds without output, attempting to kill