Closed Bug 920161 Opened 11 years ago Closed 11 years ago

Cloning of talos repo does not retry and/or output a TBPL compatible failure message ("command timed out: 3600 seconds without output, attempting to kill")

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: mozilla)

Details

(Keywords: intermittent-failure, sheriffing-P1)

Attachments

(1 file)

talos_timeout
13.38 KB, patch
jgriffin
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review
+++ This bug was initially created as a clone of Bug #920153 +++ The talos equivalent of bug 920153. eg: https://tbpl.mozilla.org/php/getParsedLog.php?id=27701351&tree=Mozilla-Inbound { 06:13:24 INFO - ##### 06:13:24 INFO - ##### Running clone-talos step. 06:13:24 INFO - ##### 06:13:24 INFO - Running pre-action listener: _resource_record_pre_action 06:13:24 INFO - Running main action method: clone_talos 06:13:24 INFO - Populating webroot /builds/slave/talos-slave/talos-data... 06:13:24 INFO - rmtree: /builds/slave/talos-slave/talos-data/talos 06:13:24 INFO - retry: Calling <function rmtree at 0xb6e6d454> with args: ('/builds/slave/talos-slave/talos-data/talos',), kwargs: {}, attempt #1 06:13:24 INFO - retry: Calling <bound method Talos._get_revision of <mozharness.mozilla.testing.talos.Talos object at 0xb7040b6c>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0xb6ea1a2c>, '/builds/slave/talos-slave/test/build/talos_repo'), kwargs: {}, attempt #1 06:13:24 INFO - Setting /builds/slave/talos-slave/test/build/talos_repo to http://hg.mozilla.org/build/talos revision ca2229a32cb6. 06:13:24 INFO - Cloning http://hg.mozilla.org/build/talos to /builds/slave/talos-slave/test/build/talos_repo. 06:13:24 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'http://hg.mozilla.org/build/talos', '/builds/slave/talos-slave/test/build/talos_repo'] 06:13:24 INFO - Copy/paste: hg --config ui.merge=internal:merge clone http://hg.mozilla.org/build/talos /builds/slave/talos-slave/test/build/talos_repo command timed out: 3600 seconds without output, attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=3658.177546 ========= Finished '/tools/buildbot/bin/python scripts/scripts/talos_script.py ...' failed (results: 2, elapsed: 1 hrs, 58 secs) (at 2013-09-11 07:13:24.901622) ========= } Expected: * Few retries of the hg clone * "Automation Error: Unable to clone talos repo" (and buildbot not having to kill the run).
Keywords: sheriffing-P1
Chris, is there someone that could take a look at this for us? :-)
Flags: needinfo?(catlee)
Attached patch talos_timeout — — Splinter Review
This patch: * allows for an "output_timeout" in the repo definition for MercurialVCS * fixes a bunch of pep8 in mozharness.mozilla.testing.talos * adds a hardcoded 1200 second talos clone timeout * adds a hardcoded mozprocess dependency; we now create the virtualenv before cloning talos, and then update the virtualenv with talos+pyyaml later. Got past the talos clone on ash with no concerns; the talos clone had an output timeout of 1200 as expected.
Assignee: nobody → aki
Attachment #8355690 - Flags: review?(jgriffin)
Flags: needinfo?(catlee)
Comment on attachment 8355690 [details] [diff] [review] talos_timeout Review of attachment 8355690 [details] [diff] [review]: ----------------------------------------------------------------- lgtm
Attachment #8355690 - Flags: review?(jgriffin) → review+
Armen: are you going to merge mozharness at some point this week, or should I?
Flags: needinfo?(armenzg)
Merged mozharness (not getting CCed to this bug). (In reply to Aki Sasaki [:aki] from comment #123) > Armen: are you going to merge mozharness at some point this week, or should > I? I had pushed new code to Cypress to make sure that default was in good shape. I had to do a lot of retries until it all looked green.
Flags: needinfo?(armenzg)
Thanks! This bug should be resolved for desktop talos. Android panda talos runs a separate workflow and may not be fixed yet, but looks to be the minority of issues posted here. I *think* I should resolve this bug, but could leave open/morph for Pandas. Do you have a preference?
Flags: needinfo?(emorley)
(In reply to Aki Sasaki [:aki] from comment #125) > I *think* I should resolve this bug, but could leave open/morph for Pandas. > Do you have a preference? Let's close this - very few of the failures were for the Pandas - we can always file another bug if needed. Thank you all for fixing this! :-D
Status: NEW → RESOLVED
Closed: 11 years ago
Flags: needinfo?(emorley)
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: