Closed Bug 940210 Opened 11 years ago Closed 11 years ago

Hazard Analysis builds need to set RETRY when failing to clone, rather than die with "Caught exception: Unable to checkout"

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: sfink)

Details

(Keywords: intermittent-failure)

Attachments

(2 files)

Retry failed checkout
5.38 KB, patch
mozilla
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
Purge first, then checkout tools
3.99 KB, patch
mozilla
: review+
sfink
: checked-in+
Details | Diff | Splinter Review 
https://tbpl.mozilla.org/php/getParsedLog.php?id=30741283&tree=Mozilla-Inbound
linux64-br-haz_mozilla-inbound_dep on 2013-11-18 20:38:16 PST for push a5a25f0e61ee
slave: bld-linux64-ec2-303

20:44:22     INFO - ##### Running checkout-source step.
20:44:22     INFO - #####
20:44:22     INFO - Running main action method: checkout_source
20:44:22     INFO - mkdir: /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:44:22     INFO - retry: Calling <bound method SpidermonkeyBuild._get_revision of <__main__.SpidermonkeyBuild object at 0x27c8dd0>> with args: (<mozharness.base.vcs.hgtool.HgtoolVCS object at 0x2710c90>, '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source'), kwargs: {}, attempt #1
20:44:22     INFO - Running command: ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:44:22     INFO - Copy/paste: tools/buildfarm/utils/hgtool.py -r a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4 --mirror http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound --bundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg http://hg.mozilla.org/integration/mozilla-inbound /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:44:22     INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared',
20:44:22     INFO -  'PATH': '/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin'}
20:44:22    ERROR - caught OS error 2: No such file or directory while running ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:44:22     INFO - rmtree: /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:44:22     INFO - retry: Calling <function rmtree at 0x23de230> with args: ('/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source',), kwargs: {}, attempt #1
20:44:22     INFO - retry: Failed, sleeping 60 seconds before retrying
20:45:22     INFO - retry: Calling <bound method SpidermonkeyBuild._get_revision of <__main__.SpidermonkeyBuild object at 0x27c8dd0>> with args: (<mozharness.base.vcs.hgtool.HgtoolVCS object at 0x2710c90>, '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source'), kwargs: {}, attempt #2
20:45:22     INFO - Running command: ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:45:22     INFO - Copy/paste: tools/buildfarm/utils/hgtool.py -r a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4 --mirror http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound --bundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg http://hg.mozilla.org/integration/mozilla-inbound /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:45:22     INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared',
20:45:22     INFO -  'PATH': '/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin'}
20:45:22    ERROR - caught OS error 2: No such file or directory while running ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:45:22     INFO - rmtree: /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:45:22     INFO - retry: Failed, sleeping 120 seconds before retrying
20:47:22     INFO - retry: Calling <bound method SpidermonkeyBuild._get_revision of <__main__.SpidermonkeyBuild object at 0x27c8dd0>> with args: (<mozharness.base.vcs.hgtool.HgtoolVCS object at 0x2710c90>, '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source'), kwargs: {}, attempt #3
20:47:22     INFO - Running command: ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:47:22     INFO - Copy/paste: tools/buildfarm/utils/hgtool.py -r a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4 --mirror http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound --bundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg http://hg.mozilla.org/integration/mozilla-inbound /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:47:22     INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared',
20:47:22     INFO -  'PATH': '/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin'}
20:47:22    ERROR - caught OS error 2: No such file or directory while running ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:47:22     INFO - rmtree: /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:47:22     INFO - retry: Failed, sleeping 240 seconds before retrying
20:51:22     INFO - retry: Calling <bound method SpidermonkeyBuild._get_revision of <__main__.SpidermonkeyBuild object at 0x27c8dd0>> with args: (<mozharness.base.vcs.hgtool.HgtoolVCS object at 0x2710c90>, '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source'), kwargs: {}, attempt #4
20:51:22     INFO - Running command: ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:51:22     INFO - Copy/paste: tools/buildfarm/utils/hgtool.py -r a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4 --mirror http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound --bundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg http://hg.mozilla.org/integration/mozilla-inbound /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:51:22     INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared',
20:51:22     INFO -  'PATH': '/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin'}
20:51:22    ERROR - caught OS error 2: No such file or directory while running ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:51:22     INFO - rmtree: /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:51:22     INFO - retry: Failed, sleeping 300 seconds before retrying
20:56:22     INFO - retry: Calling <bound method SpidermonkeyBuild._get_revision of <__main__.SpidermonkeyBuild object at 0x27c8dd0>> with args: (<mozharness.base.vcs.hgtool.HgtoolVCS object at 0x2710c90>, '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source'), kwargs: {}, attempt #5
20:56:22     INFO - Running command: ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:56:22     INFO - Copy/paste: tools/buildfarm/utils/hgtool.py -r a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4 --mirror http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound --bundle http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg http://hg.mozilla.org/integration/mozilla-inbound /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:56:22     INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared',
20:56:22     INFO -  'PATH': '/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin'}
20:56:22    ERROR - caught OS error 2: No such file or directory while running ['tools/buildfarm/utils/hgtool.py', '-r', u'a5a25f0e61ee2f1198eea0a464c1782f0d0afaa4', '--mirror', u'http://hg-internal.dmz.scl3.mozilla.com/integration/mozilla-inbound', '--bundle', u'http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'http://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/l64-br-haz_m-in_dep-0000000000/build/source']
20:56:22     INFO - rmtree: /builds/slave/l64-br-haz_m-in_dep-0000000000/build/source
20:56:22    FATAL - Can't checkout http://hg.mozilla.org/integration/mozilla-inbound!
20:56:22    FATAL - Caught exception: Unable to checkout
20:56:22    FATAL - Caught exception: Unable to checkout
20:56:22    FATAL - Caught exception: Unable to checkout
20:56:22    FATAL - Caught exception: Unable to checkout
20:56:22    FATAL - Caught exception: Unable to checkout
20:56:22    FATAL - Running post_fatal callback...
20:56:22    FATAL - Exiting -1
program finished with exit code 255

Pretty sure every other sort of build sets RETRY when it can't manage to clone.
I tried to do something fancy with @PostScriptAction, but I couldn't get it to set the return code to anything but -1.
Attachment #8334372 - Flags: review?(aki)
Assignee: nobody → sphink
Status: NEW → ASSIGNED
Attachment #8334372 - Flags: review?(aki) → review+
What's keeping this from being checked in? I'm on the verge of hiding these builds over this.
Comment on attachment 8334372 [details] [diff] [review]
Retry failed checkout

In production.
Attachment #8334372 - Flags: checked-in+
I seem to be checking out tools, then deleting it via clobber, then attempting to use it to checkout source.
Attachment #8336279 - Flags: review?(aki)
Comment on attachment 8336279 [details] [diff] [review]
Purge first, then checkout tools

I would actually put it before the mock stuff as well, but if you don't foresee that being a problem we can leave it.
Attachment #8336279 - Flags: review?(aki) → review+
Landed with purge first thing, before mock. And transplanted to production.

remote:   https://hg.mozilla.org/build/mozharness/rev/7a7c648bd4ff
remote:   https://hg.mozilla.org/build/mozharness/rev/0e4cd94839ac
Attachment #8336279 - Flags: checked-in+
something[s] here made it to production
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: