1101059 - Clobber failing with "Couldn't clobber properly, bailing out", on bld-lion-r5-051

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Daniel Holbert [:dholbert]]

Filing this bug for a busted build job, with the error being "Couldn't clobber properly, bailing out".

Log info:
{
builder: mozilla-inbound-macosx64-debug
slave: bld-lion-r5-051
starttime: 1416331850.64
results: failure (2)
buildid: 20141118092826
builduid: bb16079626b64c549064a2cc6e520872
revision: 156f8e3bc89d1385510a01335ffe526eb0005728
}

Log snippet:
{
09:30:57     INFO - ##### Running clobber step.
09:30:57     INFO - #####
09:30:57     INFO - Running main action method: clobber
09:30:57     INFO - retry: Calling <bound method FxDesktopBuild.run_command of <__main__.FxDesktopBuild object at 0x1010a7890>> with args: [['/builds/slave/m-in-osx64-d-00000000000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'https://api.pub.build.mozilla.org/clobberer/lastclobber', 'mozilla-inbound', 'OS X 10.7 64-bit mozilla-inbound leak test build', 'm-in-osx64-d-00000000000000000', 'bld-lion-r5-051', 'http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/']], kwargs: {'error_list': [{'substr': 'Error contacting server', 'explanation': 'Error contacting server for clobberer information.', 'level': 'error'}], 'cwd': '/builds/slave'}, attempt #1
09:30:57     INFO - Running command: ['/builds/slave/m-in-osx64-d-00000000000000000/scripts/external_tools/clobberer.py', '-s', 'scripts', '-s', 'logs', '-s', 'buildprops.json', '-s', 'token', '-t', '168', 'https://api.pub.build.mozilla.org/clobberer/lastclobber', 'mozilla-inbound', 'OS X 10.7 64-bit mozilla-inbound leak test build', 'm-in-osx64-d-00000000000000000', 'bld-lion-r5-051', 'http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/'] in /builds/slave
09:30:57     INFO - Copy/paste: /builds/slave/m-in-osx64-d-00000000000000000/scripts/external_tools/clobberer.py -s scripts -s logs -s buildprops.json -s token -t 168 https://api.pub.build.mozilla.org/clobberer/lastclobber mozilla-inbound "OS X 10.7 64-bit mozilla-inbound leak test build" m-in-osx64-d-00000000000000000 bld-lion-r5-051 http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/
09:30:59     INFO -  Checking clobber URL: https://api.pub.build.mozilla.org/clobberer/lastclobber?master=http%3A%2F%2Fbuildbot-master84.srv.releng.scl3.mozilla.com%3A8001%2F&slave=bld-lion-r5-051&builddir=m-in-osx64-d-00000000000000000&branch=mozilla-inbound&buildername=OS+X+10.7+64-bit+mozilla-inbound+leak+test+build
09:30:59     INFO -  m-in-osx64-d-00000000000000000:Our last clobber date:  2014-11-03 22:09:46
09:30:59     INFO -  m-in-osx64-d-00000000000000000:Server clobber date:    2014-11-18 08:30:50
09:30:59     INFO -  m-in-osx64-d-00000000000000000:Server is forcing a clobber, initiated by [e-mail]
09:30:59     INFO -  m-in-osx64-d-00000000000000000:Clobbering...
09:30:59     INFO -  Removing build/
09:30:59     INFO -  Couldn't clobber properly, bailing out.
09:30:59    ERROR - Return code: 1
09:30:59     INFO - retry: Failed, sleeping 60 seconds before retrying
}

(It retries twice more, with the same results.)

Comment 1

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-11-18T09:55:19
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-inbound&job_id=3998478
repository: mozilla-inbound
who: dholbert[at]mozilla[dot]com
machine: bld-lion-r5-051
buildname: OS X 10.7 64-bit mozilla-inbound leak test build
revision: 156f8e3bc89d

Return code: 1
Return code: 1
Return code: 1
<bound method FxDesktopBuild.run_command of <__main__.FxDesktopBuild object at 0x1010a7890>> failed after 3 tries!
failed to clobber build
Running post_fatal callback...
setting return code to 2 because fatal was called
Exiting 2
# TBPL FAILURE #
# TBPL FAILURE #

Comment 2

[Daniel Holbert [:dholbert]]

(Comment 1 is the same failed job described in comment 0, BTW. Not a different instance.)

Comment 3

[Morgan Phillips [:mrrrgn]]

This one may end up impossible to reproduce. I just logged into the same machine and tried to replicate the conditions with no success.... One possibility is that there was a permission error or an unreadable character in some filename. The best way to approach this will likely be to add more specific error messages to the clobberer client.

Here's my attempt to reproduce on the same builder: https://pastebin.mozilla.org/7367193

Comment 4

[Morgan Phillips [:mrrrgn]]

Attachment: clobberer.errors.diff

Comment 5

[Morgan Phillips [:mrrrgn]]

My proposal here is to put in this patch with more detailed messages, then close this bug. When an error happens next it will be much easier to figure out what happened. As it stands, I don't think I have enough to go on.

Comment 6

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8524768 [details] [diff] [review]
clobberer.errors.diff

Review of attachment 8524768 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Morgan Phillips [:mrrrgn] from comment #5)
> My proposal here is to put in this patch with more detailed messages, then
> close this bug. When an error happens next it will be much easier to figure
> out what happened. As it stands, I don't think I have enough to go on.

Yes, it's also possible that there is simply something wrong with this particular machine. More info is good.

Comment 7

[Morgan Phillips [:mrrrgn]]

Marking as resolved incomplete. The next time a problem happens we should have detailed errors which will allow us to debug properly and fix the issue without wasting lots of dev time trying to recreate the problem.