657448 - exceptions.KeyError: 'exepath' on test masters

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Armen [:armenzg]]

We have been receiving this exception on the emails from our testing masters.

This seems transient from running |grep exepath master/twistd.log.*| in one of the masters.

I am feeling that we might just keep this bug open for a while and then close if it doesn't happen again.

It seems the same related step failed:
2011-05-16 08:59:54-0700 [-]  step 'find_executable' complete: failure

Exception in /builds/buildbot/tests1/master/twistd.log.1:
2011-05-16 11:14:55-0700 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/buildbot-0.8.2_hg_3dc678eecd11_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 728, in startStep
	    d.addCallback(self._startStep_2)
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/twisted/internet/defer.py", line 260, in addCallback
	    callbackKeywords=kw)
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
	    self._runCallbacks()
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
	    self.result = callback(self.result, *args, **kw)
	--- <exception caught here> ---
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/buildbot-0.8.2_hg_3dc678eecd11_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 769, in _startStep_2
	    skip = self.start()
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/buildbotcustom/steps/misc.py", line 551, in start
	    value = self.value(self.build)
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/buildbotcustom/process/factory.py", line 6963, in get_exedir
	    return os.path.dirname(build.getProperty('exepath'))
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/buildbot-0.8.2_hg_3dc678eecd11_production_0.8-py2.6.egg/buildbot/process/base.py", line 93, in getProperty
	    return self.build_status.getProperty(propname)
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/buildbot-0.8.2_hg_3dc678eecd11_production_0.8-py2.6.egg/buildbot/status/builder.py", line 1196, in getProperty
	    return self.properties[propname]
	  File "/builds/buildbot/tests1/lib/python2.6/site-packages/buildbot-0.8.2_hg_3dc678eecd11_production_0.8-py2.6.egg/buildbot/process/properties.py", line 50, in __getitem__
	    rv = self.properties[name][0]
	exceptions.KeyError: 'exepath'

Comment 1

[Chris AtLee [:catlee]]

Possibly from corrupted build or unsuccessful unpacking, resulting in the subsequent steps failing to determine exepath?

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to comment #1)
> Possibly from corrupted build or unsuccessful unpacking, resulting in the
> subsequent steps failing to determine exepath?

Yes, for the next (or most recent) occurrence, can we go to the test master affected and see what actually happened? 

Either we investigate these things when they happen or we wait too long and we lose the logs that would allow us to diagnose.

Comment 3

[Armen [:armenzg]]

zwol, do you know how your package got named minefield instead of firefox?
minefield-7.0a1.en-US.linux-i686.tar.bz2?

########################################

For releng:

We got few of these yesterday and I was too busy in the morning to look at this.

After connecting a lot of dots I managed to determine that the find step for Mac or an equivalent step for Linux and Windows failed for *all* test jobs for the same push.

The failing revision is try/08c4634d589d

It seems that it wgets "minefield-7.0a1.en-US.linux-i686.tar.bz2" while other try jobs wget firefox instead "firefox-6.0a1.en-US.linux-i686.tar.bz2"

Our automation has "firefox" hardcoded many places.

### Steps to discover and symptoms
I loaded up one_line_per_build?numbuilds=2500 (yeah, I know) on bm04 and I found the purple.

The find step on Mac instead of outputting "./Nightly.app/Contents/MacOS/firefox-bin" it output nothing. The step fails on trying to set exepath and exedir and the job halts.

For Windows and Linux we fail on "get_build_info" (on this case the job attempted to carry on with all remaining steps but failed) with:
> python c:/talos-slave/test/tools/buildfarm/utils/printbuildrev.py firefox
...
> Traceback (most recent call last):
>   File "c:/talos-slave/test/tools/buildfarm/utils/printbuildrev.py", line 14, in <module>
>     buildid = appini.get('App', 'BuildID')
>   File "C:\mozilla-build\python25\lib\ConfigParser.py", line 511, in get
>     raise NoSectionError(section)
> ConfigParser.NoSectionError: No section: 'App'

Comment 4

[Armen [:armenzg]]

I found it:
http://hg.mozilla.org/try/rev/001b6b51a512
MOZ_APP_BASENAME=Minefield

zwol, is this renaming expected to land on m-c?

Comment 5

[Zack Weinberg (:zwol)]

(In reply to comment #4)
> I found it:
> http://hg.mozilla.org/try/rev/001b6b51a512
> MOZ_APP_BASENAME=Minefield
> 
> zwol, is this renaming expected to land on m-c?

No.  That is a deliberate change on my end, but it's not one which I expect ever to get to land.  (See bug 443078, which I still think should not have been WONTFIXed; but I've given up arguing.)  I would appreciate it if the try server automation could handle this, but I can cope with its not doing so.

Comment 6

[Zack Weinberg (:zwol)]

(I can cope by ensuring that that patch is removed from my mq stack before pushing to try)

Comment 7

[Armen [:armenzg]]

OK. From reading the other bugs I cannot seeing a new naming to support.

Unless there is a compelling reason to support this I am going to WONTFIX it.