723006 - Talos runs dying with "<p class="error">&lt;type 'exceptions.KeyError'&gt;: 'toolsdir'</p>"

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Phil Ringnalda (:philor)]

After the emergency reconfig to get beta working, after the bug 722991 restart to get tests scheduled, every single Talos run which has finished and reported has been purple from

https://tbpl.mozilla.org/php/getParsedLog.php?id=8996997&tree=Mozilla-Inbound

========= Started Run performance tests exception (results: 4, elapsed: 0 secs)
... (loads of CSS and HTML) ...
<p class="error">&lt;type 'exceptions.KeyError'&gt;: 'toolsdir'</p>
Traceback (most recent call last):
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 728, in startStep
    d.addCallback(self._startStep_2)
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/twisted/internet/defer.py", line 260, in addCallback
    callbackKeywords=kw)
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
    self._runCallbacks()
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/process/buildstep.py", line 769, in _startStep_2
    skip = self.start()
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/steps/shell.py", line 212, in start
    command = properties.render(self.command)
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/process/properties.py", line 108, in render
    return [ self.render(e) for e in value ]
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/process/properties.py", line 106, in render
    return value.render(self.pmap)
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/process/properties.py", line 194, in render
    s = self.fmtstring % pmap
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/process/properties.py", line 169, in __getitem__
    rv = properties[key]
  File "/builds/buildbot/tests1-linux/lib/python2.6/site-packages/buildbot-0.8.2_hg_aeaa057e9df6_production_0.8-py2.6.egg/buildbot/process/properties.py", line 50, in __getitem__
    rv = self.properties[name][0]
exceptions.KeyError: 'toolsdir'

Francisco Franco is still dead, the trees are still closed.

Comment 1

[Phil Ringnalda (:philor)]

Also? Tests are still busted - I retriggered both Android builds for https://build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/587eea31cfa1 after bug 722991, and native seems to have only scheduled talos, no other tests, while xul ran other tests, and retried them all.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Turned out to be a regression from bug 660480. Backed that out and reconfiged the whole set of masters.