804026 - Don't use slashes in builders names?

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

[buildbot-master04.build.scl1.mozilla.com] out: 2012-10-21 10:34:19-0700 [-] Unhandled Error
[buildbot-master04.build.scl1.mozilla.com] out: 	Traceback (most recent call last):
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 621, in loadTheConfigFile
[buildbot-master04.build.scl1.mozilla.com] out: 	    d = self.loadConfig(f)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 956, in loadConfig
[buildbot-master04.build.scl1.mozilla.com] out: 	    d.addCallback(lambda res: self.loadConfig_Builders(builders))
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/twisted/internet/defer.py", line 260, in addCallback
[buildbot-master04.build.scl1.mozilla.com] out: 	    callbackKeywords=kw)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
[buildbot-master04.build.scl1.mozilla.com] out: 	    self._runCallbacks()
[buildbot-master04.build.scl1.mozilla.com] out: 	--- <exception caught here> ---
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
[buildbot-master04.build.scl1.mozilla.com] out: 	    self.result = callback(self.result, *args, **kw)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 956, in <lambda>
[buildbot-master04.build.scl1.mozilla.com] out: 	    d.addCallback(lambda res: self.loadConfig_Builders(builders))
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 1103, in loadConfig_Builders
[buildbot-master04.build.scl1.mozilla.com] out: 	    statusbag = self.status.builderAdded(name, basedir, category)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/status/builder.py", line 2528, in builderAdded
[buildbot-master04.build.scl1.mozilla.com] out: 	    os.makedirs(builder_status.basedir)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib64/python2.6/os.py", line 157, in makedirs
[buildbot-master04.build.scl1.mozilla.com] out: 	    mkdir(name, mode)
[buildbot-master04.build.scl1.mozilla.com] out: 	exceptions.OSError: [Errno 17] File exists: '/builds/buildbot/tests1-macosx/master/cedar_lion-debug_test-mochitests-4/5'

Comment 1

[Rail Aliiev [:rail]]

I had to remove the problematic directories to allow the master to start

Comment 2

[Rail Aliiev [:rail]]

I think, this happens here:
hg.mozilla.org/build/buildbot/file/4aea7e2b9e1f/master/buildbot/status/builder.py#l287

buildbot checks if basedir exists, then tries to create it by os.makedirs. Since we have multiple builders with the same shared directory (cedar_lion-debug_test-mochitests-4) and 
os.makedirs running async (in parallel) we may hit the a race condition between path.exists calls in os.makedirs and mkdir in the same function.

Replacing slashes with something else (underscores) should solve the problem.

Comment 3

[Justin Wood (:Callek)]

(In reply to Rail Aliiev [:rail] from comment #2)
> Replacing slashes with something else (underscores) should solve the problem.

Sudden thought, if we change the builder name(s) we likely will need TBPL patch(es)

Comment 4

[Ed Morley [:emorley]]

(In reply to Justin Wood (:Callek) from comment #3)
> Sudden thought, if we change the builder name(s) we likely will need TBPL
> patch(es)

Depending on which buildernames change (some suites don't match on the part of the name after the slash, so we could just omit it entirely), then we may need a TBPL patch.

Ask for feedback on the patch once it is attached & I'll take a look :-)

Comment 5

[Phil Ringnalda (:philor)]

Where that particular bit of naming comes up is in http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/43b5054776ba/js/Data.js#l726 - if you just drop the /5 and match mobile's "mochitest-1", then you won't have to patch tbpl (though we'll have to patch it eventually, to make that hunk make sense again).

Comment 6

[Ed Morley [:emorley]]

(In reply to Phil Ringnalda (:philor) from comment #5)
> Where that particular bit of naming comes up is in
> http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/
> 43b5054776ba/js/Data.js#l726 - if you just drop the /5 and match mobile's
> "mochitest-1", then you won't have to patch tbpl (though we'll have to patch
> it eventually, to make that hunk make sense again).

That unfortunately won't work with the Nightly l10n builds, which prefix with a space (and bug 786314 changes that section a bit). Easily fixed, but I suspect we'll want to rename the l10n nightlies to be more consistent, rather than having to bodge it up.

Comment 7

[Aki Sasaki (not active)]

Attachment: remove slashes from mozharness mochitests

Comment 8

[Ed Morley [:emorley]]

Comment on attachment 674020 [details] [diff] [review]
remove slashes from mozharness mochitests

This should work fine with TBPL, once the aspects in bug 789652 are dealt with :-)

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 674020 [details] [diff] [review]
remove slashes from mozharness mochitests

Do you also need to change PLATFORM_UNITTEST_VARS in http://hg.mozilla.org/build/buildbot-configs/file/a6af987ce749/mozilla-tests/b2g_config.py#l108 ?

Comment 10

[Aki Sasaki (not active)]

Attachment: good call

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 674244 [details] [diff] [review]
good call

http://hg.mozilla.org/build/buildbot-configs/rev/707001661042

Comment 12

[Rail Aliiev [:rail]]

In production

Comment 13

[Ed Morley [:emorley]]

Ah, because mozharness is using mochitests-N (ie plural), this means we don't hit the regex we should have, and display the builds as "" (ie the blank columns currently on cedar).

Bug 805118 will mean that TBPL falls back to the more sensible "M" for unknown mochitest machine numbers, but we should s/mochitests/mochitest/ across the board on the buildbot side). I'll file a separate bug for this.