Closed Bug 804026 Opened 8 years ago Closed 8 years ago

Don't use slashes in builders names?

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rail, Unassigned)

References

Details

Attachments

(1 file, 1 obsolete file)

[buildbot-master04.build.scl1.mozilla.com] out: 2012-10-21 10:34:19-0700 [-] Unhandled Error
[buildbot-master04.build.scl1.mozilla.com] out: 	Traceback (most recent call last):
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 621, in loadTheConfigFile
[buildbot-master04.build.scl1.mozilla.com] out: 	    d = self.loadConfig(f)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 956, in loadConfig
[buildbot-master04.build.scl1.mozilla.com] out: 	    d.addCallback(lambda res: self.loadConfig_Builders(builders))
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/twisted/internet/defer.py", line 260, in addCallback
[buildbot-master04.build.scl1.mozilla.com] out: 	    callbackKeywords=kw)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks
[buildbot-master04.build.scl1.mozilla.com] out: 	    self._runCallbacks()
[buildbot-master04.build.scl1.mozilla.com] out: 	--- <exception caught here> ---
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks
[buildbot-master04.build.scl1.mozilla.com] out: 	    self.result = callback(self.result, *args, **kw)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 956, in <lambda>
[buildbot-master04.build.scl1.mozilla.com] out: 	    d.addCallback(lambda res: self.loadConfig_Builders(builders))
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/master.py", line 1103, in loadConfig_Builders
[buildbot-master04.build.scl1.mozilla.com] out: 	    statusbag = self.status.builderAdded(name, basedir, category)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib/python2.6/site-packages/buildbot-0.8.2_hg_bd4812420e63_production_0.8-py2.6.egg/buildbot/status/builder.py", line 2528, in builderAdded
[buildbot-master04.build.scl1.mozilla.com] out: 	    os.makedirs(builder_status.basedir)
[buildbot-master04.build.scl1.mozilla.com] out: 	  File "/builds/buildbot/tests1-macosx/lib64/python2.6/os.py", line 157, in makedirs
[buildbot-master04.build.scl1.mozilla.com] out: 	    mkdir(name, mode)
[buildbot-master04.build.scl1.mozilla.com] out: 	exceptions.OSError: [Errno 17] File exists: '/builds/buildbot/tests1-macosx/master/cedar_lion-debug_test-mochitests-4/5'
I had to remove the problematic directories to allow the master to start
I think, this happens here:
hg.mozilla.org/build/buildbot/file/4aea7e2b9e1f/master/buildbot/status/builder.py#l287

buildbot checks if basedir exists, then tries to create it by os.makedirs. Since we have multiple builders with the same shared directory (cedar_lion-debug_test-mochitests-4) and 
os.makedirs running async (in parallel) we may hit the a race condition between path.exists calls in os.makedirs and mkdir in the same function.

Replacing slashes with something else (underscores) should solve the problem.
(In reply to Rail Aliiev [:rail] from comment #2)
> Replacing slashes with something else (underscores) should solve the problem.

Sudden thought, if we change the builder name(s) we likely will need TBPL patch(es)
(In reply to Justin Wood (:Callek) from comment #3)
> Sudden thought, if we change the builder name(s) we likely will need TBPL
> patch(es)

Depending on which buildernames change (some suites don't match on the part of the name after the slash, so we could just omit it entirely), then we may need a TBPL patch.

Ask for feedback on the patch once it is attached & I'll take a look :-)
Where that particular bit of naming comes up is in http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/43b5054776ba/js/Data.js#l726 - if you just drop the /5 and match mobile's "mochitest-1", then you won't have to patch tbpl (though we'll have to patch it eventually, to make that hunk make sense again).
(In reply to Phil Ringnalda (:philor) from comment #5)
> Where that particular bit of naming comes up is in
> http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/
> 43b5054776ba/js/Data.js#l726 - if you just drop the /5 and match mobile's
> "mochitest-1", then you won't have to patch tbpl (though we'll have to patch
> it eventually, to make that hunk make sense again).

That unfortunately won't work with the Nightly l10n builds, which prefix with a space (and bug 786314 changes that section a bit). Easily fixed, but I suspect we'll want to rename the l10n nightlies to be more consistent, rather than having to bodge it up.
Comment on attachment 674020 [details] [diff] [review]
remove slashes from mozharness mochitests

This should work fine with TBPL, once the aspects in bug 789652 are dealt with :-)
Attachment #674020 - Flags: feedback+
Comment on attachment 674020 [details] [diff] [review]
remove slashes from mozharness mochitests

Do you also need to change PLATFORM_UNITTEST_VARS in http://hg.mozilla.org/build/buildbot-configs/file/a6af987ce749/mozilla-tests/b2g_config.py#l108 ?
Attached patch good callSplinter Review
Attachment #674020 - Attachment is obsolete: true
Attachment #674020 - Flags: review?(rail)
Attachment #674244 - Flags: review?(rail)
Attachment #674244 - Flags: review?(rail) → review+
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
In production
Ah, because mozharness is using mochitests-N (ie plural), this means we don't hit the regex we should have, and display the builds as "" (ie the blank columns currently on cedar).

Bug 805118 will mean that TBPL falls back to the more sensible "M" for unknown mochitest machine numbers, but we should s/mochitests/mochitest/ across the board on the buildbot side). I'll file a separate bug for this.
Blocks: 805138
Depends on: 805141
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.