794339 - Linux SpiderMonkey builds failing to find an hg to clone with

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

e.g. https://tbpl.mozilla.org/php/getParsedLog.php?id=15537136&tree=Mozilla-Inbound
bld-centos5-32-vmw-017

Upon execvpe hg ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'scripts'] in environment id 171984116
:Traceback (most recent call last):
  File "/tools/buildbot-0.8.4-pre-moz2/lib/python2.6/site-packages/twisted/internet/process.py", line 414, in _fork
    executable, args, environment)
  File "/tools/buildbot-0.8.4-pre-moz2/lib/python2.6/site-packages/twisted/internet/process.py", line 460, in _execChild
    os.execvpe(executable, args, environment)
  File "/tools/buildbot-0.8.4-pre-moz2/lib/python2.6/os.py", line 353, in execvpe
    _execvpe(file, args, env)
  File "/tools/buildbot-0.8.4-pre-moz2/lib/python2.6/os.py", line 380, in _execvpe
    func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

hg is at /tools/python/bin/hg on them, and they don't have the /tools/python27/bin or /tools/python27-mercurial which are on the path. Probably affects that whole class of machines - bld-centos5-64-vmw-NNN - but just the spidermonkey builds.

Comment 2

[Phil Ringnalda (:philor)]

Apparently a change last Thursday - https://secure.pub.build.mozilla.org/buildapi/recent/bld-centos5-64-vmw-001?numbuilds=125 says it did a green spidermonkey job at 7am, and a red one at 8:46pm.

Comment 3

[Phil Ringnalda (:philor)]

They get their env from http://hg.mozilla.org/build/buildbot-configs/annotate/f9bd9eadca7a/mozilla/config.py#l862, and http://hg.mozilla.org/build/buildbot-configs/diff/18d1ebb7084a/mozilla/config.py gave that env a path that works for the mock slaves.

Comment 4

[Chris AtLee [:catlee]]

We should switch these over to use the mock slaves as well.

Comment 5

[Steve Fink [:sfink] [:s:]]

(In reply to Chris AtLee [:catlee] from comment #4)
> We should switch these over to use the mock slaves as well.

What would this take? On the face of it, it seems like I'd just need to do something very similar to what I'm doing over in bug 775355, then copy over the mock-related arguments. That runs and produces reasonable-looking stuff, but I don't know how to properly test without some test slaves to kick around. I haven't attempted that yet; not sure how doable it is.

Comment 6

[Steve Fink [:sfink] [:s:]]

Attachment: Patch 3/3: Copy the mock-related arguments into the spidermonkey project builders' ScriptFactories

I'll put that question in the form of a set of patches

Comment 7

[Steve Fink [:sfink] [:s:]]

Attachment: Patch 2/3: Give the spidermonkey project configs the entire platform config settings

Sorry, these patches are all out of order. I'll update the descriptions with a reasonable order.

Comment 8

[Steve Fink [:sfink] [:s:]]

Attachment: Patch 1/3: Share the whole platform config chunks instead of just the env for spidermonkey project configs

This change would need to be made simultaneously to the buildbotcustom stuff

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 670944 [details] [diff] [review]
Patch 3/3: Copy the mock-related arguments into the spidermonkey project builders' ScriptFactories

Review of attachment 670944 [details] [diff] [review]:
-----------------------------------------------------------------

::: misc.py
@@ +2345,5 @@
> +            factory_platform_args = [ 'use_mock',
> +                                      'mock_target',
> +                                      'mock_packages',
> +                                      'mock_copyin_files' ]
> +            factory_kwargs = dict([ (a, pf.get(a, None)) for a in factory_platform_args ])

Can you change this to only set factory_kwargs that appear in pf? e.g.

for a in factory_platform_args:
   if a in pf:
      factory_kwargs[a] = pf[a]

This makes the diff vs. the current configuration easier to read. As this patch stands you're changing the default value use_mock to None instead of False.

Comment 10

[Steve Fink [:sfink] [:s:]]

Attachment: Copy the mock-related arguments into the spidermonkey project builders' ScriptFactories

Ok, thanks.

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 674282 [details] [diff] [review]
Copy the mock-related arguments into the spidermonkey project builders' ScriptFactories

Review of attachment 674282 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. Please add a line to initialize factory_kwargs if required.

::: misc.py
@@ +2350,5 @@
>          for variant in variants:
> +            factory_platform_args = [ 'use_mock',
> +                                      'mock_target',
> +                                      'mock_packages',
> +                                      'mock_copyin_files' ]

I think you need a "factory_kwargs = {}" here

Comment 12

[Steve Fink [:sfink] [:s:]]

(In reply to Chris AtLee [:catlee] from comment #11)
> ::: misc.py
> @@ +2350,5 @@
> >          for variant in variants:
> > +            factory_platform_args = [ 'use_mock',
> > +                                      'mock_target',
> > +                                      'mock_packages',
> > +                                      'mock_copyin_files' ]
> 
> I think you need a "factory_kwargs = {}" here

And I think you're right. Sorry about that; I should have at least done rudimentary testing. I always mess up python scoping.

I have now done my basic testing (loaded the config and dumped things out, and ran the master to make sure it started up.)

Comment 13

[Steve Fink [:sfink] [:s:]]

http://hg.mozilla.org/build/buildbot-configs/rev/d999fc274b17

Comment 14

[Steve Fink [:sfink] [:s:]]

http://hg.mozilla.org/build/buildbotcustom/rev/2104f40b645c
http://hg.mozilla.org/build/buildbotcustom/rev/ef41cdf3ef3b

Comment 15

[Steve Fink [:sfink] [:s:]]

Attachment: spidermonkey builder names need to be more unique.

Comment 16

[[e-mail] (:bhearsum)]

In production.

Comment 17

[[e-mail] (:bhearsum)]

This may have caused bug 807313.

Comment 18

[Terrence Cole [:terrence]]

Huh, I'm seeing this again on both:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&jobname=spidermonkey
https://tbpl.mozilla.org/?tree=Try&noignore=1&jobname=spidermonkey