1396266 - Intermittent OSX "The executable '/tools/buildbot/bin/python' is not found; not creating virtualenv!"

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: rvandermeulen [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=128019184&repo=mozilla-beta

https://queue.taskcluster.net/v1/task/cH8fcA9DR6-VTUIQDE5e7Q/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Phil Ringnalda (:philor)]

ITYM "t-yosemite-r7-394 is broken".

Comment 2

[Ryan VanderMeulen [:RyanVM]]

And t-yosemite-r7-0115 apparently...

Comment 3

[Intermittent Failures Robot]

2 failures in 939 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* osx-10-10: 1
* macosx64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1396266&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Can we please give r7-0115 the same treatment as 0394?

Comment 5

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

This machine t-yosemite-r7-0115 like all machines between t-yosemite-r7-[030-390] is configured to ran job in Taskcluster so it doesn't need /tools/buildbot at all.The problem here was that it seems to have been enabled (I don't know why or by whom) and this caused these issues since the machine is configured to use generic workers.
The problem is now solved,machine was disabled from buildbot so will ran under Taskcluster and I re-image it.

For more details please check the link bellow:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slavetype.html?class=test&type=t-yosemite-r7

Comment 6

[Ryan VanderMeulen [:RyanVM]]

This is happening on Taskcluster jobs.
https://treeherder.mozilla.org/logviewer.html#?job_id=128944119&repo=mozilla-beta

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

In bug 1335353 I removed all references to /tools/buildbot.  I then reimaged this machine to no longer have /tools/buildbot and observed it to run a bunch of tasks successfully.  But I didn't think to uplift the fix to beta.. that was dumb.  So this particular host (and anything else reimaged in the interim) is going to fail to build beta pushes until uplift.

I've killed the worker on the host.  Once it's uplifted (I'll reopen bug 1335353), it will be good to go.

Buildduty folks: until that's done, please don't put reimaged yosemite machines into production.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

This is not the same as t-yosemite-r7-394:

 /tools/buildbot/bin/virtualenv
vs.
 /tools/virtualenv/bin/virtualenv

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I think the easier fix will be to add some symlinks so that this keeps working, and let the change ride the trains.

A few symlinks and the virtualenv works:

[[e-mail] ~]# mkdir -p /tools/buildbot/bin
[[e-mail] ~]# ln -s /tools/python27/bin/python2.7 /tools/buildbot/bin/python
[[e-mail] ~]# mkdir -p /tools/misc-python
[[e-mail] ~]# ln -s /tools/virtualenv/bin/virtualenv /tools/misc-python/virtualenv.py
[[e-mail] ~]# /tools/buildbot/bin/python /tools/misc-python/virtualenv.py 
You must provide a DEST_DIR
Usage: virtualenv.py [OPTIONS] DEST_DIR

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Jake has agreed to write up, test, and land a patch to make the dirs and symlinks above.  I think this can go away after ESR52 is closed.

Comment 11

[Jake Watkins [:dividehex]]

Attachment: Creates dirs and symlinks

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8905194 [details] [diff] [review]
Creates dirs and symlinks

Review of attachment 8905194 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/toplevel/manifests/worker/releng/generic_worker/test.pp
@@ +33,5 @@
> +                    ensure  => link,
> +                    target  => '/tools/virtualenv/bin/virtualenv',
> +                    require => Class[ 'dirs::tools::misc_python',
> +                                      'packages::mozilla::py27_virtualenv' ];
> +            }

This refers to classes from toplevel::worker::releng.. maybe it would be better there?

I'm happy with this too - whatever you think is best.

Comment 13

[:dhouse]

I looked at changing the location, but I don't know enough context and I'm guessing that applying the fix only to toplevel::worker::releng::generic_worker::test is enough.

Comment 14

[:dhouse]

Comment on attachment 8905194 [details] [diff] [review]
Creates dirs and symlinks

Applied patch and pushed to default:
https://hg.mozilla.org/build/puppet/rev/8255684921e8ae3efbc21f643632e5a07cf71b53
Travis passed. Pushed to production:
remote:   https://hg.mozilla.org/build/puppet/rev/c9d223913b76f8779edb69a31a37a6134c818e00

Comment 15

[Intermittent Failures Robot]

15 failures in 924 pushes (0.016 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 15

Platform breakdown:
* macosx64-devedition: 8
* macosx64-nightly: 6
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1396266&startday=2017-09-04&endday=2017-09-10&tree=all

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Orange factor doesn't show anything on or after the 7th when this landed.

Comment 17

[Jake Watkins [:dividehex]]

(In reply to Dustin J. Mitchell [:dustin] from comment #12)
> Comment on attachment 8905194 [details] [diff] [review]

> This refers to classes from toplevel::worker::releng.. maybe it would be
> better there?
> 
> I'm happy with this too - whatever you think is best.

:dhouse, Dustin has a good point.  Could you make sure this moved to the toplevel::worker::releng just to clean this up?

Comment 18

[:dhouse]

(In reply to Jake Watkins [:dividehex] from comment #17)
> (In reply to Dustin J. Mitchell [:dustin] from comment #12)
> > Comment on attachment 8905194 [details] [diff] [review]
> 
> > This refers to classes from toplevel::worker::releng.. maybe it would be
> > better there?
> > 
> > I'm happy with this too - whatever you think is best.
> 
> :dhouse, Dustin has a good point.  Could you make sure this moved to the
> toplevel::worker::releng just to clean this up?

Jake, I did not move the worker symlink definitions up a level. Can I close this out or hold it until ESR52 goes away and I can remove the workaround (reverting the patch)? I'm expecting that we will keep using generic-worker on Darwin and that we do not need to move it.

Comment 19

[Jake Watkins [:dividehex]]

Leave it be.