1335353 - Remember to remove buildbot slave configuration from taskcluster machines

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task, P2)

Description

[Wander Lairson Costa]

The taskcluster-worker puppet config has a "include buildslave" clause, but in an ideal world, we should get rid of that.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I'll own this and return to it in 6mo or so.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Still present:

modules/toplevel/manifests/worker/releng.pp
 18     # not sure why these are required on Darwin..
 19     if ($::operatingsystem == 'Darwin') {
 20         include packages::mozilla::py27_virtualenv
 21         include buildslave::install
 22     }

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

In total, this is what is included:

modules/toplevel/manifests/worker/releng.pp
 18     # not sure why these are required on Darwin..
 19     if ($::operatingsystem == 'Darwin') {
 20         include packages::mozilla::py27_virtualenv
 21         include buildslave::install
 22     }
 
modules/packages/manifests/mozilla/py27_virtualenv.pp
  11     include packages::mozilla::python27
installs /tools/python27-virtualenv/bin/virtualenv

modules/packages/manifests/mozilla/python27.pp
installs /tools/python27/bin/python

modules/buildslave/manifests/init.pp
 15     class {
 16         'buildslave::install': ;
 17     } -> Anchor['buildslave::end']
 18 
 19     Anchor['buildslave::begin'] ->
 20     class {
 21         'buildslave::startup': ;
 22     } -> Anchor['buildslave::end']

modules/buildslave/manifests/install.pp
  5     include packages::mozilla::python27
...
 30     python::virtualenv {
 31         $virtualenv_path:
 32             python   => $::packages::mozilla::python27::python,
 33             require  => $py_require,
 34             packages => $packages;
creates virtualenv in /tools/buildbot

modules/python/manifests/virtualenv.pp
  7     include python::virtualenv::settings  # just variables
  8     include python::virtualenv::prerequisites

modules/python/manifests/virtualenv/prerequisites.pp
 15     python::misc_python_file {
 16         'virtualenv.py': ;
 17     } -> Anchor['python::virtualenv::prerequisites::end']

modules/buildslave/manifests/startup.pp
 12     include ::shared  # just variables
 13     include buildslave::install
 14     include users::root
...
 44         "buildslave::startup::${startuptype}":
installs /usr/local/bin/runslave.py

modules/buildslave/manifests/startup/runner.pp
 36     include runner::tasks::buildbot
 37     include runner::tasks::halt
 38     if ($::operatingsystem != Windows) {
 39         include runner::tasks::cleanslate_task
 40         include runner::tasks::post_flight
 41     }

modules/runner/manifests/tasks/buildbot.pp
  6     include runner
  7     include buildslave::install
...
 35     runner::task {
 36         "${runlevel}-buildbot.py":
 37             require => [
 38                 File[$runslave_py],
 39                 Class['buildslave::install']
 40             ],
 41             content => template("${module_name}/tasks/buildbot.py.erb");
 42     }

modules/runner/manifests/init.pp
  6     include ::config  # variables only
  7     include dirs::opt  # makes /opt
  8     include runner::service
  9     include runner::settings  # variables only
 10     include packages::mozilla::python27

modules/runner/manifests/service.pp
 49                 16.04: {    
 50                     file {
 51                         '/lib/systemd/system/runner.service':
 52                             content => template('runner/runner.service.erb');
 53                     }
 54                     service {
 55                         'runner':
 56                             provider  => 'systemd',
 57                             require   => [
 58                                 Python::Virtualenv[$runner::settings::root],
 59                                 File['/lib/systemd/system/runner.service'],
 60                             ],
 61                             hasstatus => false,
 62                             enable    => true;
 63                     }
 64                 }
...
 70         'Darwin': {
 71             file {
 72                 '/Library/LaunchAgents/com.mozilla.runner.plist':
 73                     owner   => root,
 74                     group   => wheel,
 75                     mode    => '0644',
 76                     content => template('runner/runner.plist.erb');
 77             }
 78             service {
 79                 'runner':
 80                     require   => [
 81                         Python::Virtualenv[$runner::settings::root],
 82                         File['/Library/LaunchAgents/com.mozilla.runner.plist'],
 83                     ],
 84                     hasstatus => false,
 85                     enable    => true;
 86             }

modules/runner/manifests/tasks/halt.pp
 16             runner::task {
 17                 'halt.py':
 18                     content  => template("${module_name}/tasks/halt.py.erb");
 19             }

modules/runner/manifests/tasks/cleanslate_task.pp
  5     include runner
  6     include cleanslate
  7     include buildslave::install
  8     
  9     runner::task { 
 10         "${runlevel}-cleanslate":
 11             require => [
 12                 Class['cleanslate'],
 13                 Class['buildslave::install']
 14             ],
 15             content => template("${module_name}/tasks/cleanslate.erb");
 16     }  
 
modules/cleanslate/manifests/init.pp
  7     include cleanslate::settings  # vars only
  8     include packages::mozilla::python27
...
 19     python::virtualenv {
 20         $cleanslate::settings::root:  # /opt/cleanslate
 21             python   => $packages::mozilla::python27::python,
 22             require  => Class['packages::mozilla::python27'],
 23             packages => [
 24                 'cleanslate==1.2',
 25             ];
 26     }

modules/runner/manifests/tasks/post_flight.pp
    runner::task {
        "${runlevel}-post_flight":
            source => 'puppet:///modules/runner/post_flight.py';
    }

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Ugh, we include buildslave::install, not buildslave::startup.  Suddenly things make more sense!

modules/toplevel/manifests/worker/releng.pp
 18     # not sure why these are required on Darwin..
 19     if ($::operatingsystem == 'Darwin') {
 20         include packages::mozilla::py27_virtualenv
 21         include buildslave::install
 22     }
 
modules/packages/manifests/mozilla/py27_virtualenv.pp
  11     include packages::mozilla::python27
installs /tools/python27-virtualenv/bin/virtualenv

modules/packages/manifests/mozilla/python27.pp
installs /tools/python27/bin/python

modules/buildslave/manifests/install.pp
  5     include packages::mozilla::python27
...
 30     python::virtualenv {
 31         $virtualenv_path:
 32             python   => $::packages::mozilla::python27::python,
 33             require  => $py_require,
 34             packages => $packages;
creates virtualenv in /tools/buildbot

modules/python/manifests/virtualenv.pp
  7     include python::virtualenv::settings  # just variables
  8     include python::virtualenv::prerequisites

modules/python/manifests/virtualenv/prerequisites.pp
 15     python::misc_python_file {
 16         'virtualenv.py': ;
 17     } -> Anchor['python::virtualenv::prerequisites::end']

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

[[e-mail] ~]# find /tools -maxdepth 1 -ls
446387        0 drwxr-xr-x   10 root             wheel                 340 Jun 14 07:35 /tools
482897        0 drwxr-xr-x    5 root             admin                 170 Jun 14 07:36 /tools/buildbot
451428        0 drwxr-xr-x    6 root             wheel                 204 Jun 14 07:36 /tools/misc-python
451367        8 lrwxrwxrwx    1 root             wheel                  15 Jun 14 07:33 /tools/python -> /tools/python27
451424        8 lrwxrwxrwx    1 root             wheel                  15 Jun 14 07:33 /tools/python2 -> /tools/python27
453416        0 drwxr-xr-x    5 root             wheel                 170 Oct 14  2014 /tools/python27
458454        0 drwxr-xr-x    4 root             wheel                 136 Sep 13  2016 /tools/python27-mercurial
482878       80 -rwxr-xr-x    1 root             wheel               37090 Jun 14 07:35 /tools/tooltool.py
458336        0 drwxrwxr-x    4 root             wheel                 136 Jul 19  2012 /tools/virtualenv

[[e-mail] ~]# ls -al /usr/local/bin/virtualenv
lrwxr-xr-x  1 administrator  501    32 Jun 14 07:34 /usr/local/bin/virtualenv -> /tools/virtualenv/bin/virtualenv

packages::mozilla::python27 is included elsewhere in toplevel::worker::releng, so we accept the need to install a custom python. Similarly with packages::mozilla::tooltoo.  So that leaves

 /tools/buildbot/bin/python
 /tools/virtualenv/bin/virtualenv
 /usr/local/bin/virtualenv

as suspect paths.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Only the first of those is really a concern.  Installing /tools/virtualenv/bin/virtualenv for a TC worker isn't crazy.

But it's a big problem:

>  $grep -r /tools/buildbot *
> testing/mozharness/configs/web_platform_tests/prod_config.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/web_platform_tests/prod_config.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/firefox_ui_tests/taskcluster.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/firefox_ui_tests/taskcluster.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/marionette/prod_config.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/marionette/prod_config.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/builders.py:            'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/releng_infra_configs/builders.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/builders.py:            'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/releng_infra_configs/builders.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/testers.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/testers.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/testers.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/talos/mac_config.py:PYTHON = '/tools/buildbot/bin/python'
> testing/mozharness/configs/talos/linux_config.py:PYTHON = '/tools/buildbot/bin/python'
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_fuzzing_asan_tc.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan_and_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_debug_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_debug_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan_tc_and_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_code_coverage.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_valgrind.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan_tc.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_opt.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_opt.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_debug.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_noopt_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_debug_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_debug_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_add-on-devel.py:        'PATH': '/home/worker/workspace/build/src/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_base_linux_64_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_linux_64_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_base_mac_64_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_mac_64_builds.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_cross_noopt_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_artifact.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_stat_and_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_cross_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_debug_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_debug_artifact.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_add-on-devel.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_base_linux_32_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_linux_32_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:\
> testing/mozharness/configs/builds/releng_base_mac_64_cross_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_mac_64_cross_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_base_android_64_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_android_64_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin',
> testing/mozharness/configs/unittests/mac_unittest.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/unittests/mac_unittest.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/unittests/linux_unittest.py:        "python": "/tools/buildbot/bin/python",
> testing/mozharness/configs/unittests/linux_unittest.py:        "virtualenv": ["/tools/buildbot/bin/python", "/tools/misc-python/virtualenv.py"],

So /tools/buildbot/bin/python, /tools/misc-python/virtualenv.py, and /tools/buildbot/bin/buildbot.  The latter is only used for sendchanges, so it can be safely ignored (and ripped out when buildbot is dead):

testing/mozharness/mozharness/mozilla/buildbot.py:
148     def invoke_sendchange(self, downloadables=None, branch=None,
149                    username="sendchange-unittest", sendchange_props=None):
150         """ Generic sendchange, currently b2g- and unittest-specific.
151             """
152         c = self.config
153         buildbot = self.query_exe("buildbot", return_type="list")


From a linux desktop-test loaner:

root@be16ec41c500:~# ls /tools
ls: cannot access '/tools': No such file or directory

so clearly it's possible to run most stuff without either of these paths!

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b403dafdcc9ca5a8018347c2ddab7442dc971c76

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

$ grep -r query_exe..python testing/mozharness/

It looks like nothing ever uses 'python': '/tools/buildbot/bin/python'.  That makes sense, as sys.executable is a much better choice, and

$ grep -r sys.executable testing/mozharness/ | wc -l
112

So I think we can ignore those.

As for virtualenv, it looks like mozharness preferrs the in-tree vendored version, but that's not available without a source checkout, in which case it searches for it.  It's probably safe enough to replace `["/tools/buildbot/bin/python", "/tools/misc-python/virtualenv.py"` with `"/tools/virtualenv/bin/virtualenv"` globally and ensure that we install packages::mozilla::py27_virtualenv.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Related to the python issues -- https://hg.mozilla.org/mozilla-central/rev/4a93ea01749a

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

In the try job above, I replaced /tools/buildbot with /nosuch/path.  Lots of

15:12:18    FATAL - The executable '/nosuch/path/bin/python' is not found; not creating virtualenv!

so it looks like the analysis in comment 8 is accurate.  I'll try a new push with the addition of the change mentioned in the last comment.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=804ec5c9bac4839ea04e3746a0658bc52f5505e5

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

OK, that was successful!  I added a linux build and test just to check, and they were fine too.

The happy thing is, the new paths exist on existing hardware workers (linux talos) too, so I think we can land a change in-tree to use the new virtualenv and drop the python paths, and once that's "sunk in" a bit start removing the bogus paths from the OS X testers.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1335353: remove refs to /tools/buildbot;

On Taskcluster machines, /tools/buildbot doesn't exist.  It turns out, nothing
refers to exes['python'] anyway, so we can just remove that (the preference is
to use sys.executable instead).

Lines like

   'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],

Were committing two sins: first, using a python executable from a random
virtualenv; and second, using a virtualenv.py from another random directory (in
this case, it's a utility script for a PuppetAgain module). Such lines are
replaced with a reference to /tools/virtualenv/bin/virtualenv, which is
installed for the express purpose of providing a virtualenv binary on testers
(for builds, we use the vendored copy).

Review commit: https://reviewboard.mozilla.org/r/167730/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/167730/

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8896481 [details]
Bug 1335353: remove refs to /tools/buildbot;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/167730/diff/1-2/

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 8896481 [details]
Bug 1335353: remove refs to /tools/buildbot;

https://reviewboard.mozilla.org/r/167730/#review172960

I think this is good.
We can probably tear out all references to sendchanges from testing/mozharness as well, as well as other buildbot references. This is a great start, thanks!

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

The try run reliably failed
  Cpp - https://tools.taskcluster.net/groups/fm0B3GzlT6O8AexlZD7hTg/tasks/IgmQpjcSSz-8EWl1w9E1Ig/runs/0/logs/public%2Flogs%2Flive.log
  Gtest - https://tools.taskcluster.net/groups/fm0B3GzlT6O8AexlZD7hTg/tasks/HXDKk8g7RQScA-HjVj2eIg/runs/0/logs/public%2Flogs%2Flive.log

That seems to be due to

00:31:53    FATAL - There are specified suites that are incompatible with --artifact try syntax flag: gtest

[task 2017-08-11T23:33:12.460265Z] 23:33:12    FATAL - There are specified suites that are incompatible with --artifact try syntax flag: cppunittest

so I guess that's my fault in using --artifact.  The other failures seem to be "normal" test failures.  So *crosses fingers* I think this is good to land.

Once this is landed, I will land a puppet patch that removes the buildslave::install class but keeps /tools/buildbot/bin/python and /tools/misc-python/virtualenv.py in place so that old tasks continue to run.  I'll mark that with a TODO pointing to this bug, then put it on ice for a year.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/76f32438e9f9
remove refs to /tools/buildbot; r=aki

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

This caused linux talos to fail with https://treeherder.mozilla.org/logviewer.html#?job_id=123052653&repo=autoland

Backed out in https://hg.mozilla.org/integration/autoland/rev/d49ae3164b4ed8b199b0e8317ca21c68527d9c5f

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

My try syntax was -b d -t all, which didn't trigger talos.  Oops :(

Mac talos was successful (it finds virtualenv in $PATH).  But packages::mozilla::py27_virtualenv isn't installed on linux.  Let's fix that first.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1335353: add py27_virtualenv on Darwin slaves;

Review commit: https://reviewboard.mozilla.org/r/168428/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/168428/

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 8897141 [details]
Bug 1335353: add py27_virtualenv on Darwin slaves;

https://reviewboard.mozilla.org/r/168428/#review173682

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

https://hg.mozilla.org/build/puppet/rev/8e1788f19e8b270fcf20c153604a69c881cecd17

Jake, mind merging that for me?

Comment 23

[Jake Watkins [:dividehex]]

(In reply to Dustin J. Mitchell [:dustin] from comment #22)
> https://hg.mozilla.org/build/puppet/rev/
> 8e1788f19e8b270fcf20c153604a69c881cecd17
> 
> Jake, mind merging that for me?

Merged :-)
https://hg.mozilla.org/build/puppet/rev/41491e9da8f339fb1390cc0d64716ca2cc96039e

Comment 24

[Aki Sasaki (not active)]

Mon Aug 14 16:58:39 -0700 2017 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Cannot install on Ubuntu at /etc/puppet/production/modules/packages/manifests/mozilla/py27_virtualenv.pp:34 on node talos-linux64-ix-047.test.releng.scl3.mozilla.com
Mon Aug 14 16:58:39 -0700 2017 Puppet (err): Could not retrieve catalog; skipping run

Comment 25

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks for backing that out.  I knew it was too simple :(

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

For the taskcluster-worker-based Linux Talos, I just used the distro virtualenv.  I bet we could do the same here.  So I'll re-draft that to include packages::virtualenv -- and implement that for Ubuntu 12.04 -- for linux test slaves and leave everything else unchanged.

Comment 27

[Aki Sasaki (not active)]

That will probably work, though will the path be different? in which case we'll need to point the linux talos mh configs at the correct path.

Comment 28

[Dustin J. Mitchell [:dustin] (he/him)]

It will still be in PATH, which is the critical bit (that's how it's found on OS X, per comment 19)

Comment 29

[Dustin J. Mitchell [:dustin] (he/him)]

[root@talos-linux64-ix-049 ~]# python /tools/misc-python/virtualenv.py  --version
1.7.1.2
which is the same as the version in ubuntu 12.04, so no worries there..

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8897141 [details]
Bug 1335353: add py27_virtualenv on Darwin slaves;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/168428/diff/1-2/

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

Jake, do yo mind merging the puppetagain change?

Comment 32

[Jake Watkins [:dividehex]]

(In reply to Dustin J. Mitchell [:dustin] from comment #31)
> Jake, do yo mind merging the puppetagain change?

looks like aki beat me to it :-)

Comment 33

[Dustin J. Mitchell [:dustin] (he/him)]

Looks like that landing stuck, so I'll push my patch through try again, then land it.  Once it's merged around, I'll try deleting /tools/buildbot from a few workers.  If those cause failures, we can kill the workers and address the issue.

Comment 34

[Dustin J. Mitchell [:dustin] (he/him)]

still no talos results -- I'll check back tomorrow
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=88ec1c142be1c7bc8a9ad8f09c9be154b791f850&selectedJob=124909704

Comment 35

[Dustin J. Mitchell [:dustin] (he/him)]

14 hours later, the try jobs ran and they are all green.  They are using the virtualenv in PATH:

22:12:51     INFO - Running main action method: create_virtualenv
22:12:51     INFO - Creating virtualenv /builds/slave/test/build/venv
22:12:51     INFO - Running command: ['virtualenv', '--no-site-packages', '--distribute', '/builds/slave/test/build/venv'] in

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88081c5bc847
remove refs to /tools/buildbot; r=aki

Comment 37

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/88081c5bc847

Comment 38

[Aki Sasaki (not active)]

Relanded in https://hg.mozilla.org/build/puppet/rev/97ef23d7d6d4
backed out again in https://hg.mozilla.org/build/puppet/rev/f8d448747286

Cannot install on Ubuntu at /etc/puppet/production/modules/packages/manifests/mozilla/py27_virtualenv.pp:34 on node talos-linux64-ix-067.test.releng.scl3.mozilla.com

Comment 39

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8897141 [details]
Bug 1335353: add py27_virtualenv on Darwin slaves;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/168428/diff/2-3/

Comment 40

[Dustin J. Mitchell [:dustin] (he/him)]

Ok, this one will *only* affect Darwin, and that class always works on Darwin, so i think this should be fine.

Comment 41

[Dustin J. Mitchell [:dustin] (he/him)]

[[e-mail] ~]# which virtualenv
/usr/local/bin/virtualenv
yay!  And I don't see any errors on a talos-ix run either.

Now to remove the now-unnecessary `include buildslave::install`.

Comment 42

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1335353.patch

Comment 43

[Dustin J. Mitchell [:dustin] (he/him)]

Fixed in

https://github.com/taskcluster/taskcluster-login/commit/1b3de7d5eaa4330974f5eb90e438df829399e89a

I now get creds with the appropriate scopes in the oidc demo.  YAY!

Comment 44

[Dustin J. Mitchell [:dustin] (he/him)]

I'm reimaging t-yosemite-r7-0115 to see if it still works.

Comment 45

[Dustin J. Mitchell [:dustin] (he/him)]

..ran one green task, so I'm calling it good..

Comment 46

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.