Closed Bug 1335353 Opened 3 years ago Closed 2 years ago

Remember to remove buildbot slave configuration from taskcluster machines

Categories

(Infrastructure & Operations :: RelOps: Puppet, task, P2, minor)

All
macOS

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wcosta, Assigned: dustin)

References

Details

Attachments

(3 files)

The taskcluster-worker puppet config has a "include buildslave" clause, but in an ideal world, we should get rid of that.
I'll own this and return to it in 6mo or so.
Assignee: relops → dustin
Still present:

modules/toplevel/manifests/worker/releng.pp
 18     # not sure why these are required on Darwin..
 19     if ($::operatingsystem == 'Darwin') {
 20         include packages::mozilla::py27_virtualenv
 21         include buildslave::install
 22     }
Ugh, we include buildslave::install, not buildslave::startup.  Suddenly things make more sense!

modules/toplevel/manifests/worker/releng.pp
 18     # not sure why these are required on Darwin..
 19     if ($::operatingsystem == 'Darwin') {
 20         include packages::mozilla::py27_virtualenv
 21         include buildslave::install
 22     }
 
modules/packages/manifests/mozilla/py27_virtualenv.pp
  11     include packages::mozilla::python27
installs /tools/python27-virtualenv/bin/virtualenv

modules/packages/manifests/mozilla/python27.pp
installs /tools/python27/bin/python

modules/buildslave/manifests/install.pp
  5     include packages::mozilla::python27
...
 30     python::virtualenv {
 31         $virtualenv_path:
 32             python   => $::packages::mozilla::python27::python,
 33             require  => $py_require,
 34             packages => $packages;
creates virtualenv in /tools/buildbot

modules/python/manifests/virtualenv.pp
  7     include python::virtualenv::settings  # just variables
  8     include python::virtualenv::prerequisites

modules/python/manifests/virtualenv/prerequisites.pp
 15     python::misc_python_file {
 16         'virtualenv.py': ;
 17     } -> Anchor['python::virtualenv::prerequisites::end']
[root@t-yosemite-r7-0121.test.releng.scl3.mozilla.com ~]# find /tools -maxdepth 1 -ls
446387        0 drwxr-xr-x   10 root             wheel                 340 Jun 14 07:35 /tools
482897        0 drwxr-xr-x    5 root             admin                 170 Jun 14 07:36 /tools/buildbot
451428        0 drwxr-xr-x    6 root             wheel                 204 Jun 14 07:36 /tools/misc-python
451367        8 lrwxrwxrwx    1 root             wheel                  15 Jun 14 07:33 /tools/python -> /tools/python27
451424        8 lrwxrwxrwx    1 root             wheel                  15 Jun 14 07:33 /tools/python2 -> /tools/python27
453416        0 drwxr-xr-x    5 root             wheel                 170 Oct 14  2014 /tools/python27
458454        0 drwxr-xr-x    4 root             wheel                 136 Sep 13  2016 /tools/python27-mercurial
482878       80 -rwxr-xr-x    1 root             wheel               37090 Jun 14 07:35 /tools/tooltool.py
458336        0 drwxrwxr-x    4 root             wheel                 136 Jul 19  2012 /tools/virtualenv

[root@t-yosemite-r7-0121.test.releng.scl3.mozilla.com ~]# ls -al /usr/local/bin/virtualenv
lrwxr-xr-x  1 administrator  501    32 Jun 14 07:34 /usr/local/bin/virtualenv -> /tools/virtualenv/bin/virtualenv

packages::mozilla::python27 is included elsewhere in toplevel::worker::releng, so we accept the need to install a custom python. Similarly with packages::mozilla::tooltoo.  So that leaves

 /tools/buildbot/bin/python
 /tools/virtualenv/bin/virtualenv
 /usr/local/bin/virtualenv

as suspect paths.
Only the first of those is really a concern.  Installing /tools/virtualenv/bin/virtualenv for a TC worker isn't crazy.

But it's a big problem:

>  $grep -r /tools/buildbot *
> testing/mozharness/configs/web_platform_tests/prod_config.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/web_platform_tests/prod_config.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/firefox_ui_tests/taskcluster.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/firefox_ui_tests/taskcluster.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/marionette/prod_config.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/marionette/prod_config.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/builders.py:            'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/releng_infra_configs/builders.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/builders.py:            'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/releng_infra_configs/builders.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/testers.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/testers.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/releng_infra_configs/testers.py:            'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/talos/mac_config.py:PYTHON = '/tools/buildbot/bin/python'
> testing/mozharness/configs/talos/linux_config.py:PYTHON = '/tools/buildbot/bin/python'
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_fuzzing_asan_tc.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan_and_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_debug_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_debug_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan_tc_and_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/32_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_code_coverage.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_valgrind.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan_tc.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_asan.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_opt.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_opt.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_debug.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_stat_and_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_noopt_debug.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_debug_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_debug_artifact.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_sub_linux_configs/64_add-on-devel.py:        'PATH': '/home/worker/workspace/build/src/gcc/bin:/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_base_linux_64_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_linux_64_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_base_mac_64_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_mac_64_builds.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_cross_noopt_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_artifact.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_stat_and_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_cross_debug.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_debug_artifact.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_debug_artifact.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_sub_mac_configs/64_add-on-devel.py:        'PATH': '/tools/python/bin:/tools/buildbot/bin:/opt/local/bin:/usr/bin:'
> testing/mozharness/configs/builds/releng_base_linux_32_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_linux_32_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib/ccache:\
> testing/mozharness/configs/builds/releng_base_mac_64_cross_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_mac_64_cross_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:\
> testing/mozharness/configs/builds/releng_base_android_64_builds.py:        "buildbot": "/tools/buildbot/bin/buildbot",
> testing/mozharness/configs/builds/releng_base_android_64_builds.py:        'PATH': '/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin',
> testing/mozharness/configs/unittests/mac_unittest.py:        'python': '/tools/buildbot/bin/python',
> testing/mozharness/configs/unittests/mac_unittest.py:        'virtualenv': ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py'],
> testing/mozharness/configs/unittests/linux_unittest.py:        "python": "/tools/buildbot/bin/python",
> testing/mozharness/configs/unittests/linux_unittest.py:        "virtualenv": ["/tools/buildbot/bin/python", "/tools/misc-python/virtualenv.py"],

So /tools/buildbot/bin/python, /tools/misc-python/virtualenv.py, and /tools/buildbot/bin/buildbot.  The latter is only used for sendchanges, so it can be safely ignored (and ripped out when buildbot is dead):

testing/mozharness/mozharness/mozilla/buildbot.py:
148     def invoke_sendchange(self, downloadables=None, branch=None,
149                    username="sendchange-unittest", sendchange_props=None):
150         """ Generic sendchange, currently b2g- and unittest-specific.
151             """
152         c = self.config
153         buildbot = self.query_exe("buildbot", return_type="list")


From a linux desktop-test loaner:

root@be16ec41c500:~# ls /tools
ls: cannot access '/tools': No such file or directory

so clearly it's possible to run most stuff without either of these paths!
$ grep -r query_exe..python testing/mozharness/

It looks like nothing ever uses 'python': '/tools/buildbot/bin/python'.  That makes sense, as sys.executable is a much better choice, and

$ grep -r sys.executable testing/mozharness/ | wc -l
112

So I think we can ignore those.

As for virtualenv, it looks like mozharness preferrs the in-tree vendored version, but that's not available without a source checkout, in which case it searches for it.  It's probably safe enough to replace `["/tools/buildbot/bin/python", "/tools/misc-python/virtualenv.py"` with `"/tools/virtualenv/bin/virtualenv"` globally and ensure that we install packages::mozilla::py27_virtualenv.
In the try job above, I replaced /tools/buildbot with /nosuch/path.  Lots of

15:12:18    FATAL - The executable '/nosuch/path/bin/python' is not found; not creating virtualenv!

so it looks like the analysis in comment 8 is accurate.  I'll try a new push with the addition of the change mentioned in the last comment.
OK, that was successful!  I added a linux build and test just to check, and they were fine too.

The happy thing is, the new paths exist on existing hardware workers (linux talos) too, so I think we can land a change in-tree to use the new virtualenv and drop the python paths, and once that's "sunk in" a bit start removing the bogus paths from the OS X testers.
Comment on attachment 8896481 [details]
Bug 1335353: remove refs to /tools/buildbot;

https://reviewboard.mozilla.org/r/167730/#review172960

I think this is good.
We can probably tear out all references to sendchanges from testing/mozharness as well, as well as other buildbot references. This is a great start, thanks!
Attachment #8896481 - Flags: review?(aki) → review+
The try run reliably failed
  Cpp - https://tools.taskcluster.net/groups/fm0B3GzlT6O8AexlZD7hTg/tasks/IgmQpjcSSz-8EWl1w9E1Ig/runs/0/logs/public%2Flogs%2Flive.log
  Gtest - https://tools.taskcluster.net/groups/fm0B3GzlT6O8AexlZD7hTg/tasks/HXDKk8g7RQScA-HjVj2eIg/runs/0/logs/public%2Flogs%2Flive.log

That seems to be due to

00:31:53    FATAL - There are specified suites that are incompatible with --artifact try syntax flag: gtest

[task 2017-08-11T23:33:12.460265Z] 23:33:12    FATAL - There are specified suites that are incompatible with --artifact try syntax flag: cppunittest

so I guess that's my fault in using --artifact.  The other failures seem to be "normal" test failures.  So *crosses fingers* I think this is good to land.

Once this is landed, I will land a puppet patch that removes the buildslave::install class but keeps /tools/buildbot/bin/python and /tools/misc-python/virtualenv.py in place so that old tasks continue to run.  I'll mark that with a TODO pointing to this bug, then put it on ice for a year.
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/76f32438e9f9
remove refs to /tools/buildbot; r=aki
My try syntax was -b d -t all, which didn't trigger talos.  Oops :(

Mac talos was successful (it finds virtualenv in $PATH).  But packages::mozilla::py27_virtualenv isn't installed on linux.  Let's fix that first.
Flags: needinfo?(dustin)
Comment on attachment 8897141 [details]
Bug 1335353: add py27_virtualenv on Darwin slaves;

https://reviewboard.mozilla.org/r/168428/#review173682
Attachment #8897141 - Flags: review?(aki) → review+
(In reply to Dustin J. Mitchell [:dustin] from comment #22)
> https://hg.mozilla.org/build/puppet/rev/
> 8e1788f19e8b270fcf20c153604a69c881cecd17
> 
> Jake, mind merging that for me?

Merged :-)
https://hg.mozilla.org/build/puppet/rev/41491e9da8f339fb1390cc0d64716ca2cc96039e
Flags: needinfo?(jwatkins)
Mon Aug 14 16:58:39 -0700 2017 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Cannot install on Ubuntu at /etc/puppet/production/modules/packages/manifests/mozilla/py27_virtualenv.pp:34 on node talos-linux64-ix-047.test.releng.scl3.mozilla.com
Mon Aug 14 16:58:39 -0700 2017 Puppet (err): Could not retrieve catalog; skipping run
Thanks for backing that out.  I knew it was too simple :(
For the taskcluster-worker-based Linux Talos, I just used the distro virtualenv.  I bet we could do the same here.  So I'll re-draft that to include packages::virtualenv -- and implement that for Ubuntu 12.04 -- for linux test slaves and leave everything else unchanged.
That will probably work, though will the path be different? in which case we'll need to point the linux talos mh configs at the correct path.
It will still be in PATH, which is the critical bit (that's how it's found on OS X, per comment 19)
[root@talos-linux64-ix-049 ~]# python /tools/misc-python/virtualenv.py  --version
1.7.1.2
which is the same as the version in ubuntu 12.04, so no worries there..
Jake, do yo mind merging the puppetagain change?
Flags: needinfo?(jwatkins)
(In reply to Dustin J. Mitchell [:dustin] from comment #31)
> Jake, do yo mind merging the puppetagain change?

looks like aki beat me to it :-)
Flags: needinfo?(jwatkins)
Looks like that landing stuck, so I'll push my patch through try again, then land it.  Once it's merged around, I'll try deleting /tools/buildbot from a few workers.  If those cause failures, we can kill the workers and address the issue.
14 hours later, the try jobs ran and they are all green.  They are using the virtualenv in PATH:

22:12:51     INFO - Running main action method: create_virtualenv
22:12:51     INFO - Creating virtualenv /builds/slave/test/build/venv
22:12:51     INFO - Running command: ['virtualenv', '--no-site-packages', '--distribute', '/builds/slave/test/build/venv'] in
Pushed by bjones@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/88081c5bc847
remove refs to /tools/buildbot; r=aki
https://hg.mozilla.org/mozilla-central/rev/88081c5bc847
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Relanded in https://hg.mozilla.org/build/puppet/rev/97ef23d7d6d4
backed out again in https://hg.mozilla.org/build/puppet/rev/f8d448747286

Cannot install on Ubuntu at /etc/puppet/production/modules/packages/manifests/mozilla/py27_virtualenv.pp:34 on node talos-linux64-ix-067.test.releng.scl3.mozilla.com
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Ok, this one will *only* affect Darwin, and that class always works on Darwin, so i think this should be fine.
Keywords: leave-open
[root@t-yosemite-r7-0115.test.releng.scl3.mozilla.com ~]# which virtualenv
/usr/local/bin/virtualenv
yay!  And I don't see any errors on a talos-ix run either.

Now to remove the now-unnecessary `include buildslave::install`.
Attached patch bug1335353.patchSplinter Review
Attachment #8902901 - Flags: review?(aki)
Fixed in

https://github.com/taskcluster/taskcluster-login/commit/1b3de7d5eaa4330974f5eb90e438df829399e89a

I now get creds with the appropriate scopes in the oidc demo.  YAY!
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
Status: RESOLVED → UNCONFIRMED
Ever confirmed: false
Resolution: FIXED → ---
Attachment #8902901 - Flags: review?(aki) → review+
Attachment #8902901 - Flags: checked-in+
I'm reimaging t-yosemite-r7-0115 to see if it still works.
..ran one green task, so I'm calling it good..
Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
No longer blocks: t-yosemite-r7-0115
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: FIXED → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Depends on: 1397340
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.