1328274 - [taskcluster-worker] [macosx-engine] Tests fail in mac pool of machines

Status: RESOLVED FIXED

(Taskcluster :: Operations and Service Requests, task, P1)

Description

[Wander Lairson Costa]

All tests are currently failing https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound

Comment 1

[Alin Selagea [:aselagea]]

Tests started failing on Dec 19: https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=82d90e9cc147565a24b205922957928026ba3677

Taking a look at the logs, it can be noticed that the jobs fail when trying to create the virtual environment as there's no "/tools/buildbot/bin/python" file on the machines used for running these tests:

04:38:27     INFO - Creating virtualenv /Users/worker-VtuE1wnhQ4a0jOJT08xNFQ/build/venv
04:38:27    FATAL - The executable '/tools/buildbot/bin/python' is not found; not creating virtualenv!
04:38:27    FATAL - Running post_fatal callback...
04:38:27    FATAL - Exiting -1

Snippet from a successful run from Dec 19 (https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=ff8e8c79f02902e3e5e6075736d2bd1b0da2f41f):

16:45:12     INFO - Creating virtualenv /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv
16:45:12     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv'] in /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build
16:45:12     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv
16:45:12     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
16:45:12     INFO -  Using real prefix '/tools/python27'
16:45:12     INFO -  New python executable in /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv/bin/python
16:45:13     INFO -  Installing distribute.............................................................................................................................................................................................done.
16:45:16     INFO -  Installing pip.................done.
16:45:16     INFO - Return code: 0

The t-yosemite-r7 machines used for running these tests are those in range [0040-0059]. It looks that several changes were done to the configuration of those boxes (they're currently missing several files and folders which are required for running the tests, e.g. "/tools/buildbot/", "/tools/misc-python/", etc), yet the tests that are being run are still the same. That seems to cause the failures described above.

Comment 2

[Wander Lairson Costa]

Attachment: Install tools to run tests on taskcluster-worker. r=dustin

Buildbot tools and python virtualenv are requirements to run tests
through mozharness.

Comment 3

[Wander Lairson Costa]

Comment on attachment 8823767 [details] [diff] [review]
Install tools to run tests on taskcluster-worker. r=dustin

https://treeherder.allizom.org/#/jobs?repo=try&revision=6c321d29a019&selectedJob=60685166

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8823767 [details] [diff] [review]
Install tools to run tests on taskcluster-worker. r=dustin

Review of attachment 8823767 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/toplevel/manifests/worker/releng.pp
@@ +12,5 @@
>      include packages::mozilla::py27_mercurial
>      # TODO: run mig agent on boot?
>      include mig::agent::daemon
> +    include packages::mozilla::py27_virtualenv
> +    include buildslave::install

This makes me pretty sad!

    python::virtualenv {
        "$virtualenv_path":
            python => $::packages::mozilla::python27::python,
            require => $py_require,
            packages => $packages;
    }

I suspect the only thing we really need is /tools/buildbot/bin/python, which is used as just a generic python interpreter (no idea why it's not /tools/python27/bin/python..).

Can you try removing /tools/buildbot and including code here to make a symlink from /tools/buildbot/bin/buildbot to /tools/python27/bin/python2.7?  If that works, I'll get a bug on file to try to switch those absolute paths in mozharness (later).

If it doesn't work, we can just install Buildbot :(

Comment 5

[Wander Lairson Costa]

Attachment: Install tools to run tests on taskcluster-worker. r=dustin

Python and virtualenv are requirements to run tests
through mozharness.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8823900 [details] [diff] [review]
Install tools to run tests on taskcluster-worker. r=dustin

Review of attachment 8823900 [details] [diff] [review]:
-----------------------------------------------------------------

I like it, assuming it works!

Maybe add a comment pointing to bug 1328927?

Comment 7

[Wander Lairson Costa]

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> Comment on attachment 8823900 [details] [diff] [review]
> Install tools to run tests on taskcluster-worker. r=dustin
> 
> Review of attachment 8823900 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I like it, assuming it works!
> 
> Maybe add a comment pointing to bug 1328927?

git/hg blame is your friend :-)

Comment 8

[Wander Lairson Costa]

https://hg.mozilla.org/build/puppet/rev/6b2885f0adf94eb83285365fd34f59989d38feb1

Comment 9

[Wander Lairson Costa]

https://hg.mozilla.org/build/puppet/rev/0045fa4d3766

Comment 10

[Wander Lairson Costa]

It feels like the symlink approach is not an option anymore.
https://public-artifacts.taskcluster.net/JifgskY0TF-05mdyf_k2Rw/0/public/logs/live_backing.log

Comment 11

[Wander Lairson Costa]

Attachment: Fix taskcluster-worker bustage. r=dustin

The symlink approach applied doesn't work anymore (not sure why). There
is a missing /tools/misc-python directory. Let's just play safe and
install buildslave::buildbot module to avoid future regressions, later
we can figure a more elegant way of handling this.

Comment 12

[Wander Lairson Costa]

https://hg.mozilla.org/build/puppet/rev/ac3bcc0ead3e