Closed Bug 1328274 Opened 6 years ago Closed 5 years ago

[taskcluster-worker] [macosx-engine] Tests fail in mac pool of machines

Categories

(Taskcluster :: Operations and Service Requests, task, P1)

x86_64
macOS

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wcosta, Assigned: wcosta)

References

Details

Attachments

(2 files, 1 obsolete file)

Tests started failing on Dec 19: https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=82d90e9cc147565a24b205922957928026ba3677

Taking a look at the logs, it can be noticed that the jobs fail when trying to create the virtual environment as there's no "/tools/buildbot/bin/python" file on the machines used for running these tests:

04:38:27     INFO - Creating virtualenv /Users/worker-VtuE1wnhQ4a0jOJT08xNFQ/build/venv
04:38:27    FATAL - The executable '/tools/buildbot/bin/python' is not found; not creating virtualenv!
04:38:27    FATAL - Running post_fatal callback...
04:38:27    FATAL - Exiting -1

Snippet from a successful run from Dec 19 (https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=ff8e8c79f02902e3e5e6075736d2bd1b0da2f41f):

16:45:12     INFO - Creating virtualenv /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv
16:45:12     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv'] in /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build
16:45:12     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv
16:45:12     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
16:45:12     INFO -  Using real prefix '/tools/python27'
16:45:12     INFO -  New python executable in /Users/worker-IzBZYMzTQCShQVvjUBGq2A/build/venv/bin/python
16:45:13     INFO -  Installing distribute.............................................................................................................................................................................................done.
16:45:16     INFO -  Installing pip.................done.
16:45:16     INFO - Return code: 0

The t-yosemite-r7 machines used for running these tests are those in range [0040-0059]. It looks that several changes were done to the configuration of those boxes (they're currently missing several files and folders which are required for running the tests, e.g. "/tools/buildbot/", "/tools/misc-python/", etc), yet the tests that are being run are still the same. That seems to cause the failures described above.
See Also: → 1328238
Buildbot tools and python virtualenv are requirements to run tests
through mozharness.
Comment on attachment 8823767 [details] [diff] [review]
Install tools to run tests on taskcluster-worker. r=dustin

https://treeherder.allizom.org/#/jobs?repo=try&revision=6c321d29a019&selectedJob=60685166
Attachment #8823767 - Flags: review?(dustin)
Comment on attachment 8823767 [details] [diff] [review]
Install tools to run tests on taskcluster-worker. r=dustin

Review of attachment 8823767 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/toplevel/manifests/worker/releng.pp
@@ +12,5 @@
>      include packages::mozilla::py27_mercurial
>      # TODO: run mig agent on boot?
>      include mig::agent::daemon
> +    include packages::mozilla::py27_virtualenv
> +    include buildslave::install

This makes me pretty sad!

    python::virtualenv {
        "$virtualenv_path":
            python => $::packages::mozilla::python27::python,
            require => $py_require,
            packages => $packages;
    }

I suspect the only thing we really need is /tools/buildbot/bin/python, which is used as just a generic python interpreter (no idea why it's not /tools/python27/bin/python..).

Can you try removing /tools/buildbot and including code here to make a symlink from /tools/buildbot/bin/buildbot to /tools/python27/bin/python2.7?  If that works, I'll get a bug on file to try to switch those absolute paths in mozharness (later).

If it doesn't work, we can just install Buildbot :(
Attachment #8823767 - Attachment is obsolete: true
Attachment #8823767 - Flags: review?(dustin)
Python and virtualenv are requirements to run tests
through mozharness.
Attachment #8823900 - Flags: review?(dustin)
Comment on attachment 8823900 [details] [diff] [review]
Install tools to run tests on taskcluster-worker. r=dustin

Review of attachment 8823900 [details] [diff] [review]:
-----------------------------------------------------------------

I like it, assuming it works!

Maybe add a comment pointing to bug 1328927?
Attachment #8823900 - Flags: review?(dustin) → review+
(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> Comment on attachment 8823900 [details] [diff] [review]
> Install tools to run tests on taskcluster-worker. r=dustin
> 
> Review of attachment 8823900 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I like it, assuming it works!
> 
> Maybe add a comment pointing to bug 1328927?

git/hg blame is your friend :-)
https://hg.mozilla.org/build/puppet/rev/0045fa4d3766
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
It feels like the symlink approach is not an option anymore.
https://public-artifacts.taskcluster.net/JifgskY0TF-05mdyf_k2Rw/0/public/logs/live_backing.log
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
The symlink approach applied doesn't work anymore (not sure why). There
is a missing /tools/misc-python directory. Let's just play safe and
install buildslave::buildbot module to avoid future regressions, later
we can figure a more elegant way of handling this.
Attachment #8830716 - Flags: review?(dustin)
Attachment #8830716 - Flags: review?(dustin) → review+
https://hg.mozilla.org/build/puppet/rev/ac3bcc0ead3e
Status: REOPENED → RESOLVED
Closed: 6 years ago5 years ago
Resolution: --- → FIXED
Component: Operations → Operations and Service Requests
You need to log in before you can comment on or make changes to this bug.