1395960 - t-yosemite-r7-394 is broken, permanently hitting The executable '/tools/virtualenv/bin/virtualenv' is not found; not creating virtualenv!

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=127739722&repo=autoland

07:19:19     INFO - Downloading https://queue.taskcluster.net/v1/task/GdN7QfDAThSLrdK9MOu05Q/artifacts/public/build/target.dmg to /Users/cltbld/tasks/task_1504273013/installer.dmg
07:19:19     INFO - retry: Calling _download_file with args: (), kwargs: {'url': 'https://queue.taskcluster.net/v1/task/GdN7QfDAThSLrdK9MOu05Q/artifacts/public/build/target.dmg', 'file_name': '/Users/cltbld/tasks/task_1504273013/installer.dmg'}, attempt #1
07:19:25     INFO - Downloaded 65936200 bytes.
07:19:25     INFO - Setting buildbot property build_url to https://queue.taskcluster.net/v1/task/GdN7QfDAThSLrdK9MOu05Q/artifacts/public/build/target.dmg
07:19:25     INFO - mkdir: /Users/cltbld/tasks/task_1504273013/properties
07:19:25     INFO - Writing buildbot properties ['build_url'] to /Users/cltbld/tasks/task_1504273013/properties/build_url
07:19:25     INFO - Writing to file /Users/cltbld/tasks/task_1504273013/properties/build_url
07:19:25     INFO - Contents:
07:19:25     INFO -  build_url:https://queue.taskcluster.net/v1/task/GdN7QfDAThSLrdK9MOu05Q/artifacts/public/build/target.dmg
07:19:26     INFO - Running post-action listener: _resource_record_post_action
07:19:26     INFO - Running post-action listener: find_tests_for_verification
07:19:26     INFO - Running post-action listener: set_extra_try_arguments
07:19:26     INFO - [mozharness: 2017-09-01 14:19:26.223854Z] Finished download-and-extract step (success)
07:19:26     INFO - [mozharness: 2017-09-01 14:19:26.223992Z] Running create-virtualenv step.
07:19:26     INFO - Running pre-action listener: _install_mozbase
07:19:26     INFO - Running pre-action listener: _pre_create_virtualenv
07:19:26     INFO - Running pre-action listener: _resource_record_pre_action
07:19:26     INFO - Running main action method: create_virtualenv
07:19:26     INFO - Creating virtualenv /Users/cltbld/tasks/task_1504273013/build/venv
07:19:26    FATAL - The executable '/tools/virtualenv/bin/virtualenv' is not found; not creating virtualenv!
07:19:26    FATAL - Running post_fatal callback...
07:19:26    FATAL - Exiting -1
07:19:26     INFO - Running post-action listener: _resource_record_post_action
07:19:26     INFO - Running post-action listener: _start_resource_monitoring
07:19:26    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
07:19:26    ERROR -   File "/Users/cltbld/tasks/task_1504273013/mozharness/mozharness/base/script.py", line 2009, in run_action
07:19:26    ERROR -     method(action, success=success and self.return_code == 0)
07:19:26    ERROR -   File "/Users/cltbld/tasks/task_1504273013/mozharness/mozharness/base/python.py", line 575, in _start_resource_monitoring
07:19:26    ERROR -     self.activate_virtualenv()
07:19:26    ERROR -   File "/Users/cltbld/tasks/task_1504273013/mozharness/mozharness/base/python.py", line 496, in activate_virtualenv
07:19:26    ERROR -     execfile(activate, dict(__file__=activate))
07:19:26    ERROR - IOError: [Errno 2] No such file or directory: '/Users/cltbld/tasks/task_1504273013/build/venv/bin/activate_this.py'
07:19:26     INFO - [mozharness: 2017-09-01 14:19:26.228520Z] Finished create-virtualenv step (failed)
07:19:26    FATAL - Aborting due to failure in post-action listener.
07:19:26    FATAL - Running post_fatal callback...
07:19:26    FATAL - Exiting -1

Comment 1

[Intermittent Failures Robot]

49 failures in 180 pushes (0.272 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 17
* autoland: 15
* mozilla-central: 13
* try: 4

Platform breakdown:
* osx-10-10: 42
* macosx64-stylo: 4
* macosx64-stylo-disabled: 2
* macosx64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395960&startday=2017-09-01&endday=2017-09-01&tree=all

Comment 3

[Intermittent Failures Robot]

39 failures in 62 pushes (0.629 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 12
* try: 11
* mozilla-central: 11
* mozilla-inbound: 5

Platform breakdown:
* osx-10-10: 28
* macosx64-stylo: 7
* macosx64-nightly: 2
* macosx64-devedition: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395960&startday=2017-09-02&endday=2017-09-02&tree=all

Comment 4

[Intermittent Failures Robot]

152 failures in 939 pushes (0.162 failures/push) were associated with this bug in the last 7 days. 

This is the #12 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 48
* mozilla-central: 43
* mozilla-inbound: 37
* try: 23
* mozilla-beta: 1

Platform breakdown:
* osx-10-10: 118
* macosx64-stylo: 22
* macosx64-devedition: 6
* macosx64-nightly: 5
* macosx64-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395960&startday=2017-08-28&endday=2017-09-03&tree=all

Comment 5

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

t-yosemite-r7-394 is loaned for testing pip and python upgrade on RelOps team.

Comment 6

[Intermittent Failures Robot]

55 failures in 118 pushes (0.466 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 21
* mozilla-inbound: 14
* try: 10
* mozilla-central: 7
* mozilla-beta: 3

Platform breakdown:
* osx-10-10: 41
* macosx64-stylo: 9
* macosx64-nightly: 3
* macosx64-devedition: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395960&startday=2017-09-04&endday=2017-09-04&tree=all

Comment 7

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Machine t-yosemite-r7-394 was pinned to Dragos environment and it's configured to have generic worker disabled so it shouldn't cause any more problems:

<node 't-yosemite-r7-394.test.releng.mdc1.mozilla.com' {
<    $aspects           = [ 'low-security' ]
<    $slave_trustlevel  = 'try'
<    $pin_puppet_server = 'releng-puppet2.srv.releng.scl3.mozilla.com'
<    $pin_puppet_env    = 'dcrisan'
<    include fw::profiles::osx_taskcluster_worker
<    include toplevel::base
<    include generic_worker::disabled

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

From a look at the orangefactor links, this is *not* the same problem as bug 1396266.

Comment 9

[Intermittent Failures Robot]

63 failures in 924 pushes (0.068 failures/push) were associated with this bug in the last 7 days. 

This is the #37 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 28
* mozilla-inbound: 14
* try: 11
* mozilla-central: 7
* mozilla-beta: 3

Platform breakdown:
* osx-10-10: 48
* macosx64-stylo: 10
* macosx64-nightly: 3
* macosx64-devedition: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1395960&startday=2017-09-04&endday=2017-09-10&tree=all