1393344 - Xpcshell and Mozmill tests broken on Linux (2017-08-24): The executable '/tools/virtualenv/bin/virtualenv' is not found; not creating virtualenv!

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Jorg K (CEST = GMT+2)]

The executable '/tools/virtualenv/bin/virtualenv' is not found; not creating virtualenv!

https://treeherder.mozilla.org/logviewer.html#?job_id=125384099&repo=comm-central&lineNumber=477
18:22:06    FATAL - The executable '/tools/virtualenv/bin/virtualenv' is not found; not creating virtualenv!
18:22:06    FATAL - Running post_fatal callback...
18:22:06    FATAL - Exiting -1
18:22:06     INFO - Running post-action listener: _resource_record_post_action
18:22:06     INFO - Running post-action listener: _start_resource_monitoring
18:22:06    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
18:22:06    ERROR -   File "/builds/slave/test/scripts/mozharness/base/script.py", line 2019, in run_action
18:22:06    ERROR -     method(action, success=success and self.return_code == 0)
18:22:06    ERROR -   File "/builds/slave/test/scripts/mozharness/base/python.py", line 575, in _start_resource_monitoring
18:22:06    ERROR -     self.activate_virtualenv()
18:22:06    ERROR -   File "/builds/slave/test/scripts/mozharness/base/python.py", line 496, in activate_virtualenv
18:22:06    ERROR -     execfile(activate, dict(__file__=activate))
18:22:06    ERROR - IOError: [Errno 2] No such file or directory: '/builds/slave/test/build/venv/bin/activate_this.py'
18:22:06     INFO - [mozharness: 2017-08-24 01:22:06.745819Z] Finished create-virtualenv step (failed)
18:22:06    FATAL - Aborting due to failure in post-action listener.
18:22:06    FATAL - Running post_fatal callback...
18:22:06    FATAL - Exiting -1

First seen: Thu Aug 24, 2017, 1:57:56

M-C last good: c86b7150523c
M-C first bad: f0abd25e1f4a

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=c86b7150523c&tochange=f0abd25e1f4a

Looks like bug 1335353.

Comment 1

[Jorg K (CEST = GMT+2)]

Aki, since Tom is currently not available, can you help us out here.

Comment 2

[Aki Sasaki (not active)]

Dustin: shouldn't the bb osx testers have /tools/virtualenv/bin/virtualenv ?

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

The TC OSX testers do, yes:

[[e-mail] ~]# ls /tools/virtualenv/bin/virtualenv 
/tools/virtualenv/bin/virtualenv

But I guess t-yosemite-r7-0015 is still running BB and configured by a different puppet env.

[[e-mail] ~]# ls -al /tools/virtualenv/bin/virtualenv
ls: /tools/virtualenv/bin/virtualenv: No such file or directory
[[e-mail] ~]# which virtualenv

I don't think I *removed* anything, and in fact I added that package in toplevel::slave::releng https://hg.mozilla.org/build/puppet/rev/8e1788f19e8b

[[e-mail] ~]# grep tople /var/lib/puppet/classes.txt  | sort
toplevel::base
toplevel::slave
toplevel::slave::releng
toplevel::slave::releng::test
toplevel::slave::releng::test::gpu

[[e-mail] ~]# grep puppet-agent /var/log/system.log | tail -n 1
Aug 24 11:12:47 t-yosemite-r7-0015.test.releng.scl3.mozilla.com puppet-agent[308]: Finished catalog run in 24.78 seconds

yet

[[e-mail] ~]# grep virtu /var/lib/puppet/classes.txt 
python::virtualenv::settings
python::virtualenv::prerequisites

I don't see the node pinned to an environment, or with any specific config in moco-nodes.pp.  I don't remember if there's a way to see which hg revision a catalog was generated against.  Anyway, I think this needs investigation from someone of the puppet persuasion, because this isn't doing what it should be.

Comment 4

[Aki Sasaki (not active)]

Reopened bug 1335353; we need to install virtualenv on osx.

Comment 5

[Jorg K (CEST = GMT+2)]

Something happened, so Mozmill and Xpcshell tests have returned to some Macs. So depending on the luck of the draw, some tests run on Mac, others don't.

Comment 6

[Jorg K (CEST = GMT+2)]

Looks like tests are mostly running on Mac again.

Comment 7

[Jorg K (CEST = GMT+2)]

Taking "Mac" out of the summary, since when Mac last compiled, see bug 1400533, the tests worked.

Comment 8

[Tom Prince [:tomprince]]

Attachment: Bug 1393344: Restore virtualenv paths for thunderbird buildbot builds on linux.

Review commit: https://reviewboard.mozilla.org/r/181822/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181822/

Comment 9

[Tom Prince [:tomprince]]

It looks like this was probably broken by Bug 1335353 which changed the paths that mozharness looks for for virtualenv in.

Comment 10

[Tom Prince [:tomprince]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2975d2dbdfa0dc821bbc48ff762609eb71805418
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=1f6d9d002a3f9a66f3633e38417760757684ba1a

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 8910371 [details]
Bug 1393344: Restore virtualenv paths for thunderbird buildbot builds on linux.

https://reviewboard.mozilla.org/r/181822/#review187542

Does /tools/buildbot exist anymore? If so, this should work. Might be worth a try-cc push... you'll need --buildbot in your try syntax.

Comment 12

[Tom Prince [:tomprince]]

Unfortunately, I don't *think* this can be tested on try.  Thunderbirds test builders unconditionally get mozilla-central@default's mozharness directory, so we can't test this until it lands on m-c.

Comment 13

[Jorg K (CEST = GMT+2)]

Aryx, can you please land this on M-C.

Comment 14

[Tom Prince [:tomprince]]

Can we also get this uplifted to M-B?

Comment 15

[Jorg K (CEST = GMT+2)]

Let's check whether it works first, then request uplift.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3c6d8d0c66f5
Restore virtualenv paths for thunderbird buildbot builds on linux. r=aki

Comment 17

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/mozilla-central/rev/3c6d8d0c66f5

That worked :-)

Comment 18

[Jorg K (CEST = GMT+2)]

Comment on attachment 8910371 [details]
Bug 1393344: Restore virtualenv paths for thunderbird buildbot builds on linux.

Approval Request Comment
[Feature/Bug causing the regression]: Bug 1335353 (?)

This just a build configuration thing that is needed for Thunderbird. Change here:
testing/mozharness/configs/unittests/thunderbird_buildbot.py

Firefox is not affected. It just missed the uplift to beta.

Comment 19

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8910371 [details]
Bug 1393344: Restore virtualenv paths for thunderbird buildbot builds on linux.

No impact on Firefox code, taking it to help our thunderbird friends!

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/3c6d8d0c66f5

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/ecd8fc49d159