1628498 - Frequent OSError: [Errno 40] Too many levels of symbolic links: '/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python'

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=296852481&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/flKUIfnASmiOu2KiXEalMg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-04-08T22:13:11.062Z] Running: /builds/worker/checkouts/gecko/mach python /builds/worker/workspace/mozharness/scripts/web_platform_tests.py --config-file /builds/worker/workspace/mozharness/configs/web_platform_tests/prod_config.py --config-file /builds/worker/workspace/mozharness/configs/remove_executables.py --test-type=testharness --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --allow-software-gl-layers --total-chunk=12 --this-chunk=3 --download-symbols=ondemand
[task 2020-04-08T22:13:11.503Z] Error running mach:
[task 2020-04-08T22:13:11.503Z]
[task 2020-04-08T22:13:11.503Z] ['python', '/builds/worker/workspace/mozharness/scripts/web_platform_tests.py', '--config-file', '/builds/worker/workspace/mozharness/configs/web_platform_tests/prod_config.py', '--config-file', '/builds/worker/workspace/mozharness/configs/remove_executables.py', '--test-type=testharness', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--allow-software-gl-layers', '--total-chunk=12', '--this-chunk=3', '--download-symbols=ondemand']
[task 2020-04-08T22:13:11.503Z]
[task 2020-04-08T22:13:11.503Z] The error occurred in code that was called by the mach command. This is either
[task 2020-04-08T22:13:11.503Z] a bug in the called code itself or in the way that mach is calling it.
[task 2020-04-08T22:13:11.503Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2020-04-08T22:13:11.503Z] isn't, please use |./mach busted file| to report it. If |./mach busted| is
[task 2020-04-08T22:13:11.503Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2020-04-08T22:13:11.503Z]
[task 2020-04-08T22:13:11.503Z] If filing a bug, please include the full output of mach, including this error
[task 2020-04-08T22:13:11.503Z]
[task 2020-04-08T22:13:11.503Z] OSError: [Errno 40] Too many levels of symbolic links: '/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python'
[task 2020-04-08T22:13:11.503Z]
[task 2020-04-08T22:13:11.503Z] File "/builds/worker/checkouts/gecko/python/mach_commands.py", line 66, in python
[task 2020-04-08T22:13:11.503Z] self._activate_virtualenv()
[task 2020-04-08T22:13:11.503Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 829, in _activate_virtualenv
[task 2020-04-08T22:13:11.503Z] self.virtualenv_manager.ensure()
[task 2020-04-08T22:13:11.503Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 193, in ensure
[task 2020-04-08T22:13:11.503Z] if self.up_to_date(python):
[task 2020-04-08T22:13:11.503Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/virtualenv.py", line 162, in up_to_date
[task 2020-04-08T22:13:11.503Z] if ((python, python_size) != (self.python_path, os.path.getsize(self.python_path)) and
[task 2020-04-08T22:13:11.503Z] File "/usr/lib/python2.7/genericpath.py", line 57, in getsize
[task 2020-04-08T22:13:11.503Z] return os.stat(filename).st_size
[task 2020-04-08T22:13:11.520Z] cleanup
[task 2020-04-08T22:13:11.520Z] + cleanup
[task 2020-04-08T22:13:11.520Z] + local rv=1
[task 2020-04-08T22:13:11.520Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2020-04-08T22:13:11.520Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2020-04-08T22:13:11.522Z] + true
[task 2020-04-08T22:13:11.522Z] + cleanup_xvfb
[task 2020-04-08T22:13:11.522Z] ++ pidof Xvfb
[task 2020-04-08T22:13:11.525Z] + local xvfb_pid=40
[task 2020-04-08T22:13:11.525Z] + local vnc=false
[task 2020-04-08T22:13:11.525Z] + local interactive=false
[task 2020-04-08T22:13:11.525Z] + '[' -n 40 ']'
[task 2020-04-08T22:13:11.525Z] + [[ false == false ]]
[task 2020-04-08T22:13:11.525Z] + [[ false == false ]]
[task 2020-04-08T22:13:11.525Z] + kill 40
[task 2020-04-08T22:13:11.525Z] + screen -XS xvfb quit
[task 2020-04-08T22:13:11.529Z] + exit 1
[fetches 2020-04-08T22:13:11.529Z] removing /builds/worker/fetches
[fetches 2020-04-08T22:13:11.532Z] finished
[taskcluster 2020-04-08 22:13:11.853Z] === Task Finished ===
[taskcluster 2020-04-08 22:13:13.048Z] Unsuccessful task run with exit code: 1 completed in 9.077 seconds

Comment 1

[Narcis Beleuzu [:NarcisB]]

James, the failure seems to be caused by https://hg.mozilla.org/mozilla-central/rev/c4b80e083d0b .
Can you please take a look?

Comment 2

[Intermittent Failures Robot]

29 failures in 913 pushes (0.032 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 24
• autoland: 5

Platform breakdown:

• linux1804-64: 5
• linux1804-64-qr: 16
• linux1804-64-shippable: 6
• linux1804-64-asan: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628498&startday=2020-04-08&endday=2020-04-08&tree=all

Comment 3

[Tom Prince [:tomprince]]

I suspect somehow mach is trying to setup the init virtualenv with that environment activated, and is linking python in that environment to itself.

We may also have an old version of virtualenv and newer versions may handle that case more gracefully.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Also affects awsy and GTest and only occurred on two workers: i-0f50e7448c6998851 and i-05ecddf318589cb95

This only failed commit access level 3 tasks and there it all it ran while Try ones were not failing.

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Both machines got provisioned between 22:10 and 22:15 UTC yesterday (2020-04-08).

Comment 6

[Mike Hommey [:glandium]]

There is something very wrong with that task. It has a checkout cache, and it uses it, while run-task doesn't have --gecko-checkout on its command line.

Comment 7

[James Graham [:jgraham]]

Yeah, this seems to be happening before any harness-specific code runs; it's mach python that's failing.

Comment 8

[James Graham [:jgraham]]

I'm seeing this for some tasks in my try push https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=296934982&tier=1%2C2%2C3&revision=da023a92be364c1511a87a17b019d04a9a6b5187 on different workers to the ones that Aryx mentioned. Is there a chance that this is affecting more workers over time and we're going to end up with everything broken?

Comment 9

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

James' link is a machine (i-0f7094f6e9cfe0306) which first ran tasks successfully on Try and production and later failed the ones on Try.

3 more machines which failed on production:
i-083c3cefe33c1f81f
i-0a066f51530f28ba7
i-0a702fffd06ab973f

Comment 10

[James Graham [:jgraham]]

This also happened locally:

./mach wpt --no-preload --setpref marionette.log.level=Trace testing/web-platform/tests/infrastructure/reftest/reftest_match-print.html  --reftest-screenshot=always --timeout-multiplier 10000
New python executable in /home/jgraham/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
Not overwriting existing python script /home/jgraham/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python (you must use /home/jgraham/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7)
Traceback (most recent call last):
  File "/home/jgraham/develop/gecko/third_party/python/virtualenv/virtualenv.py", line 2634, in <module>
    main()
  File "/home/jgraham/develop/gecko/third_party/python/virtualenv/virtualenv.py", line 870, in main
    symlink=options.symlink,
  File "/home/jgraham/develop/gecko/third_party/python/virtualenv/virtualenv.py", line 1162, in create_environment
    install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
  File "/home/jgraham/develop/gecko/third_party/python/virtualenv/virtualenv.py", line 1691, in install_python
    raise e
OSError: [Errno 40] Too many levels of symbolic links
Error running mach:

    ['wpt', '--no-preload', '--setpref', 'marionette.log.level=Trace', 'testing/web-platform/tests/infrastructure/reftest/reftest_match-print.html', '--reftest-screenshot=always', '--timeout-multiplier', '10000']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Failed to create virtualenv: /home/jgraham/develop/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init (virtualenv.py retcode: 1)

  File "/home/jgraham/develop/gecko/testing/web-platform/mach_commands.py", line 375, in run_wpt
    return self.run_web_platform_tests(**params)
  File "/home/jgraham/develop/gecko/testing/web-platform/mach_commands.py", line 345, in run_web_platform_tests
    self.setup()
  File "/home/jgraham/develop/gecko/testing/web-platform/mach_commands.py", line 337, in setup
    self._activate_virtualenv()
  File "/home/jgraham/develop/gecko/python/mozbuild/mozbuild/base.py", line 829, in _activate_virtualenv
    self.virtualenv_manager.ensure()
  File "/home/jgraham/develop/gecko/python/mozbuild/mozbuild/virtualenv.py", line 195, in ensure
    return self.build(python)
  File "/home/jgraham/develop/gecko/python/mozbuild/mozbuild/virtualenv.py", line 499, in build
    self.create(python)
  File "/home/jgraham/develop/gecko/python/mozbuild/mozbuild/virtualenv.py", line 239, in create
    self.virtualenv_root, result))

Comment 11

[James Graham [:jgraham]]

Bug 1627163 looks a bit suspicious in that it's the only recent change to the virtualenv code.

Comment 12

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The tree is closed for this.

Comment 13

[Emilio Cobos Álvarez (:emilio)]

I did hit this locally and a clobber helped, fwiw.

Comment 14

[Emilio Cobos Álvarez (:emilio)]

Fwiw:

$ ls -la /home/emilio/src/moz/gecko/obj-debug/_virtualenvs/init/bin/                                                                                                                                                                       
total 56
drwxr-xr-x. 2 emilio emilio 4096 Apr  9 19:11 .
drwxr-xr-x. 5 emilio emilio 4096 Apr  8 12:24 ..
-rw-r--r--. 1 emilio emilio 2243 Apr  8 12:24 activate
-rw-r--r--. 1 emilio emilio 1462 Apr  8 12:24 activate.csh
-rw-r--r--. 1 emilio emilio 3127 Apr  8 12:24 activate.fish
-rw-r--r--. 1 emilio emilio 1751 Apr  8 12:24 activate.ps1
-rw-r--r--. 1 emilio emilio 1517 Apr  8 12:24 activate_this.py
-rwxr-xr-x. 1 emilio emilio  282 Apr  8 12:24 easy_install
-rwxr-xr-x. 1 emilio emilio  282 Apr  8 12:24 easy_install-2.7
-rwxr-xr-x. 1 emilio emilio  269 Apr  8 12:24 pip
-rwxr-xr-x. 1 emilio emilio  269 Apr  8 12:24 pip2
-rwxr-xr-x. 1 emilio emilio  269 Apr  8 12:24 pip2.7
lrwxrwxrwx. 1 emilio emilio    7 Apr  9 19:11 python -> python2
lrwxrwxrwx. 1 emilio emilio    9 Apr  8 12:24 python2 -> python2.7
lrwxrwxrwx. 1 emilio emilio    7 Apr  9 19:11 python2.7 -> python2
-rwxr-xr-x. 1 emilio emilio 2372 Apr  8 12:24 python-config
-rwxr-xr-x. 1 emilio emilio  260 Apr  8 12:24 wheel

Comment 15

[Emilio Cobos Álvarez (:emilio)]

In the already-clobbered build:

$ ls -la /home/emilio/src/moz/mozilla-unified/obj-debug/_virtualenvs/init/bin/                                                                                                                                                             
total 72
drwxr-xr-x. 2 emilio emilio  4096 Apr  9 19:04 .
drwxr-xr-x. 5 emilio emilio  4096 Apr  9 19:04 ..
-rw-r--r--. 1 emilio emilio  2253 Apr  9 19:04 activate
-rw-r--r--. 1 emilio emilio  1472 Apr  9 19:04 activate.csh
-rw-r--r--. 1 emilio emilio  3137 Apr  9 19:04 activate.fish
-rw-r--r--. 1 emilio emilio  1751 Apr  9 19:04 activate.ps1
-rw-r--r--. 1 emilio emilio  1517 Apr  9 19:04 activate_this.py
-rwxr-xr-x. 1 emilio emilio   290 Apr  9 19:04 easy_install
-rwxr-xr-x. 1 emilio emilio   290 Apr  9 19:04 easy_install-2.7
-rwxr-xr-x. 1 emilio emilio   277 Apr  9 19:04 pip
-rwxr-xr-x. 1 emilio emilio   277 Apr  9 19:04 pip2
-rwxr-xr-x. 1 emilio emilio   277 Apr  9 19:04 pip2.7
lrwxrwxrwx. 1 emilio emilio     7 Apr  9 19:04 python -> python2
-rwxr-xr-x. 1 emilio emilio 16056 Apr  9 19:04 python2
lrwxrwxrwx. 1 emilio emilio     7 Apr  9 19:04 python2.7 -> python2
-rwxr-xr-x. 1 emilio emilio  2382 Apr  9 19:04 python-config
-rwxr-xr-x. 1 emilio emilio   268 Apr  9 19:04 wheel

Comment 16

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1628498 - Hackaround: Try to build virtualenv from scratch if there was a previous one. r=#build

This works for me and should allow trees to reopen.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88d221a0324a
Hackaround: Try to build virtualenv from scratch if there was a previous one. r=rstewart

Comment 18

[Mike Hommey [:glandium]]

Attachment: Bug 1628498 - Fix corner-case in virtualenv creation.

This also reverts the previous hack around the issue.

Comment 19

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/88d221a0324a

Comment 20

[Intermittent Failures Robot]

115 failures in 1016 pushes (0.113 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 13
• autoland: 15
• try: 79
• mozilla-beta: 8

Platform breakdown:

• linux1804-64-qr: 30
• linux1804-64-tsan: 14
• linux1804-64-asan: 25
• linux1804-64: 22
• linux1804-64-shippable: 7
• linux1804-64-shippable-qr: 9
• linux1804-64-devedition: 5
• linux1804-32-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628498&startday=2020-04-09&endday=2020-04-09&tree=all

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/81bc8ca6099a
Fix corner-case in virtualenv creation. r=rstewart

Comment 22

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/81bc8ca6099a

Comment 23

[Intermittent Failures Robot]

144 failures in 4823 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #2 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 37
• autoland: 20
• try: 79
• mozilla-beta: 8

Platform breakdown:

• linux1804-64: 27
• linux1804-64-qr: 46
• linux1804-64-shippable: 13
• linux1804-64-asan: 27
• linux1804-64-tsan: 14
• linux1804-64-shippable-qr: 9
• linux1804-64-devedition: 5
• linux1804-32-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628498&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 24

[Intermittent Failures Robot]

7 failures in 4528 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• try: 7

Platform breakdown:

• linux1804-64: 4
• linux1804-64-asan: 1
• linux1804-64-shippable: 1
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628498&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 25

[Intermittent Failures Robot]

2 failures in 5582 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2

Platform breakdown:

• linux1804-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1628498&startday=2020-04-20&endday=2020-04-26&tree=all