1644320 - Perma subprocess.CalledProcessError: Command '['/usr/local/bin/python3', '-m', 'pip', 'install', '--no-cache-dir', '--index-url', 'https://pypi.python.org/simple', '--upgrade', 'https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Treeherder Bug Filer]

Filed by: ccoroiu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=305549679&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/blE-iqCiSZSewcDF7CC-yw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-06-08T23:25:48.945Z] Running virtualenv with interpreter /usr/local/bin/python3
[task 2020-06-08T23:25:48.951Z] + bin/python3 condprof/main.py --geckodriver /Users/cltbld/tasks/task_1591658581/fetches/geckodriver --firefox /Users/cltbld/tasks/task_1591658581/fetches/target.dmg --scenario settled ../../archive
[task 2020-06-08T23:25:49.248Z] Looking in links: https://pypi.pub.build.mozilla.org/pub
[task 2020-06-08T23:25:49.249Z] Collecting pip==19.2.3 from https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl
[task 2020-06-08T23:25:49.301Z] Downloading https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl (1.4MB)
[task 2020-06-08T23:25:49.517Z] Collecting setuptools==41.2.0 from https://files.pythonhosted.org/packages/b2/86/095d2f7829badc207c893dd4ac767e871f6cd547145df797ea26baea4e2e/setuptools-41.2.0-py2.py3-none-any.whl
[task 2020-06-08T23:25:49.528Z] Downloading https://files.pythonhosted.org/packages/b2/86/095d2f7829badc207c893dd4ac767e871f6cd547145df797ea26baea4e2e/setuptools-41.2.0-py2.py3-none-any.whl (576kB)
[task 2020-06-08T23:25:49.612Z] Installing collected packages: pip, setuptools
[task 2020-06-08T23:25:49.612Z] Found existing installation: pip 19.0.3
[task 2020-06-08T23:25:49.754Z] Uninstalling pip-19.0.3:
[task 2020-06-08T23:25:49.839Z] Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Frameworks/Python.framework/Versions/3.7/bin/pip3'
[task 2020-06-08T23:25:49.839Z] Consider using the --user option or check the permissions.
[task 2020-06-08T23:25:49.839Z]
[task 2020-06-08T23:25:49.925Z] Traceback (most recent call last):
[task 2020-06-08T23:25:49.925Z] File "condprof/main.py", line 17, in <module>
[task 2020-06-08T23:25:49.925Z] check()
[task 2020-06-08T23:25:49.925Z] File "condprof/../condprof/check_install.py", line 108, in check
[task 2020-06-08T23:25:49.925Z] if update_pip():
[task 2020-06-08T23:25:49.925Z] File "condprof/../condprof/check_install.py", line 44, in update_pip
[task 2020-06-08T23:25:49.925Z] "https://files.pythonhosted.org/packages/b2/"
[task 2020-06-08T23:25:49.925Z] File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
[task 2020-06-08T23:25:49.925Z] raise CalledProcessError(retcode, cmd)
[task 2020-06-08T23:25:49.925Z] subprocess.CalledProcessError: Command '['/usr/local/bin/python3', '-m', 'pip', 'install', '--no-cache-dir', '--index-url', 'https://pypi.python.org/simple', '--upgrade', 'https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl', 'https://files.pythonhosted.org/packages/b2/86/095d2f7829badc207c893dd4ac767e871f6cd547145df797ea26baea4e2e/setuptools-41.2.0-py2.py3-none-any.whl']' returned non-zero exit status 1.
[fetches 2020-06-08T23:25:49.930Z] removing /Users/cltbld/tasks/task_1591658581/fetches
[fetches 2020-06-08T23:25:50.038Z] finished
[taskcluster 2020-06-08T23:25:50.049Z] Exit Code: 1
[taskcluster 2020-06-08T23:25:50.049Z] User Time: 3.990891s
[taskcluster 2020-06-08T23:25:50.049Z] Kernel Time: 1.292525s
[taskcluster 2020-06-08T23:25:50.049Z] Wall Time: 5.584323546s
[taskcluster 2020-06-08T23:25:50.049Z] Result: FAILED
[taskcluster 2020-06-08T23:25:50.049Z] === Task Finished ===
[taskcluster 2020-06-08T23:25:50.049Z] Task Duration: 5.58810122s
[taskcluster:error] Uploading error artifact public/condprof from file archive with message "Could not read directory '/Users/cltbld/tasks/task_1591658581/archive'", reason "file-missing-on-worker" and expiry 2021-06-08T21:44:08.788Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1591658581/archive'
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1591658581/archive'

Comment 1

[Intermittent Failures Robot]

25 failures in 5211 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 25

Platform breakdown:

• macosx1014-64: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644320&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I see a reference to an external site in a condprof job ( https://firefoxci.taskcluster-artifacts.net/IPYwtOWBRtuKNp6gga34VQ/0/public/logs/live_backing.log ):
[task 2020-06-13T05:18:45.804Z] Traceback (most recent call last):
[task 2020-06-13T05:18:45.804Z] File "condprof/main.py", line 17, in <module>
[task 2020-06-13T05:18:45.804Z] check()
[task 2020-06-13T05:18:45.804Z] File "condprof/../condprof/check_install.py", line 108, in check
[task 2020-06-13T05:18:45.804Z] if update_pip():
[task 2020-06-13T05:18:45.804Z] File "condprof/../condprof/check_install.py", line 44, in update_pip
[task 2020-06-13T05:18:45.804Z] "https://files.pythonhosted.org/packages/b2/"
[task 2020-06-13T05:18:45.804Z] File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
[task 2020-06-13T05:18:45.804Z] raise CalledProcessError(retcode, cmd)

this goes back to June 8th (1 week) based off this changeset:
https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=ce2095089a2a8eda9b4afc1594c06b7c72994c41

it is failing here:
https://searchfox.org/mozilla-central/source/testing/condprofile/condprof/check_install.py#44

it would be nice to keep this in house and not need installation. Maybe bake this into an image or tooltool or some solution.

possibly that needs to be updated. :tarek, I see you landed this code originally 9 months ago, could paths/versions/files have changed and need updating?

Comment 3

[Intermittent Failures Robot]

28 failures in 4035 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 28

Platform breakdown:

• macosx1014-64: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644320&startday=2020-06-15&endday=2020-06-21&tree=all

Comment 4

[Razvan Maries]

There are 32 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-06-19&endday=2020-06-26&tree=trunk&bug=1644320

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=307677214&repo=mozilla-central&lineNumber=92

Taker, are there any updates?

Comment 5

[Tarek Ziadé (:tarek)]

I have not had time to look at it, will look at it tomorrow

Comment 6

[Intermittent Failures Robot]

30 failures in 5252 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 30

Platform breakdown:

• macosx1014-64: 30

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644320&startday=2020-06-22&endday=2020-06-28&tree=all

Comment 7

[Tarek Ziadé (:tarek)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #2)

it would be nice to keep this in house and not need installation. Maybe bake this into an image or tooltool or some solution.

I think our Python infra is broken by design. Here's a proposal to clean it up

https://bugzilla.mozilla.org/show_bug.cgi?id=1649080

Comment 8

[Tarek Ziadé (:tarek)]

Attachment: Bug 1644320 - Fix how we setup the virtualenv

Fix how we update pip & al. Also updated the version

Comment 9

[Tarek Ziadé (:tarek)]

try run https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=37d40a45edfde967c0026b737616e289fc17c794

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d7c008d071f
Fix how we setup the virtualenv  r=sparky

Comment 11

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/4d7c008d071f

Comment 12

[Intermittent Failures Robot]

14 failures in 4516 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 14

Platform breakdown:

• macosx1014-64: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1644320&startday=2020-06-29&endday=2020-07-05&tree=all