1655666 - release-generate-checksums in 80.0b1 task fails to find virtualenv

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P1)

Description

[Julien Cristau [:jcristau]]

See https://firefox-ci-tc.services.mozilla.com/tasks/CxpB-_fVQY-k9JpAlUaVFA
and log: https://firefoxci.taskcluster-artifacts.net/CxpB-_fVQY-k9JpAlUaVFA/1/public/logs/live_backing.log

[task 2020-07-28T03:28:28.841Z] 03:28:28     INFO - [mozharness: 2020-07-28 03:28:28.841748Z] Running create-virtualenv step.
[task 2020-07-28T03:28:28.841Z] 03:28:28     INFO - Running main action method: create_virtualenv
[task 2020-07-28T03:28:28.841Z] 03:28:28     INFO - Creating virtualenv /builds/worker/workspace/venv
[task 2020-07-28T03:28:28.841Z] 03:28:28    FATAL - Can't find the virtualenv module
[task 2020-07-28T03:28:28.841Z] 03:28:28    FATAL - Running post_fatal callback...
[task 2020-07-28T03:28:28.841Z] 03:28:28    FATAL - Exiting -1
[task 2020-07-28T03:28:28.841Z] 03:28:28     INFO - [mozharness: 2020-07-28 03:28:28.841958Z] Finished create-virtualenv step (failed)

This is blocking releases.

Comment 1

[Hamzah Akhtar]

Can you please rpvide the command that resulted in the failure
I'll try and fix this asap

Comment 2

[Julien Cristau [:jcristau]]

The task is defined at https://hg.mozilla.org/releases/mozilla-beta/file/FIREFOX_80_0b1_BUILD1/taskcluster/ci/release-generate-checksums/kind.yml
It runs mozharness/scripts/release/generate-checksums.py.

Comment 3

[Hamzah Akhtar]

Attachment: possible_fix

I think this might fix the issue the virtualenv module not found
Can you please verify it because I'm not able to reproduce the error on my system.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Thanks for your contribution! I ran your change against our automation via a staging release task, via ./mach try scriptworker generate-checksums --release-type beta.

before: https://treeherder.mozilla.org/#/jobs?repo=try&revision=afb775679d7701fa7af808155f1be23102adcbeb&selectedTaskRun=PHowfHfAQl-0U6Rx4yJUTA.0
after: https://treeherder.mozilla.org/#/jobs?repo=try&revision=69055b5dcb1c80b0948bd55cf2bd4fa9c89d1fa2

Currently waiting for results to show up to ensure it fixes the issue.

Comment 5

[Tom Prince [:tomprince]]

Attachment: Bug 1655666: [mozharness] Always set `abs_src_path` if `GECKO_PATH` is in the environment;

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d9e59bcc3793
[mozharness] Always set `abs_src_path` if `GECKO_PATH` is in the environment; r=mtabara

Comment 7

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-beta/rev/2258ad2194cb

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/d9e59bcc3793

Comment 9

[Intermittent Failures Robot]

3 failures in 4289 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3

Platform breakdown:

• devedition-release: 2
• firefox-release: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1655666&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 10

[Ryan VanderMeulen [:RyanVM]]

This is hitting ESR78 now.
https://treeherder.mozilla.org/logviewer?job_id=330191198&repo=mozilla-esr78&lineNumber=283

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/f35972ee2800dd61e7994a3340b721df8887f8bd

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9166609 [details]
Bug 1655666: [mozharness] Always set abs_src_path if GECKO_PATH is in the environment;

Needed to unblock the 78.8esr release, approving retroactively.