1566667 - Intermittent [taskcluster:error] [mounts] reading file in zip archive: file already exists: C:\tasks\task_1563240254\mozharness\LICENSE

Status: REOPENED

(Taskcluster :: Workers, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=256797912&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/Ri9-FCbzQjWzrK8ZUNVH_A/runs/0/artifacts/public/logs/live_backing.log

---

[taskcluster 2019-07-16T19:42:32.660Z] Task ID: Ri9-FCbzQjWzrK8ZUNVH_A
[taskcluster 2019-07-16T19:42:32.660Z] === Task Starting ===
[taskcluster 2019-07-16T19:42:33.627Z] Uploading redirect artifact public/logs/live.log to URL http://aeaqcaiaaaawx7etp37rejpqgjemlxnfeo6ado5j5orxhmbg.taskcluster-worker.net:60023/log/CIRn4vKIQemFtcC_GsCqfw with mime type "text/plain; charset=utf-8" and expiry 2019-07-16T20:57:33.183Z
[taskcluster 2019-07-16T19:42:33.970Z] [mounts] Downloading task XLnC3FLySwC5cpXP9FqtQw artifact public/build/mozharness.zip to downloads\PXxoSxZAR-W4gCIGyQu_-A
[taskcluster 2019-07-16T19:42:34.811Z] [mounts] Downloaded 2338561 bytes with SHA256 baab1a779b354dd9381442a11f92e30acc517c2e33e431dc9bcf4252165eda12 from task XLnC3FLySwC5cpXP9FqtQw artifact public/build/mozharness.zip to downloads\PXxoSxZAR-W4gCIGyQu_-A
[taskcluster:warn 2019-07-16T19:42:34.811Z] [mounts] Download downloads\PXxoSxZAR-W4gCIGyQu_-A of task XLnC3FLySwC5cpXP9FqtQw artifact public/build/mozharness.zip has SHA256 baab1a779b354dd9381442a11f92e30acc517c2e33e431dc9bcf4252165eda12 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2019-07-16T19:42:34.811Z] [mounts] Creating directory C:\tasks\task_1563240254 with permissions 0700
[taskcluster 2019-07-16T19:42:34.812Z] [mounts] Extracting zip file downloads\PXxoSxZAR-W4gCIGyQu_-A to 'C:\tasks\task_1563240254'
[taskcluster 2019-07-16T19:42:35.364Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/Ri9-FCbzQjWzrK8ZUNVH_A/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-07-15T18:31:17.424Z
[taskcluster:error] [mounts] reading file in zip archive: file already exists: C:\tasks\task_1563240254\mozharness\LICENSE

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Is this a new worker behavior? If not, this is probably something that's changed in-tree and should be investigated in another component.

Comment 2

[Intermittent Failures Robot]

3 failures in 4298 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform breakdown:

• windows10-aarch64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566667&startday=2019-07-15&endday=2019-07-21&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 4608 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566667&startday=2019-07-22&endday=2019-07-28&tree=all

Comment 4

[Intermittent Failures Robot]

2 failures in 4439 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• windows10-aarch64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566667&startday=2019-07-29&endday=2019-08-04&tree=all

Comment 5

[Intermittent Failures Robot]

3 failures in 4666 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform breakdown:

• windows10-aarch64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566667&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 6

[Intermittent Failures Robot]

3 failures in 4839 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform breakdown:

• windows10-aarch64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566667&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 7

[Intermittent Failures Robot]

1 failures in 5073 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• windows10-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566667&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 8

[Intermittent Failures Robot]

2 failures in 5047 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• windows10-aarch64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566667&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 9

[Pete Moore [:pmoore][:pete]]

This seems to be happening on workers running generic-worker 13.0.2, see e.g. this failure.

This issue was fixed in bug 1551164 and released in generic-worker 14.1.1.

This issue can be solved by upgrading the affected worker(s) (e.g. t-lenovoyogac630-013) to at least generic-worker 14.1.1 (recommended: upgrade all workers to the very latest generic-worker release, which at the time of writing is generic-worker 16.0.0).

You can see the worker version of a given worker by looking at its most recent task, and then looking at the first few lines of the log file of the task. The worker version is reported in every task it runs.

Comment 10

[Mark Cornmesser [:markco]]

This seems to be specific to the Bitbar Yoga laptops. Moving the ni to Rob since he is more familiar with them.

Comment 11

[Edwin Takahashi (:egao | infrequent contributor)]

Backstory:
Rob did not handle the Bitbar setup work (I did), but he does manage the COC repo that is used to set them up. Last I tried with Bitbar folks to upgrade generic-worker on windows-aarch64 to 15.1.0, we ran into a bug on that specific version/platform combination that forced downgrade to 14.1.2.

Now, I thought we upgraded all windows10-aarch64 generic-worker to 14.1.2, but I suppose this is not the case. I'll inquire with Bitbar to see if they can attempt an upgrade. It would be helpful if there is a way to query what machine ran the job, however.

Comment 12

[Edwin Takahashi (:egao | infrequent contributor)]

I realized that I could just look this information up...silly me.

Comment 13

[Edwin Takahashi (:egao | infrequent contributor)]

Bitbar has ran the OCC script and the generic-worker on yoga-13 is now 16.0.0.

Comment 14

[Rob Thijssen [:grenade (EET/UTC+0300)]]

over the last few days i have been attempting to fix a number of problems on yoga worker instances. almost all of the yogas were in a different state of configuration, including:

• different versions of generic worker
• different versions of the generic worker wrapper script, sometimes with a wrapper script that is incompatible with the specific version of generic worker on the instance.
• different occ branch or repo sha pins in the windows registry
• different or missing versions of python 3
• different or missing versions of the system maintenance script or scheduled task
• different versions of logging configuration causing logs to be forwarded to deprecated papertrail accounts or to not forward logs at all
• different versions of the system hosts file, causing instances to identify themselves to papertrail in inconsistent ways (with and without an fqdn qualifier)

most of these problems were caused by instances not rebooting between tasks or not running occ between tasks. that's a mixed blessing or curse. occ runs on yoga instances are very slow. this is bad because it means systems running occ on each boot are able to dedicate less time to claiming and running tasks. of course, we aren't able to update systems properly in automation if they don't run occ.

because occ hasn't been running properly, folks have gotten used to fixing things manually by doing things like installing the latest generic worker on a random misbehaving instance. this is a sad state of affairs because it means we have instances behaving in different ways. instances whose gw config or wrapper is incompatible with the gw version on the same instance or instances that behave in different ways to the rest of the yoga worker pool.

occ is supposed to manage all of this and keep a worker consistent with the rest of the worker pool. since it either hasn't been running, or has been running a deprecated version, it has been basically impotent on these systems. seeing the disparate state these systems were in, i was sorta surprised they've been as productive as they have been in terms of resolving tasks. i can only apologise for not taking action sooner.

i was able to correct most of the problems above by crafting an elevated taskcluster task which looked for specific issues and modified the system to correct them. things like checking the sha hashes of significant scripts on the system and replacing anything deprecated.

this means that any manually installed versions of software that do not match the version specified in the manifest, will get automatically removed and replaced on the next successful occ run.

there are still some problems with the currently running yogas. mostly due to the gw wrapper script which is still not uniform everywhere yet. but i am working on correcting this slowly. the slowness is deliberate because a completely fixed system results in one that spends more time running occ and less time running tasks and i'm trying to develop a fix for that before i force all systems onto the patched process.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

(closed as part of mass closure of old intermittent bugs)

Comment 16

[Pete Moore [:pmoore][:pete]]

Reopening inactive bugs, because they may still need attention. Historically, inactive bugs were closed, but this hides the fact there are genuine issues which have not been resolved.