1908736 - Intermittent OS X [taskcluster:error] &errors.errorString{s:"File in cache, but not on filesystem

Status: NEW

(Taskcluster :: Workers, defect)

Description

[tszentpeteri]

• Failure Log:

[taskcluster 2024-07-18T16:56:09.394Z] Task ID: emrphpCGTK2vYalYK3ZF-A
[taskcluster 2024-07-18T16:56:09.394Z] === Task Starting ===
[taskcluster 2024-07-18T16:56:09.847Z] Uploading redirect artifact public/logs/live.log to URL https://firefoxci-websocktunnel.services.mozilla.com/mdc1.macmini-r8-51.60099/log/K02hb7KFQzSXqNQgP-lCJw with mime type "text/plain; charset=utf-8" and expiry 2024-07-18T17:51:09.779Z
[taskcluster 2024-07-18T16:56:09.884Z] [mounts] Downloading task X7BbD42sQeWkjrtsFhy6RQ artifact public/build/mozharness.zip to /opt/worker/downloads/WkyUp-1KQi6lDjZFvBuZUA
[taskcluster 2024-07-18T16:56:10.043Z] [mounts] Downloaded 3692229 bytes with SHA256 d88fee3238eaa509c251628750e358302aa3d2522390228105cad780e7ec78d1 from task X7BbD42sQeWkjrtsFhy6RQ artifact public/build/mozharness.zip to /opt/worker/downloads/WkyUp-1KQi6lDjZFvBuZUA
[taskcluster:warn 2024-07-18T16:56:10.043Z] [mounts] Download /opt/worker/downloads/WkyUp-1KQi6lDjZFvBuZUA of task X7BbD42sQeWkjrtsFhy6RQ artifact public/build/mozharness.zip has SHA256 d88fee3238eaa509c251628750e358302aa3d2522390228105cad780e7ec78d1 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2024-07-18T16:56:10.043Z] [mounts] Creating directory /opt/worker/tasks/task_172131922872581/mozharness
[taskcluster 2024-07-18T16:56:10.043Z] [mounts] Copying file '/opt/worker/downloads/WkyUp-1KQi6lDjZFvBuZUA' to '/opt/worker/tasks/task_172131922872581/WkyUp-1KQi6lDjZFvBuZUA'
[taskcluster 2024-07-18T16:56:10.045Z] [mounts] Extracting zip file /opt/worker/tasks/task_172131922872581/WkyUp-1KQi6lDjZFvBuZUA to '/opt/worker/tasks/task_172131922872581/mozharness'
[taskcluster 2024-07-18T16:56:10.171Z] [mounts] Removing file '/opt/worker/tasks/task_172131922872581/WkyUp-1KQi6lDjZFvBuZUA'
[taskcluster 2024-07-18T16:56:10.301Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2025-07-18T14:11:37.469Z
[taskcluster:error] goroutine 1 [running]:
[taskcluster:error] runtime/debug.Stack()
[taskcluster:error] 	/task_170732374097056/go/go/src/runtime/debug/stack.go:24 +0x5e
[taskcluster:error] main.(*TaskRun).Run.func2()
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/main.go:978 +0x9d
[taskcluster:error] panic({0xe37c7e0?, 0xc0002c0000?})
[taskcluster:error] 	/task_170732374097056/go/go/src/runtime/panic.go:770 +0x132
[taskcluster:error] main.ensureCached({0xe43e710, 0xc00035e500}, 0xc000435400)
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/mounts.go:665 +0xedc
[taskcluster:error] main.decompress({0xe43e710, 0xc00035e500}, {0x0, 0x0}, {0xc00581e7e0, 0x2f}, 0xc000435400)
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/mounts.go:752 +0x66
[taskcluster:error] main.(*FileMount).Mount(0xc000193ec0, 0xc000435400)
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/mounts.go:637 +0x105
[taskcluster:error] main.(*TaskMount).Start(0xc000435400)
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/mounts.go:409 +0x1b1
[taskcluster:error] main.(*TaskRun).Run(0xc00033c008)
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/main.go:1060 +0xdb1
[taskcluster:error] main.RunWorker()
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/main.go:452 +0xb05
[taskcluster:error] main.main()
[taskcluster:error] 	/task_170732374097056/taskcluster/workers/generic-worker/main.go:159 +0x565
[taskcluster:error] 
[taskcluster:error] &errors.errorString{s:"File in cache, but not on filesystem: {2024-07-18 15:04:28.523823 +0000 UTC /opt/worker/downloads/ddlTjq1tRFqydRSKVG8FRw 1 map 
<...>
[taskcluster:error] File in cache, but not on filesystem: {2024-07-18 15:04:28.523823 +0000 UTC /opt/worker/downloads/ddlTjq1tRFqydRSKVG8FRw 1 map
<...>

Could you please take a look? Thank you!

Comment 1

[Ryan Curran]

Absolutely.

This occurred because we were asked to manually clean out /opt/worker/downloads/ directory, as it had become so full rm wasn't working and test failures were occurring.

Original issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1774954

I ensured we quarantined the host before the deletion. My guess is that this failure was a side effect.

Comment 2

[Intermittent Failures Robot]

18 failures in 789 pushes (0.023 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-beta: 11
• autoland: 7

Platform and build breakdown:

• macosx1015-64-devedition-qr: 7
  • opt: 7
• macosx1015-64-shippable-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1908736&startday=2024-07-18&endday=2024-07-18&tree=all

Comment 3

[Intermittent Failures Robot]

18 failures in 3799 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 11
• autoland: 7

Platform and build breakdown:

• macosx1015-64-devedition-qr: 7
  • opt: 7
• macosx1015-64-shippable-qr: 11
  • opt: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1908736&startday=2024-07-15&endday=2024-07-21&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 3668 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform and build breakdown:

• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1908736&startday=2024-09-02&endday=2024-09-08&tree=all