1917711 - macOS webrender ("WR") tasks fail on almost all machines: abort: cannot enable sparse profile on existing non-sparse checkout

Status: RESOLVED FIXED

(Developer Services :: Mercurial: robustcheckout, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

See https://treeherder.mozilla.org/jobs?repo=autoland&revision=71eb0e99bd8debcc45397f7b5c5510e7ed90cf12&selectedTaskRun=ER30dEyOSsS1Du0kkKxIDQ.0 which had one WR(cargotest) run which succeed and many failing ones after 9 days without a run (task only scheduled if files in gfx/wr/ change).

The WR tasks use a sparse checkout profile but also try to use the existing shared checkout and bail because it is not a sparse one. It's unknown why almost every macOS hardware machine is affected now.

Connor, could you investigate?

Comment 1

[Julien Cristau [:jcristau]]

When i've seen these errors in the past it was because the task that was supposed to do the initial clone and sparse config failed at some stage after creating the repo but before setting the sparse config, and left the cache corrupted.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

This is also happening frequently with the macOS startup tests, which blocks release graphs.

Comment 3

[Andrew Erickson [:aerickson]]

https://mozilla-hub.atlassian.net/browse/RELOPS-1070 has been filed.

Comment 4

[Andrew Erickson [:aerickson]]

:pmoore, do you have any idea what's going on here? Is there a cache directory that's shared between tasks?

Do the task_dirs in the cache directory ever get cleaned up?

Comment 5

[Intermittent Failures Robot]

24 failures in 4448 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 12
• autoland: 9
• mozilla-beta: 3

Platform and build breakdown:

• macosx1015-64-qr: 19
  • debug: 9
  • opt: 10
• macosx64-shippable: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-09-09&endday=2024-09-15&tree=all

Comment 6

[Intermittent Failures Robot]

22 failures in 4860 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 8
• mozilla-beta: 5

Platform and build breakdown:

• macosx1015-64-qr: 14
  • debug: 9
  • opt: 5
• macosx64-shippable: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-09-16&endday=2024-09-22&tree=all

Comment 7

[Intermittent Failures Robot]

11 failures in 4889 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• autoland: 3
• mozilla-release: 1

Platform and build breakdown:

• macosx64-shippable: 5
  • opt: 5
• macosx1015-64-qr: 6
  • debug: 4
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-09-23&endday=2024-09-29&tree=all

Comment 8

[Intermittent Failures Robot]

15 failures in 3574 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7
• autoland: 5
• mozilla-beta: 3

Platform and build breakdown:

• macosx1015-64-qr: 10
  • opt: 6
  • debug: 4
• macosx64-shippable: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-09-30&endday=2024-10-06&tree=all

Comment 9

[Pete Moore [:pmoore][:pete]]

I'm working with Ryan on open macOS issues now, will report back.

Comment 10

[Intermittent Failures Robot]

8 failures in 4679 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 5

Platform and build breakdown:

• macosx1015-64-qr: 6
  • opt: 3
  • debug: 3
• macosx64-shippable: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-10-07&endday=2024-10-13&tree=all

Comment 11

[Intermittent Failures Robot]

5 failures in 4579 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 5

Platform and build breakdown:

• macosx1015-64-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-10-14&endday=2024-10-20&tree=all

Comment 12

[Intermittent Failures Robot]

1 failures in 4456 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 4456 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1917711&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 14

[Pete Moore [:pmoore][:pete]]

I think this is fixed now as per https://mozilla-hub.atlassian.net/browse/RELOPS-1070?focusedCommentId=955770

Comment 15

[Ryan Curran]

:aryx do we feel good about closing this?

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Okay to close it. Thanks for working on this.