1627889 - Intermittent Raptor, Talos [taskcluster:error] Aborting task... [hang while fetching artifacts]

Status: RESOLVED INCOMPLETE

(Testing :: Talos, defect, P3)

Description

[Alexandru Michis [:malexandru]]

We've seen a sudden increase of raptor and talos failures on a push whose changes we're unrelated:
https://treeherder.mozilla.org/#/jobs?repo=autoland&collapsedPushes=676749&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&searchStr=performance&tochange=828ed052b1a9ee14fe39f7235ece9398812c7f4f&fromchange=542f439cf528e5c1bc6068ebc9ff5bfec05bd9f9

Retriggers are all green, so it shows that most probably it was a hiccup.
Made a bug to track this specific issue, I'm not sure if the summary correctly describes the failures, so please change it if necessary.

Failure logs:

• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=296551390&repo=autoland&lineNumber=55
• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=296551438&repo=autoland&lineNumber=312
• https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=296551909&repo=autoland&lineNumber=755

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This happens for various artifacts like minidump_stackwalk.tar.xz and others. I wonder if it is still somewhat related to bug 1616556, but also for Windows now. Mike, any idea?

Comment 2

[Mike Hommey [:glandium]]

It doesn't seem related, and it's not only happening when downloading artifacts. Seems like problems with Windows workers?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

:markco, was there reimaging of the workers recently that could affect this? I assume this is datacenter and not bitbar.

Comment 4

[Mark Cornmesser [:markco]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3)

:markco, was there reimaging of the workers recently that could affect this? I assume this is datacenter and not bitbar.

jmaher: There have been spot reimaging of nodes that have not been taking tasks, but no mass reimaging. It looks like these are failing on missing files under c:\users\task_* directories. Those directories are not persistent between reboots since a new task user is used per task.

pmoore: Any ideas?

Comment 5

[Intermittent Failures Robot]

26 failures in 1103 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26

Platform breakdown:

• windows10-64-shippable: 8
• windows10-64-shippable-qr: 7
• linux64-shippable-qr: 4
• linux64-shippable: 5
• windows7-32-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627889&startday=2020-04-07&endday=2020-04-07&tree=all

Comment 6

[Intermittent Failures Robot]

26 failures in 4823 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 26

Platform breakdown:

• windows10-64-shippable: 8
• windows10-64-shippable-qr: 7
• linux64-shippable-qr: 4
• linux64-shippable: 5
• windows7-32-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627889&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The last failure of this type was seen on April 7th, or maybe more recent occurrences have been classified against another bug.

Comment 8

[Intermittent Failures Robot]

6 failures in 4528 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 1

Platform breakdown:

• linux64-shippable: 1
• windows10-64-shippable: 2
• windows10-64-shippable-qr: 2
• android-hw-g5-7-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627889&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 9

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

The logs in comment 0 contain "[taskcluster:error] Task aborted - max run time exceeded" so it looks like the task maxRunTime is less than the amount of time the tasks require. This could be due to the task hanging, or taking longer than it previously did, or the maxRunTime being less than it previously was.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

No, here one excerpt from the logs:

[fetches 2020-04-07T07:11:11.575Z] Removing C:\Users\task_1586238152\fetches\minidump_stackwalk.tar.xz
[taskcluster:error] Aborting task...
[taskcluster 2020-04-07T07:36:05.239Z] SUCCESS: The process with PID 356 (child process of PID 7416) has been terminated.

There is no timestamp for Aborting task..., which would be great to have btw, but as it shows it hangs in removing the minidump_stackwalk.tar.xz.

Comment 11

[Intermittent Failures Robot]

1 failures in 5276 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows7-32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627889&startday=2020-05-04&endday=2020-05-10&tree=all

Comment 12

[Intermittent Failures Robot]

2 failures in 5488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-beta: 1

Platform breakdown:

• windows10-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627889&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 13

[Intermittent Failures Robot]

2 failures in 4727 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform breakdown:

• linux64-shippable: 1
• windows10-64-ref-hw-2017: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627889&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 14

[Intermittent Failures Robot]

2 failures in 4949 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• linux64-shippable: 1
• windows10-64-shippable-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1627889&startday=2020-06-01&endday=2020-06-07&tree=all