1563662 - Windows 10 AArch64 opt bitbar workers not taking tasks

Status: RESOLVED WORKSFORME

(Taskcluster :: Workers, defect)

Description

[Andreea Pavel [:apavel]]

There are jobs in queue for ~464 minute(s)

https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&fromchange=7dbb546e613875cec06e1615c42dd7c4266fe170&searchStr=Windows%2C10%2CAArch64%2Copt&selectedJob=254856955

From IRC: tomprince> (it looks like the workers aren't taking jobs)

Edwin can you take a look?

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

Mentioned also in irc, but I suspect either:

• increased windows10-aarch64 load, or;
• some windows10-aarch64 machines have gone offline for an unknown reason

I've noted (as of 2019-07-04 22:45 PDT) that the following machines appear to be offline (not showing in the taskcluster worker listing):

009, 010, 011, 012, 013, 016, 017, 018, 019, 020, 021, 022, 024, 025, 027, 028

I've inquired with Bitbar; as they are based in Pacific timezone it may be another ~8 hours until a response is received.

Comment 2

[Andreea Pavel [:apavel]]

At this time (10h later) the jobs are still in queue https://irccloud.mozilla.com/file/c862lwGD/image.png

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

BItbar worker count is still the same as listed in Comment 1. Bitbar has not yet responded on slack.

Comment 4

[Cosmin Sabou [:CosminS]]

bc was investigating this at one point this week so ni'ing him.

Comment 5

[Bob Clary [:bc] (inactive)]

I don't have access to the projects or devices at bitbar to investigate. I have someone going to the DC this morning to check on devices. I'll ask them to look around but I don't know specifically what to tell them.

Comment 6

[Bob Clary [:bc] (inactive)]

Stanley @ Bitbar is looking at the laptops now. I see them servicing jobs from Friday evening now.

Comment 7

[Andreea Pavel [:apavel]]

Yes, jobs are green, things look good. Closing for now, please reopen if there are other issues.

Comment 8

[Intermittent Failures Robot]

4 failures in 3501 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-inbound: 4

Platform breakdown:

• windows10-aarch64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1563662&startday=2019-07-01&endday=2019-07-07&tree=all

Comment 9

[Edwin Takahashi (:egao | infrequent contributor)]

We have fluctuating worker count for windows10-aarch64. According to Bitbar, some of the machines are stuck on login screen (after rebooting post-test execution).

Bitbar suspects that at least some of the machines will have to reimaged/factory reset.

Comment 10

[Pete Moore [:pmoore][:pete]]

This issue may be resolved by upgrading to generic-worker 15.1.0 (task user management was a bit broken in 14.1.1).