1501031 - OSX jobs not starting and creating a backlog

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Posix OS, task)

Description

[Cristina Coroiu [:ccoroiu]]

Trees are closed for this
E.g. https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=9b16d71b9fc4aeaaea1771e7dd44790bf5e941e6&selectedJob=207051831

Comment 1

[Cristina Coroiu [:ccoroiu]]

Taskcluster at this time shows 2097 pending jobs for osx

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Indeed, there are just over 2000 pending jobs for OS X.  Any idea if a new worker was deployed there, or anything like that happened?

Comment 3

[:dhouse]

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #2)
> Indeed, there are just over 2000 pending jobs for OS X.  Any idea if a new
> worker was deployed there, or anything like that happened?

There hasn't been a change to the OSX worker or other infrastructure in the past few weeks. Our last update to generic-worker was more than a month ago, to 10.11.3 (https://bugzilla.mozilla.org/show_bug.cgi?id=1480412). We did upgrade mercurial 2 weeks ago to 4.7.1-1 (https://bugzilla.mozilla.org/show_bug.cgi?id=1490703), but I think this queue issue started late last week.

Comment 4

[:dhouse]

(In reply to Dave House [:dhouse] from comment #3)
> queue issue started late last week.

By "queue issue", I mean that the osx worker queue has been reaching 2000+ each weekday instead of the typical previous levels of a few hundred max each day (and 1000+ when we had the datacenter moves and had 60+ machines out of service for 1-2 weeks at a time).

From my perspective, the workers are continuing to work through the queue, but new tasks continue to come into the queue.

Comment 5

[:dhouse]

The osx jobs were starting and running, but there was increased usage.