1296077 - Jobs stuck in the "running" state

Status: ASSIGNED

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

Description

[Kyle Lahnakoski [:ekyle]]

A few jobs are old, and still "running".

http://activedata.allizom.org/tools/query.html#query_id=Q8PSy813

Here is a specific example:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=eb74a01c8dc0bc508ea8b492e6a4c179e1c2ff19&selectedJob=32358519&exclusion_profile=false

A subset of the same problem is where we can identify a specific task 

http://activedata.allizom.org/tools/query.html#query_id=hGBmQc2H

Again, an example

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=8dc198cd46fff3b1f6e39ea6e80bb4507bf2cdbe&selectedJob=32671050

Comment 1

[Kyle Lahnakoski [:ekyle]]

This comment deals with pending, so may not apply here.  But I wonder if Buildbot is not properly informing Treeherder when things go different than usual.

https://bugzilla.mozilla.org/show_bug.cgi?id=1296329#c16

Comment 2

[Ed Morley [:emorley]]

Hi!

The links in comment 0 no longer work - is this still happening?

Comment 3

[Kyle Lahnakoski [:ekyle]]

Yes, this is still happening:

SELECT 
  *
FROM 
  job 
WHERE 
  state='running' and 
  last_modified>date_add(now(), INTERVAL -30 day) AND
  last_modified<date_add(now(), INTERVAL -10 day) 
ORDER BY
  last_modified DESC
LIMIT 
  100

Comment 4

[Cameron Dawson [:camd]]

SELECT 
  j.id, j.state, j.result, j.last_modified, j.start_time, j.submit_time, rds.build_system_type
FROM 
  job j
left join reference_data_signatures rds
on j.signature_id = rds.id
WHERE 
  j.state='running' and 
  j.last_modified>date_add(now(), INTERVAL -30 day) AND
  j.last_modified<date_add(now(), INTERVAL -10 day) 
ORDER BY
  j.last_modified DESC
LIMIT 
  100

Looks like these are all from Taskcluster when I ran it just now

Comment 5

[Ed Morley [:emorley]]

Running the query from comment 4 now, I still see recent instances - so the removal of buildbot bridge et al hasn't helped.

Comment 6

[Ed Morley [:emorley]]

I'm pretty sure bug 1470243 and bug 1532255 will help prevent at least a subset of these cases.

Once the PRs in those bugs have been merged/deployed and a few weeks have passed it would be good to re-run the query in comment 4 to see if there have been any further occurrences.

Comment 7

[Karl Thiessen [:kthiessen, he/him]]

Kyle, assigning this to you for follow-up query between now and the end of March. If it persists, we'll put it on the work queue for Q2.

Comment 8

[Kyle Lahnakoski [:ekyle]]

I have confirmed that this persists.