1046642 - Job statuses not updating automatically

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

From dev.platform:

> I seem to have to refresh Treeherder to get it to update job statuses,
> whereas on TBPL the statuses update automatically.
>
> Is this expected?
>
> Thanks,
> Botond

Botond, could you explain a bit more: what job status wasn't changing? ie: a running job didn't change to completed?

Comment 1

[Nathan Froyd [:froydnj]]

I have noticed this too in my try pushes this morning.  The treeherder display does update--I think, I can't recall if I manually refreshed them recently--but they do so much slower than the tbpl display.

Comment 2

[Botond Ballo [:botond]]

(In reply to Ed Morley [:edmorley] from comment #0)
> > I seem to have to refresh Treeherder to get it to update job statuses,
> > whereas on TBPL the statuses update automatically.
> 
> Botond, could you explain a bit more: what job status wasn't changing? ie: a
> running job didn't change to completed?

Yes. I was looking at a Try push side by side in TBPL and Treeherder. As jobs completed, in TBPL, their statuses would change from running to successful or failed without any interaction; in Treeherder, the statuses would all remain running. I left it like that for 20-30 mins at which point I refreshed Treeherder and the statuses updated.

(At the beginning, I also had to refresh Treeherder to get it to show the running jobs at all, but that time I got impatient and only waited about 1 minute before refreshing, so it might just have been slow to update. 20-30 minutes, however, is definitely not just slow.)

Comment 3

[Nicholas Nethercote [inactive]]

I've seen something similar, which is that new landings don't show up on treeherder as fast as they do on TBPL. (Actually, I thought they weren't showing up at all, but it's possible that they're just slower.)

E.g. right now I have TBPL and treeherder open on inbound, and TBPL is one landing ahead.

Comment 4

[Ed Morley [:emorley]]

(In reply to Nicholas Nethercote [:njn] from comment #3)
> I've seen something similar, which is that new landings don't show up on
> treeherder as fast as they do on TBPL. (Actually, I thought they weren't
> showing up at all, but it's possible that they're just slower.)
> 
> E.g. right now I have TBPL and treeherder open on inbound, and TBPL is one
> landing ahead.

There's bug 1048043 filed for that, since it sounds like this bug is about socket.io updates not working, whereas this other issue is from the new way treeherder handles the pushlog compared to TBPL (ie it ingests the pushlog and adds it to the main DB, rather than the TBPL approach of letting the UI poll the pushlog directly)

Comment 5

[Mauro Doglio [:mdoglio]]

Attachment: Github PR #127 on treeherder-ui

I set an explicit page size for the requests to the job endpoint. It used to be 10 (the service default) but the UI was requesting 40 elements in the IN clause.
I also changed the handling of the jobs results so that if we receive a notification of new jobs and the jobs endpoint doesn't know anything yet about those jobs, a request will be retried after 10 secs. This could happen due to a delay in the master/slave replication.

Comment 6

[Mauro Doglio [:mdoglio]]

This landed on production last Friday, closing the bug

Comment 7

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/dc3ff814ba9629ed2470e41a27c5f6f295a54c5c
Bug 1046642 - fix handling of socketio notifications of new jobs

https://github.com/mozilla/treeherder/commit/3a7160ad5054bb982e4fc66e462ab06aee5c7c0a
Merge pull request #127 from mozilla/bug-1046642-job-status-not-updating

Bug 1046642 - fix handling of socketio notifications of new jobs