655314 - Tbpl can have an unsettling period of limbo where jobs disappear for up to 10 minutes after they complete

Status: RESOLVED DUPLICATE of bug 653969

(Tree Management Graveyard :: TBPL, defect)

Description

[Daniel Holbert [:dholbert]]

This morning at 11:07, mbrubeck noticed that zwol's Win debug mochitest-oth job had mysteriously vanished -- M (1 2 3 4 5), no "oth" there  at all.

At that point, I checked the buildapi page, and saw that the job had actually completed successfully at 11:01.

I then checked tbpl again (force-reloading it). Still nothing there.

At 11:11 (10 minutes after the job had finished according to buildapi), mbrubeck reported that the build had shown up on tbpl, and it was green.

I've noticed this same issue happen in the past, and it's a bit unsettling each time it happens.  Can we fix it?

Comment 1

[Phil Ringnalda (:philor)]

As a tbpl bug? Sure, it's a duplicate of "get rid of tinderbox." Otherwise, it's a duplicate of "tinderbox needs to get and read its mail faster."

Comment 2

[Daniel Holbert [:dholbert]]

Sure, I figured that there was a bug with tinderbox or some other piece of infrastructure somewhere under the hood causing this.

Still, I'd hoped that TBPL could perhaps work around the tinderbox bug -- e.g. if it had information about an in-progress job a few minutes ago, then it could set a fixed threshold (N minutes) during which to continue to showing that job as grey for a period of time, despite tinderbox having completely forgotten about it.

Of course this wouldn't work on a freshly-loaded TBPL page, but it could address this issue for people who have a long-open TBPL tab. (like most tree-watchers should)

Having said that, I don't really know how much state TBPL has to go on, so maybe the above suggestion is infeasible / more trouble than it's worth.