706229 - TBPL isn't showing builds that have finished

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

https://build.mozilla.org/buildapi/self-serve/birch/rev/414812fdb6ac has a build + 22 tests

https://tbpl.mozilla.org/?tree=Birch&rev=414812fdb6ac was only showing the one remaining job that was still running, and now is empty. Should show those 23 jobs.

---> no finished jobs are being shown

Might be multiple cronjobs importing data from builds-4hr.js.gz due to backlog for network interruption in Phoenix data center.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I've verified the "Android birch build" job from that revision is in builds-4hr.js.gz.

Comment 2

[Arpad Borsos [:Swatinem]]

I now see it in tbpl, and according to the tree admin, the (perma?)orange C R J are hidden.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Good spotting. It looks like production is catching up now, Try and Mozilla-Inbound have a lot more information on them too.

Comment 4

[Jason Thomas [:jason]]

Added lock file for tbpl production cron job running on genericadm.private.phx1. This will prevent it from running multiple times.

Comment 5

[Arpad Borsos [:Swatinem]]

Was that a code change or just deployment? We should probably add some timeouts (especially for the network stuff). I have once seen a few rogue tbpl processes as well on my server.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Probably just a 'lock <args>' prefix on the cron command.

We look to be all caught up in production and will presume that we were just bogged down due to the network blips in PHX today. The lock should help in future, and follow bugs for any network timeouts would be good.

Still might be some bustage in the dev db that needs resolving.