666367 - Builder status sometimes gets confused

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

I see this a lot with release builders (probably because I look at them a lot). Sometimes a job will finish, but on the /builders page the status will still be shown as "building". It eventually corrects itself.

[16:13] <dustin> bhearsum: can you file a bug assigned to me?  I can write the patch, although I'd rather not be in charge of deploying it
[16:13] <dustin> (patch is trivial .. call self.updateBigStatus somewhere in that fn)

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I'll get this patch sorted out shortly.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m666367-buildbot-p1-r1.patch

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Landed on 'default'.  I'll leave it to the master-wranglers to take it from here.

Comment 4

[Chris Cooper [:coop] (he/him)]

This needs master restarts to take affect. I'll handle this during the downtime tomorrow.

Comment 5

[Chris Cooper [:coop] (he/him)]

(In reply to comment #4)
> This needs master restarts to take affect. I'll handle this during the
> downtime tomorrow.

All masters have been restarted, save bm07 and bm08 which are currently being used for releases. We'll gracefully shut them down and then restart them once the releases are out the door.

Comment 6

[Chris Cooper [:coop] (he/him)]

Oops, this got done before I left for vacation.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

It looks like the patch landed on default at
  http://hg.mozilla.org/build/buildbot/rev/f5b5bf2816d4
but didn't make it onto the production-0.8 branch, so the restarts didn't pick up the change. eg bm08:/builds/buildbot/build1 has 
  Buildbot version: 0.8.2-hg-3dc678eecd11-production-0.8
which is from May 14.

Comment 8

[Chris Cooper [:coop] (he/him)]

My fault. I thought the merge to production-0.8 had already happened.

I'll take care of the merge, but we'll need another downtime to restart the masters all in one go, or else do a long, step-wise "clean shutdown" process.

Comment 9

[Chris Cooper [:coop] (he/him)]

Not sure when the next tree closure is planned. In case I'm not around, here is what needs to happen for this bug:

* merge https://hg.mozilla.org/build/buildbot/rev/f5b5bf2816d4 to production-0.8
* update buildbot clone on all masters
* restart all masters

Comment 10

[[e-mail] (:bhearsum)]

Trying to get a downtime scheduled for this week.

Comment 11

[[e-mail] (:bhearsum)]

successfully landed in the downtime