Closed Bug 666367 Opened 9 years ago Closed 9 years ago

Builder status sometimes gets confused

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

Details

(Whiteboard: [buildmasters])

Attachments

(1 file)

I see this a lot with release builders (probably because I look at them a lot). Sometimes a job will finish, but on the /builders page the status will still be shown as "building". It eventually corrects itself.

[16:13] <dustin> bhearsum: can you file a bug assigned to me?  I can write the patch, although I'd rather not be in charge of deploying it
[16:13] <dustin> (patch is trivial .. call self.updateBigStatus somewhere in that fn)
I'll get this patch sorted out shortly.
Attachment #542337 - Flags: review?(bhearsum) → review+
Landed on 'default'.  I'll leave it to the master-wranglers to take it from here.
Assignee: dustin → nobody
This needs master restarts to take affect. I'll handle this during the downtime tomorrow.
Assignee: nobody → coop
Flags: needs-treeclosure?
OS: Linux → All
Priority: -- → P2
Hardware: x86_64 → All
Whiteboard: [buildmasters]
Flags: needs-treeclosure? → needs-treeclosure+
(In reply to comment #4)
> This needs master restarts to take affect. I'll handle this during the
> downtime tomorrow.

All masters have been restarted, save bm07 and bm08 which are currently being used for releases. We'll gracefully shut them down and then restart them once the releases are out the door.
Flags: needs-treeclosure+
Oops, this got done before I left for vacation.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
It looks like the patch landed on default at
  http://hg.mozilla.org/build/buildbot/rev/f5b5bf2816d4
but didn't make it onto the production-0.8 branch, so the restarts didn't pick up the change. eg bm08:/builds/buildbot/build1 has 
  Buildbot version: 0.8.2-hg-3dc678eecd11-production-0.8
which is from May 14.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
My fault. I thought the merge to production-0.8 had already happened.

I'll take care of the merge, but we'll need another downtime to restart the masters all in one go, or else do a long, step-wise "clean shutdown" process.
Priority: P2 → P3
Flags: needs-treeclosure?
Not sure when the next tree closure is planned. In case I'm not around, here is what needs to happen for this bug:

* merge https://hg.mozilla.org/build/buildbot/rev/f5b5bf2816d4 to production-0.8
* update buildbot clone on all masters
* restart all masters
Trying to get a downtime scheduled for this week.
Assignee: coop → bhearsum
Flags: needs-treeclosure? → needs-treeclosure+
successfully landed in the downtime
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Flags: needs-treeclosure+
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.