1275428 - Weekly master restart should add comment to slavealloc while disabled

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

The recent buildbot master lag alerts were debugged by Alin to be 13 masters being disabled in slavealloc, presumably because something went wrong with their restarts. We should try to figure out the root cause (in bug 1256118) but in the meantime drop a comment in slavealloc like 'master restart YYYYMMDD' while they're disabled.

Comment 1

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug1275428_slavealloc_notification.patch

This patch should give us notifications in slavealloc about the master when was disabled.  
I will still need to test this modifications on local,can you give me any advise how can I begin testing this?and to what should I be aware of? thank you

Comment 2

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug_1275428_notification_slavealloc.patch

This patch will give us notifications in slavealloc about the master when was disabled while is been disabled.

Comment 3

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8777314 [details] [diff] [review]
bug_1275428_notification_slavealloc.patch

Review of attachment 8777314 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with nit fixed.

::: buildfarm/maintenance/restart_masters.py
@@ +263,4 @@
>      # and possibly getting hung.
>      log.debug("Disabling %s in slavealloc." % master['hostname'])
>      disable_url = furl(slavealloc_api_url + "/" + str(master_ids[master['name']]))
> +    put_data = {"enabled": 0, "notes": "master disabled at %s " %datetime.datetime.now().strftime("%d-%m-%y %H:%M")}

Nit: I prefer %Y-%m-%d as the date format because it's sortable.

Comment 4

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: bug_1275428_notification_slavealloc.patch

Comment 5

[Alin Selagea [:aselagea]]

http://hg.mozilla.org/build/tools/rev/5220a5199f6c