527994 - Setup 10 new Try Talos slaves with buildbot

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Once cloned machines are online, this is to track rest of manual setup needed. 

Note: these slaves would be in MV, so will need firewall changes to connect to try talos server in colo.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: [checked in]include new slaves

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: [checked in]add new machines to graphserver

Comment 3

[alice nodelman [:alice] [:anode]]

Pushed to staging graph server:

mysql> insert into machines values (NULL,4,0,"1.83","qm-pleopard-try12",1,unix_timestamp());
Query OK, 1 row affected (0.02 sec)

mysql> insert into machines values (NULL,4,0,"1.83","qm-pleopard-try13",1,unix_timestamp());
Query OK, 1 row affected (0.00 sec)

mysql> insert into machines values (NULL,4,0,"1.83","qm-pleopard-try14",1,unix_timestamp());
Query OK, 1 row affected (0.00 sec)

mysql> insert into machines values (NULL,4,0,"1.83","qm-pleopard-try15",1,unix_timestamp());
Query OK, 1 row affected (0.01 sec)

mysql> insert into machines values (NULL,4,0,"1.83","qm-pleopard-try16",1,unix_timestamp());
Query OK, 1 row affected (0.01 sec)

mysql> insert into machines values (NULL,5,0,"1.83","qm-pubuntu-try12",1,unix_timestamp());
Query OK, 1 row affected (0.07 sec)

mysql> insert into machines values (NULL,5,0,"1.83","qm-pubuntu-try13",1,unix_timestamp());
Query OK, 1 row affected (0.01 sec)

mysql> insert into machines values (NULL,5,0,"1.83","qm-pubuntu-try14",1,unix_timestamp());
Query OK, 1 row affected (0.00 sec)

mysql> insert into machines values (NULL,5,0,"1.83","qm-pubuntu-try15",1,unix_timestamp());
Query OK, 1 row affected (0.00 sec)

mysql> insert into machines values (NULL,5,0,"1.83","qm-pubuntu-try16",1,unix_timestamp());
Query OK, 1 row affected (0.00 sec)

Comment 4

[alice nodelman [:alice] [:anode]]

Comment on attachment 412310 [details] [diff] [review]
[checked in]include new slaves

changeset:   1754:75db33cb6953

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Manual post-clone steps for OSX now done. These slaves should be able to connect to production talos-master once the DNS entries are detangled in dep.bug.

Still need to do post-clone steps for linux slaves.

Comment 6

[alice nodelman [:alice] [:anode]]

qm-pleopard-try12/13/14/15/16 up and connected.  Had to update buildbot.tac's to point at ip instead of machines name.

Comment 7

[alice nodelman [:alice] [:anode]]

qm-pubuntu-try12/14/15/16 up and connected.  qm-pubuntu-try13 unreachable, probably just needs a kick.

Installed xrestop on all slaves, updated buildbot.tac to use ip instead of master machine name.

Comment 8

[alice nodelman [:alice] [:anode]]

qm-pubuntu-try13 is dead, phong is replacing with another mini.

Comment 9

[alice nodelman [:alice] [:anode]]

qm-pubuntu-try13 replaced, up and configured.

Comment 10

[alice nodelman [:alice] [:anode]]

All done here.

Comment 11

[alice nodelman [:alice] [:anode]]

Comment on attachment 412344 [details] [diff] [review]
[checked in]add new machines to graphserver

changeset:   256:e41e4d3f5d8d

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #6)
> qm-pleopard-try12/13/14/15/16 up and connected.  Had to update buildbot.tac's
> to point at ip instead of machines name.

(In reply to comment #7)
> Installed xrestop on all slaves, updated buildbot.tac to use ip instead of
> master machine name.

Using the IP address instead of the hostname concerns me, and I'd like to fix that. If DNS entry is still not working after a machine reboot, we should reopen blocking bug#529196.

Comment 13

[alice nodelman [:alice] [:anode]]

It appears that it can reach the master using the proper name (I pinged from one of the slaves) - might have been a temporary problem that I hit.  I don't see the point in changing the machines configuration as they are all up and connected - but good to know for future set ups.

Comment 14

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #13)
> It appears that it can reach the master using the proper name (I pinged from
> one of the slaves) - might have been a temporary problem that I hit.  I don't
> see the point in changing the machines configuration as they are all up and
> connected - but good to know for future set ups.

Please do change the buildbot.tac files in these new slaves from using IPs to using the hostname like we do in other Talos slaves. We've been bitten by small differences in config files in the past, and I'd rather not set ourselves up to hit problems like this again.

Using "graceful shutdown" should make this a quick (and boring) fix to rollout.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #14)
> Using "graceful shutdown" should make this a quick (and boring) fix to rollout.

That doesn't actually work for machines which reboot as the last step of the build - the slave reboots before the shutdown gets a chance to run, and the subsequent reconnect to the master resets the shutdown request.

Comment 16

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #15)
> (In reply to comment #14)
> > Using "graceful shutdown" should make this a quick (and boring) fix to rollout.
> 
> That doesn't actually work for machines which reboot as the last step of the
> build - the slave reboots before the shutdown gets a chance to run, and the
> subsequent reconnect to the master resets the shutdown request.

Ah, very true, Nick. Simple ssh to edit of buildbot.tac would take effect on next start. Or we could wait until they are idle. Whatever is easiest.

Anyway, original point remains, lets keep the config files on these new slaves consistent with the existing slaves by using hostnames like usual.

Comment 17

[alice nodelman [:alice] [:anode]]

buildbot.tac files fixed to point to 'talos-master' instead of ip, will be picked up on next reboot.

All done here.