1022431 - scl1 Move Train C releng config Work

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Hal Wine [:hwine] use NI!]

+++ This bug was initially created as a clone of Bug #1018284 +++

+++ This bug was initially created as a clone of Bug #1013035 +++

Train C starts June 9

Comment 1

[Hal Wine [:hwine] use NI!]

Downloaded from https://docs.google.com/a/mozilla.com/spreadsheets/d/1Pz0wSIhht1xlVLRe5Tl676iLyNAQtVqwKA5Z3rSkpmE as csv:
    SCL1-_SCL3_ Move Train C - linux build - thunderbird.csv
    SCL1-_SCL3_ Move Train C - linux build.csv
    SCL1-_SCL3_ Move Train C - linux try - thunderbird and fuzzing.csv
    SCL1-_SCL3_ Move Train C - talos-r4-snow.csv
    SCL1-_SCL3_ Move Train C - windows build - esr and b2g18 .csv
    SCL1-_SCL3_ Move Train C - windows build.csv
    SCL1-_SCL3_ Move Train C - windows try.csv

Used braindump/it-related/move_hosts to generate attachments:
    $ ~1/move_hosts SCL1-_SCL3_\ Move\ Train\ C\ -\ *csv >train_c.csv
    $ ~1/move_hosts --short SCL1-_SCL3_\ Move\ Train\ C\ -\ *csv >train_c_short.txt
    $ ~1/move_hosts --short --slavealloc SCL1-_SCL3_\ Move\ Train\ C\ -\ *csv >train_c_slavealloc.sql
    $ ~1/move_hosts --short --graphserver SCL1-_SCL3_\ Move\ Train\ C\ -\ *csv >train_c_graphs.sql

Comment 2

[Hal Wine [:hwine] use NI!]

Attachment: train_b.csv

For use as input to slavealloc scripts in braindump/it-related

Comment 3

[Hal Wine [:hwine] use NI!]

Attachment: train_b_short.txt

For use as input to braindump/it-related/rename_host_bugs

Comment 4

[Hal Wine [:hwine] use NI!]

Attachment: train_b_slavealloc.sql

SQL to update slavealloc database

Comment 5

[Hal Wine [:hwine] use NI!]

Attachment: train_c_graphs.sql

SQL to update graph servers

Comment 6

[Hal Wine [:hwine] use NI!]

Attachment: train_c.csv

For use as input to slavealloc scripts in braindump/it-related (correct hosts)

Comment 7

[Hal Wine [:hwine] use NI!]

Attachment: train_c_short.txt

For use as input to braindump/it-related/rename_host_bugs (correct hosts)

Comment 8

[Hal Wine [:hwine] use NI!]

Attachment: train_c_slavealloc.sql

SQL to update slavealloc database (correct hosts)

Comment 9

[Hal Wine [:hwine] use NI!]

Disabled hosts in slavealloc using braindump/it-related:
 ./get_enabled_hosts train_c.csv 
 ./disable_all_hosts train_c.csv

Comment 10

[Hal Wine [:hwine] use NI!]

Note: 5 hosts were already disabled (re comment 9)

hosts in nagios downtimed for 2d by manual copy/paste into #sysadmins of output from:
  ./manage_hosts train_c.csv gen_nagios_downtimes 
Note, paste in batches of about 10 for best results.

Comment 11

[Hal Wine [:hwine] use NI!]

(non-windows) hosts powered down using:
  ./manage_hosts train_c.csv -P shut_down
(-P just lets fabric run in parallel)

This step just a "nicety" as all hosts are reimaged in the new datacenter.

Comment 12

[Hal Wine [:hwine] use NI!]

All tracker bugs re-aliased for new host names using:
  braindump/it-related/rename_host_bugs train_c_short.txt

Comment 13

[Hal Wine [:hwine] use NI!]

Updated databases:

    mysql --defaults-file=slavealloc.ini -e "source train_c_slavealloc.sql"
    mysql --defaults-file=graph-stage.ini -e "source train_c_graphs.sql"
    mysql --defaults-file=graph-prod.ini -e "source train_c_graphs.sql"

Comment 14

[Hal Wine [:hwine] use NI!]

Attachment: train_c.patch

config changes - tested as follows:
 - buildbot-configs/test-masters.sh
 - pmoore script (modified) from train b:
    $ ./buildbot_slave_list.sh
    patching file mozilla-tests/production_config.py
    patching file mozilla/production_config.py
    patching file mozilla/staging_config.py
    Hosts that should have already been in buildbot but were not
    ============================================================

    Hosts that were already in buildbot but should not have been
    ============================================================
    are listed in file 'already-in-buildbot', but you can't change them:
        65 already-in-buildbot

    Hosts that should have been removed
    ===================================

    Hosts that should not have been removed
    =======================================

    Hosts that should have been added
    =================================

    Hosts that should not have been added
    =====================================

Comment 15

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8438198 [details] [diff] [review]
train_c.patch

Review of attachment 8438198 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/production_config.py
@@ +8,2 @@
>                   ['b-2008-sm-%04d' % x for x in range(33, 65)]
> +MOCK_DL120G7   = ['b-linux64-hp-%04i' % x for x in range(30,36) + range(13,17)] +\

I'm going to reverse the two ranges when I land to put them in numerical order, but otherwise fine.

Comment 16

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8438198 [details] [diff] [review]
train_c.patch

Review of attachment 8438198 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/3c37407923db

Comment 17

[Chris Cooper [:coop] (he/him)]

All Windows host have keys now. Note for train D: we need to use sftp to another host in the same VLAN to transfer Windows keys now, as documented in https://wiki.mozilla.org/ReleaseEngineering/How_To/Adjust_SSH_keys_on_a_slave#Production

All train C slaves have been enabled in slavealloc and rebooted so they connect to buildbot as soon as the reconfig is done (in theory).

Comment 18

[Simone Bruno [:simone]]

Live with reconfig on 2014-06-11 08:47 PT

Comment 19

[Chris Cooper [:coop] (he/him)]

Attachment: Fix try host range

Some of the try hosts ended up in the build group by accident. Putting them back where they should be.

Comment 20

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8438713 [details] [diff] [review]
Fix try host range

Review of attachment 8438713 [details] [diff] [review]:
-----------------------------------------------------------------

Callek r+ in IRC

Comment 21

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8438713 [details] [diff] [review]
Fix try host range

Review of attachment 8438713 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/buildbot-configs/rev/0939a9fa05fb

Merged and deployed to production.

Comment 22

[Pete Moore [:pmoore][:pete]]

Nice work Hal and Coop!