704582 - [tracking bug] deploy 83 tegras

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Taylor [:bear]]

Deploy final 83 tegras into production

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Today:
 - got five of the foopies up (bug 704590)
 - switches online, but not fully configured (need USB-to-serial to finish)
 - furniture rearranged in HAXXOR
 - CDUs deployed in HAXXOR

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

This is mostly finished.  80 tegras are ready to go.  The exceptions are:
 * foopies not all configured yet (hwine is working on this)
 * war-room tegras are not swapped out yet (bug 707584)
 * there are three more tegras which we need to look for space/power for
 * several of the new tegras aren't pingable and need some TLC - bug 710515
the latter three points will be addressed Thursday.

Comment 3

[hwine]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> This is mostly finished.  80 tegras are ready to go.  The exceptions are:
>  * foopies not all configured yet (hwine is working on this)

This was completed Thursday evening.

Comment 4

[Chris Cooper [:coop] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> This is mostly finished.  80 tegras are ready to go.  The exceptions are:
>  * foopies not all configured yet (hwine is working on this)
>  * war-room tegras are not swapped out yet (bug 707584)
>  * there are three more tegras which we need to look for space/power for
>  * several of the new tegras aren't pingable and need some TLC - bug 710515
> the latter three points will be addressed Thursday.

Anything left to do here? Can the remaining work be tracked in the follow-up bugs - bug 707584 and bug 710515?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

as far as I'm concerned, yes

Comment 6

[Mike Taylor [:bear]]

agree - all tegras are deployed and we are working out which get moved to production now

Comment 7

[hwine]

Attachment: changes to make new tegras available to production

This is inclusive of all machines, including those not yet being put into production due to being in bug 710515

Comment 8

[hwine]

closed too early - we need to add the tegras to production first (in progress)

Comment 9

[hwine]

Attachment: changes to make new tegras available to production v2

Comment 10

[hwine]

Comment on attachment 583296 [details] [diff] [review]
changes to make new tegras available to production v2

http://hg.mozilla.org/build/buildbot-configs/rev/791772e9328e

Comment 11

[hwine]

25 tegras now deployed to production, awaiting reconfig and soak of new buildbot masters used for control:

foopy18: tegra-206/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-207/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-208/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-209/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-210/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-211/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-212/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-213/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-214/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-215/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-216/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-217/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy18: tegra-219/buildbot.tac:buildmaster_host = 'buildbot-master20.build.mozilla.com'
foopy19: tegra-220/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-221/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-222/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-223/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-224/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-225/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-227/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-229/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-230/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-231/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-232/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'
foopy19: tegra-233/buildbot.tac:buildmaster_host = 'buildbot-master19.build.mozilla.org'

Comment 12

[[e-mail] (:bhearsum)]

This patch made it to production today.

Comment 13

[hwine]

remaining new foopies have been deployed to production:
foopy20,tegra-234,buildbot-master20
foopy20,tegra-235,buildbot-master20
foopy20,tegra-236,buildbot-master20
foopy20,tegra-237,buildbot-master20
foopy20,tegra-238,buildbot-master20
foopy20,tegra-240,buildbot-master20
foopy20,tegra-242,buildbot-master20
foopy20,tegra-243,buildbot-master20
foopy20,tegra-244,buildbot-master20
foopy20,tegra-245,buildbot-master20
foopy20,tegra-246,buildbot-master20
foopy20,tegra-247,buildbot-master20
foopy22,tegra-248,buildbot-master20
foopy22,tegra-249,buildbot-master20
foopy22,tegra-250,buildbot-master20
foopy22,tegra-251,buildbot-master20
foopy22,tegra-252,buildbot-master20
foopy22,tegra-253,buildbot-master20
foopy22,tegra-254,buildbot-master20
foopy22,tegra-255,buildbot-master20
foopy22,tegra-256,buildbot-master20
foopy22,tegra-257,buildbot-master20
foopy22,tegra-258,buildbot-master20
foopy22,tegra-259,buildbot-master20
foopy22,tegra-260,buildbot-master20
foopy22,tegra-261,buildbot-master20
foopy23,tegra-262,buildbot-master19
foopy23,tegra-263,buildbot-master19
foopy23,tegra-264,buildbot-master19
foopy23,tegra-265,buildbot-master19
foopy23,tegra-266,buildbot-master19
foopy23,tegra-267,buildbot-master19
foopy23,tegra-268,buildbot-master19
foopy23,tegra-269,buildbot-master19
foopy23,tegra-270,buildbot-master19
foopy23,tegra-271,buildbot-master19
foopy23,tegra-272,buildbot-master19
foopy23,tegra-273,buildbot-master19
foopy23,tegra-274,buildbot-master19
foopy23,tegra-275,buildbot-master19
foopy24,tegra-276,buildbot-master20
foopy24,tegra-277,buildbot-master20
foopy24,tegra-278,buildbot-master20
foopy24,tegra-279,buildbot-master20
foopy24,tegra-280,buildbot-master20
foopy24,tegra-281,buildbot-master20
foopy24,tegra-282,buildbot-master20
foopy24,tegra-283,buildbot-master20
foopy24,tegra-284,buildbot-master20
foopy24,tegra-285,buildbot-master20

Comment 14

[Mike Taylor [:bear]]

marking as done - tegras are in production and running fine