Closed Bug 1025335 Opened 6 years ago Closed 6 years ago

scl1 Move Train D releng config Work

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: hwine, Assigned: hwine)

References

Details

Attachments

(10 files)

+++ This bug was initially created as a clone of Bug #1013035 +++

Train D starts June 16
Downloaded from https://docs.google.com/a/mozilla.com/spreadsheets/d/1CSbCNtuY6pZMHtAs6KmOUgPholm4zMsE9SssWhZ0TfI as csv:
    SCL1-_SCL3_ Move Train D - linux build - thunderbird.csv
    SCL1-_SCL3_ Move Train D - linux build.csv
    SCL1-_SCL3_ Move Train D - linux try - thunderbird and fuzzing.csv
    SCL1-_SCL3_ Move Train D - mac-signing.csv
    SCL1-_SCL3_ Move Train D - talos-r4-snow.csv
    SCL1-_SCL3_ Move Train D - windows build - esr and b2g18 .csv
    SCL1-_SCL3_ Move Train D - windows build.csv
    SCL1-_SCL3_ Move Train D - windows try.csv

Used braindump/it-related/move_hosts to generate attachments:
    braindump/it-related/move_hosts SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d.csv
    braindump/it-related/move_hosts --short SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d_short.txt
    braindump/it-related/move_hosts --short --slavealloc SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d_slavealloc.sql
    braindump/it-related/move_hosts --short --graphserver SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d_graphs.sql
Status: NEW → ASSIGNED
Attached file train_d.csv
For use as input to slavealloc scripts in braindump/it-related
Attached file train_d_short.txt
For use as input to braindump/it-related/rename_host_bugs
Attached file train_d_slavealloc.sql
SQL to update slavealloc database
Attached file train_d_graphs.sql
SQL to update graph servers
All hosts disabled, 71 (of 81) previously enabled.
$ ./get_enabled_hosts train_d.csv 
Enter password: 
next step is to run 'disable_all_hosts train_d.csv'
0 [hwine@cruncher it-related]
$ ./disable_all_hosts train_d.csv
Enter password: 
0 [hwine@cruncher it-related]
$ wc train_d.csv.previously_enabled 
  71   71 1199 train_d.csv.previously_enabled
0 [hwine@cruncher it-related]
$ wc train_d.csv
  81   81 7024 train_d.csv
0 [hwine@cruncher it-related]
$
Attached file train_d_nagios.txt
commands for moc to disable nagios for hosts
Looks like one failed but otherwise all clear:

[rwatson@nagios1.private.releng.scl3 rwatson]# ./downtime.sh
Downtime for bld-linux64-ix-049.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for bld-linux64-ix-050.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for mac-signing3.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for mac-signing4.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-131.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-132.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-133.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-134.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-135.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-136.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-137.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-138.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-139.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-140.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-141.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-142.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-143.build.scl1.mozilla.com of 172800 seconds scheduled.
Can't find a host by that name.
Downtime for talos-r4-snow-145.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-146.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-147.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-148.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-149.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-150.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-151.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-152.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-153.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-154.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-155.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-156.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-157.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-158.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-159.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-160.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-161.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-162.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-163.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-164.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-165.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-166.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-167.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-168.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-169.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-170.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-ref.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave04.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave05.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave123.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave124.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave125.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave126.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave127.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave128.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave129.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave130.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave131.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave132.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave133.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave134.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave135.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave136.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave137.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave138.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave139.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave140.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave141.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave142.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave143.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave144.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave157.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave160.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave161.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave162.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave163.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave164.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave165.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave166.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave167.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave168.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave169.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave170.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
bug aliases & summaries changed
  $ braindump/it-releated/rename_host_bugs train_d_short.txt
Databases updated:
    $ mysql --defaults-file=graph-stage.ini -e "source train_d_graphs.sql"
    Enter password: 
    0 [hwine@cruncher it-related (fabric)]
    $ mysql --defaults-file=graph-prod.ini -e "source train_d_graphs.sql"
    Enter password: 
    0 [hwine@cruncher it-related (fabric)]
    $ mysql --defaults-file=slavealloc.ini -e "source train_d_slavealloc.sql"
    Enter password: 
    0 [hwine@cruncher it-related (fabric)]
update puppet to know where the servers are now.
Attachment #8440882 - Flags: review?(bhearsum)
Comment on attachment 8440882 [details] [diff] [review]
mac-signing.patch

Review of attachment 8440882 [details] [diff] [review]:
-----------------------------------------------------------------

::: manifests/moco-nodes.pp
@@ +161,5 @@
>  node /mac-signing[12].srv.releng.scl3.mozilla.com/ {
>      include toplevel::server::signing
>  }
>  
> +node /mac-signing[34].srv.releng.scl3.mozilla.com/ {

You should probably put this into the previous regex. r=me with that changed.
Attachment #8440882 - Flags: review?(bhearsum) → review+
Comment on attachment 8440882 [details] [diff] [review]
mac-signing.patch

Committed with requested change:
 https://hg.mozilla.org/build/puppet/rev/d8d8accf04b4
Attachment #8440882 - Flags: checked-in+
Move Train D - linux try - thunderbird and fuzing => r89157.
Move Train D - mac-signing => r89159.
Move Train D - talos-r4-snow => r89170.
Move Train D - windows build => 89171.
Move Train D - windows try => 89172.

Added missing switches => 89173. (switch1.r201-{7,8}.ops.releng.scl3.mozilla.net)

Fixing cluster checks in separate Bug 1024512.

80 hosts downtimed.

Etherpad updated with this info.
Master exception logs are complaining about this host, 10.26.64.62
(b-linux64-ix-0012.try.releng.scl3.mozilla.com).
Attachment #8441370 - Flags: review?(pmoore)
Comment on attachment 8441370 [details] [diff] [review]
b-linux64-ix range is off-by-one

Review of attachment 8441370 [details] [diff] [review]:
-----------------------------------------------------------------

I'm afraid there is a much bigger delta that needs to be processed:

Hosts that should have already been in buildbot but were not
============================================================
mac-signing3
mac-signing4
talos-r4-snow-ref

Hosts that were already in buildbot but should not have been
============================================================
b-2008-ix-0054
b-2008-ix-0055
b-2008-ix-0056
b-2008-ix-0057
b-2008-ix-0058
b-2008-ix-0059
b-2008-ix-0060
b-2008-ix-0061
b-2008-ix-0062
b-2008-ix-0063
b-2008-ix-0064
t-snow-r4-0127
t-snow-r4-0128
t-snow-r4-0129
t-snow-r4-0130
t-snow-r4-0131
t-snow-r4-0132
t-snow-r4-0133
t-snow-r4-0134
t-snow-r4-0135
t-snow-r4-0136
t-snow-r4-0137
t-snow-r4-0138
t-snow-r4-0139
t-snow-r4-0140
t-snow-r4-0141
t-snow-r4-0142
t-snow-r4-0143
t-snow-r4-0144
t-snow-r4-0145
t-snow-r4-0146
t-snow-r4-0147
t-snow-r4-0148
t-snow-r4-0149
t-snow-r4-0150
t-snow-r4-0151
t-snow-r4-0152
t-snow-r4-0153
t-snow-r4-0154
t-snow-r4-0155
t-snow-r4-0156
t-snow-r4-0157
t-snow-r4-0158
t-snow-r4-0159
t-snow-r4-0160
t-snow-r4-0161
t-snow-r4-0162
t-snow-r4-0163
t-snow-r4-0164
t-snow-r4-0165
t-snow-r4-0166

Hosts that should have been removed
===================================
bld-linux64-ix-049
bld-linux64-ix-050
talos-r4-snow-131
talos-r4-snow-132
talos-r4-snow-133
talos-r4-snow-134
talos-r4-snow-135
talos-r4-snow-136
talos-r4-snow-137
talos-r4-snow-138
talos-r4-snow-139
talos-r4-snow-140
talos-r4-snow-141
talos-r4-snow-142
talos-r4-snow-143
talos-r4-snow-144
talos-r4-snow-145
talos-r4-snow-146
talos-r4-snow-147
talos-r4-snow-148
talos-r4-snow-149
talos-r4-snow-150
talos-r4-snow-151
talos-r4-snow-152
talos-r4-snow-153
talos-r4-snow-154
talos-r4-snow-155
talos-r4-snow-156
talos-r4-snow-157
talos-r4-snow-158
talos-r4-snow-159
talos-r4-snow-160
talos-r4-snow-161
talos-r4-snow-162
talos-r4-snow-163
talos-r4-snow-164
talos-r4-snow-165
talos-r4-snow-166
talos-r4-snow-167
talos-r4-snow-168
talos-r4-snow-169
talos-r4-snow-170
w64-ix-slave04
w64-ix-slave05
w64-ix-slave123
w64-ix-slave124
w64-ix-slave125
w64-ix-slave126
w64-ix-slave127
w64-ix-slave128
w64-ix-slave129
w64-ix-slave130
w64-ix-slave131
w64-ix-slave132
w64-ix-slave133
w64-ix-slave134
w64-ix-slave135
w64-ix-slave136
w64-ix-slave137
w64-ix-slave138
w64-ix-slave139
w64-ix-slave140
w64-ix-slave141
w64-ix-slave142
w64-ix-slave143
w64-ix-slave144
w64-ix-slave157
w64-ix-slave160
w64-ix-slave161
w64-ix-slave162
w64-ix-slave163
w64-ix-slave164
w64-ix-slave165
w64-ix-slave166
w64-ix-slave167
w64-ix-slave168
w64-ix-slave169
w64-ix-slave170

Hosts that should not have been removed
=======================================

Hosts that should have been added
=================================
b-2008-ix-0137
b-2008-ix-0138
b-2008-ix-0139
b-2008-ix-0140
b-2008-ix-0141
b-2008-ix-0142
b-2008-ix-0143
b-2008-ix-0144
b-2008-ix-0145
b-2008-ix-0146
b-2008-ix-0147
b-2008-ix-0148
b-2008-ix-0149
b-2008-ix-0150
b-2008-ix-0151
b-2008-ix-0152
b-2008-ix-0153
b-2008-ix-0154
b-2008-ix-0155
b-2008-ix-0156
b-2008-ix-0157
b-2008-ix-0158
b-2008-ix-0182
b-2008-ix-0183
b-2008-ix-0184
b-linux64-ix-0013
mac-signing3
mac-signing4
t-snow-r4-ref

I'll attach the files I used to generate this report.
Attachment #8441370 - Flags: review?(pmoore) → review-
Attached file buildbot_slave_list.sh
If you download buildbot_slave_list.py and buildbot_slave_list.sh, you can run this report with, e.g.

bash buildbot_slave_list.sh

Please note it is downloading the attachments to this bug, so if you are changing these attachments, you might want to change the script to take your local version(s) of the files instead of curling them down.
Attached patch train_d.patchSplinter Review
Attachment #8441609 - Flags: review?(coop)
Comment on attachment 8441609 [details] [diff] [review]
train_d.patch

Review of attachment 8441609 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with cleanup addressed.

::: mozilla-tests/production_config.py
@@ +23,5 @@
>  for i in range(1,131):
>      SLAVES['win8']['t-w864-ix-%03i' % i] = {}
>  
> +for i in range(127,167):
> +    SLAVES['snowleopard']['t-snow-r4-%04i' % i] = {}

Just remove this range. It's covered right below.

::: mozilla/production_config.py
@@ +2,5 @@
>  LINUX_IXS      = []
>  LINUX64_IXS    = []
>  WIN32_IXS      = []
>  WIN64_IXS      = []
> +WIN64_REV2     = ['w64-ix-slave%03i' % x for x in range(145,148) + [156]] + \

There are no w64-ix slaves in slavealloc any more. If that's expected, we can remove them all here.

@@ +35,5 @@
>  TRY_WIN64_IXS  = []
>  TRY_LINUX64_EC2 = ['try-linux64-ec2-%03d' % x for x in range(1, 60) + range(301,340)] + \
>      ['try-linux64-spot-%03d' % x for x in range(1, 200) + range(300,500)] + \
>      ['try-linux64-spot-%d' % x for x in range(1000, 1100)]
> +TRY_WIN64_REV2 = ['b-2008-ix-%04i' % x for x in range(18, 65) + range(30, 42) + range(173,182)] + \

range(18, 65) includes range(30, 42).

::: mozilla/staging_config.py
@@ +4,5 @@
>  MAC_LION_MINIS = ['bld-lion-r5-%03d' % x for x in range(89, 93)]
>  WIN32_IXS      = []
>  WIN64_IXS      = []
>  WIN64_REV2     = ['ix-mn-w0864-%03d' % x for x in range(1,3)] + \
> +                 ['w64-ix-slave22'] + \

Again, no w64-ix slaves are listed in slavealloc any more. We can rip this out if that's true.
Attachment #8441609 - Flags: review?(coop) → review+
Comment on attachment 8441609 [details] [diff] [review]
train_d.patch

With changes above, still passes test-masters.sh. Some noise from Pete's scripts:

    patching file mozilla-tests/production_config.py
    patching file mozilla/production_config.py
    patching file mozilla/staging_config.py
    Hosts that should have already been in buildbot but were not
    ============================================================

    Hosts that were already in buildbot but should not have been
    ============================================================
    are listed in file 'already-in-buildbot', but you can't change them:
        51 already-in-buildbot

    Hosts that should have been removed
    ===================================

    Hosts that should not have been removed
    =======================================
    w64-ix-slave145
    w64-ix-slave146
    w64-ix-slave147
    w64-ix-slave156
    w64-ix-slave22

    Hosts that should have been added
    =================================

    Hosts that should not have been added
    =====================================

Landed:
 http://hg.mozilla.org/build/buildbot-configs/rev/b715c4dac923
Attachment #8441609 - Flags: checked-in+
looks like we putting b-2008-ix-0184.wintry.releng.scl3.mozilla.com in build-scl3 (not-try):
    - http://hg.mozilla.org/build/buildbot-configs/annotate/b715c4dac923/mozilla/production_config.py#l6
    - slavealloc has this slave set with: 'trust=core' and 'pool=build-scl3'

Just want to sanity check this host as its fqdn suggests it should be try. /me doesn't add ssh keys or enable.
Depends on: 1027394
I didn't get around to this today. The end result was that we can go ahead and add the ssh keys and enable the everything that is left bar bug 1027394 and the top three from:  https://hwine.pastebin.mozilla.org/5434018

I will do that first thing in the morning PT :)
so from the windows slaves: we had 1 not added (1027394). I didn't add 182,183 as they were staging slaves. hwine, what was the plan for these again? Make them staging but keep their 'try' fqdn/vlan?

All other win slaves looked good. They have the right keys now and are enabled. See below for breakdown.

BTW - I took a look at the darwin and two linux slaves that were part of this move. It doesn't look they are enabled. If they have been re-imaged, there are no post image steps and they can just be enabled. Would you like me to do that?

no keys added, not enabled:
    slavealloc thinks in production pool:
        b-2008-ix-0184.wintry.releng.scl3.mozilla.com

    slavealloc thinks in staging pool:
        b-2008-ix-0182.wintry.releng.scl3.mozilla.com
        b-2008-ix-0183.wintry.releng.scl3.mozilla.com

keys added, enabled:

    try:
        b-2008-ix-0054.wintry.releng.scl3.mozilla.com
        b-2008-ix-0055.wintry.releng.scl3.mozilla.com
        b-2008-ix-0056.wintry.releng.scl3.mozilla.com
        b-2008-ix-0057.wintry.releng.scl3.mozilla.com
        b-2008-ix-0058.wintry.releng.scl3.mozilla.com
        b-2008-ix-0059.wintry.releng.scl3.mozilla.com
        b-2008-ix-0060.wintry.releng.scl3.mozilla.com
        b-2008-ix-0061.wintry.releng.scl3.mozilla.com
        b-2008-ix-0062.wintry.releng.scl3.mozilla.com
        b-2008-ix-0063.wintry.releng.scl3.mozilla.com
        b-2008-ix-0064.wintry.releng.scl3.mozilla.com

    production:
        b-2008-ix-0137.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0138.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0139.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0140.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0141.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0142.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0143.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0144.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0145.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0146.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0147.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0148.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0149.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0150.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0151.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0152.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0153.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0154.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0155.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0156.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0157.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0158.winbuild.releng.scl3.mozilla.com
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.