The default bug view has changed. See this FAQ.

scl1 Move Train D releng config Work

RESOLVED FIXED

Status

Release Engineering
Platform Support
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: hwine, Assigned: hwine)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(10 attachments)

(Assignee)

Description

3 years ago
+++ This bug was initially created as a clone of Bug #1013035 +++

Train D starts June 16
(Assignee)

Comment 1

3 years ago
Downloaded from https://docs.google.com/a/mozilla.com/spreadsheets/d/1CSbCNtuY6pZMHtAs6KmOUgPholm4zMsE9SssWhZ0TfI as csv:
    SCL1-_SCL3_ Move Train D - linux build - thunderbird.csv
    SCL1-_SCL3_ Move Train D - linux build.csv
    SCL1-_SCL3_ Move Train D - linux try - thunderbird and fuzzing.csv
    SCL1-_SCL3_ Move Train D - mac-signing.csv
    SCL1-_SCL3_ Move Train D - talos-r4-snow.csv
    SCL1-_SCL3_ Move Train D - windows build - esr and b2g18 .csv
    SCL1-_SCL3_ Move Train D - windows build.csv
    SCL1-_SCL3_ Move Train D - windows try.csv

Used braindump/it-related/move_hosts to generate attachments:
    braindump/it-related/move_hosts SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d.csv
    braindump/it-related/move_hosts --short SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d_short.txt
    braindump/it-related/move_hosts --short --slavealloc SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d_slavealloc.sql
    braindump/it-related/move_hosts --short --graphserver SCL1-_SCL3_\ Move\ Train\ D\ -\ *csv >train_d_graphs.sql
Status: NEW → ASSIGNED
(Assignee)

Comment 2

3 years ago
Created attachment 8440432 [details]
train_d.csv

For use as input to slavealloc scripts in braindump/it-related
(Assignee)

Comment 3

3 years ago
Created attachment 8440433 [details]
train_d_short.txt

For use as input to braindump/it-related/rename_host_bugs
(Assignee)

Comment 4

3 years ago
Created attachment 8440435 [details]
train_d_slavealloc.sql

SQL to update slavealloc database
(Assignee)

Comment 5

3 years ago
Created attachment 8440436 [details]
train_d_graphs.sql

SQL to update graph servers
(Assignee)

Comment 6

3 years ago
All hosts disabled, 71 (of 81) previously enabled.
$ ./get_enabled_hosts train_d.csv 
Enter password: 
next step is to run 'disable_all_hosts train_d.csv'
0 [hwine@cruncher it-related]
$ ./disable_all_hosts train_d.csv
Enter password: 
0 [hwine@cruncher it-related]
$ wc train_d.csv.previously_enabled 
  71   71 1199 train_d.csv.previously_enabled
0 [hwine@cruncher it-related]
$ wc train_d.csv
  81   81 7024 train_d.csv
0 [hwine@cruncher it-related]
$
(Assignee)

Comment 7

3 years ago
Created attachment 8440683 [details]
train_d_nagios.txt

commands for moc to disable nagios for hosts
Looks like one failed but otherwise all clear:

[rwatson@nagios1.private.releng.scl3 rwatson]# ./downtime.sh
Downtime for bld-linux64-ix-049.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for bld-linux64-ix-050.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for mac-signing3.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for mac-signing4.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-131.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-132.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-133.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-134.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-135.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-136.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-137.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-138.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-139.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-140.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-141.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-142.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-143.build.scl1.mozilla.com of 172800 seconds scheduled.
Can't find a host by that name.
Downtime for talos-r4-snow-145.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-146.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-147.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-148.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-149.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-150.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-151.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-152.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-153.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-154.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-155.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-156.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-157.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-158.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-159.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-160.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-161.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-162.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-163.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-164.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-165.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-166.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-167.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-168.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-169.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-170.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for talos-r4-snow-ref.build.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave04.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave05.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave123.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave124.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave125.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave126.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave127.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave128.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave129.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave130.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave131.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave132.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave133.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave134.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave135.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave136.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave137.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave138.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave139.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave140.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave141.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave142.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave143.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave144.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave157.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave160.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave161.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave162.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave163.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave164.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave165.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave166.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave167.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave168.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave169.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
Downtime for w64-ix-slave170.winbuild.scl1.mozilla.com of 172800 seconds scheduled.
(Assignee)

Comment 9

3 years ago
bug aliases & summaries changed
  $ braindump/it-releated/rename_host_bugs train_d_short.txt
(Assignee)

Comment 10

3 years ago
Databases updated:
    $ mysql --defaults-file=graph-stage.ini -e "source train_d_graphs.sql"
    Enter password: 
    0 [hwine@cruncher it-related (fabric)]
    $ mysql --defaults-file=graph-prod.ini -e "source train_d_graphs.sql"
    Enter password: 
    0 [hwine@cruncher it-related (fabric)]
    $ mysql --defaults-file=slavealloc.ini -e "source train_d_slavealloc.sql"
    Enter password: 
    0 [hwine@cruncher it-related (fabric)]
(Assignee)

Comment 11

3 years ago
Created attachment 8440882 [details] [diff] [review]
mac-signing.patch

update puppet to know where the servers are now.
Attachment #8440882 - Flags: review?(bhearsum)
Comment on attachment 8440882 [details] [diff] [review]
mac-signing.patch

Review of attachment 8440882 [details] [diff] [review]:
-----------------------------------------------------------------

::: manifests/moco-nodes.pp
@@ +161,5 @@
>  node /mac-signing[12].srv.releng.scl3.mozilla.com/ {
>      include toplevel::server::signing
>  }
>  
> +node /mac-signing[34].srv.releng.scl3.mozilla.com/ {

You should probably put this into the previous regex. r=me with that changed.
Attachment #8440882 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 13

3 years ago
Comment on attachment 8440882 [details] [diff] [review]
mac-signing.patch

Committed with requested change:
 https://hg.mozilla.org/build/puppet/rev/d8d8accf04b4
Attachment #8440882 - Flags: checked-in+
Move Train D - linux try - thunderbird and fuzing => r89157.
Move Train D - mac-signing => r89159.
Move Train D - talos-r4-snow => r89170.
Move Train D - windows build => 89171.
Move Train D - windows try => 89172.

Added missing switches => 89173. (switch1.r201-{7,8}.ops.releng.scl3.mozilla.net)

Fixing cluster checks in separate Bug 1024512.

80 hosts downtimed.

Etherpad updated with this info.
Created attachment 8441370 [details] [diff] [review]
b-linux64-ix range is off-by-one

Master exception logs are complaining about this host, 10.26.64.62
(b-linux64-ix-0012.try.releng.scl3.mozilla.com).
Attachment #8441370 - Flags: review?(pmoore)
Comment on attachment 8441370 [details] [diff] [review]
b-linux64-ix range is off-by-one

Review of attachment 8441370 [details] [diff] [review]:
-----------------------------------------------------------------

I'm afraid there is a much bigger delta that needs to be processed:

Hosts that should have already been in buildbot but were not
============================================================
mac-signing3
mac-signing4
talos-r4-snow-ref

Hosts that were already in buildbot but should not have been
============================================================
b-2008-ix-0054
b-2008-ix-0055
b-2008-ix-0056
b-2008-ix-0057
b-2008-ix-0058
b-2008-ix-0059
b-2008-ix-0060
b-2008-ix-0061
b-2008-ix-0062
b-2008-ix-0063
b-2008-ix-0064
t-snow-r4-0127
t-snow-r4-0128
t-snow-r4-0129
t-snow-r4-0130
t-snow-r4-0131
t-snow-r4-0132
t-snow-r4-0133
t-snow-r4-0134
t-snow-r4-0135
t-snow-r4-0136
t-snow-r4-0137
t-snow-r4-0138
t-snow-r4-0139
t-snow-r4-0140
t-snow-r4-0141
t-snow-r4-0142
t-snow-r4-0143
t-snow-r4-0144
t-snow-r4-0145
t-snow-r4-0146
t-snow-r4-0147
t-snow-r4-0148
t-snow-r4-0149
t-snow-r4-0150
t-snow-r4-0151
t-snow-r4-0152
t-snow-r4-0153
t-snow-r4-0154
t-snow-r4-0155
t-snow-r4-0156
t-snow-r4-0157
t-snow-r4-0158
t-snow-r4-0159
t-snow-r4-0160
t-snow-r4-0161
t-snow-r4-0162
t-snow-r4-0163
t-snow-r4-0164
t-snow-r4-0165
t-snow-r4-0166

Hosts that should have been removed
===================================
bld-linux64-ix-049
bld-linux64-ix-050
talos-r4-snow-131
talos-r4-snow-132
talos-r4-snow-133
talos-r4-snow-134
talos-r4-snow-135
talos-r4-snow-136
talos-r4-snow-137
talos-r4-snow-138
talos-r4-snow-139
talos-r4-snow-140
talos-r4-snow-141
talos-r4-snow-142
talos-r4-snow-143
talos-r4-snow-144
talos-r4-snow-145
talos-r4-snow-146
talos-r4-snow-147
talos-r4-snow-148
talos-r4-snow-149
talos-r4-snow-150
talos-r4-snow-151
talos-r4-snow-152
talos-r4-snow-153
talos-r4-snow-154
talos-r4-snow-155
talos-r4-snow-156
talos-r4-snow-157
talos-r4-snow-158
talos-r4-snow-159
talos-r4-snow-160
talos-r4-snow-161
talos-r4-snow-162
talos-r4-snow-163
talos-r4-snow-164
talos-r4-snow-165
talos-r4-snow-166
talos-r4-snow-167
talos-r4-snow-168
talos-r4-snow-169
talos-r4-snow-170
w64-ix-slave04
w64-ix-slave05
w64-ix-slave123
w64-ix-slave124
w64-ix-slave125
w64-ix-slave126
w64-ix-slave127
w64-ix-slave128
w64-ix-slave129
w64-ix-slave130
w64-ix-slave131
w64-ix-slave132
w64-ix-slave133
w64-ix-slave134
w64-ix-slave135
w64-ix-slave136
w64-ix-slave137
w64-ix-slave138
w64-ix-slave139
w64-ix-slave140
w64-ix-slave141
w64-ix-slave142
w64-ix-slave143
w64-ix-slave144
w64-ix-slave157
w64-ix-slave160
w64-ix-slave161
w64-ix-slave162
w64-ix-slave163
w64-ix-slave164
w64-ix-slave165
w64-ix-slave166
w64-ix-slave167
w64-ix-slave168
w64-ix-slave169
w64-ix-slave170

Hosts that should not have been removed
=======================================

Hosts that should have been added
=================================
b-2008-ix-0137
b-2008-ix-0138
b-2008-ix-0139
b-2008-ix-0140
b-2008-ix-0141
b-2008-ix-0142
b-2008-ix-0143
b-2008-ix-0144
b-2008-ix-0145
b-2008-ix-0146
b-2008-ix-0147
b-2008-ix-0148
b-2008-ix-0149
b-2008-ix-0150
b-2008-ix-0151
b-2008-ix-0152
b-2008-ix-0153
b-2008-ix-0154
b-2008-ix-0155
b-2008-ix-0156
b-2008-ix-0157
b-2008-ix-0158
b-2008-ix-0182
b-2008-ix-0183
b-2008-ix-0184
b-linux64-ix-0013
mac-signing3
mac-signing4
t-snow-r4-ref

I'll attach the files I used to generate this report.
Attachment #8441370 - Flags: review?(pmoore) → review-
Created attachment 8441397 [details]
buildbot_slave_list.py
Created attachment 8441401 [details]
buildbot_slave_list.sh

If you download buildbot_slave_list.py and buildbot_slave_list.sh, you can run this report with, e.g.

bash buildbot_slave_list.sh

Please note it is downloading the attachments to this bug, so if you are changing these attachments, you might want to change the script to take your local version(s) of the files instead of curling them down.
(Assignee)

Comment 19

3 years ago
Created attachment 8441609 [details] [diff] [review]
train_d.patch
Attachment #8441609 - Flags: review?(coop)
Comment on attachment 8441609 [details] [diff] [review]
train_d.patch

Review of attachment 8441609 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with cleanup addressed.

::: mozilla-tests/production_config.py
@@ +23,5 @@
>  for i in range(1,131):
>      SLAVES['win8']['t-w864-ix-%03i' % i] = {}
>  
> +for i in range(127,167):
> +    SLAVES['snowleopard']['t-snow-r4-%04i' % i] = {}

Just remove this range. It's covered right below.

::: mozilla/production_config.py
@@ +2,5 @@
>  LINUX_IXS      = []
>  LINUX64_IXS    = []
>  WIN32_IXS      = []
>  WIN64_IXS      = []
> +WIN64_REV2     = ['w64-ix-slave%03i' % x for x in range(145,148) + [156]] + \

There are no w64-ix slaves in slavealloc any more. If that's expected, we can remove them all here.

@@ +35,5 @@
>  TRY_WIN64_IXS  = []
>  TRY_LINUX64_EC2 = ['try-linux64-ec2-%03d' % x for x in range(1, 60) + range(301,340)] + \
>      ['try-linux64-spot-%03d' % x for x in range(1, 200) + range(300,500)] + \
>      ['try-linux64-spot-%d' % x for x in range(1000, 1100)]
> +TRY_WIN64_REV2 = ['b-2008-ix-%04i' % x for x in range(18, 65) + range(30, 42) + range(173,182)] + \

range(18, 65) includes range(30, 42).

::: mozilla/staging_config.py
@@ +4,5 @@
>  MAC_LION_MINIS = ['bld-lion-r5-%03d' % x for x in range(89, 93)]
>  WIN32_IXS      = []
>  WIN64_IXS      = []
>  WIN64_REV2     = ['ix-mn-w0864-%03d' % x for x in range(1,3)] + \
> +                 ['w64-ix-slave22'] + \

Again, no w64-ix slaves are listed in slavealloc any more. We can rip this out if that's true.
Attachment #8441609 - Flags: review?(coop) → review+
(Assignee)

Comment 21

3 years ago
Comment on attachment 8441609 [details] [diff] [review]
train_d.patch

With changes above, still passes test-masters.sh. Some noise from Pete's scripts:

    patching file mozilla-tests/production_config.py
    patching file mozilla/production_config.py
    patching file mozilla/staging_config.py
    Hosts that should have already been in buildbot but were not
    ============================================================

    Hosts that were already in buildbot but should not have been
    ============================================================
    are listed in file 'already-in-buildbot', but you can't change them:
        51 already-in-buildbot

    Hosts that should have been removed
    ===================================

    Hosts that should not have been removed
    =======================================
    w64-ix-slave145
    w64-ix-slave146
    w64-ix-slave147
    w64-ix-slave156
    w64-ix-slave22

    Hosts that should have been added
    =================================

    Hosts that should not have been added
    =====================================

Landed:
 http://hg.mozilla.org/build/buildbot-configs/rev/b715c4dac923
Attachment #8441609 - Flags: checked-in+
looks like we putting b-2008-ix-0184.wintry.releng.scl3.mozilla.com in build-scl3 (not-try):
    - http://hg.mozilla.org/build/buildbot-configs/annotate/b715c4dac923/mozilla/production_config.py#l6
    - slavealloc has this slave set with: 'trust=core' and 'pool=build-scl3'

Just want to sanity check this host as its fqdn suggests it should be try. /me doesn't add ssh keys or enable.
(Assignee)

Updated

3 years ago
Depends on: 1027394
I didn't get around to this today. The end result was that we can go ahead and add the ssh keys and enable the everything that is left bar bug 1027394 and the top three from:  https://hwine.pastebin.mozilla.org/5434018

I will do that first thing in the morning PT :)
so from the windows slaves: we had 1 not added (1027394). I didn't add 182,183 as they were staging slaves. hwine, what was the plan for these again? Make them staging but keep their 'try' fqdn/vlan?

All other win slaves looked good. They have the right keys now and are enabled. See below for breakdown.

BTW - I took a look at the darwin and two linux slaves that were part of this move. It doesn't look they are enabled. If they have been re-imaged, there are no post image steps and they can just be enabled. Would you like me to do that?

no keys added, not enabled:
    slavealloc thinks in production pool:
        b-2008-ix-0184.wintry.releng.scl3.mozilla.com

    slavealloc thinks in staging pool:
        b-2008-ix-0182.wintry.releng.scl3.mozilla.com
        b-2008-ix-0183.wintry.releng.scl3.mozilla.com

keys added, enabled:

    try:
        b-2008-ix-0054.wintry.releng.scl3.mozilla.com
        b-2008-ix-0055.wintry.releng.scl3.mozilla.com
        b-2008-ix-0056.wintry.releng.scl3.mozilla.com
        b-2008-ix-0057.wintry.releng.scl3.mozilla.com
        b-2008-ix-0058.wintry.releng.scl3.mozilla.com
        b-2008-ix-0059.wintry.releng.scl3.mozilla.com
        b-2008-ix-0060.wintry.releng.scl3.mozilla.com
        b-2008-ix-0061.wintry.releng.scl3.mozilla.com
        b-2008-ix-0062.wintry.releng.scl3.mozilla.com
        b-2008-ix-0063.wintry.releng.scl3.mozilla.com
        b-2008-ix-0064.wintry.releng.scl3.mozilla.com

    production:
        b-2008-ix-0137.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0138.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0139.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0140.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0141.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0142.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0143.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0144.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0145.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0146.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0147.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0148.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0149.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0150.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0151.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0152.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0153.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0154.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0155.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0156.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0157.winbuild.releng.scl3.mozilla.com
        b-2008-ix-0158.winbuild.releng.scl3.mozilla.com

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.