966070 - s/m1.large/m3.medium/

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(dormant account)]

Amazon's newer nodes are usually more cost-efficient.

This gets us about $100 in savings :)

More seriously, it's a big step up in ECUs and disk->ssd. We can also try m3.mediums as a stretch goal for about $3K in potential savings, but that's more risky.

Comment 1

[Mike Hommey [:glandium]]

Random note: by throwing more m3.medium instances at the problem, it might work and still save money. Depends how well adding masters would work out, though.

Comment 2

[Rail Aliiev [:rail]]

I'll take a look at this.

Comment 3

[(dormant account)]

(In reply to Mike Hommey [:glandium] from comment #1)
> Random note: by throwing more m3.medium instances at the problem, it might
> work and still save money. Depends how well adding masters would work out,
> though.

I'm guessing this is a fairly fixed number of machines. Eg it doesn't grow as rapidly as number of compilation hours does. It makes sense to do something easy here, buy some reserves and call it done.

Comment 4

[(dormant account)]

So according to graphite, these nodes are pretty damn idle on average. m3.medium


https://graphite.mozilla.org/render?from=-24hours&until=now&width=1800&height=900&target=alias(scale(sum(sum(groupByNode(exclude(hosts.buildbot-master*_us*.cpu.*.cpu.*, "(steal)"), 1, "sum")),sum(groupByNode(scale(hosts.buildbot-master*_us*.cpu.*.cpu.steal,0.5),1,"sum"))),0.01), "number of slaves")&target=alias(stacked(avg(groupByNode(exclude(hosts.buildbot-master*_us*.cpu.*.cpu.*, "(steal|idle|wait)"), 1, "sum"))), "cpu busy")&target=alias(stacked(avg(groupByNode(hosts.buildbot-master*_us*.cpu.*.cpu.idle, 1, "sum"))), "idle")&target=alias(stacked(avg(groupByNode(scale(hosts.buildbot-master*_us*.cpu.*.cpu.steal,0.5), 1, "sum"))), "ec2 steal")&target=alias(stacked(avg(groupByNode(hosts.buildbot-master*_us*.cpu.*.cpu.wait, 1, "sum"))), "IO wait")&title=m1.large(buildbot) tester efficiency 24hours

Comment 5

[(dormant account)]

So according to graphite, these nodes are pretty damn idle on average. m3.medium should do fine

Comment 6

[Rail Aliiev [:rail]]

err, wrong bug

Comment 7

[Massimo Gervasini [:massimo]]

Quick update:

here's the list of out m1.large instances in the east region:

buildbot-master51
buildbot-master52
buildbot-master55
buildbot-master57
buildbot-master59
buildbot-master61
buildbot-master62
buildbot-master63
buildbot-master67
buildbot-master69
buildbot-master70
buildbot-master71
buildbot-master75
buildbot-master76
buildbot-master77

All the instances above are using less than 10 Gb disk space, beetween 3 and 8 Gb RAM and the cpus are not too much stressed (all except buildbot-master51, linux test, which uses ~25 Gb on the disk)

Tomorrow, I'll find the machine in this list that uses less resources and I'll make it a m3.medium instance and let it run over the weekend to see how it behaves.

Comment 8

[Massimo Gervasini [:massimo]]

I've found a good candidate in buildbot-master52. Unfortunately I have discovered after the graceful shutdown, that bm52 cannot be recreated as m3.medium or m3.large because these kind of instances are not available in us-east-1b availability zone; the error message says that m3.medium is available in us-east-1a, us-east-1c and us-east-1d

I am restating buildbot-master52 as m1.large (as it was before) and then I'll find another candidate in a different availability zone.

Comment 9

[Massimo Gervasini [:massimo]]

Update:

more data our m1.large masters in east region

Master   Zone    Memory  Disk
name             in use  usage
                   [Mb]
-------------------------------
bm51    east-1a    5353    25G
bm52    east-1b    2050    6.3G
bm59    east-1d    3221    3.3G
bm61    east-1c    4132    4.3G
bm62    east-1b    4522    4.3G
bm63    east-1b    4530    4.0G
bm67    east-1c    4477    7.1G
bm69    east-1a    2928    8.0G
bm70    east-1c    2937    7.2G
bm71    east-1c    2920    7.1G
bm75    east-1b    2199    4.3G
bm76    east-1a    2382    5.7G
bm77    east-1c    1984    6.8G


* Instances in east-1b cannot be converted to m3.medium or m3.large

* Memory in use is total memory - free memory (free + buffer + cached):
  + m3.medium has 3.75 Gb ram (3840 Mb)  
  + m3.large  has 7.5  Gb ram
  + m1.large  has 7.5  Gb ram

* Disk usage: 
  on build-masters 3.7 Gb are used for /dev/shm (defaults to half the RAM size)

m1.large and m3.large are just "too big" for the role of build master, m3.medium will probably just work but since there is no a wide margin (in terms of ram and disk), I prefer not to do the switch during the weekend and wait until next Monday.

Comment 10

[Massimo Gervasini [:massimo]]

Update:

bm77 is now a m3.medium instance, I'll post some data about the memory/disk/cpu usage as soon this master gets some jobs done.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Would be a good to add some swap to handle with long running master instances, which gradually accumulate memory, eg this one up since Jan 14:

[[e-mail] ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:          7467       7159        308          0        495       2379
-/+ buffers/cache:       4284       3183
Swap:            0          0          0

Using more than a m3.medium has.

Comment 12

[Massimo Gervasini [:massimo]]

some data from last 4 days:

memory: http://mzl.la/1jGbX6z
load: http:/mzl.la/1fyeUCm

there are some spikes in the cpu load. 

I have created a swap file, using the add_swap script from bug 964880, to see how this affects the performances of bm77

Comment 13

[(dormant account)]

(In reply to Massimo Gervasini [:mgerva] from comment #12)
> some data from last 4 days:
> 
> memory: http://mzl.la/1jGbX6z
> load: http:/mzl.la/1fyeUCm
> 
> there are some spikes in the cpu load. 
> 
> I have created a swap file, using the add_swap script from bug 964880, to
> see how this affects the performances of bm77

note cpu.idle and cpu.wait are the useful metrics to track

Comment 14

[Massimo Gervasini [:massimo]]

Here are the graphs for cpu idle and wait 

cpu.idle: http://mzl.la/1dQvGQz 
cpu.wait: http://mzl.la/1fBxOrM 

There's no data for the last two days. Is there anything wrong with the data collection on bm77 or am I doing something wrong with graphite?

here some data from the buildbot database: I have extracted the time required to execute the get_basedir command on the slave, from the initial request to end of the job. get_basedir runs the "pwd" command on the slave and its execution is almost immediate, so this is a good measurement of the time spent from the get_basedir command in master's queue


buildbot master 77, get_basedir command, execution time in seconds: 
insance type   period           avg      max   90th percentile
m1.large       1-13 Feb         0.6119    42   1
m3.medium      15 Feb - today   0.6642   119   1

Comment 15

[(dormant account)]

It's cos bug 971883:
https://graphite.mozilla.org/render/?width=1000&height=308&from=-48hours&until=now&target=stacked(hosts.buildbot-master77_srv_releng_use1_mozilla_com.aggregation.cpu-sum.cpu.idle)&target=stacked(hosts.buildbot-master77_srv_releng_use1_mozilla_com.aggregation.cpu-sum.cpu.wait)

Comment 16

[Massimo Gervasini [:massimo]]

I have created a new m3.medium instance from scratch (bm70), there are still a couple of things to fix:

* creating a swap file 
* add a local instance storage

Comment 17

[Massimo Gervasini [:massimo]]

Attachment: 966070-cloud-tools.patch

Hi rail,

this patch enables m3.medium masters with the following specs:
* a 15 GB disk mounted under /
* the instance storage (4GB - SSD), mounted under /mnt/instance_storage, will be used for swapping (bug 964880). 

It also sets the moz-type for this instances to "buildbot-master"

Comment 18

[Massimo Gervasini [:massimo]]

Attachment: [puppet] enable bm70 as buildbot master.patch

puppet patch to enable buildbot-master70 as a build master

Comment 19

[Massimo Gervasini [:massimo]]

Attachment: [tools] enable bm70 as buildbot master.patch

this patch adds buildbot-master70 in production-masters.json

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 8383132 [details] [diff] [review]
[tools] enable bm70 as buildbot master.patch

Review of attachment 8383132 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/maintenance/production-masters.json
@@ +631,5 @@
> +    "buildbot_branch": "production-0.8",
> +    "buildbot_python": "/builds/buildbot/build1/bin/python",
> +    "buildbot_setup": "/builds/buildbot/build1/buildbot/master/setup.py",
> +    "buildbot_version": "0.8.2",
> +    "datacentre": "scl3",

A nit, use aws-us-east-1 instead.

Comment 21

[Justin Wood (:Callek)]

Comment on attachment 8383132 [details] [diff] [review]
[tools] enable bm70 as buildbot master.patch

Review of attachment 8383132 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/maintenance/production-masters.json
@@ +653,5 @@
> +    "basedir": "/builds/buildbot/try1",
> +    "bbconfigs_branch": "production",
> +    "bbconfigs_dir": "/builds/buildbot/try1/buildbot-configs",
> +    "bbcustom_branch": "production-0.8",
> +  {

[drive-by-lucky-notice]

ugh -- this will break the whole json.

{
  ..
  stuff
  ...
{

--- basically it looks like your copy/paste included more things than you intended.

r- due to that, but if you fix it no objection

Comment 22

[Massimo Gervasini [:massimo]]

Comment on attachment 8383132 [details] [diff] [review]
[tools] enable bm70 as buildbot master.patch

Thanks rail, thanks Callek!

Comment 23

[Massimo Gervasini [:massimo]]

we have a m3.medium instance, buildbot-master70, in production. This master has two slaves locked to
it: bld-linux64-ec2-003 and bld-linux64-ec2-038. I'm going to leave it running over the weekend to see if the new instance type behaves properly.  

I will check on it on Saturday as well.

Comment 24

[Massimo Gervasini [:massimo]]

Attachment: [puppet] add buildbot-master70 in signing_new_token_allowed_ips

bm70 is not able to sign because of: "token generation failed, error message: 403 Forbidden" exception. 

This patch adds bm70 ip address in signing_new_token_allowed_ips.

Comment 25

[Massimo Gervasini [:massimo]]

Changes for buildbot-master70 have landed in production and all the builds are green. 
I am enabling bm70 in slavealloc and unlocking its slaves (bld-linux64-ec2-003 and bld-linux64-ec2-038)

Comment 26

[Massimo Gervasini [:massimo]]

bm70 is working fine, I am starting replacing existing masters in east-1 with m3.medium instances.

On east-1 we have:

* 1 try master (bm59)
* 4 test masters (bm51, bm52, bm67, bm69)
* 5 build masters (bm57, bm61, bm62, bm63, bm70)

I am going to create 4 new masters to replace bm57, bm61, bm62 and bm63

Comment 27

[Massimo Gervasini [:massimo]]

Attachment: [tools] - add bm{71..74} to production-master.json.patch

adding buildbot-master 71..74 in production-master.json

Comment 28

[Massimo Gervasini [:massimo]]

Attachment: [puppet] recreate bm{71..74} as build masters and add them in signing_new_token_allowed_ips.patch

This patch recreates buildbot-master 71..74 as build masters and adds their IP addresses in signing_new_token_allowed_ips list.

Here's the list of IP addresses:

buildbot-master71: 10.134.49.111
buildbot-master72: 10.132.50.54
buildbot-master73: 10.132.49.181
buildbot-master74: 10.132.49.94

buildbot-master 72, 73 and 74 are in aws west-2 zone

Comment 29

[Massimo Gervasini [:massimo]]

Attachment: [tools] - add bm{71..74} to production-master.json.patch

fixed wrong datacentre and hostnames for nodes in west-2 region (bm72, bm73 and 74)

Comment 30

[Simone Bruno [:simone]]

Comment on attachment 8386058 [details] [diff] [review]
[tools] - add bm{71..74} to production-master.json.patch

Hey mgerva,

The name key for each of the new entries is the same: "bm70-build1", I guess this needs to be fixed.

Comment 31

[Massimo Gervasini [:massimo]]

Attachment: [tools] - add bm{71..74} to production-master.json.patch

Thanks simone,

here is the patch with updates "names:"

Comment 32

[Simone Bruno [:simone]]

Comment on attachment 8386064 [details] [diff] [review]
[tools] - add bm{71..74} to production-master.json.patch

Now it looks ok to me.

In order to review, I compared this patch with the patches relative to buildbot-master70 (now in prod), and all relevant code parts seem to have been updated correctly.

Comment 33

[Justin Wood (:Callek)]

Comment on attachment 8386046 [details] [diff] [review]
[puppet] recreate bm{71..74} as build masters and add them in signing_new_token_allowed_ips.patch

merged to production

Comment 34

[Massimo Gervasini [:massimo]]

I am rolling out the first batch of m3.medium build masters in production.

bm70, bm71, bm72, bm73 and bm74 in west-2 are now online and are already receiving new jobs. Each of the new instances will replace a m1.large instance as follows:
* bm70 -> bm63
* bm71 -> bm61
* bm72 -> bm58
* bm73 -> bm64
* bm74 -> bm65

After a clean shutdown, m1.large instances will be stopped in aws so they can be re-enabled in case of emergency

Comment 35

[Massimo Gervasini [:massimo]]

Attachment: [tools] create 4 try masters, 1 build master, disable old masters.patch

Second batch of masters, 1 build master and 4 try masters, as follows:

try:
bm75, bm76 in east-1
bm78, bm79 in west-2

build:
bm77 in east-1

this patch sets enabled => false, for all the masters above and for bm58, bm61, bm63, bm64, bm65 as part of the migration from the first batch

Comment 36

[Massimo Gervasini [:massimo]]

Attachment: [puppet] create 4 try master and a build master.patch

Enable 4 new try masters: 

* bm75, bm76 east-1
* bm78, bm79 west-2

enable a build master in east-1: bm77 (so we have 3 build masters for region)

this patch also adds bm77 IP address (10.134.49.77) to signing_new_token_allowed_ips

Comment 37

[Massimo Gervasini [:massimo]]

Attachment: [puppet] create 4 try master and a build master.patch

added new try masters IPs to signing_new_token_allowed_ips

Comment 38

[Rail Aliiev [:rail]]

Something here broke Firefox 28.0b9.

bm71-74 are enabled in slavealloc and accept new jobs, but they are not enabled in production-masters.json, what prevents fabric reconfigs on them. release-runner uses this files to reconfig only enabled masters, so it skipped the ones listed above.

As a result we've seen a failed bouncer submitter job trying to submit jobs for 28.0b8. I also cancelled 2 xulrunner builds to be sure.

I disabled the masters in slavealloc and started a graceful shutdown

Comment 39

[Justin Wood (:Callek)]

I also want to call out, if not already done, that when they are brought back up we need to go into nagios and re-enable notifications for these hosts, otherwise they could go down and no-one would be the wiser

Comment 40

[Massimo Gervasini [:massimo]]

Attachment: [puppet] remove buildbot-master75 from moco-nodes.patch

bm75 IP address is in east-1-b subnet but m3.medium instances are not available in this subnet. Deleting this entry from puppet.

Comment 41

[Massimo Gervasini [:massimo]]

Attachment: [tools] remove buildbot-master75 from production-masters.json.patch

Deleting bm75 from production-master.json file - it cannot be puppetized as m3.medium, so we should remove this entry created in attachment 8387036 [details] [diff] [review].

Comment 42

[Massimo Gervasini [:massimo]]

Attachment: [puppet] remove buildbot-master75 from moco-nodes.patch

bm75 IP address is in east-1-b subnet but m3.medium instances are not available in this subnet. Deleting this entry from puppet... and its IP address from signing_new_token_allowed_ips

Comment 43

[Rail Aliiev [:rail]]

Comment on attachment 8387691 [details] [diff] [review]
[tools] remove buildbot-master75 from production-masters.json.patch

I'd rather change the DNS entry for bm75 instead in the inventory. Just find a free IP in other subnets, change the A and PTR entries in the inventory: https://inventory.mozilla.org/en-US/core/search/#q=10.134.48.158

Comment 44

[Massimo Gervasini [:massimo]]

A patch to enable bm71..74 and bm76..79 has just landed in tools (changeset: 4427:13eedba9148f)
The new masters have been enabled in slavealloc too.

Comment 45

[Massimo Gervasini [:massimo]]

All the new masters are on-line and working. 

I have started the shutdown procedure for the old masters.

Comment 46

[[e-mail] (:bhearsum)]

(In reply to Massimo Gervasini [:mgerva] from comment #45)
> All the new masters are on-line and working. 
> 
> I have started the shutdown procedure for the old masters.

These are starting to alert in nagios. Please downtime or otherwise silence the checks in the future. I've ack'ed the one that just went off.

Comment 47

[Massimo Gervasini [:massimo]]

Attachment: [tools] disable bm56, 57, 59 and 60 from production-masters.json.patch

Disable buildbot master 56, 57, 59 and 60 from production-masters.json before stopping the old master instances.

Comment 48

[Massimo Gervasini [:massimo]]

The following masters have been stopped:

* bm56
* bm57
* bm58
* bm59
* bm60
* bm63
* bm64

two old masters instances are still running because they have some other services

* bm61 - self serve agent
* bm65 - slave rebooter

Comment 49

[Massimo Gervasini [:massimo]]

Attachment: [puppet] add selfserve_agent to bm{70..73}; add two build masters: bm91 and bm94.patch

This patch enables selfserve_agent on bm70, bm71, bm72 and bm73 and creates two new buildmasters: bm91 and 94

Comment 50

[Massimo Gervasini [:massimo]]

Attachment: [tools] create two new buildmasters: bm91 and bm94.patch

Create two new build masters: bm90 and bm94

Comment 51

[Chris AtLee [:catlee]]

Comment on attachment 8393004 [details] [diff] [review]
[tools] create two new buildmasters: bm91 and bm94.patch

Review of attachment 8393004 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/maintenance/production-masters.json
@@ +1176,5 @@
> +    "buildbot_branch": "production-0.8",
> +    "buildbot_python": "/builds/buildbot/build1/bin/python",
> +    "buildbot_setup": "/builds/buildbot/build1/buildbot/master/setup.py",
> +    "buildbot_version": "0.8.2",
> +    "datacentre": "aws-us-east-1",

should be aws-us-west-2 I think?

Comment 52

[Massimo Gervasini [:massimo]]

Attachment: [tools] remove bm{56..65} from production-masters.json.patch

Removing bm{56..65} entries from production-masters.json - these instances are already disabled on slavealloc.

Comment 53

[Massimo Gervasini [:massimo]]

Attachment: [puppet] disable bm{56..65} and remove their ips from signing_new_token_allowed_ips.patch

remove buildbot master entries for bm{56..65} from nodes. Removed their IPs from signing_new_token_allowed_ips.

Comment 54

[Massimo Gervasini [:massimo]]

Current status:

m1.large masters
* bm{56..65} instances except for bm61 bm62 and bm65 have been stopped because they are running extra services that are not migrated yet (self-serve agent, slaverebootrbug 985088)
* bm55 and bm66 still in production

m3.medium masters
* bm{70..74}, bm{76..79} are on-line and taking jobs 
* bm75 is on hold because it has a new IP address and this requires new flows (bug 985055)
* bm91 and bm94 are running test jobs - if tests pass, they will replace bm55 and bm66

Comment 55

[Massimo Gervasini [:massimo]]

Attachment: [tools] enable bm91 and bm94 in production-master.json.patch

New masters have been running for several hours and all the builds are green. This patch enables them in production-masters.json.

When this patch lands, I'll start a graceful shutdown of bm55 and bm66

Comment 56

[Massimo Gervasini [:massimo]]

(In reply to Massimo Gervasini [:mgerva] from comment #55)

> When this patch lands, I'll start a graceful shutdown of bm55 and bm66

err bm55 is a try master, I'm not stopping it.

Comment 57

[Massimo Gervasini [:massimo]]

(In reply to Massimo Gervasini [:mgerva] from comment #56)
> (In reply to Massimo Gervasini [:mgerva] from comment #55)
> 
> > When this patch lands, I'll start a graceful shutdown of bm55 and bm66
> 
> err bm55 is a try master, I'm not stopping it.

Please ignore the above comment, I am stopping bm55.

Comment 58

[Massimo Gervasini [:massimo]]

Attachment: [puppet] disable buildbot_master and remove their ips from signing_new_token_allowed_ips for bm55 and bm66.patch

Disable buildbot master 55 and 66; removing their IP addresses from signing_new_token_allowed_ips

Comment 59

[Massimo Gervasini [:massimo]]

Attachment: [tools] remove bm66 and bm55 from production-master.json.patch

Remove buildbot master 55 and 66 from production-masters.json

Buildbot master 66 is still running some jobs so this patch (and the puppet one) will land when the master has stopped properly.

Comment 60

[Massimo Gervasini [:massimo]]

Status update:

New m3.medium instances, bm{70..79}, bm91 and bm94: 
* all the new m3.masters are active (except for buildbot master75 - bug 985055)

Old m1.large instances, bm{55..66} 
* the following instances have been terminated:  
 bm56
 bm57
 bm58
 bm59
 bm60
 bm63
 bm64
 
* leaving the following instance stopped for a while (it has been stopped for few hours)
  bm55 

* the following instances are still up and running because they are still running other services (buildbot process is stoppe)
 bm61
 bm62
 bm65
 bm66

Comment 61

[Massimo Gervasini [:massimo]]

Attachment: [tools] enable buildbot-master75 in production-masters.json.patch

Two green builds on bm75, waiting for the current job to finish to enable master in production-masters.json

Comment 62

[Massimo Gervasini [:massimo]]

Buildbot-master75 is in production. All the try/build buildbot masters on m1.large have been migrated to m3.medium instances.

Some old m1.large machines require further steps before we can terminate them because they have some running services:

bm61: self serve 
bm62: self serve
bm65: self serve, slave rebooter
bm66: self serve, gaia_bumper, b2g_bumper (Bug 985767 and Bug 986474)

* self serve is already running on m3.medium machines (bm70,71,72 and 73) but it needs to be tested.
* slave rebooter requires another puppet patch that needs to be coordinated with buildduty
* gaia_bumper and b2g_bumper will never fit on a m3.medium instance because of the disk and cpu usage, so bm66 will stay up for a while.

Comment 63

[Massimo Gervasini [:massimo]]

Attachment: [puppet] remove selfserve_agent from bm61, bm62, bm63 and bm65.patch

This patch removes selfserve_agent from bm61, bm62, bm63 and bm65

Waiting for nagios notifcations to be enabled on new self serve agents before landing this patch (bug 987173)

Comment 64

[Massimo Gervasini [:massimo]]

Attachment: [puppet] move slave rebooter service from bm65 to bm74.patch

Preparing to migrate slaverebooter from bm65 to bm74. Before landing this patch we need to wait for the nagios monitoring for slaverebooter on bm74 (bug  987793).

Comment 65

[Massimo Gervasini [:massimo]]

Attachment: [puppet] bm55: disable buildbot_master and remove its IP from signing_new_token_allowed_ips.patch

bm55 was still active in puppet master definition. This patch removes it and its IP address from signing_new_token_allowed_ips

Comment 66

[[e-mail] (:bhearsum)]

I migrated bm69 to an m3.medium today.

Comment 67

[[e-mail] (:bhearsum)]

buildbot-master-servo-01 has been migrated to an m3.medium, too.

Comment 68

[Massimo Gervasini [:massimo]]

Slave rebooter moved from bm65 to bm74.

Comment 69

[Massimo Gervasini [:massimo]]

Hi Ben,

Are there any other buidbot-masters to migrate to m3.medium?

Comment 70

[[e-mail] (:bhearsum)]

(In reply to Massimo Gervasini [:mgerva] from comment #69)
> Hi Ben,
> 
> Are there any other buidbot-masters to migrate to m3.medium?

I don't think so. The only ones left are:
51, 52, 53, 54, 67, 68 -- which are all linux64 test masters, which need more RAM than the m3.medium has.

65 - which is already stopped
66 - which I think is having its services moved elsewhere?

Comment 71

[Massimo Gervasini [:massimo]]

> 65 - which is already stopped
> 66 - which I think is having its services moved elsewhere?

(In reply to Ben Hearsum [:bhearsum] from comment #70)
> (In reply to Massimo Gervasini [:mgerva] from comment #69)
> > Hi Ben,
> > 
> > Are there any other buidbot-masters to migrate to m3.medium?
> 
> I don't think so. The only ones left are:
> 51, 52, 53, 54, 67, 68 -- which are all linux64 test masters, which need
> more RAM than the m3.medium has.
> 
> 65 - which is already stopped
> 66 - which I think is having its services moved elsewhere?

* bm65 is stopped, waiting for all the netflows/nagios notifications to be closed (bug 990775)
* bm66 is tracked here: Bug 990172