818191 - Try buildbot master functionality on AWS

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John Hopkins (:jhopkins)]

We need to know if using a buildbot master inside AWS will work.  If not, then we may need to get more in-house VM capacity, which will take a while. So the sooner we know if an AWS build master works the better.

The complication here is a combination of 1) we have no centos5.5 images in aws; 2) we have no centos6 puppet support for build masters

The existing puppet manfifests for build masters assume centos 5.x in a bunch of places...it may work with minor adjustments for centos6. This is with our old style puppet (puppet-manifests). Massimo is working on puppetagain manifests for centos6.

Comment 1

[Chris AtLee [:catlee]]

i-58957f6a (10.132.48.232) in  us-west-2a has been created for this

Comment 2

[John Hopkins (:jhopkins)]

Massimo's work will not be ready in time for me to make use of, so the options are either:
a) get the old manifests working with centos6, or
b) set up the master by hand.

Comment 3

[Rail Aliiev [:rail]]

Attachment: [configs] limit branches

The idea is to limit a master to a branch so we don't touch production quality builds. The patch helps to limit branches to be tested in staging. Worked fine for me.

Comment 4

[Rail Aliiev [:rail]]

Attachment: [configs] limit branches

A little enhancement to stringify sometimes sneaking in unicode due to json/simplejson - buildbot uses "assert isinstance(b, str)" for builderNames.

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 692966 [details] [diff] [review]
[configs] limit branches

Review of attachment 692966 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with the unicode fixes

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 692966 [details] [diff] [review]
[configs] limit branches

http://hg.mozilla.org/build/buildbot-configs/rev/e8320b597f19

Comment 7

[hwine]

in production

Comment 8

[Rail Aliiev [:rail]]

Attachment: add bm50

The patch adds and enables bm50, limiting it to the build-system branch.

Two thing to pay attention at:
* "scl3" as "datacentre"
* IP address as hostname

Comment 9

[Rail Aliiev [:rail]]

I created a new pool for slaves to be attached to this master:

INSERT INTO pools (name) VALUES ('build-aws-pp');
SELECT * FROM pools WHERE name = 'build-aws-pp'; -- got 45 for ID
INSERT INTO slave_passwords (poolid, distroid, password) VALUES (45, 15, 'xxx');

Comment 10

[Rail Aliiev [:rail]]

Attachment: add bm50

I've just realized that buildbot-configs/update-master-json.py uses `hostname` to figure out which configs should be applied.

Comment 11

[Rail Aliiev [:rail]]

Attachment: add bm50

* use IP for fabric actions

Comment 12

[Chris AtLee [:catlee]]

Comment on attachment 693987 [details] [diff] [review]
add bm50

                  _==|
             _==|   )__)  |
               )_)  )___) ))
              )___) )____))_)
         _    )____)_____))__)\
          \---__|____/|___|___-\\---
  ^^^^^^^^^\   oo oo oo oo     /~~^^^^^^^
    ~^^^^ ~~~~^^~~~~^^~~^^~~~~~
      ~~^^      ~^^~     ~^~ ~^ ~^
           ~^~~        ~~~^^~

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 693987 [details] [diff] [review]
add bm50

http://hg.mozilla.org/build/tools/rev/63efa1c676db

Comment 14

[Rail Aliiev [:rail]]

Forgot to the IP to secrets.pp/$masterIPs. Done now.

Comment 15

[Rail Aliiev [:rail]]

Attachment: Increase load on bm50

So far the finished builds have been green (since I added the IP address to signing servers). It'd be great to increase load gradually and see how it behaves for 3-4 hours, then enable it fully.

Let's start by adding Thunderbird and projects.

Comment 16

[Rail Aliiev [:rail]]

Attachment: puppet-manifests

The current changes need to bootstrap the master.

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 694290 [details] [diff] [review]
Increase load on bm50

http://hg.mozilla.org/build/tools/rev/f58c08e73bb4

Comment 18

[Rail Aliiev [:rail]]

Attachment: release the beast!

I would like to turn all branches tomorrow

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

<nagios-releng>	Thu 16:39:19 PST [403] 10.132.48.232:swap is CRITICAL: SWAP CRITICAL - 100% free (0 MB out of 0 MB) (http://m.allizom.org/swap)

[cltbld@buildbot-master50 ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:          7467       1384       6082          0        134        830
-/+ buffers/cache:        419       7047
Swap:            0          0          0

so no swap configured, I've acked the alert to this bug.

For comparison:
[cltbld@buildbot-master12 ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:          5970       5442        527          0        683       1670
-/+ buffers/cache:       3088       2881
Swap:         6149          0       6149

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 694570 [details] [diff] [review]
release the beast!

http://hg.mozilla.org/build/tools/rev/562761934a56

Comment 21

[Rail Aliiev [:rail]]

Attachment: convert bm50 into a tests master

It's time to test it as a tests masters.

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 695495 [details] [diff] [review]
convert bm50 into a tests master

http://hg.mozilla.org/build/tools/rev/420f9fd1da86

Comment 23

[Rail Aliiev [:rail]]

Since it's been under load, bm50-build1's (m1.large) max CPU utilization was ~12% on Friday. Load average was around 0.1-0.3 during that peak.

Comment 24

[Rail Aliiev [:rail]]

Attachment: rm bm50

Time to shut it down

Comment 25

[Rail Aliiev [:rail]]

Comment on attachment 696291 [details] [diff] [review]
rm bm50

http://hg.mozilla.org/build/tools/rev/76c1243fba73

Comment 26

[Rail Aliiev [:rail]]

All done here.