488368 - Set up new buildbot master for pool of talos slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

The new master should live on qm-rhel02, and go into
/build/perfmaster-pool

It should get its own files in buildbot-configs as well.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

While we are setting up this new master anyway, could we do it on a different freshly created VM or else can we rename qm-rhel02?

"production-talos-master.b.m.o" or "talos-master.b.m.o" would be much more intuitive, imho.

Comment 2

[Chris AtLee [:catlee]]

I'd be happy with talos-master.m.o.  The talos machines don't live in the build network.

We could set up a CNAME from talos-master.m.o -> qm-rhel02.m.o, or from qm-rhel02.m.o -> talos-master.m.o to make the transition easier.

Talos Try also runs on this machine, so "production-talos-master" isn't really appropriate...unless we set up another CNAME from try-talos-master.m.o -> talos-master.m.o.

Comment 3

[Chris AtLee [:catlee]]

On second thought, we should spin up new VMs for this.

Will file secondary bugs for new VMs for talos-master.m.o and talos-staging-master.m.o.

Comment 4

[Chris AtLee [:catlee]]

We need an upgraded buildbot since it has a fix for FIFO build processing when slaves connect.  Since slaves are constantly disconnecting / reconnecting in Talos, this is vital to have.

We could also consider backporting the fix to our 0.7.9 version of buildbot.

Comment 5

[[e-mail] (:bhearsum)]

Going to steal this from Catlee (with permission).

Comment 6

[[e-mail] (:bhearsum)]

Alright, post-install is done on talos-master and talos-staging-master. I also removed /builds/scratchbox since we have no use for it on a Buildbot master. I didn't bother cloning buildbot-configs since the new ones aren't there yet, anything else you'd like me to do, catlee?

Comment 7

[[e-mail] (:bhearsum)]

Off to catlee for the rest.

Comment 8

[Chris AtLee [:catlee]]

Both masters have buildbot set up and running.  The rest is should just be configuration tweaks.

Comment 9

[Chris AtLee [:catlee]]

Attachment: Buildbot configs for staging pool

Comment 10

[alice nodelman [:alice] [:anode]]

Comment on attachment 376289 [details] [diff] [review]
Buildbot configs for staging pool

+                       sendToInterestedUsers=False,
+                       extraRecipients=["[e-mail]"]

Somehow, I don't think that I want to be getting this mail.  Maybe pull this out.

Comment 11

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #10)
> (From update of attachment 376289 [details] [diff] [review])
> +                       sendToInterestedUsers=False,
> +                       extraRecipients=["[e-mail]"]
> 
> Somehow, I don't think that I want to be getting this mail.  Maybe pull this
> out.
Funny.


catlee: Could you add me to that? Or maybe build-announce? 

At least for the first while of using the new pool-of-talos-slaves, I would be interested in getting these to see how it is going. After a while of working routinely, I'd probably be back asking to get removed, and rely on some nagios alerts instead.

Comment 12

[Chris AtLee [:catlee]]

Attachment: Buildbot configs for staging pool

Pretty much the same as before, except adding more slaves to the list, disable mail to people on failure, enabling build-skipping on all branches, and fixing up the buildbot categories.

Comment 13

[Chris AtLee [:catlee]]

Attachment: Buildbot configs for production pool

Build request merging is enabled here to start with.  The plan is to disable merging for the primary branches at the very least once we have enough slaves in the pool.

Comment 14

[Chris AtLee [:catlee]]

Attachment: Buildbot configs for production pool

Same as before, except set up Tinderbox reporting properly

Comment 15

[alice nodelman [:alice] [:anode]]

Comment on attachment 378658 [details] [diff] [review]
Buildbot configs for production pool

Why does the pool need a copy of the ftppoller?

Comment 16

[alice nodelman [:alice] [:anode]]

Comment on attachment 378658 [details] [diff] [review]
Buildbot configs for production pool

Cleared up over irc, required for testing Firefox3.0.

Comment 17

[Chris AtLee [:catlee]]

Comment on attachment 378457 [details] [diff] [review]
Buildbot configs for staging pool

changeset:   1159:7dbd08467cb1

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 378658 [details] [diff] [review]
Buildbot configs for production pool

changeset:   1160:ff0e933fd060

Comment 19

[Chris AtLee [:catlee]]

Both masters are up and running.  Any future problems should go into new bugs.

Comment 20

[Chris AtLee [:catlee]]

Attachment: Fix tinderbox reporting

Comment 21

[Chris AtLee [:catlee]]

Comment on attachment 379156 [details] [diff] [review]
Fix tinderbox reporting

changeset:   1168:828d023f06e4

Comment 22

[Chris AtLee [:catlee]]

Attachment: Enable symbol fetching on nochrome tests

Comment 23

[Chris AtLee [:catlee]]

Comment on attachment 380432 [details] [diff] [review]
Enable symbol fetching on nochrome tests

changeset:   1204:31a2c07eba9f