458414 - Add new slaves to pool

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

The current pool of slaves were not able to keep up with all the jobs submitted in the rush to FF3.1b1.

Adding extra slaves to the pool, to help speed up turnaround time for developers.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Doing post-setup steps on new win32 slaves.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: add extra slaves to the pool

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: add *all* extra slaves to the pool

redoing, previous patch didnt include the new linux slaves by accident.

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 341664 [details] [diff] [review]
add *all* extra slaves to the pool

The linux slaves should get added to mobile_master.py.
You missed tracemonkey altogether.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: [checked in] add extra slaves to the pool (v3)

1) adds linux and win32 slaves to tracemonkey builders

2) adds linux slaves to mobile builders

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 341667 [details] [diff] [review]
[checked in] add extra slaves to the pool (v3) 

As per the e-mail I sent yesterday please make sure you reconfig the master like this:
vim /tools/buildbotcustom/buildbotcustom/process/factory.py
# uncomment the SetProperty import
cd /builds/buildbot
PYTHONPATH=/tools/buildbot-079:$PYTHONPATH buildbot stop moz2-master
PYTHONPATH=/tools/bulidbot-079:$PYTHONPATH buildbot start moz2-master
vim /tools/buildbotcustom/buildbotcustom/process/factory.py
# recomment the SetProperty import

(Yeah, this is far below ideal - will be fixed next week.)

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> (From update of attachment 341667 [details] [diff] [review])
> As per the e-mail I sent yesterday please make sure you reconfig the master
> like this:
> vim /tools/buildbotcustom/buildbotcustom/process/factory.py
> # uncomment the SetProperty import
> cd /builds/buildbot
> PYTHONPATH=/tools/buildbot-079:$PYTHONPATH buildbot stop moz2-master
> PYTHONPATH=/tools/bulidbot-079:$PYTHONPATH buildbot start moz2-master
> vim /tools/buildbotcustom/buildbotcustom/process/factory.py
> # recomment the SetProperty import
> 
> (Yeah, this is far below ideal - will be fixed next week.)

Sorry, cut-and-paste-o, ignore that, do this:

vim /tools/buildbotcustom/buildbotcustom/process/factory.py
# uncomment the SetProperty import
cd /builds/buildbot
PYTHONPATH=/tools/buildbot-079:$PYTHONPATH buildbot reconfig moz2-master
vim /tools/buildbotcustom/buildbotcustom/process/factory.py
# recomment the SetProperty import

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

I've fixed some omissions in the win32 slave setup docs at:
https://wiki.mozilla.org/ReferencePlatforms/Win32#Setup_extra_disk


At this point, the new slave is recognized by staging-master, and can be given jobs to process, but the slave immediately fails out the job with:
exceptions.RuntimeError: Couldn't find executable for 'hg'

Manually logging into the slave, I get:
$ which hg
which: hg: unknown command

...even though I can see /d/mercurial/hg.exe. By contrast when I login to one of the pre-existing slaves, I get:

$ which hg
/d/mercurial/hg.exe


Still working on it.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 341667 [details] [diff] [review]
[checked in] add extra slaves to the pool (v3) 

changeset:   399:039d7bf5b671

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #8)
> I've fixed some omissions in the win32 slave setup docs at:
> https://wiki.mozilla.org/ReferencePlatforms/Win32#Setup_extra_disk
> 
> 
> At this point, the new slave is recognized by staging-master, and can be given
> jobs to process, but the slave immediately fails out the job with:
> exceptions.RuntimeError: Couldn't find executable for 'hg'
> 
> Manually logging into the slave, I get:
> $ which hg
> which: hg: unknown command
> 
> ...even though I can see /d/mercurial/hg.exe. By contrast when I login to one
> of the pre-existing slaves, I get:
> 
> $ which hg
> /d/mercurial/hg.exe
> 
> 
> Still working on it.
Turns out to be problem with our refimage missing a recently added env.variable. I've updated https://wiki.mozilla.org/ReferencePlatforms/Win32.


moz2-win32-slave12 now works just fine.

Comment 11

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

moz2-win32-slave12.b.m.o now visible in production pool, and immediately started building.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

moz2-win32-slave11.b.m.o 
moz2-win32-slave13.b.m.o 
...are both now running final tests in staging pool. 

As soon as these test builds complete successfully, I'll move both of these to the production pool.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

moz2-win32-slave11.b.m.o now enabled in production pool.

Comment 14

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

moz2-win32-slave13.b.m.o now enabled in production pool.

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

we need mac slaves also!

Comment 16

[Chris AtLee [:catlee]]

which of these is going into production, and which are staying in staging?

Comment 17

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: add more win32 slaves to the pool

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 348784 [details] [diff] [review]
add more win32 slaves to the pool

These need to be added to the mozilla-central win32-debug list and the tracemonkey list

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: add more win32 slaves to the pool (v2)

Good catch, thanks.

Comment 20

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

moz2-win32-slave16.b.m.o is now in production.
moz2-win32-slave15.b.m.o is now in production.

Comment 21

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

moz2-win32-slave17.b.m.o is now in production.

Comment 22

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

moz2-win32-slave18.b.m.o is now in production. 

Thats all the slaves that *I* knew of. Is there anything left to do here, or can we now close this bug?

Comment 23

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

This bug, and all the dep.bugs are now done. Closing!