417779 - refactor/restyle b&r master.cfg files

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Robert Helmer [:rhelmer]]

We've got quite a few different styles of buildbot configuration files among the various staging and production masters; let's decide on something standard and move them all over to it without any functional changes.

for example:
depBuildFactory.addStep(ShellCommand, description='echo bootstrap.cfg',
                        command=['cat', 'bootstrap.cfg'])

versus:
depBuildFactory.addStep(ShellCommand, 
 description='echo bootstrap.cfg',
 command=['cat', 'bootstrap.cfg'],
)

The latter diffs much better, and is just easier on the eyes imho.

There is also a ton of redundancy in these files; maybe we should go ahead with a custom Factory or at least a few custom Steps to make this better (e.g. most of the config file is doing the same checkout/test/install bootstrap dance in the 1.8/1.9 configs).

Comment 1

[Robert Helmer [:rhelmer]]

Attachment: example style for 1.8 staging

Comment 2

[Robert Helmer [:rhelmer]]

This took a while, so I only did one until we get it figured out :)
I think that it's way easier to read and insert changes without making unsignificant noise in diffs, ymmv :) 

Totally open on number of spaces, etc. as long as we're consistent about it.
I just used one space.

I am quite struck, once again, by how redundant this config is. Maybe we should also move the staging bootstrap stuff into a custom factory before rolling these changes out? I think it'd be easier and better overall.

Comment 3

[Robert Helmer [:rhelmer]]

BTW, when I saw "custom factory", I am referring to a custom process-specific factory as described in http://buildbot.sourceforge.net/manual-0.7.5.html#Process_002dSpecific-build-factories

Comment 4

[Robert Helmer [:rhelmer]]

Attachment: simple BootstrapFactory example

Here's an example of a very simple BootstrapFactory, that just does the setup for you. Even this by itself is a huge savings in terms of LoC, but since we can do logic inside it, we could make it handle clobber vs. depend, or nightly vs. release, etc.

Sticking custom classes into the master.cfg is kind of ugly, but seems like a reasonable place for starters (until we start sharing buildbot-custom, or something like that).

Comment 5

[Robert Helmer [:rhelmer]]

Attachment: simple BootstrapFactory example, take 2

Sorry, included builders from my dev env by accident!

Comment 6

[Robert Helmer [:rhelmer]]

Attachment: simple BootstrapFactory example, take 3

Add staging builders back to slavelist (should really move the slave info out to an include file..)

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Can this cleanup/restyle work also include changing how we use "projectName" in master.cfg? 

Right now, for example, in the 5 build master.cfgs, we have:
c['projectName'] = "Release Automation Production"
c['projectName'] = "Release Automation Test"
c['projectName'] = "Release Automation - Trunk/1.9"
c['projectName'] = "Release Automation Test"
c['projectName'] = "Mozilla2"

...but when you put all these in a row in the bookmarks toolbar, its impossible to see which is which without hovering the mouse over them and waiting. Instead, how about using something shorter that identifies the machine on the toolbar. One idea might be the name of the machine:
c['projectName'] = "production-1.8-master"
c['projectName'] = "staging-1.8-master"
c['projectName'] = "production-1.9-master"
c['projectName'] = "staging-1.9-master"
c['projectName'] = "production-moz2-master"

Eventually, as we consolidate the various masters we could have just two build masters with the following:
c['projectName'] = "production-master"
c['projectName'] = "staging-master"

Does that seem reasonable?

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to comment #7)
> Can this cleanup/restyle work also include changing how we use "projectName" in
> master.cfg? 
> 
> Right now, for example, in the 5 build master.cfgs, we have:
> c['projectName'] = "Release Automation Production"
> c['projectName'] = "Release Automation Test"
> c['projectName'] = "Release Automation - Trunk/1.9"
> c['projectName'] = "Release Automation Test"
> c['projectName'] = "Mozilla2"
> 
> ...but when you put all these in a row in the bookmarks toolbar, its impossible
> to see which is which without hovering the mouse over them and waiting.
> Instead, how about using something shorter that identifies the machine on the
> toolbar. One idea might be the name of the machine:
> c['projectName'] = "production-1.8-master"
> c['projectName'] = "staging-1.8-master"
> c['projectName'] = "production-1.9-master"
> c['projectName'] = "staging-1.9-master"
> c['projectName'] = "production-moz2-master"
> 
> Eventually, as we consolidate the various masters we could have just two build
> masters with the following:
> c['projectName'] = "production-master"
> c['projectName'] = "staging-master"
> 
> Does that seem reasonable?
> 

Since it's per-master, we should probably have the branch name in there (we'll have more than one master on both of these, at least initially).

Comment 9

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: edit master.cfg files to tweak projectName

This changes the format of the project names to make them more readable when on BookmarkToolbar. It also make all 4 release automation master.cfg files consistent format with each other.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #8)
> (In reply to comment #7)
> > Can this cleanup/restyle work also include changing how we use "projectName" in
> > master.cfg? 
> > 
> > Right now, for example, in the 5 build master.cfgs, we have:
> > c['projectName'] = "Release Automation Production"
> > c['projectName'] = "Release Automation Test"
> > c['projectName'] = "Release Automation - Trunk/1.9"
> > c['projectName'] = "Release Automation Test"
> > c['projectName'] = "Mozilla2"
> > 
> > ...but when you put all these in a row in the bookmarks toolbar, its impossible
> > to see which is which without hovering the mouse over them and waiting.
> > Instead, how about using something shorter that identifies the machine on the
> > toolbar. One idea might be the name of the machine:
> > c['projectName'] = "production-1.8-master"
> > c['projectName'] = "staging-1.8-master"
> > c['projectName'] = "production-1.9-master"
> > c['projectName'] = "staging-1.9-master"
> > c['projectName'] = "production-moz2-master"
> > 
> > Eventually, as we consolidate the various masters we could have just two build
> > masters with the following:
> > c['projectName'] = "production-master"
> > c['projectName'] = "staging-master"
> > 
> > Does that seem reasonable?
> > 
> 
> Since it's per-master, we should probably have the branch name in there (we'll
> have more than one master on both of these, at least initially).
Summary from offline IRC with rhelmer:

1) I knew we were consolidating buildbot instances to run, still as separate instances, but on just two VMs. I didn't realize that consolidating the buildbot instances would require extra buildbot enhancements not yet present. This invalidates my "staging-master" and "production-master" idea. Would have been nice, but oh well, maybe with a newer version of buildbot.

2) My other suggestion above, about using hostname format for the projectName, was confusing. What I cared about what being able to easily tell branch (1.8/1.9) and server type (production/staging) in the BookmarkToolbar. I've posted a patch in comment#9 which would WFM in the BookmarkToolbar, and which is *not* using the hostname, to avoid any confusion. 

Sorry my question caused so much noise in this bug. This seemed like the right place to talk about a one-liner change to the 4 master.cfg files. I didnt intend to distract from the real purpose of this bug.

Comment 11

[Robert Helmer [:rhelmer]]

Comment on attachment 308312 [details] [diff] [review]
edit master.cfg files to tweak projectName

Landed, but haven't pulled/reconfig masters:

Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.21; previous revision: 1.20
done
Checking in production-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.16; previous revision: 1.15
done
Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.28; previous revision: 1.27
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.28; previous revision: 1.27
done

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 308088 [details] [diff] [review]
simple BootstrapFactory example, take 3

>Index: master.cfg
>===================================================================
>+class BootstrapFactory(factory.BuildFactory):
>+    def __init__(self):
>+        factory.BuildFactory.__init__(self)
>+        self.addStep(ShellCommand, 
>+         description='clean checkout',
>+         workdir='.', 
>+         command=['rm', '-rfv', 'build'],
>+         haltOnFailure=1)
>+        self.addStep(ShellCommand, 
>+         description='checkout', 
>+         workdir='.',
>+         command=['cvs', '-d', cvsroot, 'co', '-r', automation_tag,
>+                  '-d', 'build', cvsmodule],
>+         haltOnFailure=1,
>+        )
>+        self.addStep(FileDownload, 
>+         mastersrc="bootstrap.cfg", 
>+         slavedest="bootstrap.cfg", 
>+         workdir="build",
>+        )
>+        self.addStep(ShellCommand, 
>+         description='echo bootstrap.cfg',
>+         command=['cat', 'bootstrap.cfg'],
>+        )
>+        self.addStep(ShellCommand, 
>+         description='clean logs area',
>+         command=['bash', '-c', 'rm -rf /builds/logs/*'], 
>+         haltOnFailure=1,
>+        )
>+        self.addStep(ShellCommand, 
>+         description='unit tests',
>+         command=['make', 'test'], 
>+         haltOnFailure=1,
>+        )
>+

I think this factory would be more useful if it took in some things as variables: cvsroot, automation_tag, cvsmodule, bootstrap.cfg mastersrc, logdir seem like the important ones. This would let us use this factory for the dep builder (and probably the shark one too). Either way, I don't think these variables are defined this high up so you'll need to move them up, or make them take parameters. This looks good though; I didn't realize it would be this simple!


>+tagFactory = BootstrapFactory()
>+tagFactory.addStep(ShellCommand,
>+ description='Tag',
>+ command=['perl', './release', '-o', 'Tag'], 
>+ timeout=36000, 
>+ haltOnFailure=1,
>+)

Nice.



Even in its current state, this factory is a prime candidate for buildbotcustom. Time to create a buildbotcustom/process/factory.py I think :).

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 308088 [details] [diff] [review]
simple BootstrapFactory example, take 3

r-'ing specifically because the vars are too far down and the factory won't work.

I think you should do the following (but will leave it up to you):
* Move to buildbotcustom
* Add variables to the BootstrapFactory

Comment 14

[Robert Helmer [:rhelmer]]

(In reply to comment #13)
> (From update of attachment 308088 [details] [diff] [review])
> r-'ing specifically because the vars are too far down and the factory won't
> work.
> 
> I think you should do the following (but will leave it up to you):
> * Move to buildbotcustom
> * Add variables to the BootstrapFactory

Yeah we should totally pass variables into BootstrapFactory, and define it in an include file. I'll do that and reattach.

I did actually test this and it seemed to work as-is, fwiw.

Comment 15

[Robert Helmer [:rhelmer]]

Attachment: move to buildbotcustom/process/factory.py, use variables

Comment 16

[Mikeal Rogers [:mikeal] (mrogers@)]

One question about style.

If we're standardized on 4-space indentation for code is there a reason we aren't using the same indentation standard for marking up the large data-structures and function calls? The last diff shows them using 1-space indentation.

PEP8 http://www.python.org/dev/peps/pep-0008/ , doesn't mention this particular case so I don't think there is a "standard Python way" to do this.

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 308735 [details] [diff] [review]
move to buildbotcustom/process/factory.py, use variables

>Index: buildbot-configs/automation/staging/master.cfg
>===================================================================
> ##cvsroot = ":ext:[e-mail]:/cvsroot"
>-cvsroot = ":ext:[e-mail]:/cvsroot"
>+##cvsroot = ":ext:[e-mail]:/cvsroot"
>+cvsroot = ":pserver:[e-mail]:/cvsroot"
> cvsmodule = "mozilla/tools/release"
> automation_tag = "HEAD"

Why cvs-mirror?


>+ command=['/bin/bash', '-c', '""/bin/touch /home/ftp/pub/firefox/nightly/2.0.0.12-candidates/rc1/unsigned/win32_signing_rc1.log""'],

I think staging is using 2.0.0.13 now, can you confirm that?

>Index: buildbotcustom/process/factory.py
>===================================================================
>+class BootstrapFactory(factory.BuildFactory):
>+    def __init__(self, cvsroot=cvsroot, cvsmodule=cvsmodule, 
>+                 automation_tag=automation_tag):

These defaults don't work inside of buildbotcustom. cvsroot could default to something useful (cvs-mirror, I guess), but the others should probably be required, imho.

>+        factory.BuildFactory.__init__(self)
>+        self.addStep(ShellCommand, 
>+         description='clean checkout',
>+         workdir='.', 
>+         command=['rm', '-rfv', 'build'],
>+         haltOnFailure=1)
>+        self.addStep(ShellCommand, 
>+         description='checkout', 
>+         workdir='.',
>+         command=['cvs', '-d', cvsroot, 'co', '-r', automation_tag,
>+                  '-d', 'build', cvsmodule],
>+         haltOnFailure=1,
>+        )
>+        self.addStep(FileDownload, 
>+         mastersrc="bootstrap.cfg", 
>+         slavedest="bootstrap.cfg", 
>+         workdir="build",
>+        )
>+        self.addStep(ShellCommand, 
>+         description='echo bootstrap.cfg',
>+         command=['cat', 'bootstrap.cfg'],
>+        )
>+        self.addStep(ShellCommand, 
>+         description='clean logs area',
>+         command=['bash', '-c', 'rm -rf /builds/logs/*'], 
>+         haltOnFailure=1,
>+        )
>+        self.addStep(ShellCommand, 
>+         description='unit tests',
>+         command=['make', 'test'], 
>+         haltOnFailure=1,
>+        )

You need to import ShellCommand and FileDownload, also. These should be compatible with 0.7.6, so please use buildbot.steps.*

I still think that the logdir and bootstrap.cfg mastersrc should be variables, but I won't r- for that :). Adding both of those would make the BootstrapFactory useful to the dep builders, too.


r- because of the errors in factory.py. Would like to see logdir and mastersrc as parameters, but it's up to you.

Comment 18

[Robert Helmer [:rhelmer]]

(In reply to comment #17)
> (From update of attachment 308735 [details] [diff] [review])
> >Index: buildbot-configs/automation/staging/master.cfg
> Why cvs-mirror?


Sorry, from me testing :(


> >+ command=['/bin/bash', '-c', '""/bin/touch /home/ftp/pub/firefox/nightly/2.0.0.12-candidates/rc1/unsigned/win32_signing_rc1.log""'],
> 
> I think staging is using 2.0.0.13 now, can you confirm that?


Yeah, maybe we should move this to a var at the top. 


> >Index: buildbotcustom/process/factory.py
> >===================================================================
> >+class BootstrapFactory(factory.BuildFactory):
> >+    def __init__(self, cvsroot=cvsroot, cvsmodule=cvsmodule, 
> >+                 automation_tag=automation_tag):
> 
> These defaults don't work inside of buildbotcustom. cvsroot could default to
> something useful (cvs-mirror, I guess), but the others should probably be
> required, imho.


Ok


> >+        factory.BuildFactory.__init__(self)
> >+        self.addStep(ShellCommand, 
> >+         description='clean checkout',
> >+         workdir='.', 
> >+         command=['rm', '-rfv', 'build'],
> >+         haltOnFailure=1)
> >+        self.addStep(ShellCommand, 
> >+         description='checkout', 
> >+         workdir='.',
> >+         command=['cvs', '-d', cvsroot, 'co', '-r', automation_tag,
> >+                  '-d', 'build', cvsmodule],
> >+         haltOnFailure=1,
> >+        )
> >+        self.addStep(FileDownload, 
> >+         mastersrc="bootstrap.cfg", 
> >+         slavedest="bootstrap.cfg", 
> >+         workdir="build",
> >+        )
> >+        self.addStep(ShellCommand, 
> >+         description='echo bootstrap.cfg',
> >+         command=['cat', 'bootstrap.cfg'],
> >+        )
> >+        self.addStep(ShellCommand, 
> >+         description='clean logs area',
> >+         command=['bash', '-c', 'rm -rf /builds/logs/*'], 
> >+         haltOnFailure=1,
> >+        )
> >+        self.addStep(ShellCommand, 
> >+         description='unit tests',
> >+         command=['make', 'test'], 
> >+         haltOnFailure=1,
> >+        )
> 
> You need to import ShellCommand and FileDownload, also. These should be
> compatible with 0.7.6, so please use buildbot.steps.*
> 
> I still think that the logdir and bootstrap.cfg mastersrc should be variables,
> but I won't r- for that :). Adding both of those would make the
> BootstrapFactory useful to the dep builders, too.


Yeah I was thinking the same thing, might as well do that now. You're right, this only works because we happen to be importing FileDownload and ShellCommand in master.cfg, but that's bad behavior for anyone wanting to import this class.


> r- because of the errors in factory.py. Would like to see logdir and mastersrc
> as parameters, but it's up to you.

Comment 19

[Robert Helmer [:rhelmer]]

(In reply to comment #16)
> One question about style.
> 
> If we're standardized on 4-space indentation for code is there a reason we
> aren't using the same indentation standard for marking up the large
> data-structures and function calls? The last diff shows them using 1-space
> indentation.
> 
> PEP8 http://www.python.org/dev/peps/pep-0008/ , doesn't mention this particular
> case so I don't think there is a "standard Python way" to do this.


The master.cfg is a config file, but it's really Python :) Since it's a "config file" it tends to have lots of actual data encoded in it which makes the lines really long.

We could go with 4 spaces though, doesn't really matter to me. Ben what do you think?

Comment 20

[[e-mail] (:bhearsum)]

(In reply to comment #19)
> (In reply to comment #16)
> > One question about style.
> > 
> > If we're standardized on 4-space indentation for code is there a reason we
> > aren't using the same indentation standard for marking up the large
> > data-structures and function calls? The last diff shows them using 1-space
> > indentation.
> > 
> > PEP8 http://www.python.org/dev/peps/pep-0008/ , doesn't mention this particular
> > case so I don't think there is a "standard Python way" to do this.
> 
> 
> The master.cfg is a config file, but it's really Python :) Since it's a "config
> file" it tends to have lots of actual data encoded in it which makes the lines
> really long.
> 
> We could go with 4 spaces though, doesn't really matter to me. Ben what do you
> think?
> 

If I had to choose one I'd go with 1 space to help avoid > 80char lines. I'm pretty indifferent though.

Comment 21

[Robert Helmer [:rhelmer]]

Attachment: [checked in] move to buildbotcustom/process/factory.py take 5

Comment 22

[[e-mail] (:bhearsum)]

Comment on attachment 308962 [details] [diff] [review]
[checked in] move to buildbotcustom/process/factory.py take 5

>Index: buildbotcustom/process/factory.py
>===================================================================
>+    @type  cvsmoule: string

Nit: fix the typo

r=bhearsum with that change.

Comment 23

[Robert Helmer [:rhelmer]]

Comment on attachment 308962 [details] [diff] [review]
[checked in] move to buildbotcustom/process/factory.py take 5

Landed, with typo in docstring fixed:

RCS file: /cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v
done
Checking in buildbotcustom/process/factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
initial revision: 1.1
done
Checking in buildbot-configs/automation/staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.29; previous revision: 1.28
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/process/__init__.py,v
done
Checking in buildbotcustom/process/__init__.py;
/cvsroot/mozilla/tools/buildbotcustom/process/__init__.py,v  <--  __init__.py
initial revision: 1.1
done

Comment 24

[Robert Helmer [:rhelmer]]

Great, thanks Ben! I'll make sure this works ok on staging before rolling out these changes to the other 3 configs :P

Note that we need to keep buildbotcustom in the master dir from now on.. this is a good thing though IMHO, it opens the door to sharing more code with the rest of the buildbots (talos, unittest, moz2, etc).

Comment 25

[Robert Helmer [:rhelmer]]

Attachment: [checked in] restyle 1.9 and move to buildbotcustom/process/factory.py

Same treatment for 1.9. Note the FIXME comment, I think we could add the ability to prepend Steps to the custom factory really easily, but it's only one place and I don't want to block on getting these configs out there.

Comment 26

[Robert Helmer [:rhelmer]]

Attachment: [checked in] re-add clean target to prestage

Sorry, this was accidentally removed as part of attachment 308962 [details] [diff] [review].

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 309187 [details] [diff] [review]
[checked in] restyle 1.9 and move to buildbotcustom/process/factory.py

>Index: staging-1.9/master.cfg
>===================================================================

> prestageFactory = factory.BuildFactory()

Should be BootstrapFactory(), I think.

>+# FIXME - redownload bootstrap.cfg, because we can't prepend the above
>+prestageFactory.addStep(FileDownload,
>+ mastersrc=bootstrap_config,
>+ slavedest="bootstrap.cfg", 
>+ workdir="build",
>+)

s/bootstrap_config/bootstrap.cfg/ ?


>+signFactory.addStep(ShellCommand,
>+ description='Fake signing log',
>+ command=['/bin/bash', '-c', 
>+          '""/bin/touch /home/ftp/pub/firefox/nightly/3.0b3-candidates/rc1/unsigned/win32_signing_rc1.log""'],

I think this should be 3.0b5

>+signFactory.addStep(ShellCommand,
>+ description='Fake signing',
>+ command=['/bin/bash', '-c',
>+          '""/bin/ln -fs /home/ftp/pub/firefox/nightly/3.0b3-candidates/rc1/unsigned/* /home/ftp/pub/firefox/nightly/3.0b3-candidates/rc1/""'],

And this, too.


r=bhearsum with those changes.

Comment 28

[[e-mail] (:bhearsum)]

Comment on attachment 309271 [details] [diff] [review]
[checked in] re-add clean target to prestage

Sorry I missed that in r? =\.

Comment 29

[Robert Helmer [:rhelmer]]

Comment on attachment 309187 [details] [diff] [review]
[checked in] restyle 1.9 and move to buildbotcustom/process/factory.py

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.31; previous revision: 1.30
done

Comment 30

[Robert Helmer [:rhelmer]]

Attachment: [checked in] call BootstrapFactory correctly for 1.9

1.9 staging master.cfg seems to work correctly with this patch.

Comment 31

[[e-mail] (:bhearsum)]

Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

Don't you need to remove some of the addStep()'s below this since you're using BootstrapFactory now?

Comment 32

[Robert Helmer [:rhelmer]]

Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

nope... see FIXME comments (unless I'm missing something here).

Comment 33

[Robert Helmer [:rhelmer]]

Ben, I screwed up the r?, see comment #32

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

I was wondering about that ;)

Comment 35

[Robert Helmer [:rhelmer]]

Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.32; previous revision: 1.31
done

Comment 36

[Robert Helmer [:rhelmer]]

Ok I think all that's left is to get a full run on staging with these configs, then do the same to the prod configs.

Comment 37

[Robert Helmer [:rhelmer]]

This seems totally fine in 1.8 staging, waiting until we get good 1.9 results.

We've got the 1.9 release really soon, so I'm thinking I'll do up the patches and post them, but not land until after the release has started.

Comment 38

[Robert Helmer [:rhelmer]]

Attachment: [checked in] create logdir if it does not already exist

Always a pain when setting up a new slave.

Comment 39

[Robert Helmer [:rhelmer]]

Comment on attachment 311097 [details] [diff] [review]
[checked in] create logdir if it does not already exist

Checking in buildbotcustom/process/factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.2; previous revision: 1.1
done

Comment 40

[Robert Helmer [:rhelmer]]

Attachment: [checked in] use bash to invoke mkdir

Followup for attachment 311097 [details] [diff] [review] on win32. Turns out that just running "mkdir" finds the win32 version, because Buildbot is started by cmd.exe with the default PATH :/ 

So, call bash and have it invoke mkdir (tested on staging), for the cross-platform win.

Comment 41

[Robert Helmer [:rhelmer]]

Comment on attachment 311468 [details] [diff] [review]
[checked in] use bash to invoke mkdir 

Checking in process/factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.4; previous revision: 1.3
done

Comment 42

[Robert Helmer [:rhelmer]]

This all seems to work fine, let's merge to production after the current release (3.0b5) is over.

Comment 43

[Robert Helmer [:rhelmer]]

Once bug 415970 lands (requires a bunch of master.cfg changes), let's move all these improvements over to the production configs and close out this bug.

Comment 44

[Robert Helmer [:rhelmer]]

Attachment: [checked in] switch 1.9 over to using client-side bootstrap.cfg

Comment 45

[Robert Helmer [:rhelmer]]

I'll work on patches to merge the staging changes so far to production. 20014 is happening soon, but I'll get the patch ready anyway. We should be able to land 1.9 right away, though.

Comment 46

[Robert Helmer [:rhelmer]]

Actually, let's merge in a separate bug; there are a bunch of steps involved, and it's not all about this bug.

Comment 47

[[e-mail] (:bhearsum)]

Comment on attachment 313523 [details] [diff] [review]
[checked in] switch 1.9 over to using client-side bootstrap.cfg 

>Index: master.cfg
>===================================================================
>@@ -403,7 +390,7 @@
>  cvsmodule=cvsmodule, 
>  automation_tag=automation_tag,
>  logdir='/builds/logs', 
>- bootstrap_config='nightly-bootstrap.cfg',
>+ bootstrap_config='configs/fx-moz19-staging-bootstrap.cfg',

This is being switched from nightly -> regular config - I suppose it doesn't matter what it uses though (why are we using the BootstrapFactory for cvsmirrorFactory, anyways?)


r+, but cvsmirrorFactory needs to get un-factoried at some point.

Comment 48

[Robert Helmer [:rhelmer]]

Comment on attachment 313523 [details] [diff] [review]
[checked in] switch 1.9 over to using client-side bootstrap.cfg 

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.35; previous revision: 1.34
done