refactor/restyle b&r master.cfg files

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
11 years ago
5 years ago

People

(Reporter: rhelmer, Assigned: rhelmer)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: waiting for review)

Attachments

(8 attachments, 5 obsolete attachments)

2.73 KB, patch
rhelmer
: review+
Details | Diff | Splinter Review
42.24 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
48.38 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
670 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
711 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
1.24 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
569 bytes, patch
bhearsum
: review+
Details | Diff | Splinter Review
4.29 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
(Assignee)

Description

11 years ago
We've got quite a few different styles of buildbot configuration files among the various staging and production masters; let's decide on something standard and move them all over to it without any functional changes.

for example:
depBuildFactory.addStep(ShellCommand, description='echo bootstrap.cfg',
                        command=['cat', 'bootstrap.cfg'])

versus:
depBuildFactory.addStep(ShellCommand, 
 description='echo bootstrap.cfg',
 command=['cat', 'bootstrap.cfg'],
)

The latter diffs much better, and is just easier on the eyes imho.

There is also a ton of redundancy in these files; maybe we should go ahead with a custom Factory or at least a few custom Steps to make this better (e.g. most of the config file is doing the same checkout/test/install bootstrap dance in the 1.8/1.9 configs).
(Assignee)

Comment 1

11 years ago
Created attachment 308080 [details] [diff] [review]
example style for 1.8 staging
Assignee: nobody → rhelmer
Status: NEW → ASSIGNED
Attachment #308080 - Flags: review?(bhearsum)
(Assignee)

Comment 2

11 years ago
This took a while, so I only did one until we get it figured out :)
I think that it's way easier to read and insert changes without making unsignificant noise in diffs, ymmv :) 

Totally open on number of spaces, etc. as long as we're consistent about it.
I just used one space.

I am quite struck, once again, by how redundant this config is. Maybe we should also move the staging bootstrap stuff into a custom factory before rolling these changes out? I think it'd be easier and better overall.
(Assignee)

Comment 3

11 years ago
BTW, when I saw "custom factory", I am referring to a custom process-specific factory as described in http://buildbot.sourceforge.net/manual-0.7.5.html#Process_002dSpecific-build-factories
(Assignee)

Comment 4

11 years ago
Created attachment 308085 [details] [diff] [review]
simple BootstrapFactory example

Here's an example of a very simple BootstrapFactory, that just does the setup for you. Even this by itself is a huge savings in terms of LoC, but since we can do logic inside it, we could make it handle clobber vs. depend, or nightly vs. release, etc.

Sticking custom classes into the master.cfg is kind of ugly, but seems like a reasonable place for starters (until we start sharing buildbot-custom, or something like that).
Attachment #308080 - Attachment is obsolete: true
Attachment #308085 - Flags: review?(bhearsum)
Attachment #308080 - Flags: review?(bhearsum)
(Assignee)

Comment 5

11 years ago
Created attachment 308087 [details] [diff] [review]
simple BootstrapFactory example, take 2

Sorry, included builders from my dev env by accident!
Attachment #308087 - Flags: review?(bhearsum)
(Assignee)

Updated

11 years ago
Attachment #308085 - Attachment is obsolete: true
Attachment #308085 - Flags: review?(bhearsum)
(Assignee)

Comment 6

11 years ago
Created attachment 308088 [details] [diff] [review]
simple BootstrapFactory example, take 3

Add staging builders back to slavelist (should really move the slave info out to an include file..)
Attachment #308087 - Attachment is obsolete: true
Attachment #308088 - Flags: review?(bhearsum)
Attachment #308087 - Flags: review?(bhearsum)
Can this cleanup/restyle work also include changing how we use "projectName" in master.cfg? 

Right now, for example, in the 5 build master.cfgs, we have:
c['projectName'] = "Release Automation Production"
c['projectName'] = "Release Automation Test"
c['projectName'] = "Release Automation - Trunk/1.9"
c['projectName'] = "Release Automation Test"
c['projectName'] = "Mozilla2"

...but when you put all these in a row in the bookmarks toolbar, its impossible to see which is which without hovering the mouse over them and waiting. Instead, how about using something shorter that identifies the machine on the toolbar. One idea might be the name of the machine:
c['projectName'] = "production-1.8-master"
c['projectName'] = "staging-1.8-master"
c['projectName'] = "production-1.9-master"
c['projectName'] = "staging-1.9-master"
c['projectName'] = "production-moz2-master"

Eventually, as we consolidate the various masters we could have just two build masters with the following:
c['projectName'] = "production-master"
c['projectName'] = "staging-master"

Does that seem reasonable?
(Assignee)

Comment 8

11 years ago
(In reply to comment #7)
> Can this cleanup/restyle work also include changing how we use "projectName" in
> master.cfg? 
> 
> Right now, for example, in the 5 build master.cfgs, we have:
> c['projectName'] = "Release Automation Production"
> c['projectName'] = "Release Automation Test"
> c['projectName'] = "Release Automation - Trunk/1.9"
> c['projectName'] = "Release Automation Test"
> c['projectName'] = "Mozilla2"
> 
> ...but when you put all these in a row in the bookmarks toolbar, its impossible
> to see which is which without hovering the mouse over them and waiting.
> Instead, how about using something shorter that identifies the machine on the
> toolbar. One idea might be the name of the machine:
> c['projectName'] = "production-1.8-master"
> c['projectName'] = "staging-1.8-master"
> c['projectName'] = "production-1.9-master"
> c['projectName'] = "staging-1.9-master"
> c['projectName'] = "production-moz2-master"
> 
> Eventually, as we consolidate the various masters we could have just two build
> masters with the following:
> c['projectName'] = "production-master"
> c['projectName'] = "staging-master"
> 
> Does that seem reasonable?
> 

Since it's per-master, we should probably have the branch name in there (we'll have more than one master on both of these, at least initially).
Created attachment 308312 [details] [diff] [review]
edit master.cfg files to tweak projectName

This changes the format of the project names to make them more readable when on BookmarkToolbar. It also make all 4 release automation master.cfg files consistent format with each other.
Attachment #308312 - Flags: review?(rhelmer)
(In reply to comment #8)
> (In reply to comment #7)
> > Can this cleanup/restyle work also include changing how we use "projectName" in
> > master.cfg? 
> > 
> > Right now, for example, in the 5 build master.cfgs, we have:
> > c['projectName'] = "Release Automation Production"
> > c['projectName'] = "Release Automation Test"
> > c['projectName'] = "Release Automation - Trunk/1.9"
> > c['projectName'] = "Release Automation Test"
> > c['projectName'] = "Mozilla2"
> > 
> > ...but when you put all these in a row in the bookmarks toolbar, its impossible
> > to see which is which without hovering the mouse over them and waiting.
> > Instead, how about using something shorter that identifies the machine on the
> > toolbar. One idea might be the name of the machine:
> > c['projectName'] = "production-1.8-master"
> > c['projectName'] = "staging-1.8-master"
> > c['projectName'] = "production-1.9-master"
> > c['projectName'] = "staging-1.9-master"
> > c['projectName'] = "production-moz2-master"
> > 
> > Eventually, as we consolidate the various masters we could have just two build
> > masters with the following:
> > c['projectName'] = "production-master"
> > c['projectName'] = "staging-master"
> > 
> > Does that seem reasonable?
> > 
> 
> Since it's per-master, we should probably have the branch name in there (we'll
> have more than one master on both of these, at least initially).
Summary from offline IRC with rhelmer:

1) I knew we were consolidating buildbot instances to run, still as separate instances, but on just two VMs. I didn't realize that consolidating the buildbot instances would require extra buildbot enhancements not yet present. This invalidates my "staging-master" and "production-master" idea. Would have been nice, but oh well, maybe with a newer version of buildbot.

2) My other suggestion above, about using hostname format for the projectName, was confusing. What I cared about what being able to easily tell branch (1.8/1.9) and server type (production/staging) in the BookmarkToolbar. I've posted a patch in comment#9 which would WFM in the BookmarkToolbar, and which is *not* using the hostname, to avoid any confusion. 

Sorry my question caused so much noise in this bug. This seemed like the right place to talk about a one-liner change to the 4 master.cfg files. I didnt intend to distract from the real purpose of this bug.
(Assignee)

Comment 11

11 years ago
Comment on attachment 308312 [details] [diff] [review]
edit master.cfg files to tweak projectName

Landed, but haven't pulled/reconfig masters:

Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.21; previous revision: 1.20
done
Checking in production-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.16; previous revision: 1.15
done
Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.28; previous revision: 1.27
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.28; previous revision: 1.27
done
Attachment #308312 - Flags: review?(rhelmer) → review+
Comment on attachment 308088 [details] [diff] [review]
simple BootstrapFactory example, take 3

>Index: master.cfg
>===================================================================
>+class BootstrapFactory(factory.BuildFactory):
>+    def __init__(self):
>+        factory.BuildFactory.__init__(self)
>+        self.addStep(ShellCommand, 
>+         description='clean checkout',
>+         workdir='.', 
>+         command=['rm', '-rfv', 'build'],
>+         haltOnFailure=1)
>+        self.addStep(ShellCommand, 
>+         description='checkout', 
>+         workdir='.',
>+         command=['cvs', '-d', cvsroot, 'co', '-r', automation_tag,
>+                  '-d', 'build', cvsmodule],
>+         haltOnFailure=1,
>+        )
>+        self.addStep(FileDownload, 
>+         mastersrc="bootstrap.cfg", 
>+         slavedest="bootstrap.cfg", 
>+         workdir="build",
>+        )
>+        self.addStep(ShellCommand, 
>+         description='echo bootstrap.cfg',
>+         command=['cat', 'bootstrap.cfg'],
>+        )
>+        self.addStep(ShellCommand, 
>+         description='clean logs area',
>+         command=['bash', '-c', 'rm -rf /builds/logs/*'], 
>+         haltOnFailure=1,
>+        )
>+        self.addStep(ShellCommand, 
>+         description='unit tests',
>+         command=['make', 'test'], 
>+         haltOnFailure=1,
>+        )
>+

I think this factory would be more useful if it took in some things as variables: cvsroot, automation_tag, cvsmodule, bootstrap.cfg mastersrc, logdir seem like the important ones. This would let us use this factory for the dep builder (and probably the shark one too). Either way, I don't think these variables are defined this high up so you'll need to move them up, or make them take parameters. This looks good though; I didn't realize it would be this simple!


>+tagFactory = BootstrapFactory()
>+tagFactory.addStep(ShellCommand,
>+ description='Tag',
>+ command=['perl', './release', '-o', 'Tag'], 
>+ timeout=36000, 
>+ haltOnFailure=1,
>+)

Nice.



Even in its current state, this factory is a prime candidate for buildbotcustom. Time to create a buildbotcustom/process/factory.py I think :).
Comment on attachment 308088 [details] [diff] [review]
simple BootstrapFactory example, take 3

r-'ing specifically because the vars are too far down and the factory won't work.

I think you should do the following (but will leave it up to you):
* Move to buildbotcustom
* Add variables to the BootstrapFactory
Attachment #308088 - Flags: review?(bhearsum) → review-
(Assignee)

Comment 14

11 years ago
(In reply to comment #13)
> (From update of attachment 308088 [details] [diff] [review])
> r-'ing specifically because the vars are too far down and the factory won't
> work.
> 
> I think you should do the following (but will leave it up to you):
> * Move to buildbotcustom
> * Add variables to the BootstrapFactory

Yeah we should totally pass variables into BootstrapFactory, and define it in an include file. I'll do that and reattach.

I did actually test this and it seemed to work as-is, fwiw.
(Assignee)

Comment 15

11 years ago
Created attachment 308735 [details] [diff] [review]
move to buildbotcustom/process/factory.py, use variables
Attachment #308088 - Attachment is obsolete: true
Attachment #308735 - Flags: review?(bhearsum)
One question about style.

If we're standardized on 4-space indentation for code is there a reason we aren't using the same indentation standard for marking up the large data-structures and function calls? The last diff shows them using 1-space indentation.

PEP8 http://www.python.org/dev/peps/pep-0008/ , doesn't mention this particular case so I don't think there is a "standard Python way" to do this.
Comment on attachment 308735 [details] [diff] [review]
move to buildbotcustom/process/factory.py, use variables

>Index: buildbot-configs/automation/staging/master.cfg
>===================================================================
> ##cvsroot = ":ext:cltbld@cvs.mozilla.org:/cvsroot"
>-cvsroot = ":ext:stgbld@cvs.mozilla.org:/cvsroot"
>+##cvsroot = ":ext:stgbld@cvs.mozilla.org:/cvsroot"
>+cvsroot = ":pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot"
> cvsmodule = "mozilla/tools/release"
> automation_tag = "HEAD"

Why cvs-mirror?


>+ command=['/bin/bash', '-c', '""/bin/touch /home/ftp/pub/firefox/nightly/2.0.0.12-candidates/rc1/unsigned/win32_signing_rc1.log""'],

I think staging is using 2.0.0.13 now, can you confirm that?

>Index: buildbotcustom/process/factory.py
>===================================================================
>+class BootstrapFactory(factory.BuildFactory):
>+    def __init__(self, cvsroot=cvsroot, cvsmodule=cvsmodule, 
>+                 automation_tag=automation_tag):

These defaults don't work inside of buildbotcustom. cvsroot could default to something useful (cvs-mirror, I guess), but the others should probably be required, imho.

>+        factory.BuildFactory.__init__(self)
>+        self.addStep(ShellCommand, 
>+         description='clean checkout',
>+         workdir='.', 
>+         command=['rm', '-rfv', 'build'],
>+         haltOnFailure=1)
>+        self.addStep(ShellCommand, 
>+         description='checkout', 
>+         workdir='.',
>+         command=['cvs', '-d', cvsroot, 'co', '-r', automation_tag,
>+                  '-d', 'build', cvsmodule],
>+         haltOnFailure=1,
>+        )
>+        self.addStep(FileDownload, 
>+         mastersrc="bootstrap.cfg", 
>+         slavedest="bootstrap.cfg", 
>+         workdir="build",
>+        )
>+        self.addStep(ShellCommand, 
>+         description='echo bootstrap.cfg',
>+         command=['cat', 'bootstrap.cfg'],
>+        )
>+        self.addStep(ShellCommand, 
>+         description='clean logs area',
>+         command=['bash', '-c', 'rm -rf /builds/logs/*'], 
>+         haltOnFailure=1,
>+        )
>+        self.addStep(ShellCommand, 
>+         description='unit tests',
>+         command=['make', 'test'], 
>+         haltOnFailure=1,
>+        )

You need to import ShellCommand and FileDownload, also. These should be compatible with 0.7.6, so please use buildbot.steps.*

I still think that the logdir and bootstrap.cfg mastersrc should be variables, but I won't r- for that :). Adding both of those would make the BootstrapFactory useful to the dep builders, too.


r- because of the errors in factory.py. Would like to see logdir and mastersrc as parameters, but it's up to you.
Attachment #308735 - Flags: review?(bhearsum) → review-
(Assignee)

Comment 18

11 years ago
(In reply to comment #17)
> (From update of attachment 308735 [details] [diff] [review])
> >Index: buildbot-configs/automation/staging/master.cfg
> Why cvs-mirror?


Sorry, from me testing :(


> >+ command=['/bin/bash', '-c', '""/bin/touch /home/ftp/pub/firefox/nightly/2.0.0.12-candidates/rc1/unsigned/win32_signing_rc1.log""'],
> 
> I think staging is using 2.0.0.13 now, can you confirm that?


Yeah, maybe we should move this to a var at the top. 


> >Index: buildbotcustom/process/factory.py
> >===================================================================
> >+class BootstrapFactory(factory.BuildFactory):
> >+    def __init__(self, cvsroot=cvsroot, cvsmodule=cvsmodule, 
> >+                 automation_tag=automation_tag):
> 
> These defaults don't work inside of buildbotcustom. cvsroot could default to
> something useful (cvs-mirror, I guess), but the others should probably be
> required, imho.


Ok


> >+        factory.BuildFactory.__init__(self)
> >+        self.addStep(ShellCommand, 
> >+         description='clean checkout',
> >+         workdir='.', 
> >+         command=['rm', '-rfv', 'build'],
> >+         haltOnFailure=1)
> >+        self.addStep(ShellCommand, 
> >+         description='checkout', 
> >+         workdir='.',
> >+         command=['cvs', '-d', cvsroot, 'co', '-r', automation_tag,
> >+                  '-d', 'build', cvsmodule],
> >+         haltOnFailure=1,
> >+        )
> >+        self.addStep(FileDownload, 
> >+         mastersrc="bootstrap.cfg", 
> >+         slavedest="bootstrap.cfg", 
> >+         workdir="build",
> >+        )
> >+        self.addStep(ShellCommand, 
> >+         description='echo bootstrap.cfg',
> >+         command=['cat', 'bootstrap.cfg'],
> >+        )
> >+        self.addStep(ShellCommand, 
> >+         description='clean logs area',
> >+         command=['bash', '-c', 'rm -rf /builds/logs/*'], 
> >+         haltOnFailure=1,
> >+        )
> >+        self.addStep(ShellCommand, 
> >+         description='unit tests',
> >+         command=['make', 'test'], 
> >+         haltOnFailure=1,
> >+        )
> 
> You need to import ShellCommand and FileDownload, also. These should be
> compatible with 0.7.6, so please use buildbot.steps.*
> 
> I still think that the logdir and bootstrap.cfg mastersrc should be variables,
> but I won't r- for that :). Adding both of those would make the
> BootstrapFactory useful to the dep builders, too.


Yeah I was thinking the same thing, might as well do that now. You're right, this only works because we happen to be importing FileDownload and ShellCommand in master.cfg, but that's bad behavior for anyone wanting to import this class.


> r- because of the errors in factory.py. Would like to see logdir and mastersrc
> as parameters, but it's up to you.


(Assignee)

Comment 19

11 years ago
(In reply to comment #16)
> One question about style.
> 
> If we're standardized on 4-space indentation for code is there a reason we
> aren't using the same indentation standard for marking up the large
> data-structures and function calls? The last diff shows them using 1-space
> indentation.
> 
> PEP8 http://www.python.org/dev/peps/pep-0008/ , doesn't mention this particular
> case so I don't think there is a "standard Python way" to do this.


The master.cfg is a config file, but it's really Python :) Since it's a "config file" it tends to have lots of actual data encoded in it which makes the lines really long.

We could go with 4 spaces though, doesn't really matter to me. Ben what do you think?

(In reply to comment #19)
> (In reply to comment #16)
> > One question about style.
> > 
> > If we're standardized on 4-space indentation for code is there a reason we
> > aren't using the same indentation standard for marking up the large
> > data-structures and function calls? The last diff shows them using 1-space
> > indentation.
> > 
> > PEP8 http://www.python.org/dev/peps/pep-0008/ , doesn't mention this particular
> > case so I don't think there is a "standard Python way" to do this.
> 
> 
> The master.cfg is a config file, but it's really Python :) Since it's a "config
> file" it tends to have lots of actual data encoded in it which makes the lines
> really long.
> 
> We could go with 4 spaces though, doesn't really matter to me. Ben what do you
> think?
> 

If I had to choose one I'd go with 1 space to help avoid > 80char lines. I'm pretty indifferent though.
(Assignee)

Comment 21

11 years ago
Created attachment 308962 [details] [diff] [review]
[checked in] move to buildbotcustom/process/factory.py take 5
Attachment #308735 - Attachment is obsolete: true
(Assignee)

Updated

11 years ago
Attachment #308962 - Flags: review?(bhearsum)
Comment on attachment 308962 [details] [diff] [review]
[checked in] move to buildbotcustom/process/factory.py take 5

>Index: buildbotcustom/process/factory.py
>===================================================================
>+    @type  cvsmoule: string

Nit: fix the typo

r=bhearsum with that change.
Attachment #308962 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 23

11 years ago
Comment on attachment 308962 [details] [diff] [review]
[checked in] move to buildbotcustom/process/factory.py take 5

Landed, with typo in docstring fixed:

RCS file: /cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v
done
Checking in buildbotcustom/process/factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
initial revision: 1.1
done
Checking in buildbot-configs/automation/staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.29; previous revision: 1.28
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/process/__init__.py,v
done
Checking in buildbotcustom/process/__init__.py;
/cvsroot/mozilla/tools/buildbotcustom/process/__init__.py,v  <--  __init__.py
initial revision: 1.1
done
Attachment #308962 - Attachment description: move to buildbotcustom/process/factory.py take 5 → [checked in] move to buildbotcustom/process/factory.py take 5
(Assignee)

Comment 24

11 years ago
Great, thanks Ben! I'll make sure this works ok on staging before rolling out these changes to the other 3 configs :P

Note that we need to keep buildbotcustom in the master dir from now on.. this is a good thing though IMHO, it opens the door to sharing more code with the rest of the buildbots (talos, unittest, moz2, etc).
(Assignee)

Comment 25

11 years ago
Created attachment 309187 [details] [diff] [review]
[checked in] restyle 1.9 and move to buildbotcustom/process/factory.py

Same treatment for 1.9. Note the FIXME comment, I think we could add the ability to prepend Steps to the custom factory really easily, but it's only one place and I don't want to block on getting these configs out there.
Attachment #309187 - Flags: review?(bhearsum)
(Assignee)

Updated

11 years ago
Whiteboard: waiting for review
(Assignee)

Comment 26

11 years ago
Created attachment 309271 [details] [diff] [review]
[checked in] re-add clean target to prestage

Sorry, this was accidentally removed as part of attachment 308962 [details] [diff] [review].
Attachment #309271 - Flags: review?(bhearsum)
Comment on attachment 309187 [details] [diff] [review]
[checked in] restyle 1.9 and move to buildbotcustom/process/factory.py

>Index: staging-1.9/master.cfg
>===================================================================

> prestageFactory = factory.BuildFactory()

Should be BootstrapFactory(), I think.

>+# FIXME - redownload bootstrap.cfg, because we can't prepend the above
>+prestageFactory.addStep(FileDownload,
>+ mastersrc=bootstrap_config,
>+ slavedest="bootstrap.cfg", 
>+ workdir="build",
>+)

s/bootstrap_config/bootstrap.cfg/ ?


>+signFactory.addStep(ShellCommand,
>+ description='Fake signing log',
>+ command=['/bin/bash', '-c', 
>+          '""/bin/touch /home/ftp/pub/firefox/nightly/3.0b3-candidates/rc1/unsigned/win32_signing_rc1.log""'],

I think this should be 3.0b5

>+signFactory.addStep(ShellCommand,
>+ description='Fake signing',
>+ command=['/bin/bash', '-c',
>+          '""/bin/ln -fs /home/ftp/pub/firefox/nightly/3.0b3-candidates/rc1/unsigned/* /home/ftp/pub/firefox/nightly/3.0b3-candidates/rc1/""'],

And this, too.


r=bhearsum with those changes.
Attachment #309187 - Flags: review?(bhearsum) → review+
Comment on attachment 309271 [details] [diff] [review]
[checked in] re-add clean target to prestage

Sorry I missed that in r? =\.
Attachment #309271 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 29

11 years ago
Comment on attachment 309187 [details] [diff] [review]
[checked in] restyle 1.9 and move to buildbotcustom/process/factory.py

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.31; previous revision: 1.30
done
Attachment #309187 - Attachment description: restyle 1.9 and move to buildbotcustom/process/factory.py → [checked in] restyle 1.9 and move to buildbotcustom/process/factory.py
(Assignee)

Updated

11 years ago
Whiteboard: waiting for review
(Assignee)

Updated

11 years ago
Attachment #309271 - Attachment description: re-add clean target to prestage → [checked in] re-add clean target to prestage
(Assignee)

Comment 30

11 years ago
Created attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

1.9 staging master.cfg seems to work correctly with this patch.
Attachment #310176 - Flags: review?(bhearsum)
Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

Don't you need to remove some of the addStep()'s below this since you're using BootstrapFactory now?
Attachment #310176 - Flags: review?(bhearsum)
(Assignee)

Comment 32

11 years ago
Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

nope... see FIXME comments (unless I'm missing something here).
Attachment #310176 - Flags: review+
(Assignee)

Updated

11 years ago
Attachment #310176 - Flags: review+ → review?(bhearsum)
(Assignee)

Comment 33

11 years ago
Ben, I screwed up the r?, see comment #32
Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

I was wondering about that ;)
Attachment #310176 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 35

11 years ago
Comment on attachment 310176 [details] [diff] [review]
[checked in] call BootstrapFactory correctly for 1.9

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.32; previous revision: 1.31
done
Attachment #310176 - Attachment description: call BootstrapFactory correctly for 1.9 → [checked in] call BootstrapFactory correctly for 1.9
(Assignee)

Comment 36

11 years ago
Ok I think all that's left is to get a full run on staging with these configs, then do the same to the prod configs.
Whiteboard: testing in staging
(Assignee)

Comment 37

11 years ago
This seems totally fine in 1.8 staging, waiting until we get good 1.9 results.

We've got the 1.9 release really soon, so I'm thinking I'll do up the patches and post them, but not land until after the release has started.
(Assignee)

Comment 38

11 years ago
Created attachment 311097 [details] [diff] [review]
[checked in] create logdir if it does not already exist

Always a pain when setting up a new slave.
Attachment #311097 - Flags: review?(bhearsum)

Updated

11 years ago
Attachment #311097 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 39

11 years ago
Comment on attachment 311097 [details] [diff] [review]
[checked in] create logdir if it does not already exist

Checking in buildbotcustom/process/factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.2; previous revision: 1.1
done
Attachment #311097 - Attachment description: create logdir if it does not already exist → [checked in] create logdir if it does not already exist
(Assignee)

Comment 40

11 years ago
Created attachment 311468 [details] [diff] [review]
[checked in] use bash to invoke mkdir 

Followup for attachment 311097 [details] [diff] [review] on win32. Turns out that just running "mkdir" finds the win32 version, because Buildbot is started by cmd.exe with the default PATH :/ 

So, call bash and have it invoke mkdir (tested on staging), for the cross-platform win.
Attachment #311468 - Flags: review?(bhearsum)
(Assignee)

Updated

11 years ago
Whiteboard: testing in staging → waiting for review

Updated

11 years ago
Attachment #311468 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 41

11 years ago
Comment on attachment 311468 [details] [diff] [review]
[checked in] use bash to invoke mkdir 

Checking in process/factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.4; previous revision: 1.3
done
Attachment #311468 - Attachment description: use bash to invoke mkdir → [checked in] use bash to invoke mkdir
(Assignee)

Updated

11 years ago
Whiteboard: waiting for review → testing on staging
(Assignee)

Comment 42

11 years ago
This all seems to work fine, let's merge to production after the current release (3.0b5) is over.
Whiteboard: testing on staging → waiting for 3.0b5 release
(Assignee)

Updated

11 years ago
Depends on: 415970
(Assignee)

Comment 43

11 years ago
Once bug 415970 lands (requires a bunch of master.cfg changes), let's move all these improvements over to the production configs and close out this bug.
(Assignee)

Comment 44

11 years ago
Created attachment 313523 [details] [diff] [review]
[checked in] switch 1.9 over to using client-side bootstrap.cfg
Attachment #313523 - Flags: review?(bhearsum)
(Assignee)

Updated

11 years ago
Whiteboard: waiting for 3.0b5 release → merge to production
(Assignee)

Comment 45

11 years ago
I'll work on patches to merge the staging changes so far to production. 20014 is happening soon, but I'll get the patch ready anyway. We should be able to land 1.9 right away, though.
(Assignee)

Comment 46

11 years ago
Actually, let's merge in a separate bug; there are a bunch of steps involved, and it's not all about this bug.
Whiteboard: merge to production → waiting for review
(Assignee)

Updated

11 years ago
Blocks: 427166
Comment on attachment 313523 [details] [diff] [review]
[checked in] switch 1.9 over to using client-side bootstrap.cfg 

>Index: master.cfg
>===================================================================
>@@ -403,7 +390,7 @@
>  cvsmodule=cvsmodule, 
>  automation_tag=automation_tag,
>  logdir='/builds/logs', 
>- bootstrap_config='nightly-bootstrap.cfg',
>+ bootstrap_config='configs/fx-moz19-staging-bootstrap.cfg',

This is being switched from nightly -> regular config - I suppose it doesn't matter what it uses though (why are we using the BootstrapFactory for cvsmirrorFactory, anyways?)


r+, but cvsmirrorFactory needs to get un-factoried at some point.
Attachment #313523 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 48

11 years ago
Comment on attachment 313523 [details] [diff] [review]
[checked in] switch 1.9 over to using client-side bootstrap.cfg 

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.35; previous revision: 1.34
done
Attachment #313523 - Attachment description: switch 1.9 over to using client-side bootstrap.cfg → [checked in] switch 1.9 over to using client-side bootstrap.cfg
(Assignee)

Updated

11 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.