move prestage logic from automation builders out of Tag factory.

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
11 years ago
5 years ago

People

(Reporter: bhearsum, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments, 2 obsolete attachments)

(Assignee)

Description

11 years ago
Rob suggested having a 'prestage' factory to handle these things.
(Assignee)

Comment 1

11 years ago
As part of this, I think we should update the bootstrap-configs/ directory from CVS. This will make it harder to do local patches, but we should probably be avoiding that anyways.
Assignee: nobody → bhearsum
(Assignee)

Updated

11 years ago
Priority: -- → P2
(In reply to comment #1)
> As part of this, I think we should update the bootstrap-configs/ directory from
> CVS. This will make it harder to do local patches, but we should probably be
> avoiding that anyways.

Hmm that's a good idea, I was trying to avoid Buildbot updating itself but Buildbot updating bootstrap sounds great. 

Should we do this for production too? There's an automation tag defined in the master.cfg, maybe we should just update the dir to that?

(Assignee)

Comment 3

11 years ago
(In reply to comment #2)
> (In reply to comment #1)
> Should we do this for production too? There's an automation tag defined in the
> master.cfg, maybe we should just update the dir to that?
> 

I think that makes sense. I'm going to put together a patch for this today (hopefully).
(Assignee)

Comment 4

11 years ago
Created attachment 296579 [details] [diff] [review]
[checked in] add a prestage factory and move some buildsteps out to it

I've run all of these master.cfg's through 'buildbot checkconfig' and they pass. I haven't done an actual test of any of them. I expected to do this with staging-trunk-automation soon after these land.

Rob, I'll be leaving the office in about an hour, if I'm gone when you r+ can you land this and update staging-t-a so this gets tested on the nightly run?
Attachment #296579 - Flags: review?(rhelmer)
Comment on attachment 296579 [details] [diff] [review]
[checked in] add a prestage factory and move some buildsteps out to it

yeah looks good, feel free to remove the relevant targets from mozilla/tools/release/Makefile too..
Attachment #296579 - Flags: review?(rhelmer) → review+
(Assignee)

Comment 6

11 years ago
Comment on attachment 296579 [details] [diff] [review]
[checked in] add a prestage factory and move some buildsteps out to it

Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.10; previous revision: 1.9
done
Checking in production-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.4; previous revision: 1.3
done
Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.13; previous revision: 1.12
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.9; previous revision: 1.8
done
Attachment #296579 - Attachment description: add a prestage factory and move some buildsteps out to it → [checked in] add a prestage factory and move some buildsteps out to it
Sorry, missed a spot - need to register the new scheduler e.g.:

c['schedulers'].append(prestage_scheduler)

Otherwise nothing happens when a change is inserted, as all of the other schedulers are looking for specific branch names.
Also, for staging-build-console at least, the make_clean target can't be called until the makefile is checked out:

prestageFactory.addStep(ShellCommand, description='clean checkout',
                        workdir='.', command=['rm', '-rfv', 'build'],
                        haltOnFailure=1)
prestageFactory.addStep(ShellCommand, description='checkout', workdir='.',
                        command=['cvs', '-d', cvsroot, 'co', '-r', automation_tag,
                        '-d', 'build', cvsmodule], haltOnFailure=1)

Maybe it'd be better to do explicit cleanup in here instead of using the makefile target though?
Created attachment 297115 [details] [diff] [review]
reg. scheduler, co bootstrap, fix typo

This works for staging-build-console.

Should we move away from using the makefile for these things? Since we can't easily support multiple branches, I'm thinking so... either that or make it so we can pass version strings to the Makefile.

Fixing this somehow would be nice, because it'd mean only master.cfg needs to be touched when re-syncing the mirror.
Attachment #297115 - Flags: review?(bhearsum)
(Assignee)

Comment 10

11 years ago
Comment on attachment 297115 [details] [diff] [review]
reg. scheduler, co bootstrap, fix typo

Need to add the tag_depscheduler, too. I've got a patch that's almost-ready that covers all of the automation masters.

r+ with the tag_depscheduler added, but I'll probably have a patch up in a few mins!
Attachment #297115 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 11

11 years ago
Created attachment 297171 [details] [diff] [review]
make all of the buildbot masters work
Attachment #297115 - Attachment is obsolete: true
Attachment #297171 - Flags: review?(rhelmer)
(Assignee)

Comment 12

11 years ago
Created attachment 297186 [details] [diff] [review]
[checked in] clobber 3.0b3-candidates instead of 3.0b2-candidates

This problem got me thinking about using a Makefile some more. It would really nice to be able to just pass the version and it would Do The Right Thing. Clobber the right directories, download the right builds, etc.

I filed bug 412454 about this.
Attachment #297171 - Attachment is obsolete: true
Attachment #297186 - Flags: review?(rhelmer)
Attachment #297171 - Flags: review?(rhelmer)
Comment on attachment 297171 [details] [diff] [review]
make all of the buildbot masters work

looks ok, i've got a patch actually that removes a lot of these redundant schedulers, since we're not using them anyway; please land this now though and I'll merge
Attachment #297186 - Flags: review?(rhelmer) → review+
(Assignee)

Comment 14

11 years ago
Comment on attachment 297186 [details] [diff] [review]
[checked in] clobber 3.0b3-candidates instead of 3.0b2-candidates

Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.11; previous revision: 1.10
done
Checking in production-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.5; previous revision: 1.4
done
Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.14; previous revision: 1.13
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.12; previous revision: 1.11
done
Attachment #297186 - Attachment description: clobber 3.0b3-candidates instead of 3.0b2-candidates → [checked in] clobber 3.0b3-candidates instead of 3.0b2-candidates
Created attachment 297213 [details] [diff] [review]
[checked in] use periodic scheduler for nightlies, remove redundant schedulers

When going through the schedulers, I realize we don't really use them as intended (the idea was to be able to trigger from any point in the process using a branch name, and have dependencies work as intented).

Not sure if it still works at this point tbh. Do you think we should keep these or just remove them? I've removed them in this patch, I can reflect this in the other configs if you think that's ok.
Attachment #297213 - Flags: review?(bhearsum)
(Assignee)

Comment 16

11 years ago
Comment on attachment 297213 [details] [diff] [review]
[checked in] use periodic scheduler for nightlies, remove redundant schedulers

Personally..I comment out BuildSteps or do a "Rebuild" to kick-off specific steps. I'm fine with removing them.
Attachment #297213 - Flags: review?(bhearsum) → review+
Comment on attachment 297213 [details] [diff] [review]
[checked in] use periodic scheduler for nightlies, remove redundant schedulers

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.15; previous revision: 1.14
done
Attachment #297213 - Attachment description: use periodic scheduler for nightlies, remove redundant schedulers → [checked in] use periodic scheduler for nightlies, remove redundant schedulers
Created attachment 297239 [details] [diff] [review]
[checked in] remove redundant schedulers
Attachment #297239 - Flags: review?(bhearsum)
(Assignee)

Comment 19

11 years ago
Comment on attachment 297239 [details] [diff] [review]
[checked in] remove redundant schedulers

looks good
Attachment #297239 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 20

11 years ago
Comment on attachment 297239 [details] [diff] [review]
[checked in] remove redundant schedulers

Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.12; previous revision: 1.11
done
Checking in production-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.6; previous revision: 1.5
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.13; previous revision: 1.12
done
Attachment #297239 - Attachment description: remove redundant schedulers → [checked in] remove redundant schedulers
(Assignee)

Updated

11 years ago
Status: NEW → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.