617021 - create production branches for 0.7 and 0.8 buildbots in buildbot, buildbot-configs, and buildbotcustom repositories

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

From mail:
* We will create named branches for each major version of Buildbot. production_0_8 for 0.8.x masters, and production_0_7 for 0.7.x masters.
* We will update all masters on a given Buildbot branch when we land new Buildbot code in the repository, regardless of whether or not we believe that they only affect the Schedulers, or only affect the Builders.

Comment 1

[[e-mail] (:bhearsum)]

Also should update a few other things:
- Fabric scripts
- Puppet buildbotmaster manifests
- (maybe) bm01/02 Makefiles

Comment 2

[[e-mail] (:bhearsum)]

Note to self: we need to start pulling mozconfigs from the 'production' branch.

Comment 3

[[e-mail] (:bhearsum)]

I've created the branches in buildbot-configs and buildbotcustom (buildbot repo is blocked on bug 617041). Changes done to the masters:
production-master:
- Updated /tools/buildbotcustom/buildbotcustom to 'production-0.7' branch
- Updated /builds/buildbot/configs to 'production' branch
- No reconfig necessary

production-master01:
- Updated /tools/buildbotcustom/buildbotcustom to 'production-0.8' branch
- Updated /builds/buildbot/configs to 'production' branch
- Reconfiged /builds/buildbot/scheduler_master and /builds/buildbot/builder_master1

production-master02:
- Updated /tools/buildbotcustom/buildbotcustom to 'production-0.7' branch
- Updated /builds/buildbot/configs-default to 'production' branch
- Updated /tools/buildbotcustom-0.8.0/buildbotcustom to 'production-0.8' branch
- Updated /builds/buildbot/configs to 'production' branch
- Reconfiged /builds/buildbot/try-trunk-master and /builds/buildbot/tests-scheduler

production-master03:
- Updated /tools/buildbotcustom/buildbotcustom to 'production-0.8' branch
- Updated /builds/buildbot/configs to 'production' branch
- Reconfiged /builds/buildbot/builder_master

buildbot-master1:
- /builds/buildbot/build_master3
-- Updated buildbot-configs to 'production' branch
-- Updated buildbotcustom to 'production-0.8' branch
-- Reconfiged
- /builds/buildbot/tests_master3
-- Updated buildbot-configs to 'production' branch
-- Updated buildbotcustom to 'production-0.8' branch
-- Reconfiged
- /builds/buildbot/tests_master4
-- Updated buildbot-configs to 'production' branch
-- Updated buildbotcustom to 'production-0.8' branch
-- Reconfiged

buildbot-master2:
- /builds/buildbot/build_master4
-- Updated buildbot-configs to 'production' branch
-- Updated buildbotcustom to 'production-0.8' branch
-- Reconfiged
- /builds/buildbot/tests_master5
-- Updated buildbot-configs to 'production' branch
-- Updated buildbotcustom to 'production-0.8' branch
-- Reconfiged
- /builds/buildbot/tests_master6
-- Updated buildbot-configs to 'production' branch
-- Updated buildbotcustom to 'production-0.8' branch
-- Reconfiged

test-master01:
- Updated /tools/buildbotcustom/buildbotcustom to 'production-0.8' branch
- Updated /builds/buildbot/configs to 'production' branch
- Reconfiged /builds/buildbot/tests-master

test-master02:
- Updated /tools/buildbotcustom/buildbotcustom to 'production-0.8' branch
- Updated /builds/buildbot/configs to 'production' branch
- Reconfiged /builds/buildbot/tests-master

talos-master02:
- Updated /tools/buildbotcustom/buildbotcustom to 'production-0.8' branch
- Updated /builds/buildbot/configs to 'production' branch
- Reconfiged /builds/buildbot/tests-master

Comment 4

[[e-mail] (:bhearsum)]

Attachment: pull mozconfigs from production branch

Now that the MoCo buildbots are running the master-side code off the production branch of buildbot-configs, they should pull mozconfigs from there, too. This patch should do it, but I haven't tested it more than test-masters.sh so far.

It should also preserve the old behaviour for CC factories.

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 495941 [details] [diff] [review]
pull mozconfigs from production branch

scope creep alert!

good time to switch to hgtool for configs?

looks ok otherwise...except this is like the 375th parameter to MercurialBuildFactory

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> Comment on attachment 495941 [details] [diff] [review]
> [untested] pull mozconfigs from production branch
> 
> scope creep alert!
> 
> good time to switch to hgtool for configs?

I thought about doing it...but now that I'm 80% through testing, I'd rather not.
 
> looks ok otherwise...except this is like the 375th parameter to
> MercurialBuildFactory

Hurry now! Be the lucky one to add the 376th and win a prize!

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 495941 [details] [diff] [review]
pull mozconfigs from production branch

I tested this patch on the following builders, all of which successfully grabbed the mozconfig from the production branch:
Linux mozilla-central dep
Android R7 mozilla-central build
WINNT 5.2 mozilla-central build
OS X 10.6.2 mozilla-central build
Maemo 5 GTK places build
WINNT 5.2 Mobile Desktop places nightly
Maemo 5 QT jaegermonkey build
Linux Mobile Desktop jaegermonkey build

Comment 8

[Mark Banner (:standard8)]

Comment on attachment 495941 [details] [diff] [review]
pull mozconfigs from production branch

I find comment 0 hard to understand. Have you created the production branch so that buildbots will automatically reconfig off that branch? or something else?

In any case either gozer or jhopkins should take a look at this.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to comment #8)
> Comment on attachment 495941 [details] [diff] [review]
> pull mozconfigs from production branch
> 
> I find comment 0 hard to understand. Have you created the production branch so
> that buildbots will automatically reconfig off that branch? or something else?

The Buildbot masters track the production branch, they don't pull it automatically (yet) though.

Comment 10

[Philippe M. Chiasson (:gozer)]

Comment on attachment 495941 [details] [diff] [review]
pull mozconfigs from production branch

We are moving our buildbot master to 0.8 today, and that includes the switch to the production branches. So doing this as well is no problem at all.

Comment 11

[[e-mail] (:bhearsum)]

The way this patch is written, CC factories will still be pulling mozconfigs from default. I'm not sure if that's what you want, and whether or not pulling from 'production' will work for KaiRo. Note that this patch is only for default/production-0.8, so 0.7.x masters won't be affected. Sounds like we might be able to just default to 'production' for everyone though?

Comment 12

[Robert Kaiser]

Comment on attachment 495941 [details] [diff] [review]
pull mozconfigs from production branch

looks good, though I wonder if it might be good to have this handed in from config.py, actually.

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 495941 [details] [diff] [review]
pull mozconfigs from production branch

Pushed this on default: changeset:   1267:bbec6f692b0f
Likely to go to production today.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: [untested] update buildbot master puppet manifests

Comment 15

[Robert Kaiser]

I just found out that there seems to be some subtle bug in here - the repack factories never get self.mozconfigBranch assigned, from what I see, because they base on MozillaBuildFactory and not MercurialBuildFactory, which would set it.

I right now end up with this error:
  File "/tools/buildbotcustom/buildbotcustom/process/factory.py", line 3318, in __init__
    mozconfigBranch='default', **kwargs)
  File "/tools/buildbotcustom/buildbotcustom/process/factory.py", line 3184, in __init__
    mergeLocales=mergeLocales, **kwargs)
  File "/tools/buildbotcustom/buildbotcustom/process/factory.py", line 2482, in __init__
    self.getMozconfig()
  File "/tools/buildbotcustom/buildbotcustom/process/factory.py", line 2517, in getMozconfig
    command=['hg', 'update', '-r', self.mozconfigBranch],
AttributeError: CCReleaseRepackFactory instance has no attribute 'mozconfigBranch'


I tracked it down to the above reasoning, which I believe is what's going on. If that's the case, self.mozconfigBranch = mozconfigBranch in BaseRepackFactory::__init__ should do the trick - and it did in my case.

Comment 16

[Robert Kaiser]

Attachment: make self.mozconfigBranch defined in release repack factories

If my analysis is correct in the larger picture as well, this should fix the problem I've seen.

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 497935 [details] [diff] [review]
make self.mozconfigBranch defined in release repack factories

Landed on default and production-0.8.

Comment 18

[[e-mail] (:bhearsum)]

Clearing reconfig flag.

Comment 19

[[e-mail] (:bhearsum)]

Attachment: fix master checkouts in puppet master manifests

Comment 20

[Chris AtLee [:catlee]]

Comment on attachment 500525 [details] [diff] [review]
fix master checkouts in puppet master manifests

Do you only need to modify the two test master instances here?  What about all the other 0.8 masters?

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 500525 [details] [diff] [review]
fix master checkouts in puppet master manifests

d'oh, I don't know how I forgot the other instances.

Comment 22

[[e-mail] (:bhearsum)]

Dustin is planning to remove these entries in bug 615317, so I won't be posting an updated patch. We're all done here now.