All users were logged out of Bugzilla on October 13th, 2018

buildbot config changes to support Mountain Lion

RESOLVED FIXED

Status

RESOLVED FIXED
6 years ago
5 months ago

People

(Reporter: kmoir, Assigned: kmoir)

Tracking

Details

Attachments

(7 attachments, 1 obsolete attachment)

(Assignee)

Description

6 years ago
Created attachment 646175 [details] [diff] [review]
buildbotconfig changes to support mountain lion
(Assignee)

Updated

6 years ago
Attachment #646175 - Flags: review?(armenzg)
(Assignee)

Updated

6 years ago
Assignee: nobody → kmoir

Comment 1

6 years ago
Comment on attachment 646175 [details] [diff] [review]
buildbotconfig changes to support mountain lion

I think the patch is good but I don't have time to review it before I head out.

You also need a patch for buildbotcustom.

If you do the following you'll see what I mean:
./setup-master.py bm bm04-tests1-macosx
cd bm && buildbot checkconfig .
Attachment #646175 - Flags: review?(armenzg)
(Assignee)

Comment 2

6 years ago
Created attachment 646355 [details] [diff] [review]
buildbotcustom changes to support Mountain Lion
(Assignee)

Updated

6 years ago
Attachment #646175 - Flags: review?(coop)
(Assignee)

Updated

6 years ago
Attachment #646355 - Flags: review?(coop)
Attachment #646175 - Attachment is patch: true
Attachment #646355 - Attachment is patch: true
Comment on attachment 646175 [details] [diff] [review]
buildbotconfig changes to support mountain lion

Review of attachment 646175 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with the nits fixed.

::: mozilla-tests/config.py
@@ +85,5 @@
>  PLATFORMS['macosx64']['env_name'] = 'mac-perf'
>  PLATFORMS['macosx64']['leopard'] = {'name': "Rev3 MacOSX Leopard 10.5.8"}
>  PLATFORMS['macosx64']['snowleopard'] = {'name': "Rev4 MacOSX Snow Leopard 10.6"}
>  PLATFORMS['macosx64']['lion'] = {'name': "Rev4 MacOSX Lion 10.7"}
> +PLATFORMS['macosx64']['mountainlion'] = {'name': "Rev5 MacOSX Lion 10.8"}

Rev5 MacOSX Mountain Lion 10.8

::: mozilla-tests/preproduction_config.py
@@ +8,4 @@
>      'leopard': dict([("talos-r3-leopard-%03i" % x, {}) for x in range(1,67)]),
>      'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(1,81) + range(82,85) + [86]]),
>      'lion': dict([("talos-r4-lion-%03i" % x, {}) for x in range(1,83) + [84]]),
> +    'mountainlion': dict([("talos-mtnlion-r5-%03i" % x, {}) for x in range(1,90)]),

range(1,90) will give you 1-89 because range() is dumb. Do you mean range(1,91)?

::: mozilla-tests/production_config.py
@@ +9,4 @@
>            if x not in [7]]), # bug 655437
>      'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(4,10) + range(11,81) + [82,84]]),
>      'lion': dict([("talos-r4-lion-%03i" % x, {}) for x in range(4,10) + range(11,83) + [84]]),
> +    'mountainlion': dict([("talos-mtnlion-r5-%03i" % x, {}) for x in range(1,90)]),

Same question about range()

::: mozilla-tests/staging_config.py
@@ +8,4 @@
>      'leopard': dict([("talos-r3-leopard-%03i" % x, {}) for x in range(1,67)]),
>      'snowleopard': dict([("talos-r4-snow-%03i" % x, {}) for x in range(1,81) + range(82,85) + [86]]),
>      'lion': dict([("talos-r4-lion-%03i" % x, {}) for x in range(1,83) + [84]]),
> +    'mountainlion': dict([("talos-mtnlion-r5-%03i" % x, {}) for x in range(1,90)]),

Same question about range()
Attachment #646175 - Flags: review?(coop) → review+
Attachment #646355 - Flags: review?(coop) → review+
(Assignee)

Comment 4

6 years ago
Created attachment 649765 [details] [diff] [review]
buildbot-config changes to support mountain lion
Attachment #646175 - Attachment is obsolete: true
(Assignee)

Updated

6 years ago
Attachment #646355 - Flags: checked-in+
(Assignee)

Updated

6 years ago
Attachment #649765 - Flags: checked-in+
Comment on attachment 649765 [details] [diff] [review]
buildbot-config changes to support mountain lion

Backed out:
http://hg.mozilla.org/build/buildbot-configs/rev/cb0cbaa5d5de
Attachment #649765 - Flags: checked-in+ → checked-in-
(Assignee)

Comment 6

6 years ago
Created attachment 651479 [details] [diff] [review]
patch for puppet-manifests

Rail tried to do a reconfig and it failed because these patches were missing.  I didn't know there were changes that were required to the puppet masters to support a new platform.
Attachment #651479 - Flags: review?(coop)
I left the first patch live in buildbotcustom.

The problem with the configs patch was bm21 failing checkconfig: KeyError: 'mountainlion'

To make masters understand the new platform you need:

1) update http://hg.mozilla.org/build/puppet-manifests/file/tip/modules/buildmaster/templates/BuildSlaves-tests.py.erb and add new platform
2) update http://hg.mozilla.org/build/puppet-manifests/file/tip/secrets.pp.template and add new variable
3) update secretes.pp on master-puppet1 with real passwords
(Assignee)

Comment 8

6 years ago
So once my puppet-manifest patch in comment #6 is approved I'd like to try another reconfig tomorrow :-)  

I'm also curious about how I didn't encounter this issue when running the tests on my dev master.  What could have I done during testing to avoid this problem cropping up during the reconfig?
(Assignee)

Updated

6 years ago
Attachment #651479 - Flags: review?(coop) → review?(rail)
Attachment #651479 - Flags: review?(rail) → review+
(Assignee)

Updated

6 years ago
Attachment #651479 - Flags: checked-in+
(Assignee)

Comment 9

6 years ago
Comment on attachment 649765 [details] [diff] [review]
buildbot-config changes to support mountain lion

Checked in again, will try another reconfig this afternoon.
Attachment #649765 - Flags: checked-in- → checked-in+
Didn't get a reply in #build & need to head home soon, so posting here instead:

21:49:05 - edmorley: nthomas, coop: please may you cancel the pending 10.8 builds that are on m-c + inbound (plus presumably elsewhere), since the mountain lion patch has been backed out so philor says they'll now be stuck pending
21:49:34 - edmorley: they are also showing up as OS X opt (formerly 10.5), tbpl needs a patch (but I can take a look at that tomorrow hopefully)
I deleted 1611 10.8 test jobs from the buildbot db, since these are new anyway. kmoir has been working on enabling the slaves, but we've hit a potential issue with 'sudo reboot' not working without auth for cltbld. That's required at the end of each job.

Updated

6 years ago
Attachment #649765 - Attachment is patch: true

Updated

6 years ago
Depends on: 782826
(In reply to Nick Thomas [:nthomas] from comment #12)
> I deleted 1611 10.8 test jobs from the buildbot db

Awesome, thank you :-)
(Assignee)

Comment 14

6 years ago
I fixed the sudo reboot issue, and rebooted the other slaves early this evening. Now I'm working on some other issues after a long conversation with catlee in IRC.
Please may we kill off some more 10.8 jobs:

Pending test(s) @ Aug 15 02:15:03
mac10.8 (477)
  341 mozilla-inbound
   28 ionmonkey
   27 fx-team
   27 build-system
   27 birch
   17 services-central
   10 profiling

Pending test(s) @ Aug 15 02:15:03
mac10.8 (342)
  342 try
Ok, 1181 deleted. 

I think we should probably stop scheduling these jobs if we're blocked on setting up new masters in scl3, especially with IT all in GMT at the moment. That gives a chance to put some of these slaves on a staging master and chuck some more jobs at them to shake out any other bugs. eg, I don't see any mention of graph server sql additions here, so I think talos will have a problem posting perf results. See https://wiki.mozilla.org/ReleaseEngineering:GraphServer for clues.

Perhaps we need a checklist for adding new machine classes ? AFAIK this stuff is all in the heads of old-timers.
(Assignee)

Comment 17

6 years ago
I just deleted some more jobs.  

I'm writing a checklist of all the things that need to be done to a a new platform that I have encountered so far.  I agree that we should probably back out these changes since obviously there are a lot of issues that need to be resolved that I wasn't aware of and we need new test masters implemented.
(Assignee)

Comment 18

6 years ago
Created attachment 652186 [details] [diff] [review]
patch to revert buildbot-configs (old one was was bitrotten)

Until I can run more tests in the staging environment and we have a new test master up (bug 782870).

If I understand correctly, only the buildbot-configs patch needs to be reverted to stop the builds from queueing?
Sorry to be a pain, but please may we have another round of 10.8 job deletion :-)
(Assignee)

Comment 20

6 years ago
Ed, I just deleted some more jobs :-)
(Assignee)

Updated

6 years ago
Attachment #652186 - Flags: checked-in+
(Assignee)

Updated

6 years ago
Blocks: 731278

Updated

6 years ago
No longer depends on: 782826
(Assignee)

Comment 22

6 years ago
Created attachment 655039 [details] [diff] [review]
patch to add mountain lion key for jetpack builds

this didn't show up until I ran tests last night
(Assignee)

Updated

6 years ago
Attachment #655039 - Flags: review?(rail)
Comment on attachment 655039 [details] [diff] [review]
patch to add mountain lion key for jetpack builds

Feel free to remove all trailing spaces from buildfarm/utils/run_jetpack.py ;)
Attachment #655039 - Flags: review?(rail) → review+
(Assignee)

Updated

6 years ago
Attachment #655039 - Flags: checked-in+

Comment 24

6 years ago
Created attachment 655803 [details] [diff] [review]
unbitrotted backout of http://hg.mozilla.org/build/buildbot-configs/rev/08fa3688ec70

with the new way of defining slave ranges in {production,preproduction,staging}_config.py.

Comment 25

6 years ago
Comment on attachment 655803 [details] [diff] [review]
unbitrotted backout of http://hg.mozilla.org/build/buildbot-configs/rev/08fa3688ec70

The only differences here are the _config.py slave lists and unbitrotting.
Attachment #655803 - Flags: review?(kmoir)
(Assignee)

Updated

6 years ago
Attachment #655803 - Flags: review?(kmoir) → review+
(Assignee)

Updated

6 years ago
Attachment #655803 - Flags: checked-in+
(Assignee)

Comment 26

6 years ago
closing
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
We only have builders on esr10, mozilla-release, mozilla-beta (dep and release automation). philor points out landing race with http://hg.mozilla.org/build/buildbot-configs/rev/613642b19123 is likely to be the issue.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 655938 [details] [diff] [review]
Add mountainlion for m-c + peers

I've verified this brings up to the same number of builders as we have for 10.7, across all branches.
Attachment #655938 - Flags: review?(rail)
Comment on attachment 655938 [details] [diff] [review]
Add mountainlion for m-c + peers

lgtm
Attachment #655938 - Flags: review?(rail) → review+

Updated

6 years ago
Depends on: 786424
Comment on attachment 655938 [details] [diff] [review]
Add mountainlion for m-c + peers

http://hg.mozilla.org/build/buildbot-configs/rev/20e2fcf03f23

This will add builders for aurora, despite the request in bug 786424. I'm figuring we need to green those tests up anyway.
Attachment #655938 - Flags: checked-in+
Comment on attachment 655938 [details] [diff] [review]
Add mountainlion for m-c + peers

This made it to production today.
(Assignee)

Comment 32

6 years ago
Closing.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.