Turn off buildbot macosx64 opt builds scheduling

RESOLVED FIXED

Status

P1
normal
RESOLVED FIXED
2 years ago
6 months ago

People

(Reporter: wcosta, Assigned: kmoir)

Tracking

Details

Attachments

(12 attachments, 13 obsolete attachments)

831 bytes, patch
kmoir
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
945 bytes, patch
kmoir
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
47 bytes, text/x-github-pull-request
emorley
: review+
Details | Review | Splinter Review
1.58 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
886 bytes, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
59 bytes, text/x-github-pull-request
Details | Review | Splinter Review
1.47 KB, patch
catlee
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
1.06 KB, patch
catlee
: review+
kmoir
: checked-in+
Details | Diff | Splinter Review
1.51 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
6.84 KB, patch
catlee
: review+
Details | Diff | Splinter Review
7.28 KB, patch
kmoir
: checked-in+
Details | Diff | Splinter Review
4.11 KB, patch
Details | Diff | Splinter Review
Comment hidden (empty)
(Assignee)

Comment 1

2 years ago
Could we try this on a project branch for testing before enabling it in production for all branches to ensure this doesn't have unintended consequences?
Flags: needinfo?(wcosta)
SETA is only valid on mozilla-inbound/autoland.
(Reporter)

Updated

2 years ago
User Story: (updated)
Flags: needinfo?(wcosta)
(Reporter)

Comment 3

2 years ago
err, forgot to answer. In tests we can restrict per branch, not sure about builds. Anyway, try pushes aren't safe enough?
(Assignee)

Comment 4

2 years ago
We can't disable something in buildbot on try without disabling it for everyone else on try. That is why I was suggesting a project branch.
(Reporter)

Comment 5

2 years ago
(In reply to Kim Moir [:kmoir] from comment #4)
> We can't disable something in buildbot on try without disabling it for
> everyone else on try. That is why I was suggesting a project branch.

Since this is an in-tree config, we can land the patch in bug 1349980 only in a specific branch, or I can hack on my patch to do it too.
(Assignee)

Updated

2 years ago
Depends on: 1354284
(Assignee)

Comment 6

2 years ago
Sebastian, you mentioned that you were interested in working on this.  You could write patches to try this out on the holly branch, so once it's reset we will be ready for testing.
Assignee: nobody → spacurar
I was planning to do the following patches:

1) one for enabling buildbot builds and tests on holly branch on macosx64 
2) one for enabling taskcluster builds and tests on holly branch on macosx64 
3) one for disabling the macosx64 buidlder and tests on buildbot

I intend to test if the jobs from taskcluster are affected after step 3.
Can you give me some suggestions on how I can make a setup to stop the scheduling of buildbot opt jobs regarding macosx64, while the jobs scheduled by task cluster via buildbot-bridge won't be affected? I think that if I turn off the scheduling from buildbot-configs will affect taskcluster as well since they are both managed by buildbot.
Flags: needinfo?(kmoir)
(Assignee)

Comment 8

2 years ago
re 3) With buildbot bridge, the builders jobs have to exist on buildbot but the actual scheduling is triggered by taskcluster.

For turning off the mac tests from being scheduled in buildbot, you can look at 

https://bugzilla.mozilla.org/show_bug.cgi?id=1324915

and
https://reviewboard.mozilla.org/r/101530/diff/1#index_header

except
enable_unittest_sendchange

For the builds, I have been thinking about this, and I don't know how to do this.  I enabled talos tests for linux64-stylo by having a dummy builder but in this case we want to have the jobs scheduled via taskcluster, not just the tests.  https://bugzilla.mozilla.org/show_bug.cgi?id=1338871

I would advise asking for help in #mozbuild or #tcmigration for help on this

i.e. looking to schedule macosx opt buildbot builds on a branch via bbb via taskcluster without actually having them triggered by buildbot.

let me know if you need additional help, happy to discuss in vidyo to clarify
Flags: needinfo?(kmoir)
(Assignee)

Updated

2 years ago
Blocks: 1355945
(Assignee)

Comment 9

2 years ago
Sebastian, please post your in progress patches here so I can look at them. We would like to get these running next week.

As an aside, you can filter out running builds on a certain platform on taskcluster like they are here but modified so only macosx64 builds run on holly

https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/target_tasks.py#l297

This bug is for the buildbot changes, but you mentioned in the meeting we had this morning that you wanted to know how to filter out build platforms from running on taskcluster.  Let me know if I can help in any way!
Flags: needinfo?(spacurar)
Created attachment 8858287 [details] [diff] [review]
bug_1353376_disable buildbot macosx builds on buildbot
Attachment #8858287 - Flags: review?(kmoir)
Created attachment 8858288 [details] [diff] [review]
bug_1353376_enable holy branch on buildbot
Attachment #8858288 - Flags: review?(kmoir)
Created attachment 8858299 [details] [diff] [review]
bug_1353376_disable buildbot macosx tests on buildbot
Attachment #8858299 - Flags: review?(kmoir)
Created attachment 8858312 [details] [diff] [review]
bug_1353376_disable scheduling of macosx tests on holly taskcluster
Attachment #8858312 - Flags: review?(kmoir)
Regarding patches bug_1353376_disable buildbot macosx builds on buildbot and bug_1353376_disable buildbot macosx tests on buildbot  , are the ones which totally disable the jobs and builds from buildbot, causing problems since the jobs do not exist on buildbot anymore this way. I can mark those as obsolete since we need to stop only the scheduling for tests on taskcluster.

As for taskcluster, I cloned the holly repo and disabled the scheduling in patch bug_1353376_disable scheduling of macosx tests on holly taskcluster
Flags: needinfo?(spacurar)
Created attachment 8858326 [details] [diff] [review]
bug_1353376_filter macosx builds to run only on holly branch
Attachment #8858326 - Flags: review?(kmoir)
(Assignee)

Updated

2 years ago
Attachment #8858287 - Attachment is patch: true
Attachment #8858287 - Attachment mime type: text/x-patch → text/plain
(Assignee)

Updated

2 years ago
Attachment #8858288 - Attachment is patch: true
Attachment #8858288 - Attachment mime type: text/x-patch → text/plain
(Assignee)

Comment 16

2 years ago
Comment on attachment 8858288 [details] [diff] [review]
bug_1353376_enable holy branch on buildbot

looks good but I'm not sure you need 'mozharness_tag': 'default',

Why are you enabling win64?  Was this for your testing environment?
Attachment #8858288 - Flags: review?(kmoir) → review+
(Assignee)

Updated

2 years ago
Attachment #8858299 - Attachment is patch: true
Attachment #8858299 - Attachment mime type: text/x-patch → text/plain
(Assignee)

Updated

2 years ago
Attachment #8858312 - Attachment is patch: true
Attachment #8858312 - Attachment mime type: text/x-patch → text/plain
(Assignee)

Updated

2 years ago
Attachment #8858326 - Attachment is patch: true
Attachment #8858326 - Attachment mime type: text/x-patch → text/plain
(Assignee)

Comment 17

2 years ago
Comment on attachment 8858326 [details] [diff] [review]
bug_1353376_filter macosx builds to run only on holly branch

I think this should be 

def target_tasks_holly(full_task_graph, parameters):

instead of 
+def target_tasks_holy(gull_task_graph, parameters):
Attachment #8858326 - Flags: review?(kmoir) → review-
(Assignee)

Comment 18

2 years ago
Comment on attachment 8858287 [details] [diff] [review]
bug_1353376_disable buildbot macosx builds on buildbot

So we want the builders to still exist on the buildbot master, but just not be scheduled by changes in buildbot. Rather they should triggered by taskcluster through buildbot bridge.  So I don't think you want to delete them.
Attachment #8858287 - Flags: review?(kmoir)
(Assignee)

Comment 19

2 years ago
Comment on attachment 8858299 [details] [diff] [review]
bug_1353376_disable buildbot macosx tests on buildbot

I don't think we need to worry about changing the tests scheduling now.  This bug is just about changing the buildbot builds to scheduled through taskcluster.
Attachment #8858299 - Flags: review?(kmoir)
(Assignee)

Comment 20

2 years ago
Comment on attachment 8858312 [details] [diff] [review]
bug_1353376_disable scheduling of macosx tests on holly taskcluster

I don't think we need to worry about changing the tests scheduling now.  This bug is just about changing the buildbot builds to scheduled through taskcluster.
Attachment #8858312 - Flags: review?(kmoir)
(Reporter)

Comment 21

2 years ago
Notice only opt builds should be disabled for now.
(Reporter)

Comment 22

2 years ago
(In reply to Kim Moir [:kmoir] from comment #20)
> Comment on attachment 8858312 [details] [diff] [review]
> bug_1353376_disable scheduling of macosx tests on holly taskcluster
> 
> I don't think we need to worry about changing the tests scheduling now. 
> This bug is just about changing the buildbot builds to scheduled through
> taskcluster.

Maybe we should make the switch in a separate bug. My patch at bug 1350413 is safe to land on m-c because it doesn't turn on BBB build on.
(Assignee)

Comment 23

2 years ago
Yes, this is correct, opt builds only.  Sebastian, to recap, the requirements are to
1) enable builders on holly for macosx opt that are triggered via taskcluster via the buildbot bridge.  
2) test that these builds are triggered through taskcluster (you could land wcosta's patch from bug 1350413 on holly only to test)
3) ensure that there are not duplicate builds triggered for macosx on holly (just the taskcluster one should be triggered)

Looking at bug 1157242, it looks like the patches there set some preferences disabling builders from being 
triggered by buildbot
in
mozilla/project_branches.py

https://bug1157242.bmoattachments.org/attachment.cgi?id=8600965

Not sure if they are still relevant, but it would be worth testing

We can work out details about tests later, right now the focus should be on making sure the builds are scheduled as expected.
(In reply to Kim Moir [:kmoir] from comment #16)
> Comment on attachment 8858288 [details] [diff] [review]
> bug_1353376_enable holy branch on buildbot
> 
> looks good but I'm not sure you need 'mozharness_tag': 'default',
> 
> Why are you enabling win64?  Was this for your testing environment?

I chose mozharness_tag since I made a copy after pine. I am not quite sure what mozharness_tag does. I noticed it can be set to 'default' and to 'production', so I was thinking default is the way I need to use it. And yes I enabled win64 for the testing environment. I would like to know what mozharness_tag actually does.



(In reply to Kim Moir [:kmoir] from comment #17)
> Comment on attachment 8858326 [details] [diff] [review]
> bug_1353376_filter macosx builds to run only on holly branch
> 
> I think this should be 
> 
> def target_tasks_holly(full_task_graph, parameters):
> 
> instead of 
> +def target_tasks_holy(gull_task_graph, parameters):

Yes that was a typo mistake, will fix that!



(In reply to Wander Lairson Costa [:wcosta] from comment #21)
> Notice only opt builds should be disabled for now.

Yes I know this should be only for opt builds, but I keep noticing it happens on debug as well in treeherder. Will carry on only with the opt builds then.
Flags: needinfo?(kmoir)
Created attachment 8859140 [details] [diff] [review]
bug_1353376_v2_filter-macosx-builds-to-run-only-on-holly-branch.patch

Fixed the typo.
Attachment #8858326 - Attachment is obsolete: true
Attachment #8859140 - Flags: review?(kmoir)
(Assignee)

Comment 26

2 years ago
Comment on attachment 8859140 [details] [diff] [review]
bug_1353376_v2_filter-macosx-builds-to-run-only-on-holly-branch.patch

I think it should be target_tasks_holly instead of target_tasks_holy but other than that it looks good!
Flags: needinfo?(kmoir)
Attachment #8859140 - Flags: review?(kmoir) → review+
Created attachment 8859230 [details] [diff] [review]
bug_1353376_v3_filter-macosx-builds-to-run-only-on-holly-branch.patch

I noticed this function in taskcluster/taskgraph/target_tasks.py:

 def _target_task(name):
    def wrap(func):
        _target_task_methods[name] = func
        return func
    return wrap


And examples of other functions declarations like the one I made about holly were starting like this:

@_target_task('stylo_tasks')
def target_tasks_stylo(full_task_graph, parameters):
...lines omitted...

@_target_task('pine_tasks')
def target_tasks_pine(full_task_graph, parameters):
...lines omitted...

That is why I left it like the ones above. In this patch I modified:
@_target_task('holly_tasks')  into
@_target_task_holly('holly_tasks')
Attachment #8859140 - Attachment is obsolete: true
Attachment #8859230 - Flags: review?(kmoir)
(Assignee)

Comment 28

2 years ago
Comment on attachment 8859230 [details] [diff] [review]
bug_1353376_v3_filter-macosx-builds-to-run-only-on-holly-branch.patch

I think you need to have 

@_target_task('holly_tasks')
def target_tasks_holly(full_task_graph, parameters):

In any case, you also need a patch to disable the builds from being scheduled on buildbot, but still have the builders remain, this comment has some hints.  Let me know if you need help, happy to go over the process needed to test this

https://bugzilla.mozilla.org/show_bug.cgi?id=1353376#c23
Created attachment 8859955 [details] [diff] [review]
bug_1353376_v4_filter-macosx-builds-to-run-only-on-holly-branch.patch

I noticed in an end what you meant in comment https://bugzilla.mozilla.org/show_bug.cgi?id=1353376#c26

It was a typo regarding "holly" word.I kept thinking I wrote down the declaration of the function wrong. 
Fixed it now. Sorry for the confusion.
Attachment #8859230 - Attachment is obsolete: true
Attachment #8859230 - Flags: review?(kmoir)
Attachment #8859955 - Flags: review?(kmoir)
Created attachment 8859989 [details] [diff] [review]
bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch
Attachment #8858288 - Attachment is obsolete: true
Attachment #8859989 - Flags: review?(kmoir)
Comment on attachment 8859989 [details] [diff] [review]
bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch

I enabled holly on macosx64-debug, win64 and win64-debug just so I can test macosx64.
Created attachment 8859995 [details] [diff] [review]
bug_1353376_macosx64-opt-builds-are-triggered-through-taskcluster-on-holly-branch.patch
Attachment #8859995 - Flags: review?(kmoir)
I would need some info on how I can test if there are no duplicate builds triggered for macosx on holly. I still have no idea how to test this on my local machine.

Can you please give me some advice on how can I get this done?
Flags: needinfo?(kmoir)
(Assignee)

Updated

2 years ago
Flags: needinfo?(kmoir)
Attachment #8859955 - Flags: review?(kmoir) → review+
(Assignee)

Comment 34

2 years ago
Comment on attachment 8859989 [details] [diff] [review]
bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch

We might not need all these enable_* parameters but we can iterate on these patches in testing.
Attachment #8859989 - Flags: review?(kmoir) → review+
(Assignee)

Comment 35

2 years ago
Comment on attachment 8859995 [details] [diff] [review]
bug_1353376_macosx64-opt-builds-are-triggered-through-taskcluster-on-holly-branch.patch

I think we can use the patch in bug 1350413 to test on holly instead of this one.  Re the testing question you had earlier, we can go through the steps for that on that during our 1x1 tomorrow, does sound okay?
Attachment #8859995 - Flags: review?(kmoir)
Yes sure. Ben Hearsum helped me out with checking if the schedulers still exist in buildbot, using the dump_master.py script from braindump. bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch seems to be working: the builder is enabled on holly branch but the Schedulers regarding it do not exist ("OS X 10.7 holly build" does not appear in the "Schedulers" section).

Regarding bug_1353376_macosx64-opt-builds-are-triggered-through-taskcluster-on-holly-branch.patch I added only the first commit parts from the mozreview of Wander Costa's patch. I will update it until our 1x1 meeting.
Attachment #8858287 - Attachment is obsolete: true
Comment on attachment 8858299 [details] [diff] [review]
bug_1353376_disable buildbot macosx tests on buildbot

>diff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
>--- a/mozilla-tests/config.py
>+++ b/mozilla-tests/config.py
>@@ -3055,6 +3055,16 @@ for name, branch in items_at_least(BRANC
>                     if suite_name.startswith('mochitest-a11y'):
>                         win32['win7_ix'][test_type].remove(t)
> 
>+# Bug 1353376 - Turn off buildbot macosx64 opt builds scheduling
>+for name, branch in items_at_least(BRANCHES, 'gecko_version', 55):
>+    if name not in ['holly']:
>+        continue
>+    for platform in branch['platforms'].keys():
>+        if platform not in ['macosx64']:
>+            continue
>+        for test in ['opt_unittest_suites', 'debug_unittest_suites']:
>+            if 'yosemite_r7' in branch['platforms'][platform]:
>+                branch['platforms'][platform]['yosemite_r7'][test] = []
> 
> # bug 1343316 - add buildernames for linux64-stylo talos tests for mozilla-central and try branches
> for branch in BRANCHES.keys():
>
Attachment #8858299 - Attachment is obsolete: true
Attachment #8858312 - Attachment is obsolete: true
Attachment #8859995 - Attachment is obsolete: true
(Assignee)

Updated

2 years ago
Attachment #8859989 - Flags: checked-in+
(Assignee)

Comment 38

2 years ago
Created attachment 8860500 [details] [review]
enable holly branch for testing of macosx64 opt builds scheduled via tc
Attachment #8860500 - Flags: review?(emorley)
(Assignee)

Comment 39

2 years ago
Comment on attachment 8859955 [details] [diff] [review]
bug_1353376_v4_filter-macosx-builds-to-run-only-on-holly-branch.patch

landed on holly
Attachment #8859955 - Flags: checked-in+
(Assignee)

Comment 40

2 years ago
I also landed wcosta's patches here


Sebastian, you should be ready to test that the macosx opt builds are scheduled via taskcluster.  The next step after testing is completed is to write patches to enable this change on trunk branches.
(Assignee)

Comment 41

2 years ago
previous comment should have said, I landed wcosta's patches from bug 1350413 on holly

side note: I was thinking about the patch to disable the triggering of builds on holly.

https://bug1353376.bmoattachments.org/attachment.cgi?id=8859989

This patch sets enables enable_onchange_scheduler to False.  This is okay for initial testing on holly.

However, to write the patches to enable this on trunk branches, we want macosx opt builders to exist, but not be triggered by buildbot. So enabling a preference like enable_onchange_scheduler on an entire branch won't work because it will impact builds for all other platforms, not just macosx opt.

So we will have to do something else to disable the schedulers for macosx opt builds on trunk via buildbot on trunk branches, while keeping them as triggered on non-trunk branches.

Updated

2 years ago
Attachment #8860500 - Flags: review?(emorley) → review+
(Assignee)

Comment 43

2 years ago
Thanks, I can see the holly on the staging version of treeherder

https://treeherder.allizom.org/#/jobs?repo=holly
(Assignee)

Comment 44

2 years ago
Created attachment 8861166 [details] [diff] [review]
bug1353376graph.patch

more patches, I landed on holly
(Assignee)

Comment 45

2 years ago
Created attachment 8861177 [details] [diff] [review]
bug1353376tools.patch

had to enable branch in prodction-branches.json
(Assignee)

Comment 46

2 years ago
Created attachment 8861179 [details] [review]
pull request to add holly branch to mozilla-taskcluster
(Assignee)

Updated

2 years ago
Attachment #8861177 - Attachment is patch: true
Attachment #8861177 - Attachment mime type: text/x-patch → text/plain
Attachment #8861177 - Flags: checked-in+
(Assignee)

Comment 47

2 years ago
So the earlier problem was that builds were not being scheduled, not the issue is a different error message

https://treeherder.allizom.org/logviewer.html#?job_id=87722257&repo=holly&lineNumber=691 

seems to be an issues with the patches I landed from bug 1350413, needs investigation
(Assignee)

Updated

2 years ago
Attachment #8861166 - Flags: checked-in+
(Assignee)

Comment 48

2 years ago
I was able to fix the earlier error and now have a new one

https://treeherder.allizom.org/#/jobs?repo=holly&revision=b4caed40b9b9186821e18ce9d66ace1568c1a8f3

 HTTPError: 400 Client Error: Bad Request for url: http://taskcluster/queue/v1/task/cpwUS5EFTv6vuI_oyl7Yaw
for some reason that task has an empty worker type

  "cpwUS5EFTv6vuI_oyl7Yaw": {
    "attributes": {
      "kind": "misc"
    },
    "dependencies": {},
    "kind": "misc",
    "label": "s3-uploader-build-macosx64/opt",
    "optimizations": [],
    "task": {
      "created": {
        "relative-datestamp": "0 seconds"
      },
      "deadline": {
        "relative-datestamp": "1 day"
      },
      "metadata": {
        "description": "Upload outputs of buildbot/mozharness builds to S3",
        "name": "Buildbot/mozharness S3 uploader",
        "owner": "mshal@mozilla.com",
        "source": "http://hg.mozilla.org/build/mozharness/"
      },
      "payload": {},
      "provisionerId": "null-provisioner",
      "routes": [],
      "workerType": ""
    },
    "task_id": "cpwUS5EFTv6vuI_oyl7Yaw"
  },
(Assignee)

Comment 50

2 years ago
Okay I fixed that issue, and now have a green decision taskgraph on holly and the macosx64-opt build is using the buildbot-bridge provisioner

https://treeherder.allizom.org/#/jobs?repo=holly&revision=8cf09cae5fcfe674c32aeb5e4e59a0d89f5b54e6
(Assignee)

Comment 51

2 years ago
So I have been looking at misc.py and trying to figure out how to disable the triggering of macosx opt builders on trunk

Looking at generateBranchObjects, it generates a list of builders here

https://hg.mozilla.org/build/buildbotcustom/file/785a84be1eb7/misc.py#l887

For this bug, we want to keep macosx opt on certain trunk in the list of builders.  However, we don't it to be scheduled.

Looking at the scheduler code, 

https://hg.mozilla.org/build/buildbotcustom/file/785a84be1eb7/misc.py#l1031

and specifically here
https://hg.mozilla.org/build/buildbotcustom/file/785a84be1eb7/misc.py#l1058

the list of product builders is something like 
product_builders
['OS X 10.7 holly build', 'OS X 10.7 64-bit holly leak test build']

which has the branch and platform, but doesn't list opt or debug.  

So I'm not sure how to disable just the scheduler part of macosx opt builds on buildbot.  Even if I add  a platform specific flag  to a branch like "trigger_scheduler" to a branch, I don't know how to just remove the opt builds from the scheduler.

  'holly': {
        'enable_perproduct_builds': False,
        'lock_platforms': True,
        'enable_opt_unittests': True,
        'enable_talos': False,
        'platforms': {
            'macosx64': {
                "trigger_scheduler": False,

would appreciate any suggestions :-)

Comment 52

2 years ago
15:02 <aki> kmoir-afk: possibly editing buildObjects['schedulers'] in builder_master.cfg, which is really ugly, but may work

Amended: we may want to turn it off in the scheduler master, but allow for it in the builder master.  I'm not sure how BBB works, but the builder scheduler may be needed.
(Assignee)

Comment 53

2 years ago
Ben, since you wrote buildbot-bridge, do you know if we need to turn it off in the scheduler master but allow it in the builder master as described in comment 52?
Flags: needinfo?(bhearsum)
(In reply to Kim Moir [:kmoir] from comment #53)
> Ben, since you wrote buildbot-bridge, do you know if we need to turn it off
> in the scheduler master but allow it in the builder master as described in
> comment 52?

I'd forgotten that we had that distinction, but now that you've reminded me it sounds about right!
Flags: needinfo?(bhearsum)
(Assignee)

Comment 55

2 years ago
Created attachment 8862241 [details] [diff] [review]
bug1353376sched.patch

I tried to remove the builders on the scheduling master in staging.  From #mozbuild earlier today
bhearsum:
i forgot about universal masters
they basically do builder_master.cfg + scheduler_master.cfg
3:54 PM but we only use universal_master_sqlite.cfg in staging IIRC

It's not the working because there is a duplicate builder error

Traceback (most recent call last):
  File "/builds/buildbot/kmoir/build6/lib/python2.7/site-packages/buildbot-0.8.2_hg_c9f9b8ab349b_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/kmoir/build6/lib/python2.7/site-packages/buildbot-0.8.2_hg_c9f9b8ab349b_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/kmoir/build6/lib/python2.7/site-packages/buildbot-0.8.2_hg_c9f9b8ab349b_production_0.8-py2.7.egg/buildbot/master.py", line 812, in loadConfig
    % b['name'])
ValueError: duplicate builder name OS X 10.7 64-bit holly leak test build
make: *** [checkconfig] Error 1

Not sure if this is on the right path or some other approach is required
(Assignee)

Comment 56

2 years ago
catlee: you mentioned yesterday that you could look at this if I got stuck and I'm not sure where to go from here given comment 55.  Maybe the duplicate builder is a result of the fact that universal_master_sqlite.cfg is used in staging and it serves as both the builder_master.cfg + scheduler_master.cfg  I don't know.  I really need a good way forward on how to disable the schedulers for mac opt builds only on a branch, while keeping the debug builders + schedulers running on that same branch.
Flags: needinfo?(catlee)
(Assignee)

Comment 57

2 years ago
Created attachment 8862586 [details] [diff] [review]
bug1353376sched.patch

This is the patch that catlee wrote in our debugging session his afternoon.  I'd like to deploy it and test it on holly first and trigger some jobs.  If that works out, I'll refactor it to work on trunk.  This will include the gecko_version so we can have a dynamic list of branches to apply it to as it rides the trains, instead of a static list of branches which will inevitably break things.
Flags: needinfo?(catlee)
Attachment #8862586 - Flags: review?(catlee)
Comment on attachment 8862586 [details] [diff] [review]
bug1353376sched.patch

Review of attachment 8862586 [details] [diff] [review]:
-----------------------------------------------------------------

should we do the same for the universal master config so that dev/staging masters have the same behavior?
Attachment #8862586 - Flags: review?(catlee) → review+
(Assignee)

Comment 59

2 years ago
Created attachment 8862830 [details] [diff] [review]
bug1353376univ.patch

patch to update mozilla/universal_master_sqlite.cfg

Updated

2 years ago
Attachment #8862830 - Flags: review+
(Assignee)

Updated

2 years ago
Attachment #8862586 - Flags: checked-in+
(Assignee)

Comment 60

2 years ago
Comment on attachment 8862830 [details] [diff] [review]
bug1353376univ.patch

This didn't work, there are still duplicate builds after my reconfig and the latest runs on holly

https://treeherder.allizom.org/#/jobs?repo=holly&revision=39ee25af6a616676103431c87120bae631fef438

Not sure why, investigating, maybe the scheduler config file needs to be updated too
Attachment #8862830 - Flags: checked-in+
(Assignee)

Comment 61

2 years ago
Created attachment 8862919 [details] [diff] [review]
bug1353376sched2.patch
(Assignee)

Updated

2 years ago
Attachment #8862919 - Flags: checked-in+
(Assignee)

Comment 62

2 years ago
So that worked and now I only see one buildbot build scheduled on opt via bbb and the debug on vi tc. 

https://treeherder.allizom.org/#/jobs?repo=holly

I'll modify the patches so this scheduling fix will apply to trunk.
(Assignee)

Comment 63

2 years ago
Created attachment 8863008 [details] [diff] [review]
bug1353376trunk.patch

patch to disable macosx opt on trunk
Attachment #8863008 - Flags: review?(catlee)
Comment on attachment 8863008 [details] [diff] [review]
bug1353376trunk.patch

Review of attachment 8863008 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/builder_master.cfg
@@ +89,2 @@
>          schedulers = branchObjects['schedulers']
> +        if len(schedulers) == 1:

what is the purpose of this guard? was this failing on some branches?
(Assignee)

Comment 65

2 years ago
yes, some projects branches don't have all platforms
(In reply to Kim Moir [:kmoir] from comment #65)
> yes, some projects branches don't have all platforms

and how does that impact the number of schedulers?
(Assignee)

Comment 67

2 years ago
Comment on attachment 8863008 [details] [diff] [review]
bug1353376trunk.patch

I have a new patch to fix an issue I found.
Attachment #8863008 - Flags: review?(catlee)
(Assignee)

Comment 68

2 years ago
Created attachment 8863369 [details] [diff] [review]
bug1353376trunk-2.patch

So branches have zero schedulers since they don't have any platforms defined on them, such as cypress

'cypress': {
          'lock_platforms': True,
          'platforms': {
              # Limit Firefox to none for Bug 787208
          },
      },

Also, the previous patch assumed that the branch has 1 scheduler which was based off of holly, which actually isn't true for most trunk branches
Attachment #8863008 - Attachment is obsolete: true
Attachment #8863369 - Flags: review?(catlee)
Comment on attachment 8863369 [details] [diff] [review]
bug1353376trunk-2.patch

Review of attachment 8863369 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/builder_master.cfg
@@ +89,3 @@
>          schedulers = branchObjects['schedulers']
> +        if len(schedulers) > 0:
> +            assert len(schedulers) > 0

don't need this assert any more

@@ +89,5 @@
>          schedulers = branchObjects['schedulers']
> +        if len(schedulers) > 0:
> +            assert len(schedulers) > 0
> +            pre = branchObjects['schedulers'][0].builderNames
> +            schedulers[0].builderNames = [b for b in schedulers[0].builderNames if builder_filter(b)]

if we're supposed to handle branches with more than one scheduler, then we need to handle each scheduler here, not just the first one.
Attachment #8863369 - Flags: review?(catlee) → review-
(Assignee)

Comment 70

2 years ago
Created attachment 8863480 [details] [diff] [review]
bug1353376trunk-3.patch

The schedulers we care about modifying are all in the first one, but not sure if we care about looking at the other ones in the list since they don't apply. But the order could conceivably shift.
Attachment #8863369 - Attachment is obsolete: true
(Assignee)

Comment 71

2 years ago
Created attachment 8863561 [details] [diff] [review]
bug1353376trunk-4.patch
Attachment #8863480 - Attachment is obsolete: true
Attachment #8863561 - Flags: review?(catlee)
(Assignee)

Comment 72

2 years ago
Created attachment 8863841 [details] [diff] [review]
bug1353376trunk-5.patch

from irc this morning
catlee> Chris AtLee what is "assert schedulers[i]" for?
11:54 AM would it be easier to write that for loop as "for s in branchObjects['schedulers']:" ?
Attachment #8863561 - Attachment is obsolete: true
Attachment #8863561 - Flags: review?(catlee)
Attachment #8863841 - Flags: review?(catlee)
Comment on attachment 8863841 [details] [diff] [review]
bug1353376trunk-5.patch

Review of attachment 8863841 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/universal_master_sqlite.cfg
@@ +90,5 @@
> +                pre = s.builderNames 
> +                # builder name to be deleted
> +                bname = 'OS X 10.7 ' + branch + " build"
> +                s.builderNames = [b for b in s.builderNames if builder_filter(bname, b)]
> +                post = s.builderNames 

nit: there's some extra whitespace at the end of some lines here - please remove before pushing.
Attachment #8863841 - Flags: review?(catlee) → review+
(Assignee)

Updated

2 years ago
Attachment #8862241 - Attachment is obsolete: true
(Assignee)

Comment 74

2 years ago
Created attachment 8864362 [details] [diff] [review]
bug1353376m-i.patch

patch to limit scheduling change to all trunk branches except m-c to reduce duplicate jobs until uplift

see https://bugzilla.mozilla.org/show_bug.cgi?id=1350413#c33 for deployment details
(Assignee)

Updated

2 years ago
Attachment #8864362 - Flags: checked-in+
(Assignee)

Comment 75

2 years ago
Created attachment 8864571 [details] [diff] [review]
bug1353376m-c.patch

enable scheduling patch on m-c
(Assignee)

Updated

2 years ago
Assignee: spacurar → kmoir
(Assignee)

Comment 76

2 years ago
Looks like this is working on trunk without duplicate builds. 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=9348b76977e833f108cf77dff75b0fab887a2fc1
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Updated

6 months ago
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.