1353376 - Turn off buildbot macosx64 opt builds scheduling

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Comment 1

[Kim Moir [:kmoir] ET]

Could we try this on a project branch for testing before enabling it in production for all branches to ensure this doesn't have unintended consequences?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

SETA is only valid on mozilla-inbound/autoland.

Comment 3

[Wander Lairson Costa]

err, forgot to answer. In tests we can restrict per branch, not sure about builds. Anyway, try pushes aren't safe enough?

Comment 4

[Kim Moir [:kmoir] ET]

We can't disable something in buildbot on try without disabling it for everyone else on try. That is why I was suggesting a project branch.

Comment 5

[Wander Lairson Costa]

(In reply to Kim Moir [:kmoir] from comment #4)
> We can't disable something in buildbot on try without disabling it for
> everyone else on try. That is why I was suggesting a project branch.

Since this is an in-tree config, we can land the patch in bug 1349980 only in a specific branch, or I can hack on my patch to do it too.

Comment 6

[Kim Moir [:kmoir] ET]

Sebastian, you mentioned that you were interested in working on this.  You could write patches to try this out on the holly branch, so once it's reset we will be ready for testing.

Comment 7

[Sebastian Pacurar [:spacurar]]

I was planning to do the following patches:

1) one for enabling buildbot builds and tests on holly branch on macosx64 
2) one for enabling taskcluster builds and tests on holly branch on macosx64 
3) one for disabling the macosx64 buidlder and tests on buildbot

I intend to test if the jobs from taskcluster are affected after step 3.
Can you give me some suggestions on how I can make a setup to stop the scheduling of buildbot opt jobs regarding macosx64, while the jobs scheduled by task cluster via buildbot-bridge won't be affected? I think that if I turn off the scheduling from buildbot-configs will affect taskcluster as well since they are both managed by buildbot.

Comment 8

[Kim Moir [:kmoir] ET]

re 3) With buildbot bridge, the builders jobs have to exist on buildbot but the actual scheduling is triggered by taskcluster.

For turning off the mac tests from being scheduled in buildbot, you can look at 

https://bugzilla.mozilla.org/show_bug.cgi?id=1324915

and
https://reviewboard.mozilla.org/r/101530/diff/1#index_header

except
enable_unittest_sendchange

For the builds, I have been thinking about this, and I don't know how to do this.  I enabled talos tests for linux64-stylo by having a dummy builder but in this case we want to have the jobs scheduled via taskcluster, not just the tests.  https://bugzilla.mozilla.org/show_bug.cgi?id=1338871

I would advise asking for help in #mozbuild or #tcmigration for help on this

i.e. looking to schedule macosx opt buildbot builds on a branch via bbb via taskcluster without actually having them triggered by buildbot.

let me know if you need additional help, happy to discuss in vidyo to clarify

Comment 9

[Kim Moir [:kmoir] ET]

Sebastian, please post your in progress patches here so I can look at them. We would like to get these running next week.

As an aside, you can filter out running builds on a certain platform on taskcluster like they are here but modified so only macosx64 builds run on holly

https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/target_tasks.py#l297

This bug is for the buildbot changes, but you mentioned in the meeting we had this morning that you wanted to know how to filter out build platforms from running on taskcluster.  Let me know if I can help in any way!

Comment 10

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_disable buildbot macosx builds on buildbot

Comment 11

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_enable holy branch on buildbot

Comment 12

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_disable buildbot macosx tests on buildbot

Comment 13

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_disable scheduling of macosx tests on holly taskcluster

Comment 14

[Sebastian Pacurar [:spacurar]]

Regarding patches bug_1353376_disable buildbot macosx builds on buildbot and bug_1353376_disable buildbot macosx tests on buildbot  , are the ones which totally disable the jobs and builds from buildbot, causing problems since the jobs do not exist on buildbot anymore this way. I can mark those as obsolete since we need to stop only the scheduling for tests on taskcluster.

As for taskcluster, I cloned the holly repo and disabled the scheduling in patch bug_1353376_disable scheduling of macosx tests on holly taskcluster

Comment 15

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_filter macosx builds to run only on holly branch

Comment 16

[Kim Moir [:kmoir] ET]

Comment on attachment 8858288 [details] [diff] [review]
bug_1353376_enable holy branch on buildbot

looks good but I'm not sure you need 'mozharness_tag': 'default',

Why are you enabling win64?  Was this for your testing environment?

Comment 17

[Kim Moir [:kmoir] ET]

Comment on attachment 8858326 [details] [diff] [review]
bug_1353376_filter macosx builds to run only on holly branch

I think this should be 

def target_tasks_holly(full_task_graph, parameters):

instead of 
+def target_tasks_holy(gull_task_graph, parameters):

Comment 18

[Kim Moir [:kmoir] ET]

Comment on attachment 8858287 [details] [diff] [review]
bug_1353376_disable buildbot macosx builds on buildbot

So we want the builders to still exist on the buildbot master, but just not be scheduled by changes in buildbot. Rather they should triggered by taskcluster through buildbot bridge.  So I don't think you want to delete them.

Comment 19

[Kim Moir [:kmoir] ET]

Comment on attachment 8858299 [details] [diff] [review]
bug_1353376_disable buildbot macosx tests on buildbot

I don't think we need to worry about changing the tests scheduling now.  This bug is just about changing the buildbot builds to scheduled through taskcluster.

Comment 20

[Kim Moir [:kmoir] ET]

Comment on attachment 8858312 [details] [diff] [review]
bug_1353376_disable scheduling of macosx tests on holly taskcluster

I don't think we need to worry about changing the tests scheduling now.  This bug is just about changing the buildbot builds to scheduled through taskcluster.

Comment 21

[Wander Lairson Costa]

Notice only opt builds should be disabled for now.

Comment 22

[Wander Lairson Costa]

(In reply to Kim Moir [:kmoir] from comment #20)
> Comment on attachment 8858312 [details] [diff] [review]
> bug_1353376_disable scheduling of macosx tests on holly taskcluster
> 
> I don't think we need to worry about changing the tests scheduling now. 
> This bug is just about changing the buildbot builds to scheduled through
> taskcluster.

Maybe we should make the switch in a separate bug. My patch at bug 1350413 is safe to land on m-c because it doesn't turn on BBB build on.

Comment 23

[Kim Moir [:kmoir] ET]

Yes, this is correct, opt builds only.  Sebastian, to recap, the requirements are to
1) enable builders on holly for macosx opt that are triggered via taskcluster via the buildbot bridge.  
2) test that these builds are triggered through taskcluster (you could land wcosta's patch from bug 1350413 on holly only to test)
3) ensure that there are not duplicate builds triggered for macosx on holly (just the taskcluster one should be triggered)

Looking at bug 1157242, it looks like the patches there set some preferences disabling builders from being 
triggered by buildbot
in
mozilla/project_branches.py

https://bug1157242.bmoattachments.org/attachment.cgi?id=8600965

Not sure if they are still relevant, but it would be worth testing

We can work out details about tests later, right now the focus should be on making sure the builds are scheduled as expected.

Comment 24

[Sebastian Pacurar [:spacurar]]

(In reply to Kim Moir [:kmoir] from comment #16)
> Comment on attachment 8858288 [details] [diff] [review]
> bug_1353376_enable holy branch on buildbot
> 
> looks good but I'm not sure you need 'mozharness_tag': 'default',
> 
> Why are you enabling win64?  Was this for your testing environment?

I chose mozharness_tag since I made a copy after pine. I am not quite sure what mozharness_tag does. I noticed it can be set to 'default' and to 'production', so I was thinking default is the way I need to use it. And yes I enabled win64 for the testing environment. I would like to know what mozharness_tag actually does.



(In reply to Kim Moir [:kmoir] from comment #17)
> Comment on attachment 8858326 [details] [diff] [review]
> bug_1353376_filter macosx builds to run only on holly branch
> 
> I think this should be 
> 
> def target_tasks_holly(full_task_graph, parameters):
> 
> instead of 
> +def target_tasks_holy(gull_task_graph, parameters):

Yes that was a typo mistake, will fix that!



(In reply to Wander Lairson Costa [:wcosta] from comment #21)
> Notice only opt builds should be disabled for now.

Yes I know this should be only for opt builds, but I keep noticing it happens on debug as well in treeherder. Will carry on only with the opt builds then.

Comment 25

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_v2_filter-macosx-builds-to-run-only-on-holly-branch.patch

Fixed the typo.

Comment 26

[Kim Moir [:kmoir] ET]

Comment on attachment 8859140 [details] [diff] [review]
bug_1353376_v2_filter-macosx-builds-to-run-only-on-holly-branch.patch

I think it should be target_tasks_holly instead of target_tasks_holy but other than that it looks good!

Comment 27

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_v3_filter-macosx-builds-to-run-only-on-holly-branch.patch

I noticed this function in taskcluster/taskgraph/target_tasks.py:

 def _target_task(name):
    def wrap(func):
        _target_task_methods[name] = func
        return func
    return wrap


And examples of other functions declarations like the one I made about holly were starting like this:

@_target_task('stylo_tasks')
def target_tasks_stylo(full_task_graph, parameters):
...lines omitted...

@_target_task('pine_tasks')
def target_tasks_pine(full_task_graph, parameters):
...lines omitted...

That is why I left it like the ones above. In this patch I modified:
@_target_task('holly_tasks')  into
@_target_task_holly('holly_tasks')

Comment 28

[Kim Moir [:kmoir] ET]

Comment on attachment 8859230 [details] [diff] [review]
bug_1353376_v3_filter-macosx-builds-to-run-only-on-holly-branch.patch

I think you need to have 

@_target_task('holly_tasks')
def target_tasks_holly(full_task_graph, parameters):

In any case, you also need a patch to disable the builds from being scheduled on buildbot, but still have the builders remain, this comment has some hints.  Let me know if you need help, happy to go over the process needed to test this

https://bugzilla.mozilla.org/show_bug.cgi?id=1353376#c23

Comment 29

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_v4_filter-macosx-builds-to-run-only-on-holly-branch.patch

I noticed in an end what you meant in comment https://bugzilla.mozilla.org/show_bug.cgi?id=1353376#c26

It was a typo regarding "holly" word.I kept thinking I wrote down the declaration of the function wrong. 
Fixed it now. Sorry for the confusion.

Comment 30

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch

Comment 31

[Sebastian Pacurar [:spacurar]]

Comment on attachment 8859989 [details] [diff] [review]
bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch

I enabled holly on macosx64-debug, win64 and win64-debug just so I can test macosx64.

Comment 32

[Sebastian Pacurar [:spacurar]]

Attachment: bug_1353376_macosx64-opt-builds-are-triggered-through-taskcluster-on-holly-branch.patch

Comment 33

[Sebastian Pacurar [:spacurar]]

I would need some info on how I can test if there are no duplicate builds triggered for macosx on holly. I still have no idea how to test this on my local machine.

Can you please give me some advice on how can I get this done?

Comment 34

[Kim Moir [:kmoir] ET]

Comment on attachment 8859989 [details] [diff] [review]
bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch

We might not need all these enable_* parameters but we can iterate on these patches in testing.

Comment 35

[Kim Moir [:kmoir] ET]

Comment on attachment 8859995 [details] [diff] [review]
bug_1353376_macosx64-opt-builds-are-triggered-through-taskcluster-on-holly-branch.patch

I think we can use the patch in bug 1350413 to test on holly instead of this one.  Re the testing question you had earlier, we can go through the steps for that on that during our 1x1 tomorrow, does sound okay?

Comment 36

[Sebastian Pacurar [:spacurar]]

Yes sure. Ben Hearsum helped me out with checking if the schedulers still exist in buildbot, using the dump_master.py script from braindump. bug_1353376_v2_enable-holy-branch-on-buildbot-with-builds-triggering-disabled.patch seems to be working: the builder is enabled on holly branch but the Schedulers regarding it do not exist ("OS X 10.7 holly build" does not appear in the "Schedulers" section).

Regarding bug_1353376_macosx64-opt-builds-are-triggered-through-taskcluster-on-holly-branch.patch I added only the first commit parts from the mozreview of Wander Costa's patch. I will update it until our 1x1 meeting.

Comment 37

[Sebastian Pacurar [:spacurar]]

Comment on attachment 8858299 [details] [diff] [review]
bug_1353376_disable buildbot macosx tests on buildbot

>diff --git a/mozilla-tests/config.py b/mozilla-tests/config.py
>--- a/mozilla-tests/config.py
>+++ b/mozilla-tests/config.py
>@@ -3055,6 +3055,16 @@ for name, branch in items_at_least(BRANC
>                     if suite_name.startswith('mochitest-a11y'):
>                         win32['win7_ix'][test_type].remove(t)
> 
>+# Bug 1353376 - Turn off buildbot macosx64 opt builds scheduling
>+for name, branch in items_at_least(BRANCHES, 'gecko_version', 55):
>+    if name not in ['holly']:
>+        continue
>+    for platform in branch['platforms'].keys():
>+        if platform not in ['macosx64']:
>+            continue
>+        for test in ['opt_unittest_suites', 'debug_unittest_suites']:
>+            if 'yosemite_r7' in branch['platforms'][platform]:
>+                branch['platforms'][platform]['yosemite_r7'][test] = []
> 
> # bug 1343316 - add buildernames for linux64-stylo talos tests for mozilla-central and try branches
> for branch in BRANCHES.keys():
>

Comment 38

[Kim Moir [:kmoir] ET]

Attachment: enable holly branch for testing of macosx64 opt builds scheduled via tc

Comment 39

[Kim Moir [:kmoir] ET]

Comment on attachment 8859955 [details] [diff] [review]
bug_1353376_v4_filter-macosx-builds-to-run-only-on-holly-branch.patch

landed on holly

Comment 40

[Kim Moir [:kmoir] ET]

I also landed wcosta's patches here


Sebastian, you should be ready to test that the macosx opt builds are scheduled via taskcluster.  The next step after testing is completed is to write patches to enable this change on trunk branches.

Comment 41

[Kim Moir [:kmoir] ET]

previous comment should have said, I landed wcosta's patches from bug 1350413 on holly

side note: I was thinking about the patch to disable the triggering of builds on holly.

https://bug1353376.bmoattachments.org/attachment.cgi?id=8859989

This patch sets enables enable_onchange_scheduler to False.  This is okay for initial testing on holly.

However, to write the patches to enable this on trunk branches, we want macosx opt builders to exist, but not be triggered by buildbot. So enabling a preference like enable_onchange_scheduler on an entire branch won't work because it will impact builds for all other platforms, not just macosx opt.

So we will have to do something else to disable the schedulers for macosx opt builds on trunk via buildbot on trunk branches, while keeping them as triggered on non-trunk branches.

Comment 42

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/239d5d0aad9ff1d25eff737a9a782743fe9a29a4
Bug 1353376 - Re-enable holly repository (#2415)

Comment 43

[Kim Moir [:kmoir] ET]

Thanks, I can see the holly on the staging version of treeherder

https://treeherder.allizom.org/#/jobs?repo=holly

Comment 44

[Kim Moir [:kmoir] ET]

Attachment: bug1353376graph.patch

more patches, I landed on holly

Comment 45

[Kim Moir [:kmoir] ET]

Attachment: bug1353376tools.patch

had to enable branch in prodction-branches.json

Comment 46

[Kim Moir [:kmoir] ET]

Attachment: pull request to add holly branch to mozilla-taskcluster

Comment 47

[Kim Moir [:kmoir] ET]

So the earlier problem was that builds were not being scheduled, not the issue is a different error message

https://treeherder.allizom.org/logviewer.html#?job_id=87722257&repo=holly&lineNumber=691 

seems to be an issues with the patches I landed from bug 1350413, needs investigation

Comment 48

[Kim Moir [:kmoir] ET]

I was able to fix the earlier error and now have a new one

https://treeherder.allizom.org/#/jobs?repo=holly&revision=b4caed40b9b9186821e18ce9d66ace1568c1a8f3

 HTTPError: 400 Client Error: Bad Request for url: http://taskcluster/queue/v1/task/cpwUS5EFTv6vuI_oyl7Yaw

Comment 49

[Greg Arndt [:garndt]]

for some reason that task has an empty worker type

  "cpwUS5EFTv6vuI_oyl7Yaw": {
    "attributes": {
      "kind": "misc"
    },
    "dependencies": {},
    "kind": "misc",
    "label": "s3-uploader-build-macosx64/opt",
    "optimizations": [],
    "task": {
      "created": {
        "relative-datestamp": "0 seconds"
      },
      "deadline": {
        "relative-datestamp": "1 day"
      },
      "metadata": {
        "description": "Upload outputs of buildbot/mozharness builds to S3",
        "name": "Buildbot/mozharness S3 uploader",
        "owner": "[e-mail]",
        "source": "http://hg.mozilla.org/build/mozharness/"
      },
      "payload": {},
      "provisionerId": "null-provisioner",
      "routes": [],
      "workerType": ""
    },
    "task_id": "cpwUS5EFTv6vuI_oyl7Yaw"
  },

Comment 50

[Kim Moir [:kmoir] ET]

Okay I fixed that issue, and now have a green decision taskgraph on holly and the macosx64-opt build is using the buildbot-bridge provisioner

https://treeherder.allizom.org/#/jobs?repo=holly&revision=8cf09cae5fcfe674c32aeb5e4e59a0d89f5b54e6

Comment 51

[Kim Moir [:kmoir] ET]

So I have been looking at misc.py and trying to figure out how to disable the triggering of macosx opt builders on trunk

Looking at generateBranchObjects, it generates a list of builders here

https://hg.mozilla.org/build/buildbotcustom/file/785a84be1eb7/misc.py#l887

For this bug, we want to keep macosx opt on certain trunk in the list of builders.  However, we don't it to be scheduled.

Looking at the scheduler code, 

https://hg.mozilla.org/build/buildbotcustom/file/785a84be1eb7/misc.py#l1031

and specifically here
https://hg.mozilla.org/build/buildbotcustom/file/785a84be1eb7/misc.py#l1058

the list of product builders is something like 
product_builders
['OS X 10.7 holly build', 'OS X 10.7 64-bit holly leak test build']

which has the branch and platform, but doesn't list opt or debug.  

So I'm not sure how to disable just the scheduler part of macosx opt builds on buildbot.  Even if I add  a platform specific flag  to a branch like "trigger_scheduler" to a branch, I don't know how to just remove the opt builds from the scheduler.

  'holly': {
        'enable_perproduct_builds': False,
        'lock_platforms': True,
        'enable_opt_unittests': True,
        'enable_talos': False,
        'platforms': {
            'macosx64': {
                "trigger_scheduler": False,

would appreciate any suggestions :-)

Comment 52

[Aki Sasaki (not active)]

15:02 <aki> kmoir-afk: possibly editing buildObjects['schedulers'] in builder_master.cfg, which is really ugly, but may work

Amended: we may want to turn it off in the scheduler master, but allow for it in the builder master.  I'm not sure how BBB works, but the builder scheduler may be needed.

Comment 53

[Kim Moir [:kmoir] ET]

Ben, since you wrote buildbot-bridge, do you know if we need to turn it off in the scheduler master but allow it in the builder master as described in comment 52?

Comment 54

[[e-mail] (:bhearsum)]

(In reply to Kim Moir [:kmoir] from comment #53)
> Ben, since you wrote buildbot-bridge, do you know if we need to turn it off
> in the scheduler master but allow it in the builder master as described in
> comment 52?

I'd forgotten that we had that distinction, but now that you've reminded me it sounds about right!

Comment 55

[Kim Moir [:kmoir] ET]

Attachment: bug1353376sched.patch

I tried to remove the builders on the scheduling master in staging.  From #mozbuild earlier today
bhearsum:
i forgot about universal masters
they basically do builder_master.cfg + scheduler_master.cfg
3:54 PM but we only use universal_master_sqlite.cfg in staging IIRC

It's not the working because there is a duplicate builder error

Traceback (most recent call last):
  File "/builds/buildbot/kmoir/build6/lib/python2.7/site-packages/buildbot-0.8.2_hg_c9f9b8ab349b_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/kmoir/build6/lib/python2.7/site-packages/buildbot-0.8.2_hg_c9f9b8ab349b_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/kmoir/build6/lib/python2.7/site-packages/buildbot-0.8.2_hg_c9f9b8ab349b_production_0.8-py2.7.egg/buildbot/master.py", line 812, in loadConfig
    % b['name'])
ValueError: duplicate builder name OS X 10.7 64-bit holly leak test build
make: *** [checkconfig] Error 1

Not sure if this is on the right path or some other approach is required

Comment 56

[Kim Moir [:kmoir] ET]

catlee: you mentioned yesterday that you could look at this if I got stuck and I'm not sure where to go from here given comment 55.  Maybe the duplicate builder is a result of the fact that universal_master_sqlite.cfg is used in staging and it serves as both the builder_master.cfg + scheduler_master.cfg  I don't know.  I really need a good way forward on how to disable the schedulers for mac opt builds only on a branch, while keeping the debug builders + schedulers running on that same branch.

Comment 57

[Kim Moir [:kmoir] ET]

Attachment: bug1353376sched.patch

This is the patch that catlee wrote in our debugging session his afternoon.  I'd like to deploy it and test it on holly first and trigger some jobs.  If that works out, I'll refactor it to work on trunk.  This will include the gecko_version so we can have a dynamic list of branches to apply it to as it rides the trains, instead of a static list of branches which will inevitably break things.

Comment 58

[Chris AtLee [:catlee]]

Comment on attachment 8862586 [details] [diff] [review]
bug1353376sched.patch

Review of attachment 8862586 [details] [diff] [review]:
-----------------------------------------------------------------

should we do the same for the universal master config so that dev/staging masters have the same behavior?

Comment 59

[Kim Moir [:kmoir] ET]

Attachment: bug1353376univ.patch

patch to update mozilla/universal_master_sqlite.cfg

Comment 60

[Kim Moir [:kmoir] ET]

Comment on attachment 8862830 [details] [diff] [review]
bug1353376univ.patch

This didn't work, there are still duplicate builds after my reconfig and the latest runs on holly

https://treeherder.allizom.org/#/jobs?repo=holly&revision=39ee25af6a616676103431c87120bae631fef438

Not sure why, investigating, maybe the scheduler config file needs to be updated too

Comment 61

[Kim Moir [:kmoir] ET]

Attachment: bug1353376sched2.patch

Comment 62

[Kim Moir [:kmoir] ET]

So that worked and now I only see one buildbot build scheduled on opt via bbb and the debug on vi tc. 

https://treeherder.allizom.org/#/jobs?repo=holly

I'll modify the patches so this scheduling fix will apply to trunk.

Comment 63

[Kim Moir [:kmoir] ET]

Attachment: bug1353376trunk.patch

patch to disable macosx opt on trunk

Comment 64

[Chris AtLee [:catlee]]

Comment on attachment 8863008 [details] [diff] [review]
bug1353376trunk.patch

Review of attachment 8863008 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/builder_master.cfg
@@ +89,2 @@
>          schedulers = branchObjects['schedulers']
> +        if len(schedulers) == 1:

what is the purpose of this guard? was this failing on some branches?

Comment 65

[Kim Moir [:kmoir] ET]

yes, some projects branches don't have all platforms

Comment 66

[Chris AtLee [:catlee]]

(In reply to Kim Moir [:kmoir] from comment #65)
> yes, some projects branches don't have all platforms

and how does that impact the number of schedulers?

Comment 67

[Kim Moir [:kmoir] ET]

Comment on attachment 8863008 [details] [diff] [review]
bug1353376trunk.patch

I have a new patch to fix an issue I found.

Comment 68

[Kim Moir [:kmoir] ET]

Attachment: bug1353376trunk-2.patch

So branches have zero schedulers since they don't have any platforms defined on them, such as cypress

'cypress': {
          'lock_platforms': True,
          'platforms': {
              # Limit Firefox to none for Bug 787208
          },
      },

Also, the previous patch assumed that the branch has 1 scheduler which was based off of holly, which actually isn't true for most trunk branches

Comment 69

[Chris AtLee [:catlee]]

Comment on attachment 8863369 [details] [diff] [review]
bug1353376trunk-2.patch

Review of attachment 8863369 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/builder_master.cfg
@@ +89,3 @@
>          schedulers = branchObjects['schedulers']
> +        if len(schedulers) > 0:
> +            assert len(schedulers) > 0

don't need this assert any more

@@ +89,5 @@
>          schedulers = branchObjects['schedulers']
> +        if len(schedulers) > 0:
> +            assert len(schedulers) > 0
> +            pre = branchObjects['schedulers'][0].builderNames
> +            schedulers[0].builderNames = [b for b in schedulers[0].builderNames if builder_filter(b)]

if we're supposed to handle branches with more than one scheduler, then we need to handle each scheduler here, not just the first one.

Comment 70

[Kim Moir [:kmoir] ET]

Attachment: bug1353376trunk-3.patch

The schedulers we care about modifying are all in the first one, but not sure if we care about looking at the other ones in the list since they don't apply. But the order could conceivably shift.

Comment 71

[Kim Moir [:kmoir] ET]

Attachment: bug1353376trunk-4.patch

Comment 72

[Kim Moir [:kmoir] ET]

Attachment: bug1353376trunk-5.patch

from irc this morning
catlee> Chris AtLee what is "assert schedulers[i]" for?
11:54 AM would it be easier to write that for loop as "for s in branchObjects['schedulers']:" ?

Comment 73

[Chris AtLee [:catlee]]

Comment on attachment 8863841 [details] [diff] [review]
bug1353376trunk-5.patch

Review of attachment 8863841 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/universal_master_sqlite.cfg
@@ +90,5 @@
> +                pre = s.builderNames 
> +                # builder name to be deleted
> +                bname = 'OS X 10.7 ' + branch + " build"
> +                s.builderNames = [b for b in s.builderNames if builder_filter(bname, b)]
> +                post = s.builderNames 

nit: there's some extra whitespace at the end of some lines here - please remove before pushing.

Comment 74

[Kim Moir [:kmoir] ET]

Attachment: bug1353376m-i.patch

patch to limit scheduling change to all trunk branches except m-c to reduce duplicate jobs until uplift

see https://bugzilla.mozilla.org/show_bug.cgi?id=1350413#c33 for deployment details

Comment 75

[Kim Moir [:kmoir] ET]

Attachment: bug1353376m-c.patch

enable scheduling patch on m-c

Comment 76

[Kim Moir [:kmoir] ET]

Looks like this is working on trunk without duplicate builds. 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=9348b76977e833f108cf77dff75b0fab887a2fc1