710221 - Implement AggregatingScheduler

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Rail Aliiev [:rail]]

Attachment: AggregatingScheduler

AggregatingScheduler should wait for upstream builders and generate a buildset when they are finished successfully.

The current implementation is done by catlee.

I've removed maxTriggers TODO, and changed SourceStamp() to use revision=genBuildUID() so it doesn't ignores the buildsets with the same branch created before the actual builds are started. 

We talked about making this scheduler triggerable, but I'm not sure if we need this. This would make the scheduler more harmful during reconfigs and would require additional run time steps to trigger it.

bhearsum suggested to switch all Dependant schedulers to use this one, so we can move release schedulers to the scheduler master and release builders across all build masters. The only downside of this is the fact that we can't use a single waterfall to watch release builders (I watch them! :) ).

Comment 1

[Rail Aliiev [:rail]]

Attachment: AggregatingScheduler

* fixed the docstring
* removed unique revision parameter

interdiff:

diff -u b/scheduler.py b/scheduler.py
--- b/scheduler.py
+++ b/scheduler.py
@@ -21,7 +21,6 @@
 import util.tuxedo
 reload(util.tuxedo)
 from util.tuxedo import get_release_uptake
-from buildbotcustom.common import genBuildUID
 
 import time
 
@@ -345,9 +344,7 @@
 class AggregatingScheduler(BaseScheduler):
     """This scheduler waits until at least one build of each of
     `upstreamBuilders` completes with a result in `okResults`. Once this
-    happens, it triggers builds on `builderNames` with `properties` set. The
-    sourcestamp for the new builds will be the result of merging all the
-    sourcestamps from `upstreamBuilders`.
+    happens, it triggers builds on `builderNames` with `properties` set.
 
     `okResults` should be a tuple of acceptable result codes, and defaults to
     (SUCCESS,WARNINGS)."""
@@ -406,12 +403,8 @@
 
         if remainingBuilders:
             state['remainingBuilders'] = remainingBuilders
-            self.set_state(t, state)
-            return
-
-        if not remainingBuilders:
-            # generate a unique revision so every new buildset generates a build
-            ss = SourceStamp(branch=self.branch, revision=genBuildUID())
+        else:
+            ss = SourceStamp(branch=self.branch)
             ssid = db.get_sourcestampid(ss, t)
 
             # Start a build!
@@ -419,7 +412,8 @@
 
             # Reset the list of builders we're waiting for
             state['remainingBuilders'] = self.upstreamBuilders
-            self.set_state(t, state)
+
+        self.set_state(t, state)
 
 
 def makePropertiesScheduler(base_class, propfuncs, *args, **kw):

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 581373 [details] [diff] [review]
AggregatingScheduler

http://hg.mozilla.org/build/buildbotcustom/rev/c433e02af91d

Comment 3

[[e-mail] (:bhearsum)]

Is this code suitable for upstreaming?

Comment 4

[Chris Cooper [:coop] (he/him)]

I reconfig-ed with the landed changes from this bug today.

Comment 5

[Rail Aliiev [:rail]]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> Is this code suitable for upstreaming?

Let's test it for a couple of months, then sure!