601734 - Make cancellator.py aware of new testing masters

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[Armen [:armenzg]]

I don't even know why I assigned it to me.
I won't have time to work on it.

Comment 2

[Mike Taylor [:bear]]

removing the security flags so this can be seen by the rest of the team.

it doesn't appear to have any privileged information or data

Comment 3

[Mike Taylor [:bear]]

Attachment: add port number lookup by claimed_by host with fallback to original method

introduced a port map dictionary that allows the port number to be discovered using the claimed_by value.  if it's not found in the dictionary, fallback to hostname if/else that was originally used

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 487585 [details] [diff] [review]
add port number lookup by claimed_by host with fallback to original method

>diff --git a/buildbot-related/cancellator.py b/buildbot-related/cancellator.py
>--- a/buildbot-related/cancellator.py
>+++ b/buildbot-related/cancellator.py
>@@ -45,32 +45,46 @@ def getRunningBuilds(db, branch, revisio
> def cancelPendingBuild(brid):
>     br = db.tables['buildrequests']
> 
>     q = br.update().where(and_(br.c.id == brid, br.c.complete == 0))
>     q = q.values(complete=1, results=2, complete_at=time.time())
> 
>     res = q.execute()
> 
>+_portMap = { 'production-master01.build.mozilla.org:/builds/buildbot/builder_master1/master':  '8010',
>+             'production-master02.build.mozilla.org:/builds/buildbot/try-trunk-master/master': '8010',
I think this is should be 8011.

>+             'test-master01.build.mozilla.org:/builds/buildbot/tests-master/master':           '8012',
>+             'test-master02.build.mozilla.org:/builds/buildbot/tests-master/master':           '8012',
>+             'talos-master02.build.mozilla.org:/builds/buildbot/tests-master/master':          '8012',
>+             'test-master01.build.mozilla.org:/builds/buildbot/tests-master/master':           '8012',
>+             'test-master02.build.mozilla.org:/builds/buildbot/tests-master/master':           '8012',
Duplicate entries here?

>+             'buildbot-master1.build.mozilla.org:/builds/buildbot/tests_master3/master':       '8011',
>+             'buildbot-master1.build.mozilla.org:/builds/buildbot/tests_master4/master':       '8012',
>+             'buildbot-master2.build.mozilla.org:/builds/buildbot/tests_master5/master':       '8011',
>+             'buildbot-master2.build.mozilla.org:/builds/buildbot/tests_master6/master':       '8012',
Need entries for the build masters running on buildbot-master{1,2} on port 8010?


> def urlFromResult(result):
>     host, buildername, number = result
>     hostname = host.split(":")[0]
>     buildername = urllib.quote(buildername, "")
>+    port = '8010'
> 
>-    # try master
>-    if hostname == 'production-master02.build.mozilla.org':
>-        port = 8011
>-    # mini masters
>-    elif hostname.split('.')[0] in ('talos-master02','test-master01','test-master02'):
>-        port = 8012
>-    # pm01/pm03
>+    if host in _portMap:
>+        port = _portMap[host]
>     else:
>-        port = 8010
>+        # try master
>+        if hostname == 'production-master02.build.mozilla.org':
>+            port = '8011'
>+        # mini masters
>+        elif hostname.split('.')[0] in ('talos-master02','test-master01','test-master02'):
>+            port = '8012'
> 
>-    return "http://%(hostname)s:%(port)i/builders/%(buildername)s/builds/%(number)s" % locals()
>+    return "http://%(hostname)s:%(port)s/builders/%(buildername)s/builds/%(number)s" % locals()

Can we rely on _portMap having all the mappings correct, and then have urlFromResult raise an error if something isn't found rather than falling back to a default?

Comment 5

[Mike Taylor [:bear]]

Attachment: add port number lookup by claimed_by host

fixed errors in _portMap and made it raise an exception if the claimed_by host is not found in port map

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 487887 [details] [diff] [review]
add port number lookup by claimed_by host

>diff --git a/buildbot-related/cancellator.py b/buildbot-related/cancellator.py
>--- a/buildbot-related/cancellator.py
>+++ b/buildbot-related/cancellator.py
>@@ -45,32 +45,39 @@ def getRunningBuilds(db, branch, revisio
> def cancelPendingBuild(brid):
>     br = db.tables['buildrequests']
> 
>     q = br.update().where(and_(br.c.id == brid, br.c.complete == 0))
>     q = q.values(complete=1, results=2, complete_at=time.time())
> 
>     res = q.execute()
> 
>+_portMap = { 'production-master01.build.mozilla.org:/builds/buildbot/builder_master1/master':  '8010',
>+             'production-master02.build.mozilla.org:/builds/buildbot/try-trunk-master/master': '8011',
>+             'test-master01.build.mozilla.org:/builds/buildbot/tests-master/master':           '8012',
>+             'test-master02.build.mozilla.org:/builds/buildbot/tests-master/master':           '8012',
>+             'talos-master02.build.mozilla.org:/builds/buildbot/tests-master/master':          '8012',
>+             'buildbot-master1.build.mozilla.org:/builds/buildbot/tests_master3/master':       '8011',
>+             'buildbot-master1.build.mozilla.org:/builds/buildbot/tests_master4/master':       '8012',
>+             'buildbot-master2.build.mozilla.org:/builds/buildbot/tests_master5/master':       '8011',
>+             'buildbot-master2.build.mozilla.org:/builds/buildbot/tests_master6/master':       '8012',
>+           }

Still missing entries for buildbot-master{1,2}:8010

>+
> def urlFromResult(result):
>     host, buildername, number = result
>     hostname = host.split(":")[0]
>     buildername = urllib.quote(buildername, "")
>+    port = '8010'

No need for this now.

Comment 7

[Mike Taylor [:bear]]

Attachment: add port number lookup by claimed_by host

yea, keep forgetting this is for both builds and tests

added bm3 and bm4 to the list and removed unneeded variable init

Comment 8

[Mike Taylor [:bear]]

changes:
 e18c6225fcbd fix paths to older masters
 8930b467a10b Added to port lookup dictionary the "full" DNS host name along
 6fdfe04f3ad4 Initial commit