1196407 - add support for builder name scope in buildbot bridge

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Right now, we can only grant access to buildbot bridge jobs in an all or nothing way. Jonas suggested that adding a scope for builder names would be a good way to make this more granular, eg:
task = {
  scopes = ["buildbot-bridge:builder-name:<builderName>"],
  payload: {
    builderName: "<builderName>"
  }
}


So in order to create a task through the buildbot bridge for builder "foo", you'd need a scope like:
buildbot-bridge:builder-name:foo
buildbot-bridge:builder-name:fo*
buildbot-bridge:builder-name:*

This means the Buildbot bridge needs to look for these scopes on the task when it gets a task-pending event, and if an appropriate scope isn't found, call reportException with malformed-payload.

See also, #taskcluster conversation that spawned this bug:
15:04 <&jonasfj> selenamarie|mtg, bhearsum hwine|mtg https://etherpad.mozilla.org/bb-bridge-scopes-tasks
15:17 < bhearsum> jonasfj: so, that seems like it would make a lot of sense. am i understanding correctly that as a provisioner and worker, the bridge is responsible for validating any scopes that correspond to things in the payload?
15:17 -!- jlund is now known as jlund|food
15:30 <&jonasfj> bhearsum, yes
15:30 <&jonasfj> bhearsum, feel free to setup a quikc meeting if you want to make sure we talk this through...
15:30 < bhearsum> i think i understand, just one more question...
15:31 <&jonasfj> as long as we keep things concrete I'm sure we'll get something done :)
15:31 < bhearsum> i check for that scope in the task definition, and because only people with that scope in their credentials can create tasks with it, i can trust that whomever created the task was allowed to
15:31 < bhearsum> :)
15:35 <&jonasfj> bhearsum, exactly! :)
15:35 < bhearsum> okay, easy :)
15:35  * bhearsum files a bug
15:35 -!- hwine|mtg is now known as hwine
15:35 <&jonasfj> bhearsum, and if the task doesn't have the scope... just reportException reason: malformed-payload
15:35 < bhearsum> right
15:36 <&jonasfj> bhearsum, and if you're really nice, you upload a log too saying that some scope was missing :)
15:36 <&jonasfj> docker-worker has historically not been good at saying what was missing :)
15:36 < bhearsum> hehe
15:36 < bhearsum> okay, i'll try to be a good citizen
15:37 < bhearsum> so, i'm thinking that rail and nick and i should chat sometime soon to try and nail down what we *think* our requirements on taskcluster are, and then we can all have a chat to validate that and (hopefully) come out with a concrete list
15:38 <&jonasfj> bhearsum, sounds good...

Comment 1

[[e-mail] (:bhearsum)]

Thinking about this a bit more. The main thing we want to be able to protect against is unauthorized people starting release jobs. Unfortunately, there's no common prefix for non-release jobs, so there's no way we can easily grant access to those other than buildbot-bridge:builder-name:*. It's easy to glob release builders (they all start with release-*), however. I'm wondering if we can treat release-* specially, and explicitly require buildbot-bridge:builder-name:release-* to start them. That is to say, even if you have buildbot-bridge:builder-name:*, you'd still need buildbot-bridge:builder-name:release-* to start release jobs. I guess it's the Scheduler that would enforce this though, so maybe that's not so easy...

Another idea could be to have scopes like:
buildbot-bridge:builder:ci:[builder name]
buildbot-bridge:builder:release:[builder name]

And the Bridge would be responsible for knowing which builder names are "release" and which are "ci".

Jonas, any thoughts on these, or other ideas?

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

> And the Bridge would be responsible for knowing which builder names are "release" and which are "ci".
That's a possibility.

### Option A) Current Pattern
You can also in your config file have a list of non-restricted builders that do not require the scope:
  buildbot-bridge:builder-name:<builderName>
And then say that builderNames not explicitly listed as non-restricted requires the special scope.
Then just list the CI builders in your config file as non-restricted.

This pattern is similar to what docker-worker does when polling down docker images.
It also requires that some images (the non-public ones) have a special scope, and this
all based on configuration (ie. list of image names that requires special scope).

Note: you can have either a list of non-restricted builders, or one of restricted builders.
It's probably safest to have one of non-restricted builders :)

Oh, it'll still require the queue:create-task:builderbot-bridge/builderbot-bridge scope to create
tasks for the non-restricted builder names. It just won't require the extra scope.

### Option B) Possible Future Pattern
Or you can wait for roles to land for auth, probably a few weeks away (at least).
Then it will be possible to configure a role that has a set of scopes on tools.tc.net/auth
Roles will be scopes that grants/expands to other scopes configured server-side on auth.
e.g. assume:buildbot-bridge-builders:release-builders -> [
  buildbot-bridge:builder-name:release-builder1,
  buildbot-bridge:builder-name:release-builder2,
  ...
]
and assume:buildbot-bridge-builders:ci-builders -> [
  buildbot-bridge:builder-name:try-linux,
  buildbot-bridge:builder-name:win32-try,
  ...
]

Long term that is the elegant solution, as the scope rules offered by bb-bridge will be simple.
However, bb-bridge might have to ask auth.tc.net if scope "buildbot-bridge:builder-name:try-linux"
is granted by task.scopes, as the expansion only happens server-side.

---
I suggest option (A), as it works now. And you're really only interested in locking down a specific
set of builder-names. You probably don't need super high granularity for access to the CI builders.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #2)
> > And the Bridge would be responsible for knowing which builder names are "release" and which are "ci".
> That's a possibility.
> 
> ### Option A) Current Pattern
> You can also in your config file have a list of non-restricted builders that
> do not require the scope:
>   buildbot-bridge:builder-name:<builderName>
> And then say that builderNames not explicitly listed as non-restricted
> requires the special scope.
> Then just list the CI builders in your config file as non-restricted.

Hmmm, this might be possible by matching on things that don't start with release-* - there's no other way to identify the unrestricted builders (names vary widely, with no restrictions on what they can be).

> ### Option B) Possible Future Pattern
> Or you can wait for roles to land for auth, probably a few weeks away (at
> least).
> Then it will be possible to configure a role that has a set of scopes on
> tools.tc.net/auth
> Roles will be scopes that grants/expands to other scopes configured
> server-side on auth.
> e.g. assume:buildbot-bridge-builders:release-builders -> [
>   buildbot-bridge:builder-name:release-builder1,
>   buildbot-bridge:builder-name:release-builder2,
>   ...
> ]
> and assume:buildbot-bridge-builders:ci-builders -> [
>   buildbot-bridge:builder-name:try-linux,
>   buildbot-bridge:builder-name:win32-try,
>   ...
> ]
> 
> Long term that is the elegant solution, as the scope rules offered by
> bb-bridge will be simple.
> However, bb-bridge might have to ask auth.tc.net if scope
> "buildbot-bridge:builder-name:try-linux"
> is granted by task.scopes, as the expansion only happens server-side.

This is cool, but I don't think it helps here. It's simply not feasible to spell out _all_ of the unrestricted (or even restricted) builders - there's hundreds of each, and new ones are created relatively frequently. The only realistic thing we can look for is the pattern on the restricted builders (release-*)


I'll play around a little bit and see what I come up with. Thanks for your insights!

Comment 4

[[e-mail] (:bhearsum)]

Attachment: replace allowed builders with restricted builders + scopes

This probably needs a bit more work, but it's a first crack at something that gives us the ability to restrict release-*.

Comment 5

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8654331 [details] [review]
replace allowed builders with restricted builders + scopes

Please, adopt some saner code for scope checking...

Comment 6

[Hal Wine [:hwine] use NI!]

Comment on attachment 8654331 [details] [review]
replace allowed builders with restricted builders + scopes

Thanks for the chance to review! f+ with these comments. I'm going to comment more on the "big picture" and not at all on the code. I have two areas I'd like to see additional discussion, documentation, or bugs on.

1: Switching from fail closed to fail open:

Basically this change is switching from a "fail closed" list of accepted builders to a "fail open" list of restricted builders. I completely understand the desire to do that! It does, effectively, remove a layer from our current "defense in depth" approach.

One way to preserve a bit of that old layer is to have answers to questions like:

- How and when do we audit the assertion that "all current privileged builders start with 'release-'"?

- What changes do we make to our development process to ensure we keep that convention going forward?

2: Is this level of scope check sufficient for privileged builders?

This is a murkier question -- and may also have a phased implementation. It may be easier to talk about in phases, so let me start there.

- Today: I'd love to see this deployed with a "no scope matches" variant. That would explicitly shut the door on privileged builders until we're ready. Everyone can experiment with CI and Tests and backfill -- win-win-win \o/

- During development of build promotion: hand waving - add a 2nd requirement of "access token signed by clientID X" or some such, so that only the active developers can kick test jobs. (i.e. using current TC tooling, but blocking * scope. Not sure this functionality exists in current API.)

- Final Production: by then we'll have a "real" answer with gpg signed or whatever is agreed upon. This one likely needs it's own RRA.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Hal Wine [:hwine] (use NI) from comment #6)
> Comment on attachment 8654331 [details] [review]
> replace allowed builders with restricted builders + scopes
> 
> Thanks for the chance to review! f+ with these comments. I'm going to
> comment more on the "big picture" and not at all on the code. I have two
> areas I'd like to see additional discussion, documentation, or bugs on.
> 
> 1: Switching from fail closed to fail open:
> 
> Basically this change is switching from a "fail closed" list of accepted
> builders to a "fail open" list of restricted builders. I completely
> understand the desire to do that! It does, effectively, remove a layer from
> our current "defense in depth" approach.

Honestly, I'd rather fail closed, I just don't know of a way to do it. I don't think we can sanely pattern match all of the low risk builders, because they are of such varying names (and some of the globbing we would have to do would also match privileged builders). If you have an idea for doing this, I'd love to hear it.

I'm a little less concerned about this than you, I think, simply because there is very active and ongoing work to move things off of Buildbot. Obviously it won't be done this year, but other than for release promotion, I can't see us adding any new privileged builders to Buildbot at this point. (Do we even have any privileged builders other than release ones in Buildbot?)

> One way to preserve a bit of that old layer is to have answers to questions
> like:
> 
> - How and when do we audit the assertion that "all current privileged
> builders start with 'release-'"?

Slight nit: privileged builders don't have to start with release-*, this patch supports multiple patterns for privileged. Eg: we could also add .*nightly.* to the list of privileged builders if we wanted.

Your point remains though, and I'm not sure how we audit beyond our normal review process. This is one of the reasons why I'd rather fail closed :(.


> 2: Is this level of scope check sufficient for privileged builders?
> 
> This is a murkier question -- and may also have a phased implementation. It
> may be easier to talk about in phases, so let me start there.
> 
> - Today: I'd love to see this deployed with a "no scope matches" variant.
> That would explicitly shut the door on privileged builders until we're
> ready. Everyone can experiment with CI and Tests and backfill -- win-win-win
> \o/

I'm not sure what you mean by  "no scope matches variant" here...can you rephrase? AFAICT, deploying this as is would prevent anyone without a "*" scope or a specific builder-name scope from triggering privileged builders.

> 
> - During development of build promotion: hand waving - add a 2nd requirement
> of "access token signed by clientID X" or some such, so that only the active
> developers can kick test jobs. (i.e. using current TC tooling, but blocking
> * scope. Not sure this functionality exists in current API.)
> 
> - Final Production: by then we'll have a "real" answer with gpg signed or
> whatever is agreed upon. This one likely needs it's own RRA.

I feel like I'm missing something here. I don't really understand what you mean by having signed access tokens...it's not something I've heard about before. Can you expand an these?

Comment 8

[Hal Wine [:hwine] use NI!]

(In reply to Ben Hearsum (:bhearsum) from comment #7)
> (In reply to Hal Wine [:hwine] (use NI) from comment #6)
> > I have two
> > areas I'd like to see additional discussion, documentation, or bugs on.
> > 
> > 1: Switching from fail closed to fail open:
> > 
> > Basically this change is switching from a "fail closed" list of accepted
> > builders to a "fail open" list of restricted builders. I completely
> > understand the desire to do that! It does, effectively, remove a layer from
> > our current "defense in depth" approach.
> 
> Honestly, I'd rather fail closed, I just don't know of a way to do it. I
> don't think we can sanely pattern match all of the low risk builders,
> because they are of such varying names (and some of the globbing we would
> have to do would also match privileged builders). If you have an idea for
> doing this, I'd love to hear it.
> 
> I'm a little less concerned about this than you, I think, simply because
> there is very active and ongoing work to move things off of Buildbot.
> Obviously it won't be done this year, but other than for release promotion,
> I can't see us adding any new privileged builders to Buildbot at this point.

Agreed, and I'm not worried, I want us to make conscious choices here.

> (Do we even have any privileged builders other than release ones in
> Buildbot?)

That's the next question:

> > One way to preserve a bit of that old layer is to have answers to questions
> > like:
> > 
> > - How and when do we audit the assertion that "all current privileged
> > builders start with 'release-'"?
> 
> Slight nit: privileged builders don't have to start with release-*, this
> patch supports multiple patterns for privileged. Eg: we could also add
> .*nightly.* to the list of privileged builders if we wanted.

Great!
 
> Your point remains though, and I'm not sure how we audit beyond our normal
> review process. This is one of the reasons why I'd rather fail closed :(.

Agreed, but we need to make progress. I think a bit of discussion (beer
'n tell topic?) about what the regexp should be would be sufficient at
this point.

> > 2: Is this level of scope check sufficient for privileged builders?
> > 
> > This is a murkier question -- and may also have a phased implementation. It
> > may be easier to talk about in phases, so let me start there.
> > 
> > - Today: I'd love to see this deployed with a "no scope matches" variant.
> > That would explicitly shut the door on privileged builders until we're
> > ready. Everyone can experiment with CI and Tests and backfill -- win-win-win
> > \o/
> 
> I'm not sure what you mean by  "no scope matches variant" here...can you
> rephrase? AFAICT, deploying this as is would prevent anyone without a "*"
> scope or a specific builder-name scope from triggering privileged builders.

I mean "always deny, even if '*' scope". A hard coded "FAIL" as it were.

> > 
> > - During development of build promotion: hand waving - add a 2nd requirement
> > of "access token signed by clientID X" or some such, so that only the active
> > developers can kick test jobs. (i.e. using current TC tooling, but blocking
> > * scope. Not sure this functionality exists in current API.)
> > 
> > - Final Production: by then we'll have a "real" answer with gpg signed or
> > whatever is agreed upon. This one likely needs it's own RRA.
> 
> I feel like I'm missing something here. I don't really understand what you
> mean by having signed access tokens...it's not something I've heard about
> before. Can you expand an these?

Ah, likely terminology mismatch -- the idea :rail had about some sort of signing of requests, or maybe something inside TC auth where we can also have scopes which describe "who". Some "new thing that is not yet implemented". 

But that's in the future, and we can address with either technology or a more detailed RRA/security analysis. Not a blocker for action.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: update configs+requirements for restricted builders

Need the new Taskcluster for scope_match, and we need to add restricted_builders to the config. Once this lands in prod I can remove allowed_builders from the configs.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Hal Wine [:hwine] (use NI) from comment #8)
> (In reply to Ben Hearsum (:bhearsum) from comment #7)
> > Your point remains though, and I'm not sure how we audit beyond our normal
> > review process. This is one of the reasons why I'd rather fail closed :(.
> 
> Agreed, but we need to make progress. I think a bit of discussion (beer
> 'n tell topic?) about what the regexp should be would be sufficient at
> this point.

I'll send some mail about this.

> > > 2: Is this level of scope check sufficient for privileged builders?
> > > 
> > > This is a murkier question -- and may also have a phased implementation. It
> > > may be easier to talk about in phases, so let me start there.
> > > 
> > > - Today: I'd love to see this deployed with a "no scope matches" variant.
> > > That would explicitly shut the door on privileged builders until we're
> > > ready. Everyone can experiment with CI and Tests and backfill -- win-win-win
> > > \o/
> > 
> > I'm not sure what you mean by  "no scope matches variant" here...can you
> > rephrase? AFAICT, deploying this as is would prevent anyone without a "*"
> > scope or a specific builder-name scope from triggering privileged builders.
> 
> I mean "always deny, even if '*' scope". A hard coded "FAIL" as it were.

I had a bit of trouble understanding this, so Hal and I chatted about it a bit. It boils down to not wanting TC credentials with just a plain "*" scope to be able to trigger release builds, because those are handed out more widely than we would like. Instead, we should only accept tasks for restricted builders if there is a specific buildbot-bridge:builder-name:blah or buildbot-bridge:builder-name:bl* scope. This way we can grant those specific scopes on a limited number of credentials.

This is a short term solution until we can manage scopes to the degree we need to in Taskcluster.

Jonas, I'm curious to get your thoughts on this. We definitely need to do something here -- we need to be able to severly lock down who can trigger restricted builders.

Comment 11

[Jonas Finnemann Jensen (:jonasfj)]

Short answer is no :)
Long answer is scopes are formalized: http://docs.taskcluster.net/presentations/scopes/#/
Trying to side-step is not a solution, not even remotely, the queue will expand "*" for anyone,
in accordance with the formal definitions.
---

I understand that you are concerned "*" not being locked down sufficiently. I understand this concern
and is working towards improving that situation.

A better short term solution is to set:
  ignored_builders:     ["^((?!alder).)*$", "^release-.*$"]
  restricted_builders:  ["^release-.*$"]
Then nobody will be able to use the release builders.
When you're more comfortable with TC scopes and we've locked it down further,
you can then change the configuration to:
  ignored_builders:     ["^((?!alder).)*$"]
  restricted_builders:  ["^release-.*$"]
Then, release builders will be protected by scopes.

Note:
Minor issue, but instead of ignoring tasks with ignored_builders, it's probably better to
claim the task, and reportExcpetion({reason: malformed-payload}). It's easier when debugging.

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #11)
> Short answer is no :)
> Long answer is scopes are formalized:
> http://docs.taskcluster.net/presentations/scopes/#/
> Trying to side-step is not a solution, not even remotely, the queue will
> expand "*" for anyone,
> in accordance with the formal definitions.

I think what you're saying here is that the Buildbot Bridge doesn't have access to the credentials that created a Task, therefore it doesn't know whether a * scope was used to gain access to buildbot-bridge:builder-name:whatever?

> ---
> 
> I understand that you are concerned "*" not being locked down sufficiently.
> I understand this concern
> and is working towards improving that situation.
> 
> A better short term solution is to set:
>   ignored_builders:     ["^((?!alder).)*$", "^release-.*$"]
>   restricted_builders:  ["^release-.*$"]
> Then nobody will be able to use the release builders.
> When you're more comfortable with TC scopes and we've locked it down further,
> you can then change the configuration to:
>   ignored_builders:     ["^((?!alder).)*$"]
>   restricted_builders:  ["^release-.*$"]
> Then, release builders will be protected by scopes.

This might work, depending on the timeline for TC scopes being locked down further. I don't see any issue with ignoring all release builders for now...but obviously we need to ease that up when release promotion moves into production. Do you have any rough idea of when scopes will be more tightly locked down?

For now, your idea above is pretty good, but I think we need to tweak it a bit to take dev into account. So, for prod:
* Ignore .*alder.* (bbb dev-only branch), .*date.* (release promotion development branch), and release-.* (for security reasons)
* Restrict release-.*

For dev:
* Ignore everything _except_ .*alder.* and .*date.*
* Restrict release-.*

This should make it impossible to trigger production release jobs through prod or dev bbb, and let us work on release promotion in dev, and verify the restricted builders code there as well, because release promotion builders will have a prefix once bug 1203606 is landed. This should be pretty easy to implement. Hal, I think this addresses your concern still?

> Note:
> Minor issue, but instead of ignoring tasks with ignored_builders, it's
> probably better to
> claim the task, and reportExcpetion({reason: malformed-payload}). It's
> easier when debugging.

This won't work for us, unfortunately. The original implementation of ignored_builders is so that we could support a dev environment that uses production Taskcluster+Buildbot, but only pays attention to one branch. We have it paying attention to .*alder.*, and prod ignoring anything matching that...so if we started claiming and cancelling things matching an ignore pattern, dev would start claiming prod jobs, and prod would start claiming dev jobs :).

Comment 13

[Hal Wine [:hwine] use NI!]

Sounds great to me -- I was unaware of the per branch filtering, which addresses a concern I had.

(In reply to Ben Hearsum (:bhearsum) from comment #12)
> 
> For now, your idea above is pretty good, but I think we need to tweak it a
> bit to take dev into account. So, for prod:
> * Ignore .*alder.* (bbb dev-only branch), .*date.* (release promotion
> development branch), and release-.* (for security reasons)
> * Restrict release-.*

IIUC, if anyone tries to schedule a task for a release-.* builder on a release branch, it will sit unclaimed in the queue. Is there a already existing way to monitor such requests? (I think they'd point to misunderstandings that we'd want to straighten out prior to full production.)

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Hal Wine [:hwine] (use NI) from comment #13)
> Sounds great to me -- I was unaware of the per branch filtering, which
> addresses a concern I had.
> 
> (In reply to Ben Hearsum (:bhearsum) from comment #12)
> > 
> > For now, your idea above is pretty good, but I think we need to tweak it a
> > bit to take dev into account. So, for prod:
> > * Ignore .*alder.* (bbb dev-only branch), .*date.* (release promotion
> > development branch), and release-.* (for security reasons)
> > * Restrict release-.*
> 
> IIUC, if anyone tries to schedule a task for a release-.* builder on a
> release branch, it will sit unclaimed in the queue. Is there a already
> existing way to monitor such requests? (I think they'd point to
> misunderstandings that we'd want to straighten out prior to full production.)

You're correct, anything ignored by both dev and prod will never get claimed (which is only release-* at the moment). I'm not sure if TC has anything to monitor this.

Comment 15

[Jonas Finnemann Jensen (:jonasfj)]

> I think what you're saying here is that the Buildbot Bridge doesn't have access to the
> credentials that created a Task, therefore it doesn't know whether a * scope was used
> to gain access to buildbot-bridge:builder-name:whatever?
Exactly, don't fight "*" :)
(Note: I'm working to reduce number of * scopes, and this will happen!)

> Do you have any rough idea of when scopes will be more tightly locked down?
Infrastructure to support roles should be in place by end of this Q.
From there it's a bunch of paper pushing. When do you need it?

> This won't work for us, unfortunately. The original implementation of ignored_builders
> is so that ...... only pays attention to one branch.
Ah, I guess it's okay. TC will expire unclaimed tasks by deadline.
But the right way of doing it would be to use a different workerType name.
For example you could use "buildbot-bridge-dev" or "dummy-bbb" for the dev setup...
(when testing docker-worker locally we just use a dummy worker name)

Comment 16

[[e-mail] (:bhearsum)]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #15)
> > I think what you're saying here is that the Buildbot Bridge doesn't have access to the
> > credentials that created a Task, therefore it doesn't know whether a * scope was used
> > to gain access to buildbot-bridge:builder-name:whatever?
> Exactly, don't fight "*" :)
> (Note: I'm working to reduce number of * scopes, and this will happen!)
> 
> > Do you have any rough idea of when scopes will be more tightly locked down?
> Infrastructure to support roles should be in place by end of this Q.
> From there it's a bunch of paper pushing. When do you need it?

OK, great :). That's well ahead of our schedule.

> > This won't work for us, unfortunately. The original implementation of ignored_builders
> > is so that ...... only pays attention to one branch.
> Ah, I guess it's okay. TC will expire unclaimed tasks by deadline.
> But the right way of doing it would be to use a different workerType name.
> For example you could use "buildbot-bridge-dev" or "dummy-bbb" for the dev
> setup...
> (when testing docker-worker locally we just use a dummy worker name)

I thought about doing this, but because the provisioner/worker types are embedded into the tree, it means you'd have to locally patch these branches to make sure they continue to use dev. It's a bit a of PITA to do so, especially if you're fully overwriting them with a fresh mozilla-central from time to time =\.

Comment 17

[[e-mail] (:bhearsum)]

Attachment: update restricted/ignore builders

This updates the restricted+ignored builders per my earlier comment. I had to tweak the templates a bit to interpret a list rather than a single value, because we have that for prod now. This ends up with changes to the configs like:
--- /builds/bbb/config.json	2015-09-10 09:11:07.014604388 -0700
+++ /tmp/puppet-file20150911-27781-gsseye-0	2015-09-11 10:17:30.701210839 -0700
@@ -16,14 +16,12 @@
     "tc_worker_group": "buildbot-bridge",
     "tc_worker_id": "buildbot-bridge",
 
-    "allowed_builders": [
-        "^.*$"
-    ],
-    "restricted_builders": [
-        "^release-.*$"
-    ],
+    "allowed_builders": ["^.*$"],
+    "restricted_builders": ["^release-.*$"],
     "ignored_builders": [
-        "^.*alder.*$"
+        "^.*alder.*$",
+        "^.*date.*$",
+        "^release-.*$"
     ],
 
     "tclistener": {

Comment 18

[[e-mail] (:bhearsum)]

OK, it looks like we're almost done here. I've reverted my hack to ignore * scopes, merged the PR, and will get it deployed to dev once the puppet patch is r+'ed. If all goes well in dev I should have it pushed to prod on Monday.

Comment 19

[[e-mail] (:bhearsum)]

Attachment: upgrade buildbot bridge prod

This worked well in dev. Armen triggered some jobs earlier today that worked fine, eg: https://tools.taskcluster.net/task-inspector/#EhypDcPmRhOwpfR4_szxVA/

I tried to trigger a job for release-date_source, which was refused: https://tools.taskcluster.net/task-inspector/#YZqrjnCxTgyoRcxo3u4r0Q/
2015-09-15 08:21:51,964 - bbb.services - Payload is invalid for task YZqrjnCxTgyoRcxo3u4r0Q, refusing to create BuildRequest

I think this is ready to deploy to prod!

Comment 20

[[e-mail] (:bhearsum)]

Attachment: remove unused config vars

This worked in production: https://tools.taskcluster.net/task-inspector/#E9ZZdvziS4e4Z6_lak_K0w/ was never claimed by prod or dev, and eventually got deadline-exceeded. \o/

The only left to do here is remove all the unused config now.