862665 - restrict access to tinderbox.m.o to members of the bzr_bugzilla ldap group

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[:glob ✱]

based on discussions from bug 527038:

(John O'Duinn [:joduinn] from comment #13)
> In case it helps, here are some alternative suggestions I explored with
> other groups:
> 2) move the existing tinderbox.m.o server within VPN, and limit access to
> specific LDAP users (unknown if this is really a workable option for your
> project - it depends on the profile of the contributors involved, and how
> they interact with tinderbox. Listing for completeness.)

(Frédéric Buclin from comment #16)
> LDAP could mitigate security issues and could be used as a short term
> solution. The list of people who need access to Tinderbox for Bugzilla
> purposes is pretty short: justdave, glob, dkl, wicked and myself.

(Byron Jones ‹:glob› from comment #23)
> ldap protection looks like a reasonable short-term fix, to buy us time to
> sort out the migration to jenkins.
> joduinn: is there an issue with implementing that?

(Dave Miller [:justdave] from comment #29)
> Everyone in the project who would need access already has an LDAP account in
> order to get bzr commit access.  You could certainly restrict it to people
> in the bzr_bugzilla group rather than employees since Bugzilla's the only
> people left using it.


if all other groups have moved off tinderbox, then putting it behind ldap authentication will buy us more time to migrate bugzilla's CI to jenkins.

Comment 1

[Shyam Mani [:fox2mike]]

Is this something we can work on right away?

Comment 2

[:glob ✱]

i'd like to see confirmation from joduinn that all other groups are off tinderbox before locking them out of it.

joduinn: is there an issue with doing this right away?

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) All other groups are already long gone from tinderbox server. See bug#843383 for details.

2) If we are going to use the fallback plan of moving this server behind LDAP, we need to change the name from tinderbox.m.o to something else. Anything else. Only the bugzilla folks are using it, so they can choose the new name. We're using the announcements of EOL-of-tinderbox to train people to use new tree.m.o servers, and have bugs blocked for setting up redirects, etc from tinderbox.m.o to the new server to help with those announcements. See dep bugs linked to bug#843383 for details. Moving tinderbox behind ldap while keeping the same tinderbox.m.o name would block that work and not be ok. 

2a) I note this means bugzilla folks need to change what email address they send test results to. 
(I'll crosspost in bug#527038 to make sure this is seen by all involved.)

Comment 4

[Joe Stevensen [:joe]]

Who exactly is requesting to keep tinderbox alive still?

I want to make sure that person is ready/able/willing to maintain both the system and the applications that run on the box. There are a number of security issues that remain unresolved. Most of the issues we've seen around tinderbox have been the result of lack of ownership. 

If someone is willing to pick up the baton and fix the issues with tinderbox, I'm happy to spend the time laying out the requirements.

Comment 5

[:glob ✱]

(In reply to John O'Duinn [:joduinn] from comment #3)
> 2) If we are going to use the fallback plan of moving this server behind
> LDAP, we need to change the name from tinderbox.m.o to something else.
> Anything else. Only the bugzilla folks are using it, so they can choose the
> new name.

tinderbox.bugzilla.org sounds sane.

will the box have the same ip address?  once i have know that, i'll file a bug to get the dns entry for tinderbox.bugzilla.org created (along with the required MX entries).

i also assume sendmail/postfix/whatever on that server will need to be reconfigured to accept mail addressed to its new domain.

> 2a) I note this means bugzilla folks need to change what email address they
> send test results to. 

while i don't have any experience with bugzilla's tinderbox integration, as far as i can tell that's a simple configuration change.

> (I'll crosspost in bug#527038 to make sure this is seen by all involved.)

everyone involved is already CC'd here :)

(In reply to Joe Stevensen [:joes] from comment #4)
> Who exactly is requesting to keep tinderbox alive still?

nobody is requesting that we keep tinderbox alive with ongoing maintenance and security fixes.

this is a stop-gap measure to balance the pressure to kill tinderbox ASAP with the resources available to migrate bugzilla's CI from tinderbox to jenkins.  once bugzilla is using jenkins, tinderbox can die for good.

> I want to make sure that person is ready/able/willing to maintain both the
> system and the applications that run on the box.

i'm willing to put my hand up to babysit the tinderbox server if it helps opsec close these security issues sooner.  :wicked should also be given root access.

> If someone is willing to pick up the baton and fix the issues with tinderbox

i'm not willing to fix any issues with tinderbox. once access is restricted and configs updated, hopefully the only command i'll run as root will be 'poweroff' :)

Comment 6

[Mark Côté [:mcote]]

Ping?

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Byron Jones ‹:glob› from comment #5)
> (In reply to John O'Duinn [:joduinn] from comment #3)
> > 2) If we are going to use the fallback plan of moving this server behind
> > LDAP, we need to change the name from tinderbox.m.o to something else.
> > Anything else. Only the bugzilla folks are using it, so they can choose the
> > new name.
> 
> tinderbox.bugzilla.org sounds sane.
:joes - does having the name not associated with mozilla helps? 


> will the box have the same ip address?  once i have know that, i'll file a
> bug to get the dns entry for tinderbox.bugzilla.org created (along with the
> required MX entries).

:joes, fix2mike: would know this best here. (My suspicion is that if this is to stay alive, then I believe getting it off Mozilla networks and machines should meet OpSec requirements about solving Mozilla exposure from the currently exploited security bugs... hence I do expect a new IP would be needed.)


> i also assume sendmail/postfix/whatever on that server will need to be
> reconfigured to accept mail addressed to its new domain.
Yep.


> > 2a) I note this means bugzilla folks need to change what email address they
> > send test results to. 
> while i don't have any experience with bugzilla's tinderbox integration, as
> far as i can tell that's a simple configuration change.
Agreed, that matches my understanding also.


[snip]


> (In reply to Joe Stevensen [:joes] from comment #4)
> > Who exactly is requesting to keep tinderbox alive still?
> nobody is requesting that we keep tinderbox alive with ongoing maintenance
> and security fixes.
> 
> this is a stop-gap measure to balance the pressure to kill tinderbox ASAP
> with the resources available to migrate bugzilla's CI from tinderbox to
> jenkins.  once bugzilla is using jenkins, tinderbox can die for good.
> 
> > I want to make sure that person is ready/able/willing to maintain both the
> > system and the applications that run on the box.
> 
> i'm willing to put my hand up to babysit the tinderbox server if it helps
> opsec close these security issues sooner.  :wicked should also be given root
> access.

OpSec have a list of security-issues-currently-being-exploited that continue to need fixing. OpSec are looking for people to code-fix those issues. :joes can give more details offline, or forward you those bugs, if that info would help clarify the urgency of the situation. 




> > If someone is willing to pick up the baton and fix the issues with tinderbox
> i'm not willing to fix any issues with tinderbox. once access is restricted
> and configs updated, hopefully the only command i'll run as root will be
> 'poweroff' :)

There seems to be no shortage of volunteers willing to "poweroff" tinderbox server! With any luck, after working towards this milestone for a few years now, I hope I'm still at the head of the line.

Comment 8

[:glob ✱]

(In reply to John O'Duinn [:joduinn] from comment #7)
> OpSec have a list of security-issues-currently-being-exploited that continue
> to need fixing. OpSec are looking for people to code-fix those issues. 

i'm not interested in fixing any tinderbox issues, nor do i think that should be a goal here.  if we can restrict access to tinderbox to just those with bugzilla commit access, we should do that without hesitation - the security holes will be effectively "closed" (or at least only exploitable by people who we already trust).

however, even with this restricted access in place, we should continue to work on migrating bugzilla to jenkins with the upmost importance.

> .. then I believe getting it off Mozilla networks and machines ..

migrating tinderbox of its current machine isn't what i'm proposing here.

> :joes can give more details offline, or forward you those bugs, if that info would
> help clarify the urgency of the situation. 

i understand the urgency of the situation.

Comment 9

[David Lawrence [:dkl]]

+1 for LDAP protecting the current server right away. Shyam, who could do this for us now? 
Then we can rename it or whatever while I work to still get up and running with Jenkins or something else.

dkl

Comment 10

[:glob ✱]

(From Dave Miller [:justdave] from bug 527038 comment #46)
> So, here's a thought on how to secure the existing tinderbox if we need to
> lock it down short term until the jenkins stuff gets worked out.
> 
> Assumptions:
> 1. The main pages that need to be publicly-visible are the waterfall pages. 
> There are static cached versions of those pages which are generated by
> tinderbox to allow for speedy page delivery and so forth.
> 2. The people who need to access the pages for notes and tree control and
> such all have committer rights and thus LDAP access.
> 
> The idea:
> 1. Move the tinderbox server behind the firewall.
> 2. Point the tinderbox domain name at a proxy server.
>    Or whatever domain it ends up sitting behind - it won't be t.m.o :)
> 3. Restrict the proxy server so that it will only pass through the URLs to
> the static waterfall pages, and maybe an index page
> 4. People who need to edit things can log into the VPN (perhaps on the
> contributor VLAN, we have one there, right?) to access the real machine
> directly.
>
> Actually, if we go with the proxy route, we *could* keep the t.m.o domain
> name, as the proxy could redirect everything other than the
> Bugzilla-specific URLs to the new site.
> 
> Thoughts?

Comment 11

[Joe Stevensen [:joe]]

(In reply to Byron Jones ‹:glob› from comment #10)
> (From Dave Miller [:justdave] from bug 527038 comment #46)
> > The idea:
> > 1. Move the tinderbox server behind the firewall.
> > 2. Point the tinderbox domain name at a proxy server.
> >    Or whatever domain it ends up sitting behind - it won't be t.m.o :)
> > 3. Restrict the proxy server so that it will only pass through the URLs to
> > the static waterfall pages, and maybe an index page
> > 4. People who need to edit things can log into the VPN (perhaps on the
> > contributor VLAN, we have one there, right?) to access the real machine
> > directly.

We're fine with this.

Comment 12

[Brandon Burton [:solarce]]

(In reply to Joe Stevensen [:joes] from comment #11)
> (In reply to Byron Jones ‹:glob› from comment #10)
> > (From Dave Miller [:justdave] from bug 527038 comment #46)
> > > The idea:
> > > 1. Move the tinderbox server behind the firewall.
> > > 2. Point the tinderbox domain name at a proxy server.
> > >    Or whatever domain it ends up sitting behind - it won't be t.m.o :)
> > > 3. Restrict the proxy server so that it will only pass through the URLs to
> > > the static waterfall pages, and maybe an index page
> > > 4. People who need to edit things can log into the VPN (perhaps on the
> > > contributor VLAN, we have one there, right?) to access the real machine
> > > directly.
> 
> We're fine with this.

If we can put everyone who needs access in an LDAP group then we can do everything with a couple of Apache location blocks, no need for proxies or VPNs

Comment 13

[:glob ✱]

(In reply to Brandon Burton [:solarce] from comment #12)
> If we can put everyone who needs access in an LDAP group then we can do
> everything with a couple of Apache location blocks, no need for proxies or
> VPNs

i agree, and i also think it would be safer to just block all access to tinderbox rather than just the admin pages (i've seen security issues with the public interface).

we don't need to make a new group - bzr_bugzilla (people with commit access to bugzilla) would be suitable.

Comment 14

[:glob ✱]

(In reply to John O'Duinn [:joduinn] from bug 527038 comment #51)
> (In reply to Dave Miller [:justdave] from comment #46)
> > 2. The people who need to access the pages for notes and tree control and
> > such all have committer rights and thus LDAP access.
> unknown. bugzilla folks could confirm these assumptions.

as per bug 527038 comment 29, and comment 0 of this bug, this is ok.

> joes (in OpSec) would need to comment on whether this proposal resolves his
> open security concerns, flagging him for needs-info. 

joe has already ok'ed this from a security point of view, in comment 11.

> aiui, unaddressed in the above proposal are: 
> 1) who will maintain this tinderbox server while the rest of the transition
> work is completed? 

as per comment 5 i'm happy to babysit it, assuming it's behind ldap protection.

> 2) I have no info on how much work remains to complete transition from
> tinderbox to jenkins, so dont know if this workaround is best use of our
> limited human time, or whether we'd spend as-much-time on the workaround as
> on the "real" fix. 

i can't image the work to edit .htaccess to require ldap authentication taking longer than migrating to a new test suite, especially given the uncertainty surrounding jenkins vs travis.

> Another proposal could be to export/move the tinderbox instance onto a
> bugzilla-community machine owned and hosted outside of Mozilla. This would
> not solve the security problems with the tinderbox instance, but would
> reduce the time-pressure from Opsec and RelEng. The security problems would
> now become those of whomever agreed to host the instance outside of Mozilla.
> I'm not sure this proposal is a good idea, but I want to list it in case
> others think it helps.

Comment 15

[Brandon Burton [:solarce]]

I'm ready to turn this on, anyone that should be notified first?

Comment 16

[David Lawrence [:dkl]]

(In reply to Brandon Burton [:solarce] from comment #15)
> I'm ready to turn this on, anyone that should be notified first?

I realize that everyone on the cc list of this bug has known this was potentially coming for a while now, but let's just give it a day or so for all involved to realize that this is happening
now and that they are aware.

LpSolit, wicked, anyone else? Anyone object to turning this on immediately and/or do you know of anyone not on the cc of this bug that should be given notice as well? I suspect not but doesn't hurt to ask. Should we send a notice to the bugzilla developer's list for example?

Thanks
dkl

Comment 17

[Brandon Burton [:solarce]]

(In reply to David Lawrence [:dkl] from comment #16)
> (In reply to Brandon Burton [:solarce] from comment #15)
> > I'm ready to turn this on, anyone that should be notified first?
> 
> I realize that everyone on the cc list of this bug has known this was
> potentially coming for a while now, but let's just give it a day or so for
> all involved to realize that this is happening
> now and that they are aware.
> 
> LpSolit, wicked, anyone else? Anyone object to turning this on immediately
> and/or do you know of anyone not on the cc of this bug that should be given
> notice as well? I suspect not but doesn't hurt to ask. Should we send a
> notice to the bugzilla developer's list for example?
> 
> Thanks
> dkl

I'll check this in the morning and if no one has replied otherwise, will proceed, as OpSec would like this change to be completed

Comment 18

[Frédéric Buclin]

(In reply to David Lawrence [:dkl] from comment #16)
> LpSolit, wicked, anyone else? Anyone object to turning this on immediately
> and/or do you know of anyone not on the cc of this bug that should be given
> notice as well?

No need to warn anyone else, at least not now. As long as I can still access Tinderbox as usual, I'm fine with the move.

Comment 19

[Brandon Burton [:solarce]]

Unless I hear otherwise I'll turn this on at 1:15PM PDT today, 2013-05-30

Comment 20

[Frédéric Buclin]

This makes me think: will IRC still be notified when the state of a box changes?

Comment 21

[David Lawrence [:dkl]]

(In reply to Frédéric Buclin from comment #20)
> This makes me think: will IRC still be notified when the state of a box
> changes?

Hmm. Does the irc bot use http to scrape the current tree status or does the tinderbox system send a notification to irc itself? If the former then I would think that the irc bot will need to be updated to be able to access the tinderbox UI over LDAP. 

Brandon. Also the community server that runs the tests sends the status and logs via email. I assume this will still work as before since you are just LDAP protecting the web UI?

dkl

Comment 22

[Brandon Burton [:solarce]]

(In reply to David Lawrence [:dkl] from comment #21)
> (In reply to Frédéric Buclin from comment #20)
> > This makes me think: will IRC still be notified when the state of a box
> > changes?
> 
> Hmm. Does the irc bot use http to scrape the current tree status or does the
> tinderbox system send a notification to irc itself? If the former then I
> would think that the irc bot will need to be updated to be able to access
> the tinderbox UI over LDAP. 
> 
> Brandon. Also the community server that runs the tests sends the status and
> logs via email. I assume this will still work as before since you are just
> LDAP protecting the web UI?
> 
> dkl

I have no knowledge of the IRC or Email portions but I don't see how this change would affect email, IRC depends on the bot

An .htpasswd file could be added to allow the bot to auth, while users still use LDAP, but irc bot changes are out of my scope of knowledge

Comment 23

[Dave Miller [:justdave]]

Yeah, the bot runs on one of the Bugzilla project servers, it http-scrapes to get the status (it polls one of the machine-readable static output pages).  mkanat used to maintain it, but he's not around anymore.  Not sure if wicked has hacked on it or not.  If nobody's actually been taking care of it lately I can. :)

Comment 24

[David Lawrence [:dkl]]

(In reply to Dave Miller [:justdave] from comment #23)
> Yeah, the bot runs on one of the Bugzilla project servers, it http-scrapes
> to get the status (it polls one of the machine-readable static output
> pages).  mkanat used to maintain it, but he's not around anymore.  Not sure
> if wicked has hacked on it or not.  If nobody's actually been taking care of
> it lately I can. :)

Do you know where he has the source hosted from? 

All I was able to find documentation wise was this:

https://wiki.mozilla.org/Bugzilla:Infrastructure#cg-bugs02.mozilla.org

I do not have access to that server myself, but wicked might.

Once we can get some clarification that the IRC bot can be made to still work, we can go ahead and lock this down right away.

dkl

Comment 25

[David Lawrence [:dkl]]

Brandon. Feel free to lock down Tinderbox behind LDAP at any time. We have located the source for the bugbot Tinderbox plugin and will need LDAP in place to be able to test out changes. We will need to make sure that [e-mail] has an LDAP account and is the bzr_bugzilla group.

Thanks
dkl

Comment 26

[Brandon Burton [:solarce]]

This has been completed.

Currently the following groups can auth

* bzr_bugzilla
* sysadmins

I've also added a htpasswd file for bugbot, the username is bugbot, see me on IRC for the password