Closed Bug 871231 Opened 11 years ago Closed 11 years ago

m-cMerge should be hosted on a mozilla.org server, over https:

Categories

(Tree Management :: Bugherder, defect)

Product:
Tree Management
Component:
Bugherder
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dbaron, Assigned: emorley)

References

Details

http://www.graememcc.co.uk/m-cmerge/ is an essential tool for our current landing workflow (with mozilla-inbound); it's what sheriffs use to mark bugs as fixed without going through hours of manual work.  Source is in https://bitbucket.org/graememcc/m-cmerge .

It's currently served over http on a non-mozilla.org server, and requires sheriffs to enter their bugzilla username and password.  (If the source is received intact without any MitM attacks, I believe the password goes only to mozilla.org's bugzilla over https.  But there's no guarantee of that.)

I believe sheriffs regularly use this tool over http: and enter their bugzilla passwords; I believe many of these sheriffs also have security-bug access.  This is a bad combination since it represents a *major* weak point in our protection of security bugs; if somebody breaks in to Graeme's server, or MitMs one of our sheriffs, they can get access to our security-sensitive bugs.

Hosting this site on a mozilla.org server, and over https, should be a high priority.

(I could have sworn there was a bug on this the last time I looked into it, but I can't find it.)
Dropping severity on the bug and passing to OpSec, which is probably more appropriate.

OpSec: I could find information about the tool at http://www.graememcc.co.uk/2012/06/12/introducing-m-cmerge/ and graemecc's info at https://mozillians.org/en-US/u/graememcc/
Assignee: server-ops-devservices → nobody
Severity: major → normal
Component: Server Operations: Developer Services → Security Assurance: Operations
QA Contact: shyam
Graeme, what are the server-side requirements? Just wondering whether we could pop onto the TBPL box, or if using one of the A-Team's AWS VMs would work?
(In reply to David Baron [:dbaron] (don't cc:, use needinfo? instead) from comment #0)
> It's currently served over http on a non-mozilla.org server, and requires
> sheriffs to enter their bugzilla username and password.  (If the source is
> received intact without any MitM attacks, I believe the password goes only
> to mozilla.org's bugzilla over https.  But there's no guarantee of that.)

Correct the bzapi calls go directly to bugzilla over HTTPs. Agree that MitM is a definite vector that we need to mitigate by moving to a Mozilla box using HTTPs.

> (I could have sworn there was a bug on this the last time I looked into it,
> but I can't find it.)

We've definitely discussed the desire to move m-cMerge onto Mozilla hardware before - and were planning on doing so [1] as part of the treeherder project, but that's looking a bit too far off at this point.

[1] https://www.pivotaltracker.com/story/show/49615299
(In reply to Ed Morley [:edmorley UTC+1] from comment #2)
> Graeme, what are the server-side requirements? Just wondering whether we
> could pop onto the TBPL box, or if using one of the A-Team's AWS VMs would
> work?

PHP with the cURL extension, for the "what version is m-c at?"/"what status flags are applicable?" code, which makes GET requests to https://hg.mozilla.org - all other network activity is client-side.
Assignee: nobody → jstevensen
Resetting this to major.

Sheriffs have access to sensitive bugs in bugzilla and their credentials are being exposed through this tool.
Assignee: jstevensen → nobody
Severity: normal → major
David Baron,

Do you have a list of Bugzilla users that used this site?
Who currently owns this site?
Graeme McCutcheon (CCed to this bug already)
I have started setting up mcmerge.mozilla.org - I think it's mostly done except we're having an unrelated problem with DNS changes not propagating.
Flags: needinfo?(graememcc_firefox)
Graeme,

When you are available, please contact me via irc "joes" or by email jstevensen@mozilla.com.

I need a few things from you. 

1. List of users that entered their Bugzilla accounts credentials in cleartext.
2. Shut down this site, or at least the ability to enter Bugzilla credentials.

If we don't hear from you in a few hours, we're going to have to take additional measures to prevent your server from accessing Mozilla systems.
It looks like the code uses Heather Arthur's bz.js library and submits to bugzilla directly from the user's client. Bugzilla passwords are not transmitted in the clear to the server at www.graememcc.co.uk/m-cmerge so there is no opportunity for passive MITM password sniffing (e.g. public wifi).

The primary danger would be that the code would be modified by an active MITM. Still not all that hard, but at least it would require a motivated attacker to specifically target m-cMerge and not simply someone capturing interesting things as they go by.

Lowers my worry level a bit. This has been live for months, however, and it would still be prudent to ask everyone who has used it to change their bugzilla passwords.
Hosting this tool also makes Graeme's machine an attractive target for exploitation, too. That would be a problem regardless of whether it was hosted over SSL or not since it may not be monitored as actively as mozilla infrastructure.
(In reply to Joe Stevensen [:joes] from comment #11)
> 1. List of users that entered their Bugzilla accounts credentials in
> cleartext.

The account credentials don't go to Graeme's server, so he wouldn't have that.  The only logs would be whatever logs result from bugzilla access through the bugzilla API.

> 2. Shut down this site, or at least the ability to enter Bugzilla
> credentials.
> 
> If we don't hear from you in a few hours, we're going to have to take
> additional measures to prevent your server from accessing Mozilla systems.

His server doesn't access Mozilla systems.  Also, given the domain name, I'd expect he's in UK time.





Also, once mcmerge.mozilla.org is working, tinderboxpushlog should be updated to point to it instead of graememcc.co.uk, and the update deployed to tbpl.mozilla.org.
(In reply to Joe Stevensen [:joes] from comment #11)
> Graeme,
> 
> When you are available, please contact me via irc "joes" or by email
> jstevensen@mozilla.com.
> 
> I need a few things from you. 
> 
> 1. List of users that entered their Bugzilla accounts credentials in
> cleartext.
> 2. Shut down this site, or at least the ability to enter Bugzilla
> credentials.
> 
> If we don't hear from you in a few hours, we're going to have to take
> additional measures to prevent your server from accessing Mozilla systems.

Others have partially covered this - but for clarity...

Credentials are not sent over plaintext (please read comment 0 and comment 3 amongst others). In addition, since the bzapi calls are directly from the client, it's not going to be possible for you to block it, nor for Graeme to tell who has used m-cMerge. For a rough approximation, search mozilla-central for commit messages matching "^[Mm]erge" but that do not contain "backout". That user list will contain myself, Ryan, a couple of other sheriffs and also people who merge project repos such as fx-team, services-central etc.

Graeme, I haven't had a chance to dig into the m-cMerge repo, but is there any reason we have to use server-side components (rather than in the client) to fetch the current tree version? It would make deploying easier if it were entirely client-side.

Joe/Jake, I think we'd rather host this on tbpl.mozilla.org rather than it's own domain (since TBPL links to it, and is the primary route of access anyway).
Flags: needinfo?(graememcc_firefox)
(In reply to Ed Morley [:edmorley UTC+1] from comment #15)
> Graeme, I haven't had a chance to dig into the m-cMerge repo, but is there
> any reason we have to use server-side components (rather than in the client)
> to fetch the current tree version? It would make deploying easier if it were
> entirely client-side.

Probably bug 658614 / bug 774766, I'd think.
(In reply to Joe Stevensen [:joes] from comment #11)
> 1. List of users that entered their Bugzilla accounts credentials in
> cleartext.
As others have said, credentials are not transmitted in the clear, and certainly not transmitted to a machine I control. m-cMerge simply wouldn't exist if it had been necessary for credentials to be transmitted to me. Once credentials are entered, the only communication is between the client's machine and Bugzilla over https. Client-side, access to the Javascript object bz.js creates is proxied using Javascript lexical scoping to prevent the rest of the codebase from reading the details entered.  

I can grep server logs for GET requests to m-cmerge/index.html etc. and attempt to correlate this with commits to the repository. This will be both inexact and incomplete.

> 2. Shut down this site, or at least the ability to enter Bugzilla
> credentials.
Done.

(In reply to David Baron [:dbaron] (don't cc:, use needinfo? instead) from comment #16)
> Probably bug 658614 / bug 774766, I'd think.
Indeed - this can be rewritten when hg.mozilla.org returns a permissive Access-Control-Allow-Origin header.
(In reply to Joe Stevensen [:joes] from comment #11)
> Graeme,
> 
> When you are available, please contact me via irc "joes" or by email
> jstevensen@mozilla.com.
> 
> I need a few things from you. 
> 
> 1. List of users that entered their Bugzilla accounts credentials in
> cleartext.
> 2. Shut down this site, or at least the ability to enter Bugzilla
> credentials.
> 
> If we don't hear from you in a few hours, we're going to have to take
> additional measures to prevent your server from accessing Mozilla systems.

Now that this is officially blocking our work flow, I assume that getting this running on Mozilla's servers is top priority?
So, just to be clear here, we shut down a tool critical to our workflow out of a concern for theoretical vulnerabilities without having a replacement?
Depends on: 873099
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #19)
> So, just to be clear here, we shut down a tool critical to our workflow out
> of a concern for theoretical vulnerabilities without having a replacement?

Ted, this concern was reported to Security Assurance and upon closer investigation it is now clear that credentials are not trasmitted in the clear. 

Upon our initial investigation, IT (jakem of webops) started getting this setup on mozilla's servers. I've been told that mcmerge.mozilla.org is now online. Please let me or jakem know if this is not true.

Big thanks to jakem and his team for helping out.
(In reply to Joe Stevensen [:joes] from comment #20)
> Ted, this concern was reported to Security Assurance and upon closer
> investigation it is now clear that credentials are not trasmitted in the
> clear. 

It would have been nice if that closer investigation had been done before ordering it to be taken down.
(In reply to Ryan VanderMeulen [:RyanVM] from comment #21)
> (In reply to Joe Stevensen [:joes] from comment #20)
> > Ted, this concern was reported to Security Assurance and upon closer
> > investigation it is now clear that credentials are not trasmitted in the
> > clear. 
> 
> It would have been nice if that closer investigation had been done before
> ordering it to be taken down.

It would be, however, when we're dealing with bugzilla accounts with access to extremely sensitive information, we need to balance the amount of time we spend figuring out what is/isn't happening, with the need to respond quickly in the event of possible account credential disclosure.
Ted/Ryan/Grame/[anyone else concerned], 

On an unrelated note, I think it might be a good idea to give the IT team a heads up about other tools 'critical to your workflow' that run on personal systems or 3rd party webhosts. Getting these hosted on Mozilla infrastructure (when appropriate) has a number of advantages. 

From a security point of view, those are:

* OS hardening
* OS patching
* Access control management
* Vulnerability scanning and remediation
* Attack detection through system, application, and network monitoring
* Ability to perform security incident response
I think we're all on the same page about that: tools critical to our workflow should be hosted on Mozilla infra. There's always going to be a gray area where new tools are prototyped outside of Mozilla infra (because they're written by a volunteer contributor, say, which was the case here, and was the case for TBPL originally) and they become critical but have not been transitioned to Mozilla hosting.
(In reply to Joe Stevensen [:joes] from comment #22)
> It would be, however, when we're dealing with bugzilla accounts with access
> to extremely sensitive information, we need to balance the amount of time we
> spend figuring out what is/isn't happening

This information was contained in comment 0 and comment 3, no time needed to figure it out beyond reading the bug - however, on a more constructive note - thank you for setting up mcmerge.m.o so quickly :-)

I think I'm still going to move m-cMerge into TBPL's repo (and host it at tbpl.m.o/mcMerge/ or similar), since it makes deployment easier for us.

In parallel, it would be good if opsec could drive bug 774766 forwards, so that we can make m-cMerge client-side only.
To be frank -- I'm surprised and impressed by the reaction in this bug.  I was expecting the reaction to be more like the usual silence seen in bug 774766.  (If that bug had been fixed, by the way, the m-cMerge tool could have easily been run *locally* rather than over http:, after downloading from a secure source, without the MitM risks -- and we might not have had this issue at all.)
(In reply to David Baron [:dbaron] (don't cc:, use needinfo? instead) from comment #26)
> To be frank -- I'm surprised and impressed by the reaction in this bug.  I
> was expecting the reaction to be more like the usual silence seen in bug
> 774766.  (If that bug had been fixed, by the way, the m-cMerge tool could
> have easily been run *locally* rather than over http:, after downloading
> from a secure source, without the MitM risks -- and we might not have had
> this issue at all.)

David,

I just pinged Yvan (webappsec) about bug 774766. The bug wasn't assigned to a security person, nor was a secreview bug filed, nor was a needinfo set, so it wasn't seen.
On process, now that I no longer control deployment of my code: in the event of updates I assume I file against IT/Server Ops, and they can pick up the changes from the Bitbucket repository?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #24)
> There's always going to be a
> gray area where new tools are prototyped outside of Mozilla infra (because
> they're written by a volunteer contributor, say, which was the case here,
> and was the case for TBPL originally) and they become critical but have not
> been transitioned to Mozilla hosting.

We're working on infrastructure that may help with this (platform-as-a-service). I don't know about the volunteer contributor aspect, but certainly it could help with employees at least (anyone with LDAP, for now). It should hopefully make development of new apps, particularly web apps, easier to get up and running on a supported infrastructure, but without a lot of IT blockage.

(In reply to Ed Morley [:edmorley UTC+1] from comment #25)
> on a more constructive note
> - thank you for setting up mcmerge.m.o so quickly :-)

You're welcome! :)

> I think I'm still going to move m-cMerge into TBPL's repo (and host it at
> tbpl.m.o/mcMerge/ or similar), since it makes deployment easier for us.

This sounds good to me. You've already got Chief for that, so this seems like a very logical step.


(In reply to Graeme McCutcheon [:graememcc] from comment #28)
> On process, now that I no longer control deployment of my code: in the event
> of updates I assume I file against IT/Server Ops, and they can pick up the
> changes from the Bitbucket repository?

For mcmerge.mozilla.org, yes... you can file in "mozilla.org :: Server Operations: Web Operations" for now. For tbpl.m.o/mcMerge/, once it exists, no... Ed can handle that directly. :)


I think we're good here. Closing this out, but feel free to re-open if we've overlooked anything. Thanks all!
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
(In reply to Graeme McCutcheon [:graememcc] from comment #17)
> As others have said, credentials are not transmitted in the clear, and
> certainly not transmitted to a machine I control.

Because I didn't see this clarified, just commenting to make sure everyone got the point.

It doesn't matter if the application instructs the client to send the passwords encrypted to the real server if the application itself was delivered over an unencrypted channel.  Anyone with appropriate access (wifi access point, or other more malicious network insertions) could have MiTMed the application itself and instructed it to send those credentials somewhere else, just one time, and nobody would have known any better.

Doesn't matter anymore since this has all been appropriately dealt with now, but just wanted to make sure everyone understood that.
Severity: major → normal
Blocks: 875323
I believe this bug can be opened up now - please may someone do the honours? :-)
Blocks: 891875
Please can this bug be opened up (currently websites-sec)
Group: websites-security
Assignee: nobody → emorley
No longer blocks: 875323
Component: Security Assurance: Operations → mcMerge
Depends on: 875323
Product: mozilla.org → Tree Management
Version: other → unspecified
You need to log in before you can comment on or make changes to this bug.