1490148 - NDA Slack Users Can Manage Apps/Channels & Compromise Potential Mozilla User Accounts

Status: RESOLVED FIXED

(Websites :: Other, defect)

Description

[Griffin Francis]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36

Steps to reproduce:

https://mozilla.slack.com/services/BBW5A5BTM (Slack App Integration)


Emails sent to [e-mail] will end up in this channel #keepmozilla safe


From
Firefox Accounts<[e-mail]>
To
<[e-mail]>
Subject
Verify your Firefox Account

Welcome!
Please activate your account by confirming this email address.

Activate now

This is an automated email; if you received it in error, no action is required. For more information, please visit Mozilla Support.



Two-step authentication enabled
You have successfully enabled two-step authentication on your Firefox Account from the following device:

Chrome on Windows 10
Armidale, NSW, Australia (estimated)
IP address: 220.233.83.142
8:25:10 AM (AEST) Tuesday, Sep 11, 2018

Security codes from your authentication app will now be required at each sign-in.



Hi there,

You have been vouched on http://Mozillians.org!

You are now able to search through all Mozillians profiles and you can also join groups.

Vouch description: An automatic vouch for being a Mozilla employee.

View your vouches on your profile https://mozillians.org/it/u/danger/

Take a moment to join a functional area or group right now:

- https://mozillians.org/it/functional-areas/
- https://mozillians.org/it/groups/

Keep rocking the open Web!

The http://Mozillians.org Team

-- 
You received this message because you are subscribed to the Google Groups "Danger" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [e-mail].

Comment 1

[Griffin Francis]

Attachment: Mozilla 1.png

SSO Dashboard

Comment 2

[Griffin Francis]

Attachment: Mozilla 2.PNG

Mozillian Account

Comment 3

[Griffin Francis]

Attachment: Mozilla 3.PNG

Slack Channel Notifications

Comment 4

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

Impact: 

1.) NDA/Staff can manage (add/delete/modify) all slack integration configurations that are not private
2.) They cannot add new integration
3.) They can presumably take control of email addresses that are bound to email auth flows and impersonate that user

Mitigating Factors:

1.) Limited to NDA/Staff
2.) Post process would allow account to exist for 30min before being disabled

Containment Plan:

# Short-term
1.) Disable danger@ integration and in g-suite (from July 24th and forward)
2.) Review all email configurations in email integration

# Long-term
3.) Review logs to see if a non-mozillian managed slack configurations
4.) Determine long-term strategy for configuration capability plans to prevent in the future

Comment 5

[Griffin Francis]

User: [e-mail]
Password: 2aY4JfxQcM

Comment 6

[Griffin Francis]

https://mozilla.slack.com/messages/C4E7SLU72/convo/C4EG4URRS-1536411946.000100/

Just found some AWS Notification emails within the #meao-alerts channel. There might be some impact here if we could reset the AWS password associated with the account?


AWS Notifications<[e-mail]>
To
email-alert
Subject
ALARM: "SUMO_Prod_Oregon-A-awsroute53-cbd6cd55-ab9d-4d90-80f8-7b6be2da7..." in US East (N. Virginia)
You are receiving this email because your Amazon CloudWatch Alarm "SUMO_Prod_Oregon-A-awsroute53-cbd6cd55-ab9d-4d90-80f8-7b6be2da7c71-Low-HealthCheckStatus" in the US East (N. Virginia) region has entered the ALARM state, because "Threshold Crossed: 1 datapoint [0.0 (23/05/18 14:59:00)] was less than the threshold (1.0)." at "Wednesday 23 May, 2018 15:00:00 UTC".

View this alarm in the AWS Management Console:
https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#s=Alarms&alarm=SUMO_Prod_Oregon-A-awsroute53-cbd6cd55-ab9d-4d90-80f8-7b6be2da7c71-Low-HealthCheckStatus

Alarm Details:
- Name: SUMO_Prod_Oregon-A-awsroute53-cbd6cd55-ab9d-4d90-80f8-7b6be2da7c71-Low-HealthCheckStatus
- Description:
- State Change: OK -> ALARM
- Reason for State Change: Threshold Crossed: 1 datapoint [0.0 (23/05/18 14:59:00)] was less than the threshold (1.0).
- Timestamp: Wednesday 23 May, 2018 15:00:00 UTC
- AWS Account: 236517346949

Threshold:
- The alarm is in the ALARM state when the metric is LessThanThreshold 1.0 for 60 seconds. 

Monitored Metric:
- MetricNamespace: AWS/Route53
- MetricName: HealthCheckStatus
- Dimensions: [HealthCheckId = cbd6cd55-ab9d-4d90-80f8-7b6be2da7c71]
- Period: 60 seconds
- Statistic: Minimum
- Unit: not specified

State Change Actions:
- OK: 
- ALARM: [arn:aws:sns:us-east-1:236517346949:MozillaMarketingSlack]
- INSUFFICIENT_DATA: 

--
If you wish to stop receiving notifications from this topic, please click or visit the link below to unsubscribe:
https://sns.us-east-1.amazonaws.com/unsubscribe.html?SubscriptionArn=arn:aws:sns:us-east-1:236517346949:MozillaMarketingSlack:cd69a157-4e80-478b-b5b4-71b1686b9c77&Endpoint=[e-mail]

Please do not reply directly to this email. If you have any questions or comments regarding this email, please contact us at https://aws.amazon.com/support

Comment 7

[Caglar Ulucenk [:cag]]

Quick update here:

- Re: the potential risk in comment 6, I talked to the person who created that specific Slack integration (https://mozilla.slack.com/services/B8M2VV3DX). An AWS IAM user is used to perform this integration. It's not root account, and this account is (most likely) protected by MFA. Since it's also an IAM user, the ability to self-perform password reset without logging in is not possible, and I wasn't able to find a venue to have AWS send a password reset email/link to this address.

I also did a quick pass on other public non-private Slack integrations to see if any others may be using a similar notification mechanism (hook via AWS users/accounts sending notifications), therefore may be potentially affected. I was not able to find any.

Comment 8

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

Jen and I caught up this morning, the understanding here was that Staff/NDA users could request new configurations, but what we did not know is that a Staff/NDA could edit existing configurations.  Additionally, it was unknown that a user could use the email integration (such as danger@) to register new accounts outside of Mozilla's control.  This also means the user could register accounts with say GitHub (we're still not sure to the extend of the impact there).

So, based on our understanding the issues are:

1.) NDA/Staff have the ability to add/edit/delete some integration configurations (the belief was that they could add, but it would trigger a vetting process by admins).  Examples of this include: email, Jenkins-CI, New Relic, etc.

2.) A slack email integration with a public email address and a non-private slack room is a pattern that could result in any NDA/Staff to register that email account with an external provider (examples: GitHub, FxA, etc.).  However when using those accounts against Mozilla SSO, this will result in a block in auth0 but the user can still use 3rd party account (of which we have no control).

Comment 9

[Griffin Francis]

FYI. The email [e-mail] is also affected. Updates will be posted to the #hubs-support channel. I will cease testing these @mozilla.com email accounts as I am not sure on the impact for the service owners.

Comment 10

[Griffin Francis]

Volunteers also appear to be able to delete/archive channels within the Mozilla slack instance. Screenshot attached. 

Is this intended functionality? Looking over some other organisation slack channels which I am a member of I do not appear to be able to do this.

Comment 11

[Griffin Francis]

Attachment: Example 1.png

Comment 12

[Griffin Francis]

Ok. Just an update following the discussion we have had within the Slack Channel. Using the email [e-mail] which Jen had provided us with I was able to create a Firefox account, Mozillians account (linked to my NDA account) Github account and also a Bugzilla email account. The Buzilla has access to the following confidential groups:

You have the following permission bits set on your account:
adobe-confidential    Adobe Confidential Group
bz_canusewhineatothers    Can configure whine reports for other users
bz_canusewhines    User can configure whine reports for self
canconfirm    Can confirm a bug.
editbugs    Can edit all aspects of any bug.
editbugs-team    People with 'editbugs' privilege
everyone    Everyone with a Bugzilla account
grants    Confidential grant-related bugs
kddi-confidential    KDDI Confidential Group
legal-involved    users that are allowed to be involved with legal bugs
mfa-required    Users that must enable 2FA
mozilla-confidential    Confidential Mozilla Project Bug (use another group if possible)
mozilla-corporation    Mozilla Corporation employee
mozilla-employee-confidential    Confidential Mozilla Employee Bug
panasonic-confidential    Panasonic Confidential Bug
qualcomm-confidential    Qualcomm Confidential Group
tai-confidential    Tai Confidential Bug
tai-team    Tai Partner Team
tako-confidential    TAKO Confidential Bug
tako-team    TAKO Partner Team
woodduck-confidential    Woodduck Confidential Bug
woodduck-team    Woodduck Partner Team

Attached is a screenshot demonstrating this access.

Comment 13

[Griffin Francis]

Attachment: Bugzilla Access.png

Comment 14

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

Jen and I have been working with Griffin closely on this to assess it's impact, here's a summary of where we're at...

1.) [*UPDATED*] NDA/Staff have the ability to add/edit/delete some integration configurations (the belief was that they could add, but it would trigger a vetting process by admins).  Examples of this include: email, Jenkins-CI, New Relic, etc.  They additionally have the ability to delete/archive public channels.

2.) A slack email integration with a public email address and a non-private slack room is a pattern that could result in any NDA/Staff to register that email account with an external provider (examples: GitHub, FxA, etc.).  However when using those accounts against Mozilla SSO, this will result in a block in auth0 but the user can still use 3rd party account (of which we have no control).

3.) [*NEW*] Abusing #2, a user who has control over a @mozilla.com account, will also gain access to BMO groups (noted in comment 12) that they shouldn't have access to due to some business logic on BMO.  Because BMO is not integrated into SSO, it doesn't result in the same types of protections for external providers notes above in #2.

Comment 15

[Sarah Huffman]

A user can't do this without it being visible to everyone who can see the channel the email integration is with, right? They could delete the channel to hide it, but that seems awfully visible as well.

Comment 16

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Sarah Huffman from comment #15)
> A user can't do this without it being visible to everyone who can see the
> channel the email integration is with, right? They could delete the channel
> to hide it, but that seems awfully visible as well.

Yes, the registration process to claim the account, as well as 2FA registrations would be visible to the respective channel.

Comment 17

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

We agreed to let the BMO account for blueberry@ continue so we could explore the impact a bit more (because Griffin is an NDA'd contributor as well), but as I rethink this and look at MoCo bug examples, I'm not confident that users are making best decisions around security flags and at the risk of being called paranoid, I've asked Dylan to disable the account.  I have a discussion later this week with Jen to discuss further and identify next steps.

Comment 18

[Dylan Hardison [:dylan] (he/him)]

(In reply to Griffin Francis from comment #12)

> adobe-confidential    Adobe Confidential Group
> bz_canusewhineatothers    Can configure whine reports for other users
> bz_canusewhines    User can configure whine reports for self
> canconfirm    Can confirm a bug.
> editbugs    Can edit all aspects of any bug.
> editbugs-team    People with 'editbugs' privilege
> everyone    Everyone with a Bugzilla account
> grants    Confidential grant-related bugs
> kddi-confidential    KDDI Confidential Group
> legal-involved    users that are allowed to be involved with legal bugs
> mfa-required    Users that must enable 2FA
> mozilla-confidential    Confidential Mozilla Project Bug (use another group
> if possible)
> mozilla-corporation    Mozilla Corporation employee
> mozilla-employee-confidential    Confidential Mozilla Employee Bug
> panasonic-confidential    Panasonic Confidential Bug
> qualcomm-confidential    Qualcomm Confidential Group
> tai-confidential    Tai Confidential Bug
> tai-team    Tai Partner Team
> tako-confidential    TAKO Confidential Bug
> tako-team    TAKO Partner Team
> woodduck-confidential    Woodduck Confidential Bug
> woodduck-team    Woodduck Partner Team


Just a note -- all of those groups are "confidential" but not security.
That is, any security issues discussed in those bugs are inappropriate and something that we (bmo admins) are probably going to do something about (by, for instance, requiring bugs in some products to use a more appropriate group).
Quite importantly, bug activity that is secured with those groups is sent out in the clear,
in contrast with security groups which is encrypted with either S/MIME or GPG.

Comment 19

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

We have restricted the workspace for Slack email integrations such that new integrations can't be added or existing integrations edited without admin approval.  This will also prevent arbitrary users from viewing the details of each email configuration, which allowed Griffin to explore and learn the weak spots faster.  Existing email integrations will be grandfathered in and may get further scrutiny if we deem them dangerous (basically if it matches the blueberry/danger/hubs pattern they are probably at risk).

That said, we are still consciously choosing to not restrict work-spaces for other Slack integration types (such as Jenkins-CI, Circle-Ci, etc.) just yet (though we may in the near future), mainly because it increases the scope at a time where Slack itself doesn't have a lot of granularity in the permissions model, currently it's sort of all or nothing. It's also unclear what the risks beyond a DoS or notification label issue for these other integrations, which are not as clear as the email risk demonstrated.

We're also hopeful bug 1501693 will serve as a catalyst to see whether it's worth exploring the idea of not giving implicit BMO groups (specially confidential groups) to any person who has gotten control over a *@mozilla.com account.  Perhaps it would be more appropriate to make that a request item (or at least verify they are an employee first) to decrease the risk of an email account takeover resulting in a confidential data leak.  It's also possible this fix be implicit with a possible auth0 integration for BMO in the future that would allow the authentication process for employee emails.

I am closing this bug out now, as resolved.  Please again note that we acknowledge there are still deficiencies in our Slack permission model, but we deem that as a mix of wanting to be flexible with out trusted users and also hope that Slack will offer more granular controls in the future to strike the type of balance we want for our users.  That said, this is evolving, and we may choose to increase the security controls here, as we become aware of specific risks.

Comment 20

[Griffin Francis]

Thanks for resolving this issue. I have taken the time to delete the Blueberry Github account as per my conversation with Jonathan.

Comment 21

[Griffin Francis]

Attachment: Github Acc Deleted.PNG

Comment 22

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

A few contextual items to add to this regarding bounty decision:

1.) Slack isn't in our eligible property list (so we typically wouldn't offer a bounty for it)
2.) Griffin has a slight advantage here that your typical bounty hunter doesn't have (he's an NDA's contributor with Slack access, we only give Slack access to employees/nda'd contribs at the moment, this may be more open in the future).
3.) This helped us identify and fix a weakness in our Slack integration permissions setup (we thought it worked a different way).
4.) Abusing this issue allowed any employee/nda contrib to impersonate bot email accounts and gain access to confidential information.
5.) This serves as a catalyst for us to consider how we grant implicit access to BMO in the future and that may reduce the impact of an email takeover scenario for that application.

Comment 23

[Caglar Ulucenk [:cag]]

It looks like the blueberry account ([e-mail]) created for testing was not disabled?

If it's no longer required, can this be disabled?

NI'ing :claudijd and :kang.

Comment 24

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Caglar Ulucenk [:Cag] from comment #23)

It looks like the blueberry account ([e-mail]) created for testing was not disabled?

If it's no longer required, can this be disabled?

NI'ing :claudijd and :kang.

cag: in what system? (BMO, FxA, LDAP, etc.)

Comment 25

[Griffin Francis]

Account is still active within FxA.

Comment 26

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

cag/griffin: that work is tracked in bug 1502869, but no progress has been made, I'll ping relevant folks for help.

Comment 27

[Caglar Ulucenk [:cag]]

I'm not sure if it's only FxA, our CIS API returns:
"{\"user_email\": \"[e-mail]\", \"connection_method\": \"ad\"}" which seems to indicate it's active in LDAP as well.

Comment 28

[Caglar Ulucenk [:cag]]

NI'ing kang again.

Comment 29

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

I spoke to Griffin about this yesterday, he noted specifically the FxA accounts were still active, the tracking for that is in bug 1502869

Comment 30

[Guillaume Destuynder [:kang] [this account is no longer active]]

this user is in LDAP, i can't delete it (permission-wise)
@jabba can you delete it?

Regarding https://bugzilla.mozilla.org/show_bug.cgi?id=1502869 <= i don't have access, if i'm not needed thats fine though

for what its worth [e-mail] is not in auth0 directly (ie its not been used since it was deleted there)

Comment 31

[Griffin Francis]

I am attempting to sign into LDAP with the password which I set on the FXA account without success, I would say it has been set to something else?

Comment 32

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Guillaume Destuynder [:kang] (NEEDINFO to ensure replies) (TZ: PDT) from comment #30)

this user is in LDAP, i can't delete it (permission-wise)
@jabba can you delete it?

Regarding https://bugzilla.mozilla.org/show_bug.cgi?id=1502869 <= i don't have access, if i'm not needed thats fine though

You've been CC'd (jabba as well).

Comment 33

[Justin Dow [:jabba]]

I'm not seeing this account in LDAP, nor do I see any evidence that it ever was in LDAP?

Comment 34

[Griffin Francis]

Just double checking back on this report, it appears the email [e-mail] is still affected.

Comment 35

[Griffin Francis]

Attachment: Hubs Twitter Reset.png

Comment 36

[Caglar Ulucenk [:cag]]

Griffin reached out to me about this and we had discussed and explored it for a fair bit. Looks like [e-mail] is a google groups email.

I think there are a couple of things to mention here:

1. hubs integration is still affected by this, in a very similar - if not the same - fashion as the original report from last year. Since this is an existing integration, I am not sure what to do or suggest, also in light of comment 19.

• I see the original account used to report ([e-mail]) was disabled, perhaps we can disable this one as well, although it looks like it has a valid use for Hubs/WebVR folks. Perhaps making this integration private may help conceal the issue? Although not sure if such change may break the integration (I do not know much about Slack integrations).
• Note, this also means I asked Griffin to explore the same path again, using [e-mail] i.e. create an FxA account, then a GitHub account and then try to access BMO again. The BMO access appears the same as was stated in comment 12 and comment 18. Both Griffin and I have access to these accounts created with [e-mail].

2. hubs.mozilla.com can be logged into with this, using [e-mail] email, since login to this website is possible with an email address only (A one time login link is sent to the email address). However the impact of this is not clear. There is an admin page which can be navigated (at /admin), but effectively no content is shown/loaded.

Will add screenshots shorty to make it clearer.

Jonathan, what do you think? Is there anything specific we should do for this instance? Also re: BMO access, looks like nothing changed since last time, likely due to the departure of Dylan, who was supposed to work on bug 1501693.

I will also loop in edunham, who appears to setup the Slack integration and may have input for the points above.

Comment 37

[Caglar Ulucenk [:cag]]

Attachment: hubs.mozilla.com-login-link.png

Comment 38

[Caglar Ulucenk [:cag]]

Attachment: hubs.mozilla.com-admin.png

Comment 39

[Caglar Ulucenk [:cag]]

edunham, I will try to summarise the issue for you:

Emails sent to [e-mail] is posted in the #hubs-support Slack channel due to a Slack integration. Users who can see the details of the integrations (as it is not private) - currently this includes Mozilla employees and NDA'ed contributors (like Griffin) - can abuse this to have access to the emails sent to this address (via the Slack channel), including password reset emails like the one from Twitter (as attached). This was an issue we identified last year for another Slack integration, and had investigated quite a bit for an impact since it can be used as a venue to take partial control of an @mozilla.com address and access places/websites that typical NDA'ed contributors should not be able to access.

My specific questions to you are:

• Do you think it's possible to make this integration private, without breaking its functionality? I am asking this based on the assumption that we cannot entirely disable this integration, since it looks like it is actively in use for hubs.mozilla.com.
• Is it a concern that this email can be used to login to hubs.mozilla.com, and access to seemingly an admin part of the website? You may not know the answer of this, if not please let me know who I should contact (or feel free to loop them in).

Comment 40

[Caglar Ulucenk [:cag]]

Quick update: Griffin has disabled Bugzilla, FxA and GitHub accounts created with [e-mail].

Comment 41

[edunham]

I'm no longer involved with Research ops -- needinfo-ing Lars, who can put you in touch with whoever's currently responsible for that integration.

Comment 42

[Lars Bergstrom (:larsberg)]

Connecting with gfodor, who set up the ops around [e-mail] and mails and other things going to the #hubs-support channel.

Comment 43

[Greg Fodor]

thanks for raising this issue.

• wrt locking down access, I think that seems fine to me. we can restrict that slack channel (#hubs-support) to the necessary parties.

• the ability to log in as that user on hubs.mozilla.com does not compromise admin access since we have a secondary layer of protection that requires a SSH key and 2fa tunnel to access the administrative resources. however, we should turn off the administrator bit on that account, it seems unnecessary and does get an attacker past the first layer.

it looks like I don't have the ability to convert the #hubs-support channel to a private channel. as per a quick google it seems that this should be possible to do via the additional channel options, but i don't see it. is that a privileged action? here are the slack handles that should have access:

gfodor
klee
jconrad
jshaughnessy
rlong
mqp
bpeiris
dom
oerickson
emclaren
larsberg
avrignaud

thank you!

Comment 44

[Greg Fodor]

actually, the [e-mail] account on hubs.mozilla.com does not have administrative access at all. i misunderstood the previous comment to imply that we were showing the link to /admin on the homepage (which would imply the account has access, since we display the link for those that do) but it sounds like /admin was just navigated to manually. hitting /admin itself to show the UI is not a privileged action, it's the access to the REST APIs which is secured both via an administrative bit on the account as well as the need to open an SSH tunnel to a AWS server.

so the only remediation here i believe is to lock down channel access.

Comment 45

[Caglar Ulucenk [:cag]]

Hi Greg, thanks for looking into this and confirming [e-mail] does not have admin access on https://hubs.mozilla.com.

Re: Slack access, I think turning hubs-support channel into a private one could fix this issue. In case this can't be done for whatever reason, you could also consider making the Slack integration for the Email app (https://mozilla.slack.com/apps/A0F81496D-email) a private one, which basically conceals the details of the integration (however does not restrict access to the channel itself).

I will see if I can have a Slack admin perform either.

Comment 46

[Griffin Francis]

Flags: sec-bounty-hof+ ?

Comment 47

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Griffin Francis from comment #46)

Flags: sec-bounty-hof+ ?

Just a heads up, when we 'sec-bounty+' a bug, the hof credit is implied. The 'sec-bounty-hof+' flag is used when we 'sec-bounty-', but still want to offer an hof mention.

Comment 48

[Griffin Francis]

Ok thanks mate. Hopefully we can publicly disclose this issue soon? I eventually want to do a writeup on it.

Comment 49

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Caglar Ulucenk [:Cag] from comment #45)

Hi Greg, thanks for looking into this and confirming [e-mail] does not have admin access on https://hubs.mozilla.com.

Re: Slack access, I think turning hubs-support channel into a private one could fix this issue. In case this can't be done for whatever reason, you could also consider making the Slack integration for the Email app (https://mozilla.slack.com/apps/A0F81496D-email) a private one, which basically conceals the details of the integration (however does not restrict access to the channel itself).

I will see if I can have a Slack admin perform either.

Cag: where do we stand on the status of this? Assuming it's resolved, we can like make this public so Griffin can do a write up on it.

Comment 50

[Caglar Ulucenk [:cag]]

I need to confirm with a Slack admin whether or not the particular instance (hubs) has been addressed. Once I do, I will update the status here.

Comment 51

[Greg Fodor]

I can confirm that the channel #hubs-support has been made private.

Comment 52

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

Lifting sec flags per Griffin's request.