Closed Bug 1250950 Opened 8 years ago Closed 5 years ago

Automate checking Heroku org members have 2FA enabled

Categories

(mozilla.org :: Heroku: Administration, task)

Product:
mozilla.org
Component:
Heroku: Administration
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: hwine)

References

Details

Attachments

(2 files, 2 obsolete files)

Script that outputs members missing 2FA (v3)
2.76 KB, text/plain
Details
Example reminder email template
860 bytes, text/plain
Details 
At the moment I periodically check the org access page by hand:
https://dashboard.heroku.com/orgs/<org_name>/access

...and contact users who are listed as not having 2FA enabled, asking them to switch it on. 

However this is just something I do in my spare time - it would be ideal to automate it instead.

I've filed a ticket against Heroku asking if they would add an option to enforce that 2FA is enabled before people can access org apps:
https://help.heroku.com/tickets/338164

...however I imagine the reply is going to be along the lines of "for that you need our more expensive enterprise offering" - and even if it isn't, we really should automate this sooner than they'll implement the feature.

Automating this is quite easy, using their API:
https://devcenter.heroku.com/articles/platform-api-reference#organization-member-list

Auth guides:
https://devcenter.heroku.com/articles/platform-api-quickstart#authentication
https://devcenter.heroku.com/articles/oauth
Example usage of their API, that outputs a list of emails for members that don't have 2FA set.

Requires that HEROKU_ORG_NAME and HEROKU_API_KEY be set.
The latter can be populated using the output from `heroku auth:token`, however for a production script this may want to use OAUTH instead (see links in comment 0).
Attachment #8723104 - Attachment mime type: text/x-python → text/plain
Hal, am I correct in thinking there's a similar automated job that runs somewhere for GitHub 2FA? Where does that run?
Flags: needinfo?(hwine)
(In reply to Ed Morley [:emorley] from comment #2)
> Hal, am I correct in thinking there's a similar automated job that runs
> somewhere for GitHub 2FA? Where does that run?

Unfortunately, it's not automated atm -- as it requires "owner" credentials.

I manually run it weekly. I do have on my "to do" list to see if a lower auth API key is sufficient for github.
Flags: needinfo?(hwine)
Can we put heroku behind Okta?
There is https://blog.heroku.com/archives/2015/12/17/SSO-for-Heroku and I've filed bug 1251180 to get you a Heroku account so you can inspect the userlist for o=com,net,org eval.
Unfortunately SSO is currently only available for users of their premium Enterprise service. The Mozilla account is not an enterprise account, but a standard org account.
Another solution i was thinking about - since this is the same problem as Github - was that only 2FA'd users would be allowed to be added to the mozilla org.
This means, a script auto-removes you if you do not have 2FA, remembers that, and re-adds you when 2FA is enabled.

This also means you'd add your github email to phonebook for automatic-add/removal (manual adds would still work for compatibility with non-moco accounts).
We can't auto-remove, because that risks killing deployed apps in place, which is a very bad thing. I think compliance and notification is the correct approach here. If someone is unresponsive / vacation, we can assess and take more significant action, but for now it would be perfectly sufficient (from a Webops perspective) to report-only on this.

If you were to use the mozdef vuln2bugs and have one for Heroku 2FA and another for Github 2FA, that would be terrific. It would give us a place to track the work in progress to resolve it and have an audit trail over time, as well as escalate if the bug isn't closed within 1 business day and so on.
Of the four people who didn't have 2FA enabled when I emailed out last (24th Feb), two still haven't enabled. One was an org member so I removed access, the other just a collaborator on a single app (so lower risk) - so I sent the latter another email reminder instead.

Since then there are three more new users without 2FA enabled, have sent them an email today.
The users in comment 9 have all now enabled 2FA.

However there is now 1 new member and 2 new contributors who don't have 2FA enabled. Emailed them today.
There appears to be a regression in the Heroku API for this - the `two_factor_authentication` attribute is no longer returned, so my script has nothing to check. In addition, the web UI hasn't shown anyone missing 2FA for a while.

I have a sneaking suspicion the Heroku switch from us as an "edge case non-enterprise org", to their new "team org" feature might have meant we can no longer see who has 2FA enabled.

I've filed:
https://help.heroku.com/tickets/375765
(Ask if you want CC)

Another reason we should move to Heroku Enterprise IMO :-/
Response from Heroku:

"""
Sorry about the confusion. This value is only being returned for organizations and not for teams. Sadly, the API docs didn't keep apace with the addition of teams. But we're working on it!
"""

I read that as meaning "they are working on the docs", rather than "we're going to re-add support for checking 2FA status when using teams, rather than Enterprise" - however I will request clarification.
Their next reply:

"""
Ah, I could've been more specific. For the short term, a doc fix is pending indicating that the field is absent for Teams. Longer term, there is more work to be done on the Teams feature before we can expose whether a Team member has 2FA enabled for Teams.
"""
I hope to address this with an enterprise agreement, which will allow us to use SSO. Review is underway in https://bugzilla.mozilla.org/show_bug.cgi?id=1297222
Ah great! :-)
The enterprise feature bits have appeared to now been flipped - amongst other things the 2FA status of each org member is visible again.

There are currently 0 admin, 0 member and 8 collaborators without 2FA enabled.
Let me know when we can get the ball rolling on SSO and we can have a quick meeting about how and when to implement.
Checking today I found an admin without 2FA enabled (bimsland@moco).

I've demoted that account to 'member' temporarily - Blake, let me know when 2FA is enabled and I'll bump it back to 'admin'. 

I'll email all of the admins to remind them of their responsibility to:
* check people have 2FA enabled before adding new admins
* self-edit user role from admin to member if needing to remove one's own 2FA at any point
Assignee: infra → nobody
Component: Infrastructure: Other → Heroku: Administration
Product: Infrastructure & Operations → mozilla.org
QA Contact: cshields
:emorley ugh, thought I had it enabled but it was for the wrong account, sorry about that.
No problem; switch account back to role admin :-)
s/switch/I've switched/
Updated script.

It now:
* uses netrc rather than requiring the environment variable be set (since netrc is already set up just by logging into the Heroku CLI, so for most users there is now zero set-up)
* lists/groups users by their role (admin/member/contributor)
Attachment #8723104 - Attachment is obsolete: true
Running the script just now found:
0 admins and members but 8 collaborators with 2FA not enabled.

I've emailed those 8 users.
Dustin, I see 4 collaborators to taskcluster-* apps that don't have 2FA enabled:
https://dashboard.heroku.com/orgs/mozillacorporation/access

These users have full access to environment variables such as AWS access keys/taskcluster access tokens.

Could you reach out to those collaborators (or remove them from the apps until 2FA enabled), and also make checking for 2FA part of the process of adding new collaborators to the apps?
Flags: needinfo?(dustin)
(In reply to Ed Morley [:emorley] from comment #0)
> I've filed a ticket against Heroku asking if they would add an option to
> enforce that 2FA is enabled before people can access org apps:
> https://help.heroku.com/tickets/338164

In that ticket they promised to follow up by email, but never did so.

I've filed again (no way to reopen old tickets):
https://help.heroku.com/tickets/466004
That sounds like a job for Greg (I can't modify access)
Flags: needinfo?(dustin) → needinfo?(garndt)
Leaving ni? for now, removed 3 of the 4 collaborators as they no longer need access.  Requesting the 4th collaborator to add 2fa.
Contributor has enabled 2fa.  There shouldn't be anymore collaborators that do not have 2fa for taskcluster apps.
Flags: needinfo?(garndt)
Many thanks :-)
Running the script again now found:
0 admins, 1 members and 2 collaborators with 2FA not enabled.

The 2 collaborators were already emailed 10 weeks ago in comment 23, so I've now removed them from the Mozilla org and emailed explaining how to request re-addition after enabling 2FA.

I've emailed the 1 new member with the standard reminder.
(In reply to Ed Morley [:emorley] from comment #30)
> I've emailed the 1 new member with the standard reminder.

They've just enabled 2FA, so we now have zero users without 2FA.
Running the script again now found:
0 admins, 0 members and 2 collaborators with 2FA not enabled.

Both users belonged to the same set of apps: owg, owg-stage, owg-dev.

Email sent.
See Also: → 1387250
Checking again now found:
0 admins, 0 members and 11 collaborators with 2FA not enabled.

Between them the users have access to the following 14 apps:
 - moz-experiments-viewer
 - moz-experiments-viewer-staging
 - mozilla-pontoon
 - mozilla-pontoon-staging
 - normandy-cors-proxy
 - owg
 - owg-dev
 - owg-stage
 - release-notifications
 - releng-production-tooltool
 - releng-production-treestatus
 - releng-staging-tooltool
 - releng-staging-treestatus
 - shipit-staging-pulse-listener

2 of those collaborators were the same as from comment 32, so I've removed them from the org and emailed them explaining how to be re-added. They were collaborators for the owg* apps - which as a result now have no owners -> filed bug 1399487.

I've emailed the remaining 9 collaborators.

Of note, I had bhearsum replying saying he wasn't aware he had a Heroku account. Looking into this more it seems that both the Heroku web access page and their API don't provide a way to differentiate between an active user and a not accepted invitation. It's also not clear if invitations expire at any point, which doesn't seem ideal.

For this I've filed:
https://help.heroku.com/tickets/512467
(Ticket wording: https://emorley.pastebin.mozilla.org/9032304)
Checking again found 2 collaborators with 2FA not enabled.

Affected apps:
normandy-cors-proxy
releng-production-tooltool
releng-production-treestatus
releng-staging-database
releng-staging-tooltool
releng-staging-treestatus

Since both were in the group from comment 33, I've just gone ahead and deleted them.

(In reply to Ed Morley [:emorley] from comment #33)
> Of note, I had bhearsum replying saying he wasn't aware he had a Heroku
> account. Looking into this more it seems that both the Heroku web access
> page and their API don't provide a way to differentiate between an active
> user and a not accepted invitation. It's also not clear if invitations
> expire at any point, which doesn't seem ideal.
> 
> For this I've filed:
> https://help.heroku.com/tickets/512467
> (Ticket wording: https://emorley.pastebin.mozilla.org/9032304)

Heroku's response:

"""
With enterprise organizations, there was an intentional product design to have members implicitly trusted. There is a feature flag that exists that would enable invitations to work as you expect for your organization, returning data from the invitations endpoint. The bad news is that this will only return any outstanding invitations, which may or may not give you what you want in terms of your use case. It wont show you any currently invited & outstanding invitations. Those would have to be re invited.

Let us know if you'd like to try enabling that feature flag for your organization. Again we appreciate the feedback, please don't hesitate to reach out to us in the future!
"""

(I haven't replied to that yet.)
Checking again today found 2 collaborators without 2FA enabled.

Affected apps:
deepspeech-scriptworker
mozilla-pontoon

Email sent to both.
Checking again today found 1 member and 2 collaborators without 2FA enabled. One collaborator was that from comment 35 (for mozilla-pontoon), so I've removed them from the account and notified via email. The other two users were new this month - the standard initial email has been sent. 

The affected apps are:
cloud-mirror
cloud-mirror-us-east-1
ec2-manager
ec2-manager-staging
jsonschemalint
mozilla-taskcluster
provisioner-monitor
provisioner-staging
queue-taskcluster-net
scheduler-taskcluster-net
taskcluster-auth
taskcluster-auth-staging
taskcluster-aws-provisioner2
taskcluster-events
taskcluster-github
taskcluster-github-staging
taskcluster-hooks
taskcluster-httpbin
taskcluster-index
taskcluster-pulse
taskcluster-secrets
taskcluster-stats-collector
taskcluster-task-analysis
taskcluster-tools
taskcluster-treeherder
taskcluster-treeherder-staging
tc-queue-load-test
The two users didn't respond, so have now been removed from the Mozilla org (which helps with bug 1429354 / bug 1429335) and told to file a but to request re-addition:
apereira@mozilla.com
jhford@mozilla.com

There were no new users with 2FA not enabled since comment 36.
See Also: → 1429430
Checking now showed 5 users without 2FA enabled.

Two were related to bug 1432185 so have been removed pending their reply. Another two have been emailed, and the 5th is an email typo ('@mozlila.com`).
The latest version of the script that I've been using (it also calculates the apps affected by a particular user's lack of 2FA).
Attachment #8845507 - Attachment is obsolete: true
We now have this running in the Security Operations automation platform (slight mod to the script attached).

I'll wait to close this out until more of the email parts are documented on our end.

:emorely - can you email me (or attach here) the "standard email" you've referred to above?
Assignee: nobody → hwine
Flags: needinfo?(emorley)
Flags: needinfo?(emorley)

    
    

    
  
And great to hear! :-)

    
  
See Also: 1387250 → 

    
    

    
  
Our Heroku account now uses SSO, which handles 2FA for us.


Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: