1550366 - AWS SSRF to Pull AWS Metadata and Keys on - https://wpt.stage.mozaws.net/

Status: RESOLVED FIXED

(Cloud Services :: QA: Test Automation, defect)

Description

[Griffin Francis]

Hello.

You can access this domain through Github SSO at - https://wpt.stage.mozaws.net/

Within the Enter a Website URL Form submit the domain http://0251.00376.000251.0000376/latest/meta-data/public-keys/0/openssh-key the regular aws meta data URL is blocked.

Observe the attached screenshot where the keys are pulled from the webservice.

Regards,
Griffin.

Comment 1

[Griffin Francis]

Attachment: AWS Keys.png

Comment 2

[Caglar Ulucenk [:cag]]

Thanks for the report Griffin!

I've been talking to Griffin on Slack about this. As he mentioned the website requires SSO. Griffin is an NDA'd contributor, which also brings up this question: Should this website be accessible to NDA'd Mozillians, or only Mozilla staff?

Also, authenticated users can see the test history for all users, which shows some attempts from Griffin as a part of his test. This is fine, however we should probably clean them up to reduce exposure of AWS metadata content.

Let me try to find a contact for this.

Comment 3

[Caglar Ulucenk [:cag]]

: psiinon, :ulfr, do you know much about this service, and who/which team manages it?

It looks like there was an attempt to block the AWS metadata URL, however sounds like it's a soft control (we did not consider all encoding mechanisms).

In addition to the SSH key of the instance, someone could also retrieve the AWS credentials with this, so I suggest we should check the permissions on the account, perform an audit of abuse, and rotate credentials, as well as removing the test screenshot attempts from Griffin's tests.

Comment 4

[Caglar Ulucenk [:cag]]

Looking at the older bugs for a similar host, I was able to (hopefully) find the right team/component/contact etc.

Also cc'ing :relud as apparently he helped in a prior issue affecting a similar service (wpt-vpn.stage.mozaws.net)

Comment 5

[Griffin Francis]

Hi All.

Just thought I would fix my steps to reproduce this issue as reading over them they are a bit unclear. Impact is SSRF to RCE.

1. Authenticate via Github SSO at - https://wpt.stage.mozaws.net/ (Access appears limited to NDA Contributors, other instances of this install have not been behind SSO within the past.)

2. Within the Website URL form submit the following: http://0251.00376.000251.0000376/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance

3. Click Start Test. Allow it to load.

4. Once this has finished the Screenshot section of the service will display the AWS metadata ACCESS_KEY_ID and SECRET_KEY (Screenshot attached)

Will assume this has RCE impact and won't continue testing as per my discussion with Cag. Thanks.

Regards,
Griffin.

Comment 6

[Griffin Francis]

Attachment: AWS Secrets.jpg

Comment 7

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

I can also confirm the behavior that Griffin has suggested. Also NI'ing :relud.

Comment 8

[Greg G]

ckolos said wpt is :jbuck's these days

Comment 9

[Jon Buckley [:jbuck]]

WPT blocks certain IPs from being tested in https://github.com/WPO-Foundation/webpagetest/blob/ce90c2e56f4dd6d0098a1a116bf971fd7a61b909/www/runtest.php#L1663-L1664 . This including 169.254.169.254 but it doesn't handle octal-encoded IPs (I didn't even know that was a thing), or a DNS entry pointing at 169.254.169.254.

On the bright side, there's no IAM instance profile attached to these instances (wpt-driver-linux-stage-default instances in stage/us-east-1) so the AWS access keys don't let you do anything with them.

Fixing this will require an upstream fix, since the wpt-driver-linux-stage-default instances use an AMI provided by WPT.

:stepehend - Could you contact Patrick Meenan and let him know about this? I don't want to test this on the public WPT servers but I suspect if any of them are running in AWS they'll have this vulnerability too.

Comment 10

[Griffin Francis]

Hi. Thanks for the update. Would it be possible to apply for a CVE on this issue?

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Jon Buckley [:jbuck] from comment #9)

<snip>

Fixing this will require an upstream fix, since the wpt-driver-linux-stage-default instances use an AMI provided by WPT.

:stepehend - Could you contact Patrick Meenan and let him know about this? I don't want to test this on the public WPT servers but I suspect if any of them are running in AWS they'll have this vulnerability too.

Sorry for the lag; I've emailed Patrick with a PDF of this issue, and have also pinged him in a WebPerformance Slack channel.

Once he's had a chance to respond, and if it's OK to do so then, we can cc: him on this issue directly, to help expedite and clarify.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

I've just added Patrick to the bug; sorry for the volleys (email, bugmail, Slack) thus far about this, Pat!

Comment 13

[Patrick Meenan]

Fix pushed a few minutes ago: https://github.com/WPO-Foundation/wptagent/commit/9106fc42d08ab430c2b04becfa4a4c40624ee8a5

All of the agents will pick it up over the next hour naturally or if you reboot them all they will get it immediately.

There are so many ways to hide a URL fetch from a browser that I had to apply a fairly heavy hammer. After the test runs on the agent, it scans through all of the URLs that were requested by the browser and if any of them have a path that starts with /latest/meta-data it deletes all of the data instead of uploading and fails the test.

Here is what the result will look like: https://www.webpagetest.org/result/190509_J6_af6da68b9c4cdf279a81d63ddc2de2ce/

The requests are still made though so if it is possible to get the content of the metadata URL from JavaScript in a page (I'm hoping CORS prevents it) then it can in theory be exfiltrated as part of the test itself and posted to a 3rd-party server. If that is possible then the only solution that comes to mind is to add a route to the metadata IP during the test that makes the IP completely unroutable (I might look into doing that anyway).

Comment 14

[Patrick Meenan]

Change of plans. I null-route the metadata IP on the EC2 agents now so there is no way to access it at all from the agents once they are running: https://www.webpagetest.org/result/190509_TxJM_df0b550f952d523edeef2165d9c4b0e7/

Also just pushed so it will be live within the hour (or upon reboot).

If you are running Windows agents this doesn't help with that and the Windows EC2 images require routes to even be able to reach the metadata server and won't work if I block/enable them so I'd generally recommend avoiding Windows on EC2 until I kill the routes there as well.

Comment 15

[Griffin Francis]

Appreciate the prompt fix. Once this issue is resolved. Would we be able to publicly disclose this issue?

Comment 16

[Jon Buckley [:jbuck]]

Patrick, thank you very much for your quick and thorough turn-around on this bug!

I have confirmed that the fix has been applied to our runners and I get "Sorry, 169.254.169.254 is blocked from testing" now.

Griffin - I'd defer to Patrick and our security folks for public disclosure. Greg, Jonathan, what say you?

Comment 17

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

The fix has been live for a week, so I think we're all set here. Nice find :griffin!

Comment 18

[Patrick Meenan]

Griffin - No objection on my side to disclosing. I like to keep the project as transparent as possible.

Comment 19

[Griffin Francis]

Awesome. Appreciate the turn around with this fix. Would Mozilla be able to apply for a CVE on this issue for me once it has been made public?

Comment 20

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Griffin Francis from comment #19)

Awesome. Appreciate the turn around with this fix. Would Mozilla be able to apply for a CVE on this issue for me once it has been made public?

Griffin: Typically a CVE is assigned for a CPE (Customer Premise Equipment), or in other words, things that are installed with a customer (client/server) and thus need broad communication about the fix so consumers know to update. If I'm understanding this case correct, this is simply a test page where we're using a technique to bypass a string-based access control allowing access to MetaData info. I'd almost be inclined to put this into the territory of a TTP (Tactics, Techniques, and Procedures), specifically a technique for bypass. I think you'd likely find that this technique will be successful in other services and installation, but I'm not sure a CVE is the right fit here (please let me otherwise if you disagree). I lump this in a similar category to a subdomain takeover and having the technique being looking for 404's to identify their presence.

Comment 21

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

Additionally, I find the technique novel and very interesting. It is a great find!

Comment 22

[Griffin Francis]

Hi Jonathan. I contacted Mitre and my friend @sml555 who advised me that I should apply for a CVE. I was assigned CVE-2019-12161 for the issue. Appreciate Mozilla working with me on this!

Comment 23

[Griffin Francis]

Would this be eligible for a blanket RCE bounty?

Comment 24

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

(In reply to Griffin Francis from comment #23)

Would this be eligible for a blanket RCE bounty?

It wasn't marked resolved, so it wasn't making it to bounty panel discussion meeting. I have since fixed, we'll likely get to it next week. Thanks for the bump!

Comment 25

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

This was discussed with the bounty committee. We're classifying this as an auth bypass (or authr bypass) equivalent. Because the property is not an eligible property and we believe the impact to be small, we're going to HoF this submission. That said, we wanted to note that the technique was novel and interesting!