1283675 - Large search suggestions can hang or crash browser

Status: RESOLVED WORKSFORME

(Firefox :: Search, defect, P3)

Description

[research]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0
Build ID: 20160606113944

Steps to reproduce:

If a search provider is installed that serves suggestions without SSL (such as ebay as we filed in a separate bug #1283672), an attacker can serve a large response and crash the browser. It may also be possible to inject data to cause an RCE but we did not test that use case. 

1. Install DNSMASQ:
sudo apt-get install dnsmasq

2. Install NGINX:
sudo apt-get install nginx

3. Modify the /etc/hosts file to add the following entries to map to the IP of the local computer (varies by vendor of the phone):

192.168.1.x autosug.ebay.com

4. Configure /etc/dnsmasq.conf file to listed on the IP:
listen-address=192.168.1.x

5. Restart DNSMASQ:
sudo /etc/init.d/dnsmasq restart

6. Use fallocate to create a file in "/var/www/html/"
sudo fallocate -l 5G autosug

7. Modify DNS settings on the test machine or the same machine to point to "192.168.1.x". If same machine, modify resolve.conf as follows:
nameserver 192.168.1.x

8. Start FireFox, go to search settings and choose "Ebay" as the default search provider. Enable suggestions.
9. Open new tab and try to type something in the search box.


Actual results:

On Linux, the FireFox process eventually runs out of memory and hangs the system. We did not try to exploit this to get RCE.

On Android, the phone crashes and reboots but not consistently.


Expected results:

The response processing should be capped at a reasonable number and stop processing the response after that number.

Comment 1

[:Gijs (away/busy until Mar 10)]

Florian, can you investigate?

Comment 2

[Florian Quèze [:florian]]

What happens if you just load http://192.168.1.x/autosug from the url bar?

Comment 3

[research]

When served as an HTML file, the browser does not crash or hang. When not served as an HTML file, the browser asks to download the file, and does not hang or crash,

Comment 4

[Daniel Veditz [:dveditz]]

Do you have a test server set up on the internet we could try?

Could you paste some of the crash ID's you get when trying this? (found by opening about:crashes )

Comment 5

[research]

We don't have either right now, but will re-try and add info here

Comment 6

[research]

We retested this and can provide the following crash ID for Android:
https://crash-stats.mozilla.com/report/index/cb9a85e5-5c7c-4264-8193-8e0332160711

HOWEVER, on desktop Linux (ubuntu), the browser never crashes - instead it hangs and hangs the entire OS requiring a reboot of the computer.

We do not have a public server for this since this was tested in a local env.

Also, to replicate this for Android, it requires installing the Ebay search and enabling suggestions since it is only enabled by default on desktop.

Comment 7

[research]

For the Linux hang, to clarify, we are testing on a system with an SSD drive and swap disabled. It is possible that with swap enabled, it would get really slow but would not hang.

Comment 8

[Florian Quèze [:florian]]

We are parsing suggestions at http://searchfox.org/mozilla-central/source/toolkit/components/search/SearchSuggestionController.jsm#283 it's a simple JSON.parse. Does a JSON.parse call on a 5GB file also cause OOM errors? If so the bug isn't search specific.

Comment 9

[research]

It could also be an error in XMLHTTPRequest running out of memory as per this line:
http://searchfox.org/mozilla-central/source/toolkit/components/search/SearchSuggestionController.jsm#252

From a security standpoint, we are highlighting this specifically related to search because it can be triggered by a remote attacker without visiting a specific page but by simply intercepting non-SSL search suggestion traffic and modifying it. To replicate this on a regular webpage, the attacker would need to find someother place JSON.parse is used or push their own Javascript in.

What is also interesting is that this DOES NOT happen on either Firefox for Windows or MacOS. There seems to be an implementation difference between those platforms and Linux/Android.

Comment 10

[research]

It is also possible that we do not release the XHR object here:
http://searchfox.org/mozilla-central/source/toolkit/components/search/SearchSuggestionController.jsm#285

as we do later on near line 378:

this._request = null;

Comment 11

[:Gijs (away/busy until Mar 10)]

(In reply to research from comment #9)
> It could also be an error in XMLHTTPRequest running out of memory as per
> this line:
> http://searchfox.org/mozilla-central/source/toolkit/components/search/
> SearchSuggestionController.jsm#252
> 
> From a security standpoint, we are highlighting this specifically related to
> search because it can be triggered by a remote attacker without visiting a
> specific page but by simply intercepting non-SSL search suggestion traffic
> and modifying it. To replicate this on a regular webpage, the attacker would
> need to find someother place JSON.parse is used or push their own Javascript
> in.
> 
> What is also interesting is that this DOES NOT happen on either Firefox for
> Windows or MacOS. There seems to be an implementation difference between
> those platforms and Linux/Android.

What build are you using on Linux? An offical 64-bit build? I'm not aware of any platform-specificity in any of the code involved here, so this seems very unlikely. :-\

Comment 12

[research]

We are using this build for Linux:
https://hg.mozilla.org/releases/mozilla-release/rev/b0310cb90fd0158abd0e92850a47768649ba3d77

And for Android, we are using the version from the Play Store.

What we noticed when testing on Windows and MacOS, is that the big search suggestion file does not get entirely retrieved by FireFox, it stops around 30 MBs.

Also, for Linux, on every search suggestion it gets the big 1 GB file and consumes 1 GB of extra memory. That memory is not released according to top.

Comment 13

[research]

We did some more testing and the error is certainly not in JSON.parse. Here is what we did with a very big test2.json file. The error we got in the dev console was "allocation size overflow" on the JSON.parse call. This seems to revolve around XHR, perhaps not being released?

    <html>
    <head>
    <title></title>
    <script>
    function listener() {
    console.log('got request');
    var json = JSON.parse(this.responseText);
    console.log('parsed request');
    }

    function test() {
    var req = new XMLHttpRequest();
    req.addEventListener("load", listener);
    req.open("GET", "http://localhost/test2.json");
    req.send();

    }
    </script>
    </head>
    <body>
    <h1>test</h1>
    <button onClick="test()">test</button>
    </body>
    </html>

Comment 14

[research]

We are able to replicate this problem by doing lots of XHR requests and serving a large file. The memory hang issue happens in Linux, the crash happens in Android. For Windows and MacOS, the memory consumption and hang happens also, but not as bad as Linux (browser hangs, not the OS). The previous differences we discussed seem to have been artifacts of our network setup so the Windows machine triggered a timeout due to a slower connection. This may also be a mitigating factor for the attack.

The replication steps would be easy then:
1. Setup a webserver.
2. Use fallocate to create a large "test2.json" file.
3. Add an HTML file as follows:

<html><head><title></title>
<script>
function listener() {  console.log('got request');  }
function test() {
    var req = new XMLHttpRequest();
    req.addEventListener("load", listener);
    req.open("GET", "http://localhost/test2.json");
    req.send();
}
</script></head><body>
<button onClick="test()">test</button>
</body></html>

4. Load in firefox and click on the button lots of times to trigger many requests. Observe usage of memory go high.

Comment 15

[research]

We have publicly disclosed this:
https://wwws.nightwatchcybersecurity.com/2016/07/26/research-crashing-browsers-remotely-via-insecure-search-suggestions/

Comment 16

[Florencia Di Ciocco, NI to vbalducci]

Hi, I'm closing this bug as WFM since I think this has been done. If I'm wrong, please do reopen it.