761043 - Query timing attack allows extraction of private bug data

Status: RESOLVED FIXED

(Bugzilla :: Query/Bug List, defect)

Description

[Jаmes Kettle]

Attachment: proof of concept timing attack

Bugzilla's search function does not use a CSRF token. By using the onload() handler, an attacker can see how long a particular search takes. Searches that return one result consistently take longer than those that return no results. By repeatedly timing specially crafted searches an attacker may (slowly) extract the entirety of a bug's summary, or any other field. I've attached a proof of concept which will, with a bit of luck, map out the entire summary of any bugid given to it. It's currently keyed to #686451 but it ought to work with any bug you have permission to view. It's unpolished but I think it's reliable enough to make the point. 

The proof of concept attack could be prevented just by adding X-Frame-Options but I think adding a CSRF token to searches would be a far safer patch.

Comment 1

[Reed Loden [:reed]]

Your the second person today that commented that X-Frame-Options would solve a problem. However, Bugzilla (at least bugzilla.mozilla.org) has been sending X-Frame-Options: SAMEORIGIN for quite some time (bug 475894). Do you see of some page or some case where that header is not being sent?

CSRF tokens are meant to prevent the writing of data, not reading... Too many things would break if we required tokens for searches.

Comment 2

[Reed Loden [:reed]]

Ah, I just did this:

$ curl -I https://bugzilla.mozilla.org/buglist.cgi
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: pp-app-bugs01
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Strict-transport-security: max-age=2629744; includeSubDomains, max-age=2629744; includeSubDomains
Date: Mon, 04 Jun 2012 02:00:14 GMT
Refresh: 10; URL=query.cgi
Transfer-Encoding: chunked
Connection: Keep-Alive
X-frame-options: SAMEORIGIN, SAMEORIGIN

That seems like a problem... Note the repeating of Strict-Transport-Security and X-Frame-Options headers.

Comment 3

[Reed Loden [:reed]]

I can't reproduce that upstream, so I'll open an IT bug to look into the repeating issue. I don't think there's any bmo-specific code that could be causing it.

Comment 4

[Reed Loden [:reed]]

(In reply to Reed Loden [:reed] from comment #3)
> I can't reproduce that upstream, so I'll open an IT bug to look into the
> repeating issue. I don't think there's any bmo-specific code that could be
> causing it.

Filed bug 761046.

Comment 5

[Frédéric Buclin]

This is not exploitable, for the following reasons:

1) the noise is too high to get significant data. Here is what I get after 2 minutes with your PoC:

[Calibrating]
Average false time: 1831.3333333333333
Average true time: 1991.6666666666667
Threshold: 1911.5
Confirming: k
k
Confirming: kn
Reverted to k
Confirming: kn
kn
Confirming: knb
knb
Confirming: knby
Reverted to knb
Confirming: knba
Reverted to knb
Confirming: knbw
Reverted to knb
Confirming: knb
Confirming: knb
Reverted to knb
Confirming: knbg
Reverted to knb
Confirming: knb.
knb.

The script is going nowhere.

2) The script will run only as long as the user remains on the crafted HTML page, which isn't long enough to get significant data.

3) More importantly, as reed said in comment 1, Bugzilla is protected against this type of attacks thanks to the X-Frame-Options header.


I tested all Bugzilla installations listed on http://lpsolit.wordpress.com/bugzilla-usage-worldwide/ which are running 4.0 or newer, and there are 3 installations which have the same problem as described by reed in comment 2:

https://bugzilla.mozilla.org/
http://netbeans.org/bugzilla/
https://bugs.eclipse.org/bugs/

All other installations are fine and have the correct X-Frame-Options header. It would be interesting to know what these 3 installations have in common.

I'm leaving this bug as UNCO for now, but will probably mark it as WFM or INVALID later today, when I'm back from work.

Comment 6

[Jаmes Kettle]

Here is my output from running the poc for a couple of minutes:

[Calibrating]
Average false time: 1078.3333333333333
Average true time: 2398.3333333333335
Threshold: 1738.3333333333333
Confirming: c
c
Confirming: cs
cs
Confirming: csr
csr
Confirming: csrf
csrf
Confirming: csrf
Confirming: csrf
Reverted to csrf
Confirming: csrf.
csrf.
Confirming: csrf.i
csrf.i
Confirming: csrf.in
csrf.in
Confirming: csrf.in
Confirming: csrf.in
Reverted to csrf.in
Confirming: csrf.in.
csrf.in.
Confirming: csrf.in.b
csrf.in.b
Confirming: csrf.in.bu
csrf.in.bu
Confirming: csrf.in.bui
csrf.in.bui
Confirming: csrf.in.buil
csrf.in.buil
Confirming: csrf.in.build
csrf.in.build
Confirming: csrf.in.builde
csrf.in.builde
Confirming: csrf.in.builder
csrf.in.builder
Confirming: csrf.in.builder
Confirming: csrf.in.builder
Reverted to csrf.in.builder
Confirming: csrf.in.builder.
csrf.in.builder.
Confirming: csrf.in.builder.a
csrf.in.builder.a
Confirming: csrf.in.builder.ad
csrf.in.builder.ad
Confirming: csrf.in.builder.add
csrf.in.builder.add
Confirming: csrf.in.builder.addo
csrf.in.builder.addo

As you can see, there is a large gap between the time taken between true and false requests, and as such the attack collects information accurately and reasonably quickly. I'm not sure why your 'average true time' is so small. Perhaps you ran the poc on the Mozilla internal network, and it's local to the bmo server? Or maybe you don't have read-permission on bug #686451? Perhaps someone else could try the poc and see how successful it is for them.


Re: 2). I don't think it is terribly hard to get a user to leave a page open for 10 minutes. Just provide a link that opens to another tab, or a video, or some kind of realtime updating mechanism. 

Re 3) Yes, this is specific to bmo, I filed it in the wrong category.

Comment 7

[Frédéric Buclin]

(In reply to Frédéric Buclin from comment #5)
> 3) More importantly, as reed said in comment 1, Bugzilla is protected
> against this type of attacks thanks to the X-Frame-Options header.

Hum, I wrote a small PoC which tries to load buglist.cgi and index.cgi into an iframe, and while index.cgi is correctly blocked, buglist.cgi is always loaded into the iframe, even for installations which correctly pass X-Frame-Options: SAMEORIGIN. If I disable server push, buglist.cgi is no longer loaded into the iframe. So we must be doing something wrong here.

Comment 8

[Jаmes Kettle]

Attachment: TIming poc

As mentioned before, the poc only works if the bugID is set to one the target has permission to view. For a reason entirely opaque to me, it also fails if the bug is not private. Neither of these have any impact of the effectiveness of the attack. I have changed the target bug id to 761043 to ensure that the poc will work for anyone who can view it.

Comment 9

[Jаmes Kettle]

I have tested this poc in Firefox, IE8 and Chrome. It works fine in firefox and IE, and fails in chrome as chrome interprets the XFO header as disallowing framing.

Comment 10

[Frédéric Buclin]

Cc'ing Denis for his input about comment 5: Eclipse's Bugzilla also returns the wrong header for X-Frame-Options. It seems this only affects intallations running mod_perl (NetBeans also runs mod_perl, as well as bmo). It seems that disabling server push fixes the issue, at least partially.

Comment 11

[Denis Roy]

We're indeed running BZ on mod_perl.

Comment 12

[Jаmes Kettle]

Could you confirm whether the update poc works for you?

Comment 13

[Denis Roy]

I've copied the code over to bugs.eclipse.org and modified the queries somewhat, but It doesn't seem to go anywhere.

Comment 15

[Daniel Veditz [:dveditz]]

Confirmed in theory, not convinced it's entirely practical. It took gobs of time to get anywhere:

10 minutes to extract "csrf.timing.attack"
20 minutes to extract "csrf.timing.attack.allows.extraction"
25 minutes to extract "csrf.timing.attack.allows.extraction.ofzprivate."
30 minutes to extract "csrf.timing.attack.allows.extraction.ofzprivate.bug.b"

Whenever I started using the network success went to hell so I mostly left the machine alone but I'm not sure that will be true in practice. In the end I'm not sure very many bug titles tell you what you need to know (you wouldn't get from that title to your POC very easily).

I removed CSRF from the summary because that describes a different kind of vulnerability.

There's definitely a BMO (or load balancer?) configuration problem with the duplicate headers (bug 761043) and a Firefox problem failing to handle that case (bug 761655).

Comment 16

[Brandon Burton [:solarce]]

(In reply to Daniel Veditz [:dveditz] from comment #15)
> Confirmed in theory, not convinced it's entirely practical. It took gobs of
> time to get anywhere:
> 
> 10 minutes to extract "csrf.timing.attack"
> 20 minutes to extract "csrf.timing.attack.allows.extraction"
> 25 minutes to extract "csrf.timing.attack.allows.extraction.ofzprivate."
> 30 minutes to extract "csrf.timing.attack.allows.extraction.ofzprivate.bug.b"
> 
> Whenever I started using the network success went to hell so I mostly left
> the machine alone but I'm not sure that will be true in practice. In the end
> I'm not sure very many bug titles tell you what you need to know (you
> wouldn't get from that title to your POC very easily).
> 
> I removed CSRF from the summary because that describes a different kind of
> vulnerability.
> 
> There's definitely a BMO (or load balancer?) configuration problem with the
> duplicate headers (bug 761043) and a Firefox problem failing to handle that
> case (bug 761655).

In another bug, https://bugzilla.mozilla.org/show_bug.cgi?id=761046#c10 , we determined that the duplicate headers are coming from BMO, it's not a Zeus or Apache issue

Comment 17

[Denis Roy]

> In another bug, https://bugzilla.mozilla.org/show_bug.cgi?id=761046#c10 , we
> determined that the duplicate headers are coming from BMO, it's not a Zeus
> or Apache issue

It's not specific to BMO either, bugs.eclipse.org is also sending out duplicate headers.  Is there a separate bug for this, or can I get cc'd on bug 761046, which seems to be private?  Thanks

Comment 19

[Frédéric Buclin]

The duplicated header issue has been fixed in bug 761046 and has nothing to do with this bug. The problem described here is caused by bug 761667.

Comment 20

[Frédéric Buclin]

Attachment: iframe check

(this is just a check, ignore it)

Comment 21

[Frédéric Buclin]

That's interesting; it seems that Opera is still able to use this PoC to guess the bug summary despite it cannot load the page in a frame. It seems that the time taken to display the error message about Bugzilla not being allowed to be loaded in a frame is enough to guess what the bug summary is.

Comment 22

[Frédéric Buclin]

FYI, this bug has been fixed in Firefox 22. Is there still a browser which is affected by this problem?

Comment 23

[Jаmes Kettle]

The timing poc appears to still work fine in IE9, and it just about works in Opera 12.1

Comment 24

[Jаmes Kettle]

Hi, just a heads up that I'm going to publicly disclose this bug in April (2022). I'm not sure whether it's been fixed or not.

Comment 25

[Frederik Braun [:freddy]]

AFAICT, the PoC does not reproduce anymore due to various prevention mechanisms (rate limiting, X-Frame-Options, etc.)

Thanks for the heads up, James!

Comment 26

[Jаmes Kettle]

Could you make this bug public? Thanks!