939431 - Persistent connections to Google servers

Status: RESOLVED INVALID

(Firefox :: Untriaged, defect)

Description

[Mark Straver]

I've noticed some really odd behavior when using the browser:
Whenever a Google-affiliated or google-owned site is visited, the browser starts making persistent connections to Google-operated servers (74.125.232.*). 

These connections continue *even after the tab is closed*. Most of them are SSL/TLS connections and HTTP connections.

I have verified (Using NetMon):
* This is not related to safebrowsing. I have completely disabled safebrowsing and the connections still occur. The target servers are also not SB servers s far as I can tell.
* These are not persistent HTTP connections. Waiting patiently on a blank page after closing a Google tab sees new connections made to new IPs and new conversations started on previously used IPs.
* Starting a new browser session and not visiting a Google site (instead visiting a website with no Google-owned elements) prevents these connections from ever being made, no matter how long one waits.

This is potentially exploitable behavior - if Google can do it, then so can others (including malicious site owners).
I'm not sure how this is done, I assume through some scripting creating background processes from a webpage that continue to run even after the tab is closed.

Comment 1

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

Is this a stock browser or are add-ons installed?

Comment 2

[Mark Straver]

Definitely a stock browser - I made sure to verify this issue when dropping it in a clean Windows VM with no add-ons installed (base Windows install + Microsoft NetMon + Firefox in the VM, nothing else installed)

Comment 3

[Mark Straver]

An idea: is it possible that this is done through (or a side effect of) Google's SPDY protocol?

Comment 4

[:Gavin Sharp [email: [e-mail]]]

When did you start noticing this, and which build are you running?

Seer may also be related (bug 881804).

Comment 5

[Mark Straver]

It was brought to my attention by one of my forum users very recently who saw this in his firewall as part of an investigation into background traffic eating up his monthly allotment of bandwidth. I don't know how long this behavior has been present in the browser, though.

This is on the 24-ESR branch.

Comment 6

[Patrick McManus [:mcmanus]]

(In reply to Mark Straver from comment #0)
> I've noticed some really odd behavior when using the browser:
> Whenever a Google-affiliated or google-owned site is visited, the browser
> starts making persistent connections to Google-operated servers
> (74.125.232.*). 

> * These are not persistent HTTP connections. Waiting patiently on a blank
> page after closing a Google tab sees new connections made to new IPs and new
> conversations started on previously used IPs.


I think that's the most interesting part of your report.. otherwise most of it could be explained by the global persistent connection pool and perhaps some keep alive traffic. But new connections are clearly something different. (and I would not thing spdy related in any particular way).

Given that you can reproduce this (I cannot) - the most interesting advice I can give is to make an HTTP log:

https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging?redirectlocale=en-US&redirectslug=HTTP_Logging

That will log all the network activity you do through the firefox networking stack. We should be able to at least see in there what the transactions think they are doing.

Comment 7

[Mark Straver]

Attachment: logs.zip

Attached a log pair (mozilla logging + netmon logging of the same session) of a google search, then closing the google tab and leaving just an about:blank tab left.

It's possible that I haven't properly read the NetMon output as far as connections to new IPs goes (some may have been background traffic of normal Firefox stuff. Of note I noticed that Firefox still connects to an fhr server while fhr is disabled in Options... maybe something for a different bug.)

The data I see -could- be some form of keepalive connections but what still makes it suspicious is that the only connections with chatter on them this way are connections to google servers. With exchange of TLS application data, not just TCP keepalive packets. I looked at similar connections to another https server and it simply shows the closing of the TLS and then the TCP channels (which would be the pool you talked about, I gather). No exchange of application data there.

In addition: Restoring the previous session (with just the blank tab) in a new browser session saw it immediately connect to Google again, with TLS chatter all throughout the session - color me confused, since there was nothing in the session but a blank tab, unless the session stored whatever is making these connections...?
If I don't restore the session, nothing like this happens.

(sorry if anything is unclear)

Comment 8

[Patrick McManus [:mcmanus]]

These are all the urls from the log

2013-11-20 02:42:16.378000 UTC - 0[f0f140]: uri=http://www.google.com/
2013-11-20 02:42:16.425000 UTC - 0[f0f140]: uri=http://www.google.com/
2013-11-20 02:42:16.534000 UTC - 0[f0f140]: uri=https://www.google.com/
2013-11-20 02:42:16.550000 UTC - 0[f0f140]: uri=https://www.google.com/
2013-11-20 02:42:16.628000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:16.643000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:16.753000 UTC - 0[f0f140]: uri=http://gtglobal-ocsp.geotrust.com/
2013-11-20 02:42:16.753000 UTC - 0[f0f140]: uri=http://gtglobal-ocsp.geotrust.com/
2013-11-20 02:42:17.331000 UTC - 0[f0f140]: uri=https://www.gstatic.com/og/_/js/k=og.og.en_US.bXik5
2013-11-20 02:42:17.331000 UTC - 0[f0f140]: uri=https://www.google.com/images/icons/product/chrome-
2013-11-20 02:42:17.331000 UTC - 0[f0f140]: uri=https://www.google.com/images/srpr/logo11w.png
2013-11-20 02:42:17.347000 UTC - 0[f0f140]: uri=https://ssl.gstatic.com/gb/images/v1_53a1fa6a.png
2013-11-20 02:42:17.362000 UTC - 0[f0f140]: uri=https://www.gstatic.com/og/_/js/k=og.og.en_US.bXik5
2013-11-20 02:42:17.362000 UTC - 0[f0f140]: uri=https://www.google.com/xjs/_/js/k=xjs.s.en_US.dtkly
2013-11-20 02:42:17.378000 UTC - 0[f0f140]: uri=https://apis.google.com/_/scs/abc-static/_/js/k=gap
2013-11-20 02:42:17.425000 UTC - 0[f0f140]: uri=https://www.google.com/images/icons/product/chrome-
2013-11-20 02:42:17.425000 UTC - 0[f0f140]: uri=https://www.google.com/images/srpr/logo11w.png
2013-11-20 02:42:17.456000 UTC - 0[f0f140]: uri=https://ssl.gstatic.com/gb/images/v1_53a1fa6a.png
2013-11-20 02:42:17.472000 UTC - 0[f0f140]: uri=https://www.google.com/xjs/_/js/k=xjs.s.en_US.dtkly
2013-11-20 02:42:17.472000 UTC - 0[f0f140]: uri=https://apis.google.com/_/scs/abc-static/_/js/k=gap
2013-11-20 02:42:17.565000 UTC - 0[f0f140]: uri=https://www.google.com/extern_chrome/4031ae856d725a
2013-11-20 02:42:17.565000 UTC - 0[f0f140]: uri=https://www.google.com/xjs/_/js/k=xjs.s.en_US.dtkly
2013-11-20 02:42:17.581000 UTC - 0[f0f140]: uri=https://www.google.com/textinputassistant/tia.png
2013-11-20 02:42:17.597000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:17.597000 UTC - 0[f0f140]: uri=https://www.google.com/extern_chrome/4031ae856d725a
2013-11-20 02:42:17.675000 UTC - 0[f0f140]: uri=https://www.google.com/xjs/_/js/k=xjs.s.en_US.dtkly
2013-11-20 02:42:17.706000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:17.706000 UTC - 0[f0f140]: uri=https://www.google.com/textinputassistant/tia.png
2013-11-20 02:42:17.706000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:17.706000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:17.800000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:17.800000 UTC - 0[f0f140]: uri=https://www.google.com/images/nav_logo170.png
2013-11-20 02:42:17.815000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?v=3&s=webhp&action=&
2013-11-20 02:42:17.815000 UTC - 0[f0f140]: uri=http://clients1.google.com/ocsp
2013-11-20 02:42:17.815000 UTC - 0[f0f140]: uri=https://www.google.com/images/nav_logo170.png
2013-11-20 02:42:17.815000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?v=3&s=webhp&action=&
2013-11-20 02:42:17.831000 UTC - 0[f0f140]: uri=https://www.google.com/favicon.ico#-moz-resolution=
2013-11-20 02:42:17.831000 UTC - 0[f0f140]: uri=https://www.google.com/favicon.ico#-moz-resolution=
2013-11-20 02:42:46.003000 UTC - 0[f0f140]: uri=https://fhr.data.mozilla.com/1.0/submit/metrics/ab7
2013-11-20 02:42:46.003000 UTC - 0[f0f140]: uri=https://fhr.data.mozilla.com/1.0/submit/metrics/ab7
2013-11-20 02:42:46.518000 UTC - 0[f0f140]: uri=http://gtssl-ocsp.geotrust.com/
2013-11-20 02:42:46.518000 UTC - 0[f0f140]: uri=http://gtssl-ocsp.geotrust.com/
2013-11-20 02:42:46.925000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=1
2013-11-20 02:42:46.925000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=1
2013-11-20 02:42:47.034000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?atyp=i&ct=1&cad=1&rs
2013-11-20 02:42:47.050000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=2
2013-11-20 02:42:47.050000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?atyp=i&ct=1&cad=1&rs
2013-11-20 02:42:47.050000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=2
2013-11-20 02:42:47.237000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=3
2013-11-20 02:42:47.237000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=3
2013-11-20 02:42:47.347000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=4
2013-11-20 02:42:47.347000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=4
2013-11-20 02:42:47.550000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=5
2013-11-20 02:42:47.550000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=5
2013-11-20 02:42:47.675000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=6
2013-11-20 02:42:47.675000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=6
2013-11-20 02:42:47.893000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=7
2013-11-20 02:42:47.893000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=7
2013-11-20 02:42:48.081000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=8
2013-11-20 02:42:48.081000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=8
2013-11-20 02:42:48.190000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=9
2013-11-20 02:42:48.190000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=9
2013-11-20 02:42:48.331000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=1
2013-11-20 02:42:48.331000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=1
2013-11-20 02:42:48.456000 UTC - 0[f0f140]: uri=https://www.google.com/images/nav_logo170_hr.png
2013-11-20 02:42:48.518000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=1
2013-11-20 02:42:48.518000 UTC - 0[f0f140]: uri=https://www.google.com/xjs/_/js/k=xjs.s.en_US.dtkly
2013-11-20 02:42:48.534000 UTC - 0[f0f140]: uri=https://www.google.com/images/nav_logo170_hr.png
2013-11-20 02:42:48.550000 UTC - 0[f0f140]: uri=https://www.google.com/s?gs_rn=32&gs_ri=psy-ab&cp=1
2013-11-20 02:42:48.550000 UTC - 0[f0f140]: uri=https://www.google.com/xjs/_/js/k=xjs.s.en_US.dtkly
2013-11-20 02:42:48.550000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?v=3&s=web&action=&ei
2013-11-20 02:42:48.550000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?v=3&s=web&action=&ei
2013-11-20 02:42:50.784000 UTC - 0[f0f140]: uri=https://www.google.com/search?sclient=psy-ab&q=test
2013-11-20 02:42:50.784000 UTC - 0[f0f140]: uri=https://www.google.com/search?sclient=psy-ab&q=test
2013-11-20 02:42:51.143000 UTC - 0[f0f140]: uri=https://www.google.com/search?sclient=psy-ab&q=test
2013-11-20 02:42:51.159000 UTC - 0[f0f140]: uri=https://www.google.com/search?sclient=psy-ab&q=test
2013-11-20 02:42:51.706000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?atyp=i&ct=phandle&ca
2013-11-20 02:42:51.706000 UTC - 0[f0f140]: uri=https://www.google.com/images/phd/gfl.gif
2013-11-20 02:42:51.706000 UTC - 0[f0f140]: uri=https://www.google.com/images/phd/px.gif
2013-11-20 02:42:51.722000 UTC - 0[f0f140]: uri=https://www.google.com/gen_204?atyp=i&ct=phandle&ca
2013-11-20 02:42:51.722000 UTC - 0[f0f140]: uri=https://www.google.com/images/phd/gfl.gif
2013-11-20 02:42:51.722000 UTC - 0[f0f140]: uri=https://www.google.com/images/phd/px.gif
2013-11-20 02:43:43.706000 UTC - 0[f0f140]: uri=https://addons.mozilla.org/blocklist/3/%7Bec8030f7-
2013-11-20 02:43:43.706000 UTC - 0[f0f140]: uri=https://addons.mozilla.org/blocklist/3/%7Bec8030f7-
2013-11-20 02:43:44.206000 UTC - 0[f0f140]: uri=http://ocsp.digicert.com/
2013-11-20 02:43:44.206000 UTC - 0[f0f140]: uri=http://ocsp.digicert.com/
2013-11-20 02:43:44.362000 UTC - 0[f0f140]: uri=http://ocsp.digicert.com/
2013-11-20 02:43:44.362000 UTC - 0[f0f140]: uri=http://ocsp.digicert.com/

Comment 9

[Patrick McManus [:mcmanus]]

None of the uris in the log jump out as unusual for doing a search or for happening a long time after your search. Its hard to make a good correlation between your 2 logs because the netmon timestamps do not match, but I presume that 2:42:51.722 request from the previous comment is the last one associated with your search (and then you close the tab). There doesn't seem to be any web related activity after that other than the background task that appears to be updating the addon security list.

I've grepped the log for all activity on those sessions after the 2:42:51 mark and then excluded responses that were still finishing up in the next second or so.

After that point all I find is a small amount of normal chatter than can be explained by the global connection pool and a trivial amount of keepalive chatter.

Specifically it is made up of the closing of persistent connections, SPDY PING requests and replies (this is the spdy specific keep alive mechanism), some SPDY SETTINGS frames (which are used to optimize the network protocol.. they say things like "my sending window grew to 128KB over the last minute"), and the SPDY GOAWAY closing handshake. It doesn't add up to any important bandwidth - less than 200 bytes in total (plus the tcp/ip/ssl headers) spread over 3 minutes.. you can read the log to see what is in the frames if you like.

This is a self limiting condition - the sockets do close down after about this period of inactivity and stop talking at all.

I think this can be marked as INVALID - let me know if you disagree. We should also remove the security screen here but I want to give a round trip of time before doing that in case there is some reason not to.

Comment 10

[Patrick McManus [:mcmanus]]

Thanks Mark!

> Of note I noticed that Firefox still connects to an fhr
> server while fhr is disabled in Options... maybe something for a different
> bug.)
> 

yes - please file a different bug. the fhr folks surely aren't looking here.

I appreciate you filing bugs on stuff that seems wrong to you even if it turned out to be expected.

Comment 11

[Mark Straver]

All right, I'll file a different bug for the fhr issue.

If you say it's normal for the global connection pool (I wasn't aware Firefox used this mechanism, TBH, before this bug) in combination with SPDY, then that's as good an explanation as any for what struck me as odd.