628197 - (DNT) Implement do-not-track HTTP header

Status: VERIFIED FIXED

(Core :: Networking: HTTP, enhancement)

Description

[Sid Stamm [:geekboy or :sstamm]]

Attachment: proposed patch

Opting-out of tracking for behavioral advertisements is currently
suboptimal: either you set a bunch of "opt-out" cookies or you block the
ad networks using an ad-blocking add-on.  As a step towards improving
this, we should implement a way for users to send a universal opt-out
signal to advertising networks (and others) expressing their desire to
opt out.

Attached is a rough first-round patch that, when the boolean preference
"privacy.trackingheader.enabled" is set to "true", sends an opt-out HTTP
header:

 X-Tracking-Choice: do-not-track

Perhaps it's also worth considering an opt-in statement ("ok-to-track")?

Comment 1

[Giorgio Maone [:ma1]]

Why inventing yet another header ("X-Tracking-Choice") rather than reusing the "X-Do-Not-Track" proposal, which is already implemented by NoScript and Adblock Plus, and also endorsed by yourself in the "Universal Behavioral Advertising Opt-out" add-on?

Comment 2

[Mike Beltzner [:beltzner, not reading bugmail]]

Giorgio: is X-Do-Not-Track implemented by many sites? The biggest advantage I can see to Sid's proposed header is that it clearly allows multiple values, "do-not-track" being one of them, which may provide for a more robust and collaborative development of the header.

Comment 3

[Giorgio Maone [:ma1]]

@Mike: the http://donottrack.us/ guys are maybe in a better position to answer, but even if, as I suspect, the implementing sites were not many, if any, there was already lots of discussion/feedback making "Do Not Track" a recognizable "brand".

On the other hand, I can see how "do-not-track" is still kept as a possible header value, so I'd say it's bad but not *so* bad from a marketing perspective as dropping it entirely. What other values are realistically foreseeable ("ok-to-track" being obviously not one of them)?

Comment 4

[Christopher Soghoian]

First off, I really appreciate the fact that Mozilla has embraced the idea of the do not track header. This shows bold leadership, and a willingness to take risks in order to do everything possible to protect user privacy.

However, from a policy and political perspective, I think that changing the header spec at this point is a serious mistake.

The header as implemented in the add-on that Sid and I created is a year and a half old. The Stanford guys who run www.donottrack.us have had their site up for at least 6 months, if not longer. During that time, no one complained about the specific format of the header -- in fact, no one really argued about it at all.

Sure, the header as we originally proposed it could be better, and, after going through a lengthy technical committee process, will likely look very different. 

However, by changing the header today, Mozilla is giving the advertising industry a propagnda gift, wrapped in a pretty bow. Once the industry gives up on their current approach (which essentially comes down to putting their head in the sand, and pretending that there isn't a need for any improvement to the current opt out / tracking controls), it will be able to argue that the browser vendors and other technologists cannot agree on the specific format of the header. They will be able to truthfully state that the most popular implementations (Adblock Plus and NoScript) differ in format from Mozilla, and so therefore, the industry will want to wait until the technical community, via a lengthy committee process, agrees upon the format of a header.

Remember, the ad networks are not engaging in good faith negotiations. They do everything within their power to slow things down, because each day without action is another day that they can track the hell out of users.

During the last week, when Mozilla was quietly briefing policy groups in DC, it could have had a quiet word with Giorgio, Vladimir, Arvind and Jonathan and asked them to all change their code to comply with Mozilla's preferred header format.. and as Mike correctly notes, no sites currently look for the header, so I doubt if those guys would have had any problem with changing their code.

Comment 5

[Mike Shaver (:shaver emeritus)]

Why would that quiet word be any less effective now?  Did a lot of sites adopt over the weekend?

Comment 6

[Mike Beltzner [:beltzner, not reading bugmail]]

While I don't want this to distract from the first and most obvious value of the proposed header (do-not-track) I can see other values being proposed to allow different types of tracking, such as:

 - tracking for unique visitor counts
 - tracking within domain but not shared to third parties
 - tracking for session only
 - tracking for N days only

Again, I'm spitballing here, and just saying that there may well be values between "yes" and "no" for the header, so I like how Sid's proposed header name allows for that future flexibility. If we're not losing a bunch of existing work by renaming it, I feel like there's value in doing so, and as you say, "do-not-track" is maintained as a "brand" in the existing structure.

Comment 7

[Giorgio Maone [:ma1]]

@Mike:
the examples you're giving could be expressed as well as X-Do-Not-Track values, such as:

- X-Do-Not-Track: behavior
  (may track visits and explicit preferences, but not behavioral patterns and implicit preferences)
- X-Do-Not-Track: shared
  (don't share tracking information with 3rd party)
- X-Do-Not-Track: persistent
  (don't track across session)
- X-Do-Not-Track: 3days
  (don't store tracking info longer than 3 days)

or mixed values, such as

X-Do-Not-Track: behavior+shared,15days
(don't share behavioral info with 3rd parties, and anyway don't store any info longer than 15 days)

The legacy value "1" would be synonym of "anything".

I suspect allowing all this expressiveness would quickly become messy, though...

Just to clarify, I've no vested interest in keeping X-Do-Not-Track as the header name: I can easily switch to any other name in 30 seconds and turn off the option on supporting Firefox builds. I just feel this is a suboptimal move from a communication standpoint.

Comment 8

[Wladimir Palant]

I have no problem with changing my code either, it didn't even make it into the stable release yet. But I also feel that this creates a need for communication that is not justified. As Giorgio correctly noticed, "please do track" isn't something that anybody will want to use (it's the "default" anyway). We might have different levels of "no not track" - but that isn't a problem with the current header as well. And there might be more parties involved here, e.g. Microsoft with IE9. They announced privacy protection features which might mean X-Do-No-Track header as well. If they release with a different technical solution than Firefox that will create just the situation that Christopher Soghoian is worried about (as if XDomainRequest wasn't enough fun). So I would rather stick with the existing proposal unless there is a really important reason why it is not acceptable.

Comment 9

[Jonathan Mayer]

Arvind and I would prefer to stick with "X-Do-Not-Track: 1" for now.  Our aim is a one-click, universal opt-out that conforms to user expectations, and we think the right way to achieve that is a binary choice.  As users learn more about web privacy, we're certainly open to considering new values.  We're also very concerned that a multi-valued header could splinter the proposal, leading to unneeded competition, user confusion, and gaming.

Comment 10

[Mike Beltzner [:beltzner, not reading bugmail]]

Everyone: the suggestion to use the old header name has been lodged. +1 comments no longer welcome.

Comment 11

[Peter Eckersley]

Mike: with respect to "fine grained" tracking controls, such as "only domains x y and z may track me" or "tracking for time period X is okay": those controls should not be built into the client-side request.  Keep it simple.

Nothing prevents sites from offering these more complicated policies or responses if the header is in place.  For instance, a 1st party web page can say: "Dear cookied/logged in user X: we see you've asked not to be tracked; we will respect that request, and you can read the <details here>.  We have some features which require tracking to work; you can reenable those by clicking <here>; if you do that, we promise we'll delete our logs after 7 days".  Etc Etc.  We can also develop some standards for server-to-client header messages about this stuff, but those don't need to be in the first client side implementations.

Comment 12

[Sid Stamm [:geekboy or :sstamm]]

(In reply to comment #11)
> Mike: with respect to "fine grained" tracking controls, such as "only domains x
> y and z may track me" or "tracking for time period X is okay": those controls
> should not be built into the client-side request.  Keep it simple.

I agree, and think maybe in the *future* the client software could broker/manage this information ... sending out the header as needed.  First step: send header.  Step for later: figure out good UI for deciding when to send header.

Comment 13

[Brendan Eich [:brendan]]

Shorter is better -- I laughed at the TE: (Transfer-Encoding) header when I heard of it, but it wins.

http://en.wikipedia.org/wiki/List_of_HTTP_header_fields

/be

Comment 14

[chris hofmann]

re: comments 0, 6 and 7 

Sounds like we are pretty committed to the default factory setting be something low impact like "ok-to-track", but we are also committed to raising the visibility of this user choice, and in finding ways to get users to participate in evaluating and making that choice.  

Maybe with something like a "do not tracking/targeted ads choice day"

Sid and I talked the other day about a few different states that could reflect if the user had actually been involved in making the choice of the state of the header or were just going along with factory defaults of each browser.

Metrics will be needed to understand the effectiveness of default settings user choice evangelism campaigns.  For these metrics we need to consider something like:

                      factor default     user set
do not track -                             B
ok-to-track  -             A               C
(other tracking states?)

A matrix like that would enable stats collection like:

A + B + C  = number of visits to a site or an ad network were from browsers that supported the do-not-track feature.

C = % of users that asked for targeted ads

B = % of users that asked not to be tracked.

A = % of users that were unaware they had this option 
      or taken the time to make a choice

This would help out with understanding the effect of any evangelism efforts, and help content providers and ad networks to understand the importance of this feature to their viewers, and to help them make informed decisions on how much work to put into the their backend to support notions of do-not-track.

Comment 15

[chris hofmann]

Actually the choices look more like this, with an 'unknown' status being used to express that the user had not made a choice.  'Unknown' might be the absence of the header, or it could be a header with 'unknown' as the setting.  I think both have advantages.

                      factory default     user set
unknown                    A
do-not-track -                             B
ok-to-track  -                             C

Comment 16

[Anthon Pang]

Any thoughts on a read-only property added to the navigator object (or document, if it's going to be per site) that would allow JavaScript to query this setting?

Comment 17

[David E. Ross]

Does anyone really believe that the most abusive sites relative to tracking would actually honor a do-not-track request?  Somehow, I feel that this RFE -- it is definitely NOT a "normal" bug -- will prove to be a waste of effort.

Comment 18

[Peter Adams]

(In reply to comment #16)
> Any thoughts on a read-only property added to the navigator object (or
> document, if it's going to be per site) that would allow JavaScript to query
> this setting?

I fully second this as most third party tracking and Analytics (Google Analytics, omniture, Open Web Analytics) serve up static JavaScript libs that inspect the Dom and then issue GET requests for logging.

Those solutions are going to have a hard time moving from simply serving up static js libs

Comment 19

[Peter Adams]

(In reply to comment #16)
> Any thoughts on a read-only property added to the navigator object (or
> document, if it's going to be per site) that would allow JavaScript to query
> this setting?

I fully second this as most third party tracking and Analytics tools (Google Analytics, omniture, Open Web Analytics) serve up static JavaScript libs that inspect the Dom and then issue GET requests for logging.

Those solutions are going to have a hard time moving from simply serving up static js libs to now having to inspect headers (which is not insignificant overhead when you are serving up the lib on high volume sites).

However a read only JavaScript variable would be very easy for these tools to inspect and deal with.

Comment 20

[Gen Kanai [:gen]]

Lauren Weinstein's comments on the Do-not-track HTTP header are posted to his blog:

http://lauren.vortex.com/archive/000803.html

Comment 21

[Sid Stamm [:geekboy or :sstamm]]

Attachment: proposed patch

Patch refresh to include feedback such as shorten the header immensely and remove X-.  ("DNT: 1")

Comment 22

[Johnny Stenback (:jst)]

Comment on attachment 507663 [details] [diff] [review]
proposed patch

Honza, could you review this patch? I'm no necko peer...

Comment 23

[[e-mail]]

Comment on attachment 507663 [details] [diff] [review]
proposed patch

Wow, this patch is a hot potato. I'll jump in since I'm a necko peer. ;)

>diff -r bdfa2b67da3e -r 91c8dc761b51 modules/libpref/src/init/all.js

>+// Enable "don't track me" signal via HTTP header
>+pref("privacy.donottrackheader.enabled",    false);

'Enable' is disingenuous since you're actually disabling it. Perhaps 'Disable by default'?

You also don't need this at all, since nsHttpHandler defaults to disabled. That'd make it a hidden pref rather than one the user can see in about:config by default. I don't mind either way.

r=dwitte!

Comment 24

[Sid Stamm [:geekboy or :sstamm]]

Attachment: patch

Thanks dwitte!  Updated comment appropriately.

Comment 25

[Johnny Stenback (:jst)]

Comment on attachment 507671 [details] [diff] [review]
patch

a=jst!

Comment 26

[Johnny Stenback (:jst)]

Not blocking on this, but we should land the approved patch!

Comment 27

[Anthon Pang]

Just an 11th hour plea to please add a readable property via JavaScript.

Consider the following use case:
 * user visits site A with DNT disabled
 * site A loads a tracking script from third party site G
 * G's script creates first party tracking cookies (i.e., domain A)
 * user revisits site A with DNT enabled
 * site A wants to comply with the user's request...

Without a property:
 * site A could dig into the implementation of G's obfuscated script to delete the tracking cookies, or
 * site G could send different versions of the script depending on the presence of the DNT header; this script would not be cacheable by UAs or proxies

With a property:
 * site G's script can check for the property before sending the tracking request to its servers; if the first party tracking cookie(s) exist, then it will delete them

Comment 28

[Sid Stamm [:geekboy or :sstamm]]

Anthon: that's bug 629535.

Comment 29

[Jonathan Watt [:jwatt]]

Pushed http://hg.mozilla.org/mozilla-central/rev/6963333a74d1

Comment 30

[[Baboo]]

I wonder if this will break router configuration pages…

Comment 31

[Mike Shaver (:shaver emeritus)]

I would be quite surprised if routers did anything with that header, and browsers send additional headers all the time.

Comment 32

[Giorgio Maone [:ma1]]

(In reply to comment #31)
> I would be quite surprised if routers did anything with that header, and
> browsers send additional headers all the time.

He's asking probably because NoScript's implementation needed to whitelist private addresses in order to prevent basic authentication issues with some home router web consoles (apparently based on Apache, which makes this even weirder), see http://forums.informaction.com/viewtopic.php?f=7&t=5626

Comment 33

[Wladimir Palant]

Giorgio, please note that Adblock Plus 1.3.3 doesn't implement X-Do-Not-Track - for now the implementation is only in the development builds and requires a filter "*$donottrack" (included in various privacy protection subscriptions).

Comment 34

[Peter Eckersley]

Ugh, bad news about those router bugs.  Sounds like there will need to be an IP address exception list in about:config that defaults to the private IP address ranges.

Comment 35

[[Baboo]]

What about remote control?

Comment 36

[David E. Ross]

Re comment #34:  Should not this bug be reopened until that issue is resolved?  Or is there a new bug report for that issue?

Comment 37

[Doug Turner (:dougt)]

new bugs please!

Comment 38

[David E. Ross]

Although this bug report is Resolved, please post a comment with any new related bug reports for those of us who are tracking this bug report.

Comment 39

[Janet Swisher]

"Section two details case studies from four different types of companies; we step through the resources the companies needed and the decisions they made as they implemented DNT, starting on page 10, then conclude that section with a flow chart of decisions for your company to consider during your implementation. Section three provides annotated code samples in a DNT tutorial, starting on page 17;"

Those cross-references need to be updated to page 12 and page 21, respectively. Hot links would be even better, but might not be possible in Word.

Comment 40

[Janet Swisher]

"Section two details case studies from four different types of companies; we step through the resources the companies needed and the decisions they made as they implemented DNT, starting on page 10, then conclude that section with a flow chart of decisions for your company to consider during your implementation. Section three provides annotated code samples in a DNT tutorial, starting on page 17;"

Those cross-references need to be updated to page 12 and page 21, respectively. Hot links to section names and page numbers would be even better, but might not be possible in Word.

Comment 41

[Janet Swisher]

Please disregard comments 39 and 40.

Comment 42

[Ioana (away)]

Verified as fixed with manual tests (added here http://bit.ly/sXdHaE) on Firefox 9.0 beta 1:
Mozilla/5.0 (Windows NT 5.1; rv:9.0) Gecko/20100101 Firefox/9.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0
Mozilla/5.0 (Windows NT 6.1; rv:9.0) Gecko/20100101 Firefox/9.0
Mozilla/5.0 (X11; Linux i686; rv:9.0) Gecko/20100101 Firefox/9.0

Also verified that the automated test (dom/tests/mochitest/general/test_bug629535.html) involving the privacy.donottrackheader.enabled passes on all OSs:
https://tbpl.mozilla.org/php/getParsedLog.php?id=7322343&full=1&branch=mozilla-beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=7322331&full=1&branch=mozilla-beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=7322030&full=1&branch=mozilla-beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=7320492&full=1&branch=mozilla-beta

Comment 43

[Ioana (away)]

Verified as fixed on Aurora(Fx 10) and Central(Fx11) with the same tests as in the above comment:
http://bit.ly/sXdHaE
+
https://tbpl.mozilla.org/php/getParsedLog.php?id=7374068&full=1&branch=mozilla-aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=7368649&full=1&branch=mozilla-aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=7368492&full=1&branch=mozilla-aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=7374991&full=1&branch=mozilla-aurora

https://tbpl.mozilla.org/php/getParsedLog.php?id=7383147&full=1&branch=services-central
https://tbpl.mozilla.org/php/getParsedLog.php?id=7382741&full=1&branch=services-central
https://tbpl.mozilla.org/php/getParsedLog.php?id=7383091&full=1&branch=services-central
https://tbpl.mozilla.org/php/getParsedLog.php?id=7383079&full=1&branch=services-central

Comment 44

[Ioana (away)]

Aurora builds this bug has been verified on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a2) Gecko/20111109 Firefox/10.0a2
Mozilla/5.0 (X11; Linux i686; rv:10.0a2) Gecko/20111109 Firefox/10.0a2
Mozilla/5.0 (Windows NT 5.1; rv:10.0a2) Gecko/20111110 Firefox/10.0a2
Mozilla/5.0 (Windows NT 6.1; rv:10.0a2) Gecko/20111113 Firefox/10.0a2

Central builds this bug has been verified on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0a1) Gecko/20111109 Firefox/11.0a1
Mozilla/5.0 (X11; Linux i686; rv:11.0a1) Gecko/20111109 Firefox/11.0a1
Mozilla/5.0 (Windows NT 5.1; rv:11.0a1) Gecko/20111110 Firefox/11.0a1
Mozilla/5.0 (Windows NT 6.1; rv:11.0a1) Gecko/20111113 Firefox/11.0a1