1601496 - Firefox does not set the value of `document.referrer` to that of the Referer header sent if it's modified by an extension

Status: REOPENED

(Core :: DOM: Security, defect, P3)

Description

[Erin Yuki Schlarb [:ntninja]]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

1. Install my extension Smart Referer (https://addons.mozilla.org/de/firefox/addon/smart-referer/)
2. Search for “BrowserLeaks JavaScript”
3. Open the first search result (should be https://browserleaks.com/javascript)
4. Observe the value of the Document Referrer row:
   • On Firefox 68- (like don't Firefox 10 or less at least!) it will show “https://browserleaks.com/javascript”
   • On Firefox 69+ it will instead show the name of the search engine you came from

Actual results:

The Smart Referer browser extension will detect that navigating from search-engine.example to browserleaks.com is a cross-domain navigation and will hence replace the Referer header value with the URL of the page that is being navigated to (by default anyways).

Expected results:

In all Firefox version prior to 69 this would be enough to also have Firefox report a different document.referrer JS value on the target page (and the classic version of that extension worked at least all the way down to Firefox 10 – maybe earlier) while relying on this behaviour.

This behaviour has now been silently been broken and hence the extension was turned from a privacy extension to a privacy-aware-user beacon…

Will this behaviour be fixed/restored? Knowing BMO I won't hold my breath, so I'll be adding a workaround instead, asap.

Comment 1

[Gingerbread Man]

Can you find the exact regression range?
https://mozilla.github.io/mozregression/quickstart.html

Comment 2

[Alice0775 White]

I think that latest version 0.2.12 of the add-on fix the issue.
see https://addons.mozilla.org/en-US/firefox/addon/smart-referer/versions/

ntninja, Can you please re-test this with version 0.2.12?

Comment 3

[Gingerbread Man]

(In reply to Alice0775 White from comment #2)

ntninja, Can you please re-test this with version 0.2.12?

Reporter is co-author of the add-on. The intention of adding a workaround was mentioned in the description:

(In reply to ntninja from comment #0)

Will this behaviour be fixed/restored? Knowing BMO I won't hold my breath, so I'll be adding a workaround instead, asap.

Comment 4

[Alice0775 White]

So, Version 0.2.11 is not include the workaround.

Regression window using Version 0.2.11:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=c259c1017aa193b8b4a2031488aab0dc779b3a2f&tochange=b5df2e6d8478cb4aea88d109259714ca2772b6cd

Comment 5

[Gingerbread Man]

Thank you for the regression window, Alice!

Comment 6

[Alex Cornestean]

Hello,

I have managed to reproduce the issue using the provided STR on the latest Beta (72.0b4/20191206183317) and Release (71.0/20191202093317) under Windows 10 Pro 64-bit and MacOS Catalina 10.15 using the Smart Referrer add-on version 0.2.11.

With the latest version of the add-on (0.2.13) the issue no longer occurs, with the Document Referrer field showing the correct value i.e. https://browserleaks.com/javascript (the URL of the page that is being navigated to) on Beta and Release.

On Nightly, with both the 0.2.11 and 0.2.13 versions of the add-on, the Document Referrer field is initially empty (when loading the page), however, upon using the “[reload to check]” option at the right of the field, the correct value will be displayed i.e. https://browserleaks.com/javascript.

With these findings in mind, I will change the status of the issue to New, however, since the issue no longer occurs through using the newest iteration of the add-on, please feel free to resolve it as you see fit.

Comment 7

[Alice0775 White]

(In reply to Alex Cornestean from comment #6)

Hello,

I have managed to reproduce the issue using the provided STR on the latest Beta (72.0b4/20191206183317) and Release (71.0/20191202093317) under Windows 10 Pro 64-bit and MacOS Catalina 10.15 using the Smart Referrer add-on version 0.2.11.

With the latest version of the add-on (0.2.13) the issue no longer occurs, with the Document Referrer field showing the correct value i.e. https://browserleaks.com/javascript (the URL of the page that is being navigated to) on Beta and Release.

On Nightly, with both the 0.2.11 and 0.2.13 versions of the add-on, the Document Referrer field is initially empty (when loading the page), however, upon using the “[reload to check]” option at the right of the field, the correct value will be displayed i.e. https://browserleaks.com/javascript.

With these findings in mind, I will change the status of the issue to New, however, since the issue no longer occurs through using the newest iteration of the add-on, please feel free to resolve it as you see fit.

The empty issue on Nightly71.0a1 is Bug 1601743.

Comment 8

[Shane Caraveo (:mixedpuppy)]

Switching to the component of the bug that regressed this.

Comment 9

[Erin Yuki Schlarb [:ntninja]]

Hi! – OP / Smart Referer dev here!

My thanks to Alice for doing regression testing for this!

I like to clarify some incorrect / confusing statements from the previous comment however:

On Nightly, with both the 0.2.11 and 0.2.13 versions of the add-on, the Document Referrer field is initially empty (when loading the page), however, upon using the “[reload to check]” option at the right of the field, the correct value will be displayed i.e. https://browserleaks.com/javascript.

I'm not sure “when initially loading the page” means in this context, but if you just copied https://browserleaks.com/javascript into the address bar and pressed enter, then the “Document Referrer field [being] initially empty” is expected since that is by definition a “direct-hit”. There is nothing unusual or unexpected about this. You should instead, as the STR mentions, search for “BrowserLeaks JavaScript” and click on the first result instead as that will create cross-domain navigation that sets a referrer header normally blocked/rewritten by Smart Referer.

Clicking “[reload to check]” tells you nothing as it will only create a navigation from https://browserleaks.com/javascript → https://browserleaks.com/javascript – even if Smart Referer were trying to override the referrer in this case it would still end up resulting in the exact same value as the default referrer value and the URL of the target page coincide in this particular case.

With the latest version of the add-on (0.2.13) the issue no longer occurs, with the Document Referrer field showing the correct value i.e. https://browserleaks.com/javascript (the URL of the page that is being navigated to) on Beta and Release.

Indeed and the reason of this is of course, as I mentioned in the initial post I would do, is because I pushed a extension-side mitigation for this. Version 0.2.14 tries to not make this easily finger-printable anymore, but so far it covers only the obvious case of using Reflect.getOwnPropertyDescriptor(document, "referrer") to detect that value was manipulated (when not overriding document.referrer that operation would return undefined for whatever weird reasons), I'm not sure if there are other ways to detect such meaning.

TL;DR: All reproducibility testing should be done with Smart Referer version 0.2.11 only!

Comment 10

[Alex Cornestean]

Attachment: screenrecording.gif

Hello!

I have followed the STR you provided to the letter, meaning that I installed the extension (both version 0.2.11 and the latest available one, to have a basis for comparison and provide accurate results), searched for “BrowserLeaks JavaScript” and opened the first provided result which was https://browserleaks.com/javascript (this is what I meant by “loading the page”).

You can see in the attached video exactly what I did and the corresponding results. This was done with version 0.2.11 of the extension installed beforehand, on a fresh Nightly profile. Using the “[reload to check]” option will do what I mentioned in my previous comment.

Comment 11

[Alice0775 White]

(In reply to Alex Cornestean from comment #10)

Created attachment 9114808 [details]
screenrecording.gif

Hello!

I have followed the STR you provided to the letter, meaning that I installed the extension (both version 0.2.11 and the latest available one, to have a basis for comparison and provide accurate results), searched for “BrowserLeaks JavaScript” and opened the first provided result which was https://browserleaks.com/javascript (this is what I meant by “loading the page”).

You can see in the attached video exactly what I did and the corresponding results. This was done with version 0.2.11 of the extension installed beforehand, on a fresh Nightly profile. Using the “[reload to check]” option will do what I mentioned in my previous comment.

The empty issue on Nightly71.0a1 is Bug 1601743.

Comment 12

[Erin Yuki Schlarb [:ntninja]]

So the fix for bug 1601743 does not address this issue then?

Comment 13

[Jonathan Kingston [:jkt] he/him]

I think this is perhaps an issue with this specific extension.
I modified the extension background/policy.js to fix the this.list is undefined error in the "browser console":

From:

        c.fromJSON = function(data) {
                let self = new c("");
                for(let matcherStrings of this.list) {

To:

        c.fromJSON = function(data) {
                let self = new c("");
                for(let matcherStrings of self.list) {

:ntninja are you able to verify if this fixes the issue, also?

The results were that I wasn't able to reproduce the error anymore.

Comment 14

[Jonathan Kingston [:jkt] he/him]

I tested with 71 and a central build.

Comment 15

[Erin Yuki Schlarb [:ntninja]]

Thanks for finding this! Don't know how I manged to break that again and not notice the breakage :-/

Version 0.2.15 has the proper fix: https://addons.mozilla.org/de/firefox/addon/smart-referer/versions/

Sorry for the confusion!

Comment 16

[Jonathan Kingston [:jkt] he/him]

Closing the bug as it appears both stable and nightly are working again. I think there perhaps was an issue resolved in bug 1601743 but I haven't tested.

Feel free to re-raise or reopen if there are edge cases that I'm not finding.

Thanks.

Comment 17

[Erin Yuki Schlarb [:ntninja]]

Jonathan Kingston: While what you found was indeed a bug in the current version of the extension. This issue can only be reproduced with Smart Referer version 0.2.11 or earlier. – Newer versions contain a workaround for this Firefox bug and are not relevant to this discussion. The bug report was created shortly before 0.2.12 was released, so the OP description does not contain that crucial bit of information and I cannot update either.

So unless you can reproduce the Firefox 68- behaviour described in the OP with latest nightly this issue is not fixed.

Comment 18

[uBlock-user]

Firefox 73 is affected too, bug reported with uMatrix[1] extension where spoofing referrer header doesn't work at a certain website and the detection is successfull.

[1] - https://github.com/uBlockOrigin/uMatrix-issues/issues/78

Comment 19

[Jonathan Kingston [:jkt] he/him]

Thomas are you able to take a look at this please?
Thanks!

Comment 20

[Thomas Nguyen (:tnguyen)]

I see the problem here is: document.referrer is the result from what we computed referrer header based on referrer policy. And these extensions injected (or changed) referrer header in HTTP request directly, therefore these 2 referrer values will not be matched.
I can do some slightly changes, however, in terms of referrer policy, for me, it does not make sense if we ignore referrer policy and force document.referrer = request.referrer (modified). Given the case, document's referrer policy is no-referrer then the extension set referrer header to https://browserleaks.com/javascript
The result we should get from document.referrer is an empty string, not the target "https://browserleaks.com/javascript"

Comment 21

[uBlock-user]

This is a regression, so why change the behavior now instead of bringing back the old behavior ?

Comment 22

[Thomas Nguyen (:tnguyen)]

I think the regression range here is only pointing out where the issue found, in this case it points out where the change starts. It does not 100% mean we should bring back the behavior (particularly if the old behavior is not good as new one).
I suggest in this case, we should adapt document.referrer to the new one from Smart Referrer, but still apply document's referrer policy
You might get the result likes "https://browserleaks.com" but not "https://browserleaks.com/javascript" (as the document's referrer policy is "origin"). Ignoring referrer-policy seems not to be good to me, as we don't want to fix this for one extension but may leak more important things later.

Comment 23

[Erin Yuki Schlarb [:ntninja]]

Speaking for Smart Referer, this change would be fine by me. Smart Referer actually bases all of its policy decisions on the Referrer header/value we are given by the browser anyways. So if the browser's policy is more restrictive then the extension's one the browser's policy will always win.

I don't know if it was originally planned to be this way, but at this point I consider it a feature so if you'd like to enforce the browser's policy on top of that of Smart Referer, I have no objections!

Comment 24

[uBlock-user]

if the old behavior is not good as new one

I'm not aware of the new planned behavior that you mentioned, but the old one didn't leak the document.referrer value like it does currently and so what's the new proposed behavior ?

Comment 25

[Thomas Nguyen (:tnguyen)]

(In reply to uBlock-user from comment #24)

if the old behavior is not good as new one

I'm not aware of the new planned behavior that you mentioned, but the old one didn't leak the document.referrer value like it does currently and so what's the new proposed behavior ?

There're 2 options we can do:

• 1 - Bring back document.referrer always the same as channel's referrer header (even when an addon modifies referrer header of the channel).
• 2 - document.referrer is the result of the modified referrer applies referrer policy.
  1 means we will put more responsibilities to users or developers who install the addons as they will have to carefully decide to modify (or spoof) referrer header. But it seems 1 is still the right one here.

Comment 26

[uBlock-user]

Which one will stop the leak and use the value set by the extension ?

Comment 27

[Thomas Nguyen (:tnguyen)]

Both, but 2 you may get different referrer than before, e.g, you might get "https://browserleaks.com" instead of "https://browserleaks.com/javascript"

Comment 28

[uBlock-user]

Does the old behavior also sent the full page url like that ?

Comment 29

[Kevin Locke]

I just encountered a different manifestation of this issue which might be a less confusing repro case:

1. Visit https://browserleaks.com/javascript
2. Open the Network tab (Ctrl+Shift+E)
3. Click "click on self-link to recheck" link on page
4. Right-click the first request on the Network tab and choose "Edit and Resend"
5. Change Referer header to "https://example.com", then press "Send"
6. Observe that the Referer header in the new request is https://example.com while the displayed "Document Referrer" (from document.referrer) on the page is https://example.com

(In reply to Thomas Nguyen (:tnguyen) from comment #25)

There're 2 options we can do:

• 1 - Bring back document.referrer always the same as channel's referrer header (even when an addon modifies referrer header of the channel).
• 2 - document.referrer is the result of the modified referrer applies referrer policy.
  1 means we will put more responsibilities to users or developers who install the addons as they will have to carefully decide to modify (or spoof) referrer header. But it seems 1 is still the right one here.

I agree. I'd argue the responsibility of respecting referrer policy (where desired) is already present when modifying the header.

Comment 30

[Kevin Locke]

(In reply to Kevin Locke from comment #29)

6. Observe that the Referer header in the new request is https://example.com while the displayed "Document Referrer" (from document.referrer) on the page is https://example.com

Ugh. That should have read:

6. Observe that the Referer header in the new request is https://example.com while the displayed "Document Referrer" (from document.referrer) on the page is NOT https://example.com

Comment 31

[Christoph Kerschbaumer [:ckerschb]]

Putting this one in the backlog for now.

Comment 32

[fiona.klute]

Is there a chance to get this resolved before Manifest v3? The reason I'm asking is that my Referer Modifier extension currently uses a content script generated at runtime to inject the modification config before page scripts get to run, as a workaround for this bug. From looking at its code, the Smart Referer extension mentioned above does the same. All the other options I see (messaging with the background script or loading config from storage) are asynchronous. My tests show page scripts will very likely get to run and potentially read document.referrer before the promises resolve. As generated content scripts won't be available with Manifest v3, the workaround will break if/when it becomes mandatory.

Comment 33

[uBlock-user]

Hello, are there any plans to get this fixed ?

Comment 34

[Daniel Veditz [:dveditz]]

There is no one available to work on this at this time, that is why it is listed as "backlog" in the whiteboard field

Comment 35

[agen138coms]

What happens is,

If I redirect from some page to this page, it gives the proper url (previous page), that's fine.
If I directly type in the URL it gives me a blank string, that's also fine.
But, if the page gets reloaded, then it gives me the same page URL in Chrome (thats my expected result), but in Firefox it still gives the previous page URL. In some cases it also gives a blank URL. https://agen138.com/
What would be the issue? Are there any recommendations or alternatives?

Firefox version: 37.0