1429556 - webRequest.onBeforeRequest returns incorrect url for redirects

Status: RESOLVED DUPLICATE of bug 1448599

(WebExtensions :: Request Handling, defect, P2)

Description

[Alexander]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36

Steps to reproduce:

1. WebExtension's background script:
browser.webRequest.onBeforeRequest.addListener(function (details) {
	console.log(details.url);
}, {
	urls : ['<all_urls>'],
	types : ['main_frame']
});

2. Following the link: https://www.ebates.com/gap.com_3864-xfas?ebtoken=eyJhbGciOiJIUzI1NiJ9.eyJsbHQiOjE1MTM2NzI0OTA4NTIsIm1JZCI6IjE5MzRFMzIwOUE4ODI0N0QxQUMzRjlDIn0.e0Le8EppSaVqTSzHQ-UkVMve3tpIiPDow_B60TOMe5s&tb=yes&sourceName=toolbar&eeid=23509&store_url=http%3A%2F%2Fwww.gap.com%2F


Actual results:

Firefox redirects through some redirect URLs to http://www.gap.com
List of URLs:
- https://www.ebates.com/gap.com_3864-xfas?ebtoken=eyJhbGciOiJIUzI1NiJ9.eyJsbHQiOjE1MTM2NzI0OTA4NTIsIm1JZCI6IjE5MzRFMzIwOUE4ODI0N0QxQUMzRjlDIn0.e0Le8EppSaVqTSzHQ-UkVMve3tpIiPDow_B60TOMe5s&tb=yes&sourceName=toolbar&eeid=23509&store_url=http%3A%2F%2Fwww.gap.com%2F
- http://www.dpbolvw.net/click-46157-10433992?url=http%3A%2F%2Fwww.gap.com%2F&sid=ebs1643353301sbe
- http://cj.dotomi.com/i1103vpyxF/pw0/EDHGGMMF/HJEIK/D/D/D?h=yI83%3D41IRWUTTVTTQRI14%26KHB%3D7JJF%25Ta%25Sf%25SfMMM.60F.2EC%25Sf%3c%3c7JJF%3A%2F%2FMMM.3F1EBLM.D4J%3AYQ%2F2B82A-UWRVX-RQUTTZZS%3c%3cg%3c%3c%3c
- http://www.emjcd.com/dh108p-87S/-6B/POSRRXXQ/SUPTV/O/QRVROSQQSURQTWWVXQ:Kc5U8pRDYOyb/y0R-RPU22UQRPP1VWQO2OOOPOyP-O1O1?l=byoj%3Dkhy7CA99B9967yhk%260xr%3Dnzzv%259G%258L%258L222.mgv.ius%258L%3cipu!2CDz-zrlu9wz2%3cnzzv%3A%2F%2F222.jvhur12.tkz%3AE6%2Firoiq-AC7BD-76A99FF8%3c%3cM%3c%3c%3c
- http://www.gap.com/?tid=goaff46157&ap=2&siteID=goafcid10433992&cvosrc=affiliate.cj.46157


Expected results:

We expect that webRequest.onBeforeRequest will handle each redirect URL
But the result of above code:
> https://www.ebates.com/gap.com_3864-xfas?ebtoken=eyJhbGciOiJIUzI1NiJ9.eyJsbHQiOjE1MTM2NzI0OTA4NTIsIm1JZCI6IjE5MzRFMzIwOUE4ODI0N0QxQUMzRjlDIn0.e0Le8EppSaVqTSzHQ-UkVMve3tpIiPDow_B60TOMe5s&tb=yes&sourceName=toolbar&eeid=23509&store_url=http%3A%2F%2Fwww.gap.com%2F
> http://www.tkqlhce.com/click-46157-10433992?url=http%3A%2F%2Fwww.gap.com%2F&sid=ebs1643346486sbe
> http://www.tkqlhce.com/click-46157-10433992?url=http%3A%2F%2Fwww.gap.com%2F&sid=ebs1643346486sbe
> http://www.tkqlhce.com/click-46157-10433992?url=http%3A%2F%2Fwww.gap.com%2F&sid=ebs1643346486sbe
> http://www.tkqlhce.com/click-46157-10433992?url=http%3A%2F%2Fwww.gap.com%2F&sid=ebs1643346486sbe

The same code works in Chrome as expected

Comment 1

[vitaliy.iakuba]

Reproduced this FF 57.0.4

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0

NOTICED the issue is NOT reproducible when running FF 56

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:56.0) Gecko/20100101 Firefox/56.0

Comment 2

[Emil Pasca [:emilpasca]]

I am assigning a component to this issue in order to involve the development team and get an opinion on this.

Comment 3

[Dan Moffat]

This still seems to happen on Firefox Quantum 60.0.1 ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:60.0) Gecko/20100101 Firefox/60.0").

Strangely enough, I am able to fix it by adding an empty event handler to `webRequest.onBeforeRedirect`, like so:

browser.webRequest.onBeforeRedirect.addListener(function(){}, {
    urls: ['<all_urls>']
});

After adding this, onBeforeRequest shows the correct redirected URLs. Hopefully this helps someone else out as I've just spent a silly amount of time trying to resolve this :)

Comment 4

[Rob Wu [:robwu]]

I'm bumping the priority of this bug, because the underlying primitive is used in many places, with the underlying assumption that the url reflects the requested URL. Using a different URL can result in bad (security-sensitive) decisions.

I ran into this issue while writing a unit test that confirms that a ChannelWrapper's URL (instantiated at http-on-modify-request) is showing the redirect URL. It looks like the redirect-awareness of ChannelWrapper is handled via the channel.channel assignment in onChannelReplaced, which is called by an asyncOnChannelRedirect handler, which in turn is only registered when any webRequest.onBeforeRedirect listener is present in Firefox. Presently, there is a built-in extension with such a listener, but we should not be relying on that as it is very brittle.

And for the record, I also note the use of newChannel.originalURI in onChannelReplaced (called by asyncOnChannelRedirect), but the IDL file claims that originalURI is unusable at that time. The comment was added in bug 1732250. If this bug gets fixed without unit tests looking at this originalURI usage etc, then we should have a new bug filed.

Comment 5

[Rob Wu [:robwu]]

The webRequest.onBeforeRequest bug has been fixed by bug 1448599, specifically the change at https://hg.mozilla.org/mozilla-central/rev/5c9c6f5a4ca3#l2.30

The underlying issue still exists for the ChannelWrapper primitive, so I will file a new bug for that.