1312460 - Some webRequest events not fired for FTP

Status: RESOLVED WONTFIX

(WebExtensions :: Request Handling, defect, P3)

Description

[mmun]

Attachment: EventsComparison.txt

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0
Build ID: 20100101

Steps to reproduce:

Using in-house tests tied to our servers I generated different types of requests while capturing webRequest events.


Actual results:

I see differences between Firefox and Chrome:

1) Firefox isn't firing onResponseStarted for ftp: requests
2) In Firefox, requests redirected to ftp do not end with the firing of OnComplete or OnErrorOccurred
3) 301, 302, 303, 307, and 308 redirects from http: to ftp: were successful in both Firefox and Chrome (each img retrieved from the respective FTP server and displayed).  However, Chrome fires a second onBeforeRequest for the ftp phase and Firefox does not.  Both Chrome and Firefox fire a second onBeforeRequest for http to http redirects.  So that appears to be the most common behavior, and one which would make some features easier to implement and lower overhead (destination logging and/or blocking through onBeforeRequest listening alone).



Expected results:

Firefox's behavior should match Chromes(?)

Comment 1

[mmun]

Attachment: WebRequest-EventLogger.xpi

Comment 2

[u408661]

Looks like this more belongs in the web extension framework - necko isn't the bit that sends webRequest notifications.

Comment 3

[Shane Caraveo (:mixedpuppy)]

Attachment: webrequestFTP test

Given changes in bug 1314492, this test illustrates the failures for ftp.

Comment 4

[mmun]

(In reply to Shane Caraveo (:mixedpuppy) from comment #3)
> Given changes in bug 1314492, this test illustrates the failures for ftp.

I'm not familiar with the Mochitest framework but I think I see what that is trying to do.  Should the test have a second part, where an FTP error condition is created and it verifies that OnErrorOccurred is fired?

Comment 5

[jf]

Attachment: index.html, page2.html, and image.png referenced in my comment

Comment 6

[jf]

Some events aren't firing in my installed version of Firefox ESR 52.2.1.  So I grabbed PortableApps versions of Chrome 59, Firefox 54.0.1, Firefox 55.0b2, and Firefox Nightly 56.0a1 (2017-07-06) and ran some tests.  There were some differences, but all of the Firefox versions had some issues in this area.  Even when networkPredictionEnabled=false.  FTP doesn't seem to be of much interest, but:

* ftp: schemed urls can be used for web bugging purposes, to exfiltrate other data, embed malicious script, etc.
* Firefox allows ftp: subresources in ftp:, http:, and file: pages.  It also allows https: -> ftp: redirects.
* Protective addons may be bypassed if they don't receive the necessary events.  Some are relying upon blocking onBeforeRequest for <all_urls>.

I'll use Firefox 55.0b2 as an example.  I uploaded the attached test files to my home server, navigated to ftp://ftp.test/index.html (which includes an image via http: -> ftp: redirection), and then clicked on the link to ftp://ftp.test/page2.html.  Chrome events look OK to me, but in Firefox:

* No events for the top-level ftp: load that was initiated via address bar (or bookmarks bar).
* No onBeforeRequest for the ftp: phase of the image load.
* The image load began with requestId of 1 (http: phase) but the (ftp: phase) onResponseStarted and onCompleted have a requestId of 2.
* No onResponseStarted for page2 load.
* The onBeforeRequest and onCompleted for page2 have a tabId of -1 rather than 1.

I hope someone who can confirm a bug will look into this.  If it helps, there is ftp://speedtest.tele2.net/ and https://httpbin.org/ has a redirect-to feature.  My captures:


*** Event log from Chrome 59
*** User action: Navigate to ftp://ftp.test/index.html via address bar (or bookmarks bar)
[WREL] #103 onBeforeRequest 2 ftp://ftp.test/index.html
[WREL] #103 onResponseStarted 2 ftp://ftp.test/index.html
[WREL] #103 onCompleted 2 ftp://ftp.test/index.html
[WREL] #104 onBeforeRequest 2 http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png
[WREL] #104 onBeforeSendHeaders 2 http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png
[WREL] #104 onSendHeaders 2 http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png
[WREL] #104 onHeadersReceived 2 http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png
[WREL] #104 onBeforeRedirect 2 http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png
[WREL] #104 onBeforeRequest 2 ftp://ftp.test/image.png
[WREL] #104 onResponseStarted 2 ftp://ftp.test/image.png
[WREL] #104 onCompleted 2 ftp://ftp.test/image.png
*** User action: Click on index.html link to ftp://ftp.test/page2.html
[WREL] #107 onBeforeRequest 2 ftp://ftp.test/page2.html
[WREL] #107 onResponseStarted 2 ftp://ftp.test/page2.html
[WREL] #107 onCompleted 2 ftp://ftp.test/page2.html


*** Event log from Firefox 55.0b2
*** User action: Navigate to ftp://ftp.test/index.html via address bar (or bookmarks bar)
[WREL] #1 onBeforeRequest 1 "http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png"
[WREL] #1 onBeforeSendHeaders 1 "http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png"
[WREL] #1 onSendHeaders 1 "http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png"
[WREL] #1 onHeadersReceived 1 "http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png"
[WREL] #1 onBeforeRedirect 1 "http://http.test/?redirect-to=ftp%3a%2f%2fftp.test%2fimage.png"
[WREL] #2 onResponseStarted 1 "ftp://ftp.test/image.png"
[WREL] #2 onCompleted 1 "ftp://ftp.test/image.png"
*** User action: Click on index.html link to ftp://ftp.test/page2.html
[WREL] #3 onBeforeRequest -1 "ftp://ftp.test/page2.html"
[WREL] #3 onCompleted -1 "ftp://ftp.test/page2.html"

Comment 7

[Shane Caraveo (:mixedpuppy)]

jf: That is basically what I would expect, much of WebRequest is internally very http specific.

Comment 9

[Rob Wu [:robwu]]

I just checked, by adding a webRequest listener for all events. None of the events are triggered when I access ftp://speedtest.tele2.net/ or ftp://ftp.hp.com/ (two random sites that I found when looking for a public ftp server for testing).

I think that we can WONTFIX this, because FTP support is likely going to be phased out - see bug 1574475 .

And Chrome has dropped the ability to load FTP from extensions (https://www.chromestatus.com/feature/5709390967472128). As someone who maintains a popular extension (almost 1M users) that relies on webRequest, I have received only a handful of complaints about the lack of FTP support.

Philipp, can we close this?

Comment 10

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Yes I agree this can go away. Thanks for the analysis!