1714809 - Copy as cURL returns invalid headers

Status: RESOLVED FIXED

(DevTools :: Netmonitor, defect, P3)

Description

[Steven Penny]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0

Steps to reproduce:

Copy as cURL returns invalid headers

1. New Private Window
2. Tools, Web Developer, Network
3. browse to http://rarbg.to/threat_defence.php
4. right click final HTML request, Copy, Copy as cURL (POSIX)

Result is something like this:

curl
'http://rarbg.to/threat_defence.php?defence=2&sk=f3xl5b8c6u&cid=45916380&i=2536743499&ref_cookie=rarbg.to&r=32767847'
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0)
Gecko/20100101 Firefox/84.0' -H 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8' -H
'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer:
http://rarbg.to/threat_defence.php' -H 'Connection: keep-alive' -H 'Cookie:
sk=f3xl5b8c6u' -H 'Upgrade-Insecure-Requests: 1'

Notice that "--compressed" is in the command. This is not valid. If you view the
actual Request Headers, you will see this:

Accept-Encoding: gzip, deflate

This server is using this difference to its advantage, as a way to determine if
request is coming from a browser. If you naively use the cURL command as is, you
get a redirect:

HTTP/1.1 302 Found
Location: /threat_defence.php?defence=nojc

If you change to proper header, you get expected response:

HTTP/1.1 200 OK

Comment 1

[Hubert Boma Manilla (:bomsy)]

Hi Honza,
Can you reproduce this on Windows?

Thanks

Comment 2

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Thank you for the report Steven.

I tried the STRs from comment #0 on my machine and I can see the --compressed

Note that it's added when the request contains accept-encoding header.
You might also checkout the source code here:
https://searchfox.org/mozilla-central/rev/67945e4b7316dfb6914986213ed5b8a7df700958/devtools/client/shared/curl.js#151

In my case the complete cURL is as follows:

curl 'http://rarbg.to/threat_defence.php?defence=nojc' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: http://rarbg.to/threat_defence.php' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: sk=v3r7dqclz6' -H 'Upgrade-Insecure-Requests: 1' -H 'Cache-Control: max-age=0'

When executing it on the command line (I am also adding -i argument) I am getting HTTP/1.1 200 OK from the server (no redirect)

Some questions:

• Are you suggesting that we should change the logic and not add --compressed when accept-encoding header is available?
• I can't reproduce the redirect. Did the server change?
• Could you also test with the latest Firefox Nightly 91? https://www.mozilla.org/en-US/firefox/channel/desktop/

Honza

Comment 3

[Steven Penny]

Sorry, I think I left out a step, as the URL you are working with is not correct, try this:

1. New Private Window
2. Tools, Web Developer, Network
3. browse to http://rarbg.to/threat_defence.php
4. after page loads, click the "Click here"
5. right click final HTML request, Copy, Copy as cURL (POSIX)

resultant cURL command should be similar to what I put in the original post. I can still repro the original issue consistently.

Comment 4

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Thanks for the update.

I tried STRs from comment #3 and I am getting the following:
(I am right clicking on the last HTML request after clicking the "Click here" link)

curl 'http://rarbg.to/threat_defence.php?defence=2&sk=fw0zmxedot&cid=46046102&i=2544389859&ref_cookie=rarbg.to&r=38672076' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: http://rarbg.to/threat_defence.php?defence=1' -H 'Cookie: sk=fw0zmxedot' -H 'Upgrade-Insecure-Requests: 1'

Since the request contains Accept-Encoding: gzip, deflate header the --compressed argument is automatically added, this is how the logic works (see also my comment #2). Are you suggesting that we shouldn't add it in some cases?

Honza

Comment 5

[Steven Penny]

You should only add "--compressed", if it can replace the original "Accept-Encoding" header, without changing the resultant request at all. That is not what is happening here. By your own admission (and my testing), the original request uses:

Accept-Encoding: gzip, deflate

While using "--compressed" uses something else:

PS C:\> curl -I -v --compressed example.com
> Accept-Encoding: deflate, gzip, br

The server in question recognizes this difference, and proceeds to reject the request.

Comment 6

[Jan Honza Odvarko [:Honza] (always need-info? me)]

(In reply to Steven Penny from comment #5)

You should only add "--compressed", if it can replace the original "Accept-Encoding" header, without changing the resultant request at all. That is not what is happening here. By your own admission (and my testing), the original request uses:

Accept-Encoding: gzip, deflate

While using "--compressed" uses something else:

PS C:\> curl -I -v --compressed example.com
> Accept-Encoding: deflate, gzip, br

The server in question recognizes this difference, and proceeds to reject the request.

Thank for the update.

So, we should check out what cURL is generating for the Accept-Encoding header (the value) before producing the final cURL command.

Honza

Comment 7

[Steven Penny]

Honestly it would be better to just never use "--compressed" at all. Its currently just a synonym for:

Accept-Encoding: deflate, gzip, br

but that could change if cURL adopts other decompression methods. Since Firefox knows the exact request headers, just use those directly. Yes, the resultant commands will be slightly longer, but the logic should be simpler, and the resulting cURL commands will be accurate.

Comment 8

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I see, thanks for the note. I agree, this sounds like the way to go.

Some instructions for anyone interested in fixing this issue:

1. The place where the --compressed header is added is here:
   https://searchfox.org/mozilla-central/rev/67945e4b7316dfb6914986213ed5b8a7df700958/devtools/client/shared/curl.js#151

2. You can execute that function through the request list context menu Copy -> Copy, Copy as cURL (POSIX)

3. The --compressed argument should not be added to the generated command. Instead, we should copy the Accept-Encoding header. So, looks like removing the if (header.name.toLowerCase() === "accept-encoding") { condition could be enough.

Honza

Comment 9

[bugzilla.zyaep]

Can I work on this?

I understand it probably involves removing the if (header.name.toLowerCase() === "accept-encoding") { statement in the curl.js file and testing it afterwards.

Comment 10

[bugzilla.zyaep]

I do not mean to be rude or impatient but I already started working on this bug. Removing the if-statement above (comment #8) did indeed remove the --compressed argument.

I now get the following cURL command when following the instructions from comment #3:

curl 'https://rarbg.to/threat_defence_ajax.php?sk=mp81t074yr&cid=46957464&i=2598546805&r=41557728&_=1625425618512' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: text/plain' -H 'X-Requested-With: XMLHttpRequest' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: https://rarbg.to/threat_defence.php?defence=1' -H 'Cookie: sk=mp81t074yr' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin'

Does that completely fix the issue? And if so, who could be the reviewer or reviewer group for my commit? (hg log /home/user/Mozilla/mozilla-central/devtools/client/shared/curl.js show 'Honza' as most common but I see a few others as well.)

Comment 11

[Jan Honza Odvarko [:Honza] (always need-info? me)]

@bugzilla.zyaep: thank you for helping with this one!
And yes, I can review the patch.

Please post your patch to Phabricator
You might want to see some instructions here:
https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html

Honza

Comment 12

[bugzilla.zyaep]

Attachment: Bug 1714809 - Remove cURL --compressed argument. r=Honza

Comment 13

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I tested the attached patch and it seems to be working well for me.

When following the STRs from comment #0 I am getting:

curl 'http://rarbg.to/threat_defence.php?defence=2&sk=le9cjm3hgt&cid=46983381&i=2600114623&ref_cookie=rarbg.to&r=80587410' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate' -H 'Connection: keep-alive' -H 'Referer: http://rarbg.to/threat_defence.php' -H 'Cookie: sk=le9cjm3hgt' -H 'Upgrade-Insecure-Requests: 1' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache'

Steven, can you please test the patch on your machine and double check it works before we land it?

I pushed to Mozilla Try server here:
https://treeherder.mozilla.org/jobs?repo=try&revision=1059a20e21554747314b2b7a55ba4ede97bce616&selectedTaskRun=PvNlxRMVS8GE3W9yQ_YlcQ.0
(go to the Artifacts tab at the bottom of the page)

You can download Win Build (exe Firefox installer or bundle to unzip to a dir and run Firefox directly)
(let me know if you need different OS build)

Thank you,
Honza

Comment 14

[Steven Penny]

I dont know what artifact to download, I see like 40 files to choose from

Comment 15

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Two options:

1. target.zip - unzip and run firefox.exe
2. target.installer.exe - run and install Firefox.

I'd go with #1 since you can easily remove everything after testing.
You should run with a new profile: firefox.exe -P test-profile

Honza

Comment 16

[Steven Penny]

OK yes, with the patch everything seems to work now. At first I was still having trouble, but that was my own fault for adding "-I" to do a HEAD request, which that server also doesnt support. I removed that (which gave me the original command from Firefox), and only added "-v" for troubleshooting, and I get a 200 OK.

Comment 17

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Perfect, thank you for testing!

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

This good-first-bug hasn't had any activity for 2 months, it is automatically unassigned.
For more information, please visit auto_nag documentation.

Comment 19

[Onyinye Ezike]

Attachment: Bug 1714809 - [devtools] fix copy as cURL that returns invalid header. r=Honza,bomsy

Comment 20

[Onyinye Ezike]

I've submitted a patch to this bug. Regarding the failing test, please kindly guide me on how I can fix the 2 test cases that are failing after the bug fix,

I tried removing the "--compressed" that is inside the "BASE_RESULT" array on devtools/client/netmonitor/test/browser_net_copy_as_curl.js,
but that didn't fix it.

Kindly guide me on how I can fix this test successfully so I can update the patch.
@Honza @bomsy

Thanks.

Comment 21

[Hubert Boma Manilla (:bomsy)]

Add to outreachy

Comment 22

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Patch in Phab is ready to land.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88ca2c5e4030
[devtools] fix copy as cURL that returns invalid header. r=Honza,bomsy

Comment 24

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/88ca2c5e4030