669759 - 'Auto-detect proxy settings' causes some HTTP headers to be dropped

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Lindsey Poole]

Attachment: firefox_cloudplayer_bug.rtf

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1

Steps to reproduce:

1. Open 'Preferences > Advanced > Connection' and ensure that 'No proxy' is selected.
2. Restart Firefox so there is a clean instance.
3. Log into Amazon Cloud Player. http://www.amazon.com/cloudplayer/
4. Open 'Preferences > Advanced > Connection', select 'Auto-detect proxy settings for this network', and click 'OK'.
5. Either initiate an API call to Amazon Cloud Player by interacting with the UI, or wait for a periodic call to 'getGlobalLastUpdatedDate' API to fire.


Actual results:

API call reliably fails with '401 Unauthorized' due to authentication header fields being stripped from the request.

Looking at outgoing requests we see:

1. a POST to https://www.amazon.com/cirrus/ that is never received on the server-side (I presume that it is either paused or dropped).
2. a GET to http://wpad/wpad.dat.
3. a POST to https://www.amazon.com/cirrus/ that is missing the following header fields:
  a. X-Requested-With=XMLHttpRequest
  b. x-adp-token={enc:dxH169kcCX66OQy65gqfpgX1xP/GPUK3B1emV21cEgb5GS2WSOHqi3GE5bJrABHN0t+XqblxDN6ruQ/IdsxzXE6//Q4dBXb6xc2+AkEd0jIOJhvhYbo0A/FRzwWJcGL+M3ouR7Cw9mIiaveGs90nMvCtBwTjgepEMt3jk9eb/bcs3t+zQQN3veOqodN7JQGh75xpo1sxopoQq6guEwL7MoHndNTn9SyBZGnUNd/99maAQSQaOSkB76Hsk7zLcnC4oPBbONHMb1/6NuEMME4lsuimOEbhfFvEyjLu7vcF/m/qGqTxwnoXspDl4VHiVkizogNx99FPflXl/qwkxsLu0+4vMeuBmdLef2h5P2/3VY631jjAH5VPZcR9gC7YT551df8WwRm9g2xW8aoqbz0+iMkO3YN+VNPyfi/kWsSHbRLLHRMvy7+VnlHo7MVDHEJQQOjRCNRiXpaCfhKf0XxIWNsyWElCG2Bwo7prk06muA4KGFNppLQ0sf9ClpFPAinjnGbsQDSKnPmP26VJqX6I/+6Tnhi8ao6T2R3edRhkqcwqf9KanS3ECg9QPrZ8e5/GzAJMjehYuzUdKfRdYcpSYGpyHuiSouVHwGlIk7iwAAiKqIwGEALLZMDDWlpLT0Vm}{iv:/5k8LTV/bYpSF9Ng3N/vJA==}{key:PSdsJ50L1RARaooUhbPwckKYPG5MZzImKoq6c+DhQ6DNU8u+iqhlgoLB/YStzshZGRl8K1zPvVSl52JNMl6vz9SOJxwYmBHodmVAJZ+xa7ApqA0ipGAWdvthYHB5SqZ21WAWiHEBLDKUDymNKdwX167RqVRfyuqibqGrMVxnwXwrN3PbRr1Tw7T5t8bFFwpA++v5WQ8oAqdmWClCOugRVdJ121o75tKKnel3vm8AIA5c01oTv6oyIBpeiucWUUZQL9ylgxZah4jcok8DREkjI1Lwj9kZForPr4DFJ9nadEDF8xfJ8olDeeSJZp4XtWX6D2WmbiC0uPfg/fSot8XjZQ==}{name:QURQVG9rZW5FbmNyeXB0aW9uS2V5}{serial:MQ==}
  c. x-amzn-RequestId=5c4df99f-ca67-dmcp-4eff-60670c8aeea1

I presume that request '3' is the resumption of the paused instance of request '1', since the 'getGlobalLastUpdatedDate' API call should have only fired once in the given timeframe.


Expected results:

1. Request '1' should have included the missing header fields.
2. The behavior of Firefox when interacting with Amazon Cloud Player, should be identical whether using 'No proxy' or 'Auto-detect proxy settings for this network'

Comment 1

[Lindsey Poole]

Sorry, that should read:

Expected results:

1. Request '3' should have included the missing header fields.

Comment 2

[Justin Dolske [:Dolske]]

When you're using the autodetect mode, does Firefox actually end up finding and using some proxy? A more minimal testcase would also be helpful, eg can a simple page making a XHR reproduce this?

Comment 3

[Boris Zbarsky [:bzbarsky]]

I was 99% sure we had fixed this, but apparently not.

Proxy autodetect performs an internal redirect to the proxy.  And then you hit bug 216828: custom request headers are not preserved across redirects.

Arguably, all headers should be preserved for the proxy redirect....

Comment 4

[Lindsey Poole]

@dolske No, there is no proxy that it can use in our test scenario.  Unsure about the XHR request - I'm a back-end Cloud Player guy, I'd have to pull in someone from the web player team to investigate further.

Comment 5

[Lindsey Poole]

Hey, just checking in to see if anyone has been able to reproduce this, or has otherwise taken a look.

Is there anything I can do to assist?

Comment 6

[Boris Zbarsky [:bzbarsky]]

Lindsey, the problem is understood: see comment 3....

Comment 7

[Lindsey Poole]

Hi Boris, I understand that the problem is understood :)

I'm just poking the thread to see if there is a developer who can work on a fix, and if so, if there is an ETA on merging the fix into a FF release.  If not, I'd like to see whether it makes sense to find a resource to write a FF patch, or whether we should implement a workaround on the server-side.

Anyways, sorry to be a pain, we just need the information to make correct engineering decisions moving forward.

Comment 8

[Boris Zbarsky [:bzbarsky]]

Ah, ok.  I sent mail to the necko developer mailing list pointing to this bug.  Hopefully someone will take a look at it...

Comment 9

[Lindsey Poole]

Cool, thanks!

Comment 10

[Boris Zbarsky [:bzbarsky]]

Actually, for the specific case of XMLHttpRequest (which is what comment 0 seems to be about) this seems like it might be fixed by bug 553888 as of late June.

Lindsey, could you try an Aurora build from http://www.mozilla.com/firefox/channel/ and see whether the problem is gone there?

Comment 11

[Lindsey Poole]

Sure, I'll try to reproduce the problem now.

Comment 12

[Lindsey Poole]

I cannot reproduce the problem with Aurora 7.0a2 (2011-07-26).

Posting sample request data to attachments.

Looks like this is fixed in this nightly.

Comment 13

[Lindsey Poole]

Attachment: Sample requests

Comment 14

[Boris Zbarsky [:bzbarsky]]

Thanks!  In that case, we'll be shipping that fix in final builds in late September.  So at that point the problem will go away.

Comment 15

[Lindsey Poole]

Super - thanks for looking into it for us!

Comment 16

[Josh Aas]

Can we resolve this bug?

Comment 17

[Boris Zbarsky [:bzbarsky]]

This bug is still a problem as filed.  It won't be a problem for XHR anymore, but it could still break other cases where the caller sets request headers (plug-ins, say).

Comment 18

[Patrick McManus [:mcmanus]]

769764 changes this so the proxy isn't implemented with an internal redirect, so this should be fixed up.

Comment 19

[Patrick McManus [:mcmanus]]

fixed by 769764 
https://hg.mozilla.org/mozilla-central/rev/828f91de7143