Closed Bug 1207233 Opened 9 years ago Closed 8 years ago

XHR setRequestHeader("Content-Type", "") incorrectly provides a default content-type header

Categories

(Core :: DOM: Core & HTML, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla44
Tracking Status
firefox44 --- fixed
firefox50 --- fixed

People

(Reporter: nika, Assigned: wisniewskit)

References

(Blocks 1 open bug)

Details

Attachments

(2 files, 17 obsolete files)

4.55 KB, patch
Details | Diff | Splinter Review
31.99 KB, patch
Details | Diff | Splinter Review
Assignee: nobody → michael
Comment on attachment 8664912 [details] [diff] [review]
Don't provide a default content-type header if user calls XHR.setRequestHeader('Content-Type', '')

Review of attachment 8664912 [details] [diff] [review]:
-----------------------------------------------------------------

This looks fine to me.
Attachment #8664912 - Flags: review?(jduell.mcbugs) → review+
https://hg.mozilla.org/mozilla-central/rev/b0d7f0fd067c
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla44
Depends on: 891433
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
So what we could consider is setting the default type to a void string, not just empty, when the spec says it should be null, and having the HTTP code check for nonvoid before setting the header.
This has since been fixed in bug 918742, and the web platform tests confirm it: http://w3c-test.org/XMLHttpRequest/send-content-type-charset.htm

I'm not sure about the dependent Cleopatra bug, though.
> This has since been fixed in bug 918742

Are you sure?

When I run my test from bug comment 10 in a current nightly, I see:

  Content-Type: text/plain;charset=UTF-8

being sent.  The patch in this bug sent:

  Content-Type: 

instead.

Note that the test you link to in comment 8 does NOT test the behavior for setRequestHeader("Content-Type", ""), because it never makes such a call.
Oh, and current spec still requires sending:

  Content-Type: 

afaict.  The dependent Cleopatra bug was caused by the fact that we currently have internal code that passes "" for the content type and expects it to cause no Content-Type header to be sent, and the patch in this bug changed that behavior to sending an empty Content-Type header instead.
You're quite correct. I misread the test-case in that WPT, my bad.

After some investigating I'm not 100% sure what the correct fix should be, but based on your comments and the XHR spec, it sounds like what needs to happen is:

1. HttpChannel::ExplicitSetUploadStream() needs to allow passing aContentType in as a pointer instead, and not set a content-type at all if it's nullptr (in addition to allowing "" as per this bug).

2. nsXMLHttpRequest::Send() needs to be altered accordingly to follow step 4 in the spec (https://xhr.spec.whatwg.org/#the-send%28%29-method).


Do you think that will be enough to prevent the Cleopatra bug again? I can't quite tell based on the comments.

Also, note that Chrome seems to also set a default content-type if it's set to "". I'm guessing they're just off-spec too and need to change to match it, but I thought I'd mention it just in case it bears consideration.
> needs to allow passing aContentType in as a pointer instead

Or use a void string (the equivalent of null for a nullable string in idl) to represent "don't set the header".

> Do you think that will be enough to prevent the Cleopatra bug again?

I think so, yes.

> Also, note that Chrome seems to also set a default content-type if it's set to ""

Yes.  See bug 891433 comment 10 for what various browsers did at the time and bug 1207233 comment 24 for what the spec editor said in response to my suggestion that we change the spec here to align with at least one browser....
Alright, the upcoming patchset brings request content-type header more up to spec, as per the comments in the tickets and the spec links:
- https://xhr.spec.whatwg.org/#the-send%28%29-method
- https://fetch.spec.whatwg.org/#concept-bodyinit-extract

As such the patches cover more than the ticket was created to fix (sending a blank content-type properly), since I noticed that the specified default values weren't always being sent.

URLSearchParams was the biggest change. It was being passed around as a string nsIVariant, and so it couldn't be distinguished as a URLSearchParams anymore when it came time to assign its specced default content type. This required giving it a proper IDL interface and making it structured clonable so workers could pass it around, and adjusting the XHR code to accept it properly.

As for part 3, bz mentioned on IRC several days ago that nsXHRSendable is no longer necessary since binary XPCOM isn't a thing anymore, so since it's not particularly useful anyhow I opted to just remove it entirely.

I'll run these patches through a full try later when I'm granted access.
Part 0 - an updated version of the web platform test XMLHttpRequest/setrequestheader-content-type.htm. It now covers the default values of the header that should be sent for various data-types as per spec, if setRequestHeader isn't called at all. It also checks that "" can be set for each appropriate type, since it's possible (say) that send(string) will sent a "" properly, but send(document) will not.

Hallvors, I'm not sure who should review this, so I'm going to needinfo you for now.
Flags: needinfo?(hsteen)
Part 1 - change the way Content-Type headers are sent so that it's possible to send no header at all (via a void string) or a blank header (via the string "").
Attachment #8762727 - Flags: review?(jonas)
Part 2.1 - give URLSearchParams an IDL interface, nsDOMURLSearchParams, so they can be pulled back out of nsIVariants properly rather than just being passed around as strings.
Attachment #8762728 - Flags: review?(jonas)
Part 2.2 - fix the XHR code to properly accepts URLSearchParams.
Attachment #8762729 - Flags: review?(jonas)
Part 2.3 - make URLSearchParams structured clonable so workers can pass them to the main thread for XHRs.

Note that I'm not sure where to place the new SCTAG, as SCTAG_MAX is named in a way that makes it sound like it should be kept at the end. So I placed the new tag second-to-last.
Attachment #8762730 - Flags: review?(jonas)
Part 3 - remove nsIXHRSendable.
Attachment #8762731 - Flags: review?(jonas)
Part 4 - mark the existing version of the web platform test as passing, since it does.
Perhaps Anne or James wants to review the test changes? James will "upstream" them to W3C/web-platform-tests.
Flags: needinfo?(james)
Flags: needinfo?(hsteen)
Flags: needinfo?(annevk)
Comment on attachment 8762726 [details]
1207233-part0-setrequestheader-content-type-wpt.htm

>      function request() {

The arguments logic you have for this function strikes me as a little convoluted. Splitting it out into separate functions or just not passing certain values sometimes seems clearer than branching on arguments.length.

Individual tests look okay.
Flags: needinfo?(annevk)
Alright, here's a version that changes the function to be a little less convoluted.
Flags: needinfo?(annevk)
The patches appear to have passed a try run just fine: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b0058a3678da
Thank you, that is a lot more readable and looks fine.
Flags: needinfo?(annevk)
Some drive-by notes on the code:

1)  SetIsVoid(true) truncates the string.  So no need for the separate Truncate() call.
2)  You don't need an IDL nsIDOMURLSearchParams thing.  Just give
    URLSearchParams an IID if you want to QI to it.
3)  You have tabs in the code (e.g. in URLSearchParams::GetSendInfo's function header).
4)  Instead of this:

  nsCString converted = NS_ConvertUTF16toUTF8(serialized);

your should do:

  NS_ConvertUTF16toUTF8 converted(serialized);

4)  It seems like part 2.2 won't actually work without part 2.3.
    Presumably they should be folded together before checkin.
5)  The changes in part 4 should presumably be folded before checkin into
    whatever the changeset is that makes the test pass.
6)  You don't need to make the GetSendInfo methods virtual in part 3, right?  And certainly 
    you don't want them to be NS_IMETHODIMP in the implementation if they're not
    declared NS_IMETHOD.  Just make the impl "nsresult", whether the decl is
    virtual or not.
Hmm, I tried 2 (using NS_DEFINE/DECLARE_STATIC_IID_ACCESSOR), but it wasn't working. I'll try again (and address the other notes as well). Thanks!
Alright, here's a version of the patchset with the notes in comment 26 addressed. Try run is successful: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9cd36d138d38
Attachment #8762726 - Attachment is obsolete: true
Attachment #8762727 - Attachment is obsolete: true
Attachment #8762732 - Attachment is obsolete: true
Attachment #8762727 - Flags: review?(jonas)
Attachment #8763375 - Flags: review?(khuey)
Attachment #8762728 - Attachment is obsolete: true
Attachment #8762729 - Attachment is obsolete: true
Attachment #8762730 - Attachment is obsolete: true
Attachment #8762728 - Flags: review?(jonas)
Attachment #8762729 - Flags: review?(jonas)
Attachment #8762730 - Flags: review?(jonas)
Attachment #8763376 - Flags: review?(khuey)
Attachment #8762731 - Attachment is obsolete: true
Attachment #8762731 - Flags: review?(jonas)
Attachment #8763377 - Flags: review?(khuey)
If you want review, please r? not needinfo? :)

Seems like annevk already reviewed the tests.
Flags: needinfo?(james)
Will do. I was really just needinfoing you to find out what would be involved in "upstreaming" the patches. In the future, should I file bugs on the web-platform-test github and link them to any related conversation here on Bugzilla? Or would it be better to just ask you for a final review and you'll take it from there?
Ah OK. The idea is that upstreaming the patches is transparent; you don't have to do anything apart from commit the changes in the right place. Whoever is qualified to review the patch is also assumed to be trusted to stop you upstreaming nonstandard stuff. Apart from the upstreaming itself, I am not on the critical path for anything (but I can ofc review things I know about when that helps).
Ok, thanks, makes sense. Then I guess my question should be whether such patches should be against the Firefox source tree or the WPT github source tree, or if it doesn't really make a difference to you?
If you make them against the Firefox source tree then you will have an easier time getting reviewers. That's the only real reason to care.
Alright then, I'll do that. Once they do pass review, I'll ni? you and let you know that's the case and that they need upstreaming (assuming I don't ask you for the review at the time, of course).
Thomas, once they pass review, you just get them checked into the Firefox source tree.  James does sync both ways every so often (well, quite often), so things that land in the Firefox source tree get upstreamed automatically when that happens.
Comment on attachment 8763375 [details] [diff] [review]
1207233-part1-allow-sending-empty-or-not-content-type-header.diff

Review of attachment 8763375 [details] [diff] [review]:
-----------------------------------------------------------------

A networking peer needs to look at this one.
Attachment #8763375 - Flags: review?(khuey)
Attachment #8763375 - Flags: review?(dd.mozilla)
Attachment #8763375 - Flags: review+
Comment on attachment 8763376 [details] [diff] [review]
1207233-part2-give-URLSearchParams-an-IID-and-make-XHRs-use-it-properly.diff

How does this work today?
Flags: needinfo?(wisniewskit)
> How does this work today?

The URLSearchParams object is just stringified and passed to the XHR code, which dutifully sends it as a string with content type text-plain/charset=UTF8.
(In reply to Thomas Wisniewski from comment #40)
> > How does this work today?
> 
> The URLSearchParams object is just stringified and passed to the XHR code,
> which dutifully sends it as a string with content type
> text-plain/charset=UTF8.

Ok.
Flags: needinfo?(wisniewskit)
Comment on attachment 8763376 [details] [diff] [review]
1207233-part2-give-URLSearchParams-an-IID-and-make-XHRs-use-it-properly.diff

Review of attachment 8763376 [details] [diff] [review]:
-----------------------------------------------------------------

r=me on the XHR parts, I'd like baku to look at the structured cloning.
Attachment #8763376 - Flags: review?(khuey)
Attachment #8763376 - Flags: review?(amarchesini)
Attachment #8763376 - Flags: review+
Comment on attachment 8763377 [details] [diff] [review]
1207233-part3-remove-nsIXHRSendable.diff

Review of attachment 8763377 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/base/File.h
@@ +139,5 @@
>  
> +  virtual nsresult GetSendInfo(nsIInputStream** aBody,
> +                               uint64_t* aContentLength,
> +                               nsACString& aContentType,
> +                               nsACString& aCharset);

This shouldn't need to be virtual.

::: dom/base/FormData.h
@@ +140,5 @@
>  
>      return true;
>    }
>  
> +  virtual nsresult GetSendInfo(nsIInputStream** aBody,

Same.
Attachment #8763377 - Flags: review?(khuey) → review+
Review comments addressed. Carrying over r+.
Attachment #8763377 - Attachment is obsolete: true
Comment on attachment 8763376 [details] [diff] [review]
1207233-part2-give-URLSearchParams-an-IID-and-make-XHRs-use-it-properly.diff

Review of attachment 8763376 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/base/URLSearchParams.cpp
@@ +301,5 @@
>  NS_IMPL_CYCLE_COLLECTING_RELEASE(URLSearchParams)
>  
>  NS_INTERFACE_MAP_BEGIN_CYCLE_COLLECTION(URLSearchParams)
>    NS_WRAPPERCACHE_INTERFACE_MAP_ENTRY
> +  NS_INTERFACE_MAP_ENTRY(URLSearchParams)

to remove.

@@ +306,4 @@
>    NS_INTERFACE_MAP_ENTRY(nsISupports)
>  NS_INTERFACE_MAP_END
>  
> +URLSearchParams::URLSearchParams(nsISupports* aParent)

remove this CTOR.

@@ +316,2 @@
>  URLSearchParams::URLSearchParams(nsISupports* aParent,
>                                   URLSearchParamsObserver* aObserver)

set aObjserver = nullptr so that we have only 1 CTOR and 1 default value.

@@ +456,5 @@
>  
> +// Helper functions for structured cloning
> +inline bool
> +ReadString(JSStructuredCloneReader* aReader, nsString& aString)
> +{

MOZ_ASSERT(aReader);

@@ +484,5 @@
> +}
> +
> +bool
> +URLParams::WriteStructuredClone(JSStructuredCloneWriter* aWriter) const
> +{

MOZ_ASSERT(aWriter);

@@ +490,5 @@
> +  if (!JS_WriteUint32Pair(aWriter, nParams, 0)) {
> +    return false;
> +  }
> +  for (uint32_t i = 0; i < nParams; ++i) {
> +    if (!WriteString(aWriter, mParams[i].mKey)) return false;

if (!WriteString(A) || !WriteString(B)) {
  return false;
}

@@ +498,5 @@
> +}
> +
> +bool
> +URLParams::ReadStructuredClone(JSStructuredCloneReader* aReader)
> +{

MOZ_ASSERT(aReader);

@@ +505,5 @@
> +  uint32_t nParams, dummy;
> +  nsString key, value;
> +  if (!JS_ReadUint32Pair(aReader, &nParams, &dummy)) {
> +    return false;
> +  }

MOZ_ASSERT(dummy == 0);
and rename dummy to 'zero'

@@ +534,5 @@
> +{
> +  aContentType.AssignLiteral("application/x-www-form-urlencoded");
> +  aCharset.AssignLiteral("UTF-8");
> +
> +  nsString serialized;

nsAutoString

::: dom/base/URLSearchParams.h
@@ +17,5 @@
>  
>  namespace mozilla {
>  namespace dom {
>  
> +#define NS_URLSEARCHPARAMS_IID  \

Remove this.

@@ +138,5 @@
>  {
>    ~URLSearchParams();
>  
>  public:
> +  NS_DECLARE_STATIC_IID_ACCESSOR(NS_URLSEARCHPARAMS_IID)

remove this.

@@ +225,5 @@
>    nsCOMPtr<nsISupports> mParent;
>    RefPtr<URLSearchParamsObserver> mObserver;
>  };
>  
> +NS_DEFINE_STATIC_IID_ACCESSOR(URLSearchParams, NS_URLSEARCHPARAMS_IID)

Remove.

::: dom/base/nsXMLHttpRequest.cpp
@@ +2425,5 @@
>        return GetRequestBody(stream, aResult, aContentLength, aContentType, aCharset);
>      }
>  
> +    // URLSearchParams?
> +    nsCOMPtr<URLSearchParams> usp = do_QueryInterface(supports);

This is wrong. URLSearchParams is not passed as nsIVariant but as RequestBody.

::: dom/workers/XMLHttpRequest.cpp
@@ +18,5 @@
>  #include "mozilla/ArrayUtils.h"
>  #include "mozilla/dom/Exceptions.h"
>  #include "mozilla/dom/File.h"
>  #include "mozilla/dom/FormData.h"
> +#include "mozilla/dom/URLSearchParams.h"

alphabetic order.
Attachment #8763376 - Flags: review?(amarchesini) → review-
> This is wrong. URLSearchParams is not passed as nsIVariant but as
> RequestBody.

The nsIVariant path is currently used by worker XHR code. The main thread case does indeed get a RequestBody, but I currently need this patch hunk and the IID and structuring cloning stuff for workers to work as the tests expect.

However, I just noticed bug 1269162... were you suggesting that I just ignore the worker case for now, as it will end up making the nsIVariant XHR stuff obsolete? If so, I'll just omit all of the worker-related stuff from my patches (including the structured cloning stuff, unless you think that's worth keeping?)
Unfortunately, baku, even with your recent XHR changes the IID stuff still seems necessary. If I omit the code you suggest, workers still end up sending URLSearchParams to the XHR as an nsIVariant as before, where they end up being treated as a string (and so the correct default content-type isn't used, but rather the one for strings). And when I try to query the URLSearchParams object back out of the nsIVariant, I'm of course told to give the class an IID. Am I missing something?

At any rate, I've rebased the patches to m-i to perhaps make subsequent reviews a bit easier (given the changes you've made to the XHR code recently). I've also addressed your other review comments, except that I couldn't tell how much more alphabetized I could get that include, since the headers aren't alphabetized in that file to begin with. Where would you prefer that line?

I also tossed the web platform test change into the patch you were reviewing as well, since those changes make most sense to land with that patch. The WPT bits were already reviewed by :annevk anyhow, and nothing has really changed aside from the rebase and other review comments being addressed, so I'll have to wait for your word on this patch (and :dragana's on this first patch as :khuey suggested).
Attachment #8763375 - Attachment is obsolete: true
Attachment #8763375 - Flags: review?(dd.mozilla)
Attachment #8766169 - Flags: review?(dd.mozilla)
Attachment #8762923 - Attachment is obsolete: true
Attachment #8763376 - Attachment is obsolete: true
Attachment #8766170 - Flags: review?(amarchesini)
Carrying over r=khuey on this one, since it's just a rebase to m-i.
Attachment #8763654 - Attachment is obsolete: true
Comment on attachment 8766170 [details] [diff] [review]
1207233-part2-give-URLSearchParams-an-IID-and-make-XHRs-use-it-properly.diff

Review of attachment 8766170 [details] [diff] [review]:
-----------------------------------------------------------------

I see why we need nsIVariant. But I don't like to have a NS_URLSEARCHPARAMS_IID. What about if URLSearchParams implements nsIXHRSendable?

Doing this we end up here:
https://dxr.mozilla.org/mozilla-central/source/dom/xhr/XMLHttpRequestMainThread.cpp#2317

and we don't need anything else.
Plus, you are already implementing GetSendInfo() in URLSearchParams and GetSendInfo() is part of nsIXHRSensable interface.
All the rest looks good.

Ah! And you will hate me again: I moved URLSearchParams in dom/url/...

::: dom/base/URLSearchParams.cpp
@@ +503,5 @@
> +
> +  DeleteAll();
> +
> +  uint32_t nParams, zero;
> +  nsString key, value;

nsAutoString

::: dom/xhr/XMLHttpRequestMainThread.cpp
@@ +2241,5 @@
>  }
>  
>  static nsresult
> +GetRequestBodyInternal(URLSearchParams* aURLSearchParams, nsIInputStream** aResult, uint64_t* aContentLength,
> +               nsACString& aContentType, nsACString& aCharset)

2 things:

1. 80chars max.
2. indentation
Attachment #8766170 - Flags: review?(amarchesini) → review-
> I see why we need nsIVariant. But I don't like to have a
> NS_URLSEARCHPARAMS_IID. What about if URLSearchParams implements
> nsIXHRSendable?

Ah, makes sense. I was operating on the premise that it would be better to get rid of nsIXHRSendable and just add the IID. Thanks for the patience.

Here's a new second patch. I'll cancel part 3.

Btw, would it be feasible to have workers just pull in the XHR code and use it, rather than using nsIVariants to clone the data to the main thread just to send it? Is it mostly out of a desire to not pull too much code into the worker threads?
Attachment #8766170 - Attachment is obsolete: true
Attachment #8766172 - Attachment is obsolete: true
Attachment #8766355 - Flags: review?(amarchesini)
Attachment #8766355 - Flags: review?(amarchesini) → review+
Indeed. But this is out of the scope of this bug: we should have 1 single XHR implementation able to work on main-thread and workers.
I'm unassigning myself because this isn't really my bug anymore.
Assignee: michael → nobody
Assignee: nobody → wisniewskit
Attachment #8766169 - Flags: review?(dd.mozilla) → review+
Trivial rebase of part 1. Carrying over r+.
Attachment #8664912 - Attachment is obsolete: true
Attachment #8766169 - Attachment is obsolete: true
Trivial rebase of part 2, now with an appropriate meta file for the modified web platform test. Carrying over r+.
Attachment #8766355 - Attachment is obsolete: true
A fresh try-run seems fine (failures seem unrelated): https://treeherder.mozilla.org/#/jobs?repo=try&revision=d54301995dd8

Requesting checkin.
Keywords: checkin-needed
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/643bfa8b72ba
Part 1: Allow not sending a Content-Type header, or sending a blank string for it, and have XMLHttpRequest.setRequestHeader honor those possibilities. r=dragana
https://hg.mozilla.org/integration/mozilla-inbound/rev/ff7dbda6a6d0
Part 2: Make URLSearchParams nsIXHRSendable as well as clonable, and have XHRs set the correct request content type for them. r=baku
Keywords: checkin-needed
Are you sure these failures were caused by these patches? Aside from a bunch of (what seem to be) known intermittents, I can't see how these changes could cause any of the missing file/python module issues, or "unused willResolvePlugins" and other build failures...
Flags: needinfo?(cbook)
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ca67297f6c62
Part 1: Allow not sending a Content-Type header, or sending a blank string for it, and have XMLHttpRequest.setRequestHeader honor those possibilities. r=dragana
https://hg.mozilla.org/integration/mozilla-inbound/rev/01c1359222ea
Part 2: Make URLSearchParams nsIXHRSendable as well as clonable, and have XHRs set the correct request content type for them. r=baku
(In reply to Thomas Wisniewski from comment #60)
> Are you sure these failures were caused by these patches? Aside from a bunch
> of (what seem to be) known intermittents, I can't see how these changes
> could cause any of the missing file/python module issues, or "unused
> willResolvePlugins" and other build failures...

Hi Thomas,

yeah i think you're right, checked this in again, i guess the bustage was from the push before yours. sorry!
Flags: needinfo?(cbook)
Oh no problem, I just wanted some reassurance before I embarked on a possible wild-goose chase (especially if it meant another patch was being given time to waste more try-runs). Thanks!
Pushed by amccreight@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c6bd7036ef52
Add explicit to URLSearchParams ctor. r=bustage
Depends on: 1328761
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: