1629921 - CORS preflight triggered for User-Agent header when GeckoSessionSettings.setUserAgentOverride() is used for setting a custom user agent

Wicher Minnaard

Reporter

Description

•

5 years ago

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

Initialize a GeckoView session with some variation of:

new GeckoSession(new GeckoSessionSettings.Builder()
.userAgentOverride("Pannekoek")
.build()
)

Perform a CORS request that should be simple — simple as per CORS spec, which is to say, no need for an OPTIONS preflight, as only browser-added headers will be sent . JS console:

await fetch('https://admin.asteroid.tl/media/images/elbow_sneeze.width-300.png', {"mode": "cors", "credentials": "omit"})

CORS request fails, error in JS console is:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://admin.asteroid.tl/media/images/elbow_sneeze.width-300.png. (Reason: CORS request did not succeed).

Actual results:

Capturing the HTTP conversation (with Apache's log_forensic) shows that Geckoview preflights this request, as follows:

As this particular server does not allow the OPTIONS method (doesn't matter for this issue, as the point is that there shouldn't be a preflight at all), the preflight and thus the rest of the request fails. Note two things:

a) The CORS header: Access-Control-Request-Headers:user-agent
This is odd, as user-agent is not a header that needs to be preflighted, as it's set by the browser and not in JS-land.

b) Of much less concern, the actual user agent sent with this OPTIONS request is not "Pannekoek".

Expected results:

No CORS preflight OPTIONS request, as it should be a simple request (per https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#Simple_requests).
"Pannekoek" sent as User-Agent header in any request.

Note that a simple request (sans preflight) is exactly what happens if I don't apply
.userAgentOverride() to the Geckoview session settings builder.

James Willcox (:snorp) (jwillcox@mozilla.com) (he/him)

Comment 1

•

5 years ago

Honza, any ideas here?

Flags: needinfo?(honzab.moz)

Firefox Bug Husbandry Bot

Comment 2

•

5 years ago

Resetting severity to default of --.

Honza Bambas (:mayhemer)

Comment 3

•

5 years ago

Could be a regression from bug 1615967, but I'm not 100% sure.

Anyway, can you please either provide a simple test case (never work with GeckoView directly) I can test with locally or bisect using mozregresison?

Thanks.

Flags: needinfo?(honzab.moz) → needinfo?(wicher)

Emily Toop (:fluffyemily)

Updated

•

5 years ago

Priority: -- → P5

James Willcox (:snorp) (jwillcox@mozilla.com) (he/him)

Comment 4

•

5 years ago

Honza, you should be able to reproduce this by building (or installing from treeherder) the example app and change the useragent in the settings.

Flags: needinfo?(honzab.moz)

Wicher Minnaard

Reporter

Comment 5

•

5 years ago

I've done some more digging and this is what came up:

I've tried a number of builds (from https://maven.mozilla.org/?prefix=maven2/org/mozilla/geckoview/geckoview/) and the problem occurs as early as geckoview-70.0.20191022130254 (and possibly with 69.0.20190903125908 as well, but I haven't tested that one).
In geckoview-75.0.20200403170909 and geckoview-76.0.20200429185419 I can peruse .useMultiprocess(false) on a GeckoRuntimeSettings.Builder, which prevents the problem from occurring; the resource is not preflighted. (Also, the .userAgentOverride() on the GeckoSessionSettings.Builder does not have the desired effect of actually setting the UA header; it only has the effect of triggering the bug (iff .useMultiProcess(true)). So that's the "user agent override does not work" part of this bug; the other part is "user agent override triggers preflight CORS".). This suggests that the bug might have to do something with e10s.
When (tested with geckoview-76.0.20200429185419) I set the user agent through a user.js pref (general.useragent.override), preflight is not triggered, and the custom UA actually shows up in the request headers! Yay. At least there's a workaround!

Now, the plot thickens:

Firefox Desktop itself also displays this behaviour; it triggers a CORS preflight when you set a user agent (for instance with https://addons.mozilla.org/en-US/firefox/addon/user-agent-string-switcher/). (but, in contrast to android-geckoview's .userAgentOverride(), doing so actually has the desired effect of setting the User-Agent request header, apart from triggering the CORS preflight bug). The bug is not triggered when you set a UA through the general.useragent.override pref.
I've started digging into the Firefox source and added some debug logging to nsCORSListenerProxy::CheckPreflightNeeded() in netwerk/protocol/http/nsCORSListenerProxy.cpp (of rev 528536:98040184b6c0 of mozilla-central). Now when the bug is triggered I can see that loadInfo->CorsUnsafeHeaders() (at netwerk/protocol/http/nsCORSListenerProxy.cpp:988) contains the User-Agent header, which is inconsistent with the CORS spec (the User-Agent header is not an unsafe header, as it can't be set from page context JS).

Flags: needinfo?(wicher)

Honza Bambas (:mayhemer)

Comment 6

•

4 years ago

(In reply to Wicher Minnaard from comment #5)

Now when the bug is triggered I can see that loadInfo->CorsUnsafeHeaders() (at netwerk/protocol/http/nsCORSListenerProxy.cpp:988) contains the User-Agent header, which is inconsistent with the CORS spec (the User-Agent header is not an unsafe header, as it can't be set from page context JS).

https://searchfox.org/mozilla-central/rev/09b8072a543c145de2dc9bb76eddddd4a6c09adc/netwerk/protocol/http/nsCORSListenerProxy.cpp#987-991

The UA header bubbles up from either fetch or xhr.

I won't be able to look at this soon, leaving ni? on me to not loose it.

Wicher Minnaard

Reporter

Comment 7

•

4 years ago

Since it has now become apparent that this is a general Firefox bug instead of an Android GeckoView specific bug, it should probably be reclassified as such instead of staying in the niche "The GeckoView components for Android" category I filed it in.

Honza Bambas (:mayhemer)

Comment 8

•

4 years ago

Wicher, can you provide a way to reproduce this with Firefox Desktop browser?

Flags: needinfo?(wicher)

Wicher Minnaard

Reporter

Comment 9

•

4 years ago

Yes. See earlier comments. If you'd set a custom user agent through https://addons.mozilla.org/en-US/firefox/addon/user-agent-string-switcher/ you should see the spurious OPTIONS request appearing, and disappearing again when you disable that addon. That is, if it's still doing it the same way as a month ago. For details on exactly which versions I see it happening on, and where to insert debug logging that shows the symptom in more detail, please refer to my earlier comments explanations.

Flags: needinfo?(wicher)

Honza Bambas (:mayhemer)

Comment 10

•

4 years ago

Testing with the latest Nightly (desktop) and https://addons.mozilla.org/en-US/firefox/addon/user-agent-string-switcher/ set to Custom:

{
  "www.google.com": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36",
  "www.bing.com, www.yahoo.com, www.wikipedia.org": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0",
  "www.example.com": [
    "random-useragent-1",
    "random-user-agent-2"
  ],
  "*": "useragent-for-all-hostnames"
}

I'm getting:

From a 3rd party web page's console:

await fetch('https://admin.asteroid.tl/media/images/elbow_sneeze.width-300.png', {"mode": "cors", "credentials": "omit"})

getting:

GET /media/images/elbow_sneeze.width-300.png HTTP/2
Host: admin.asteroid.tl
User-Agent: useragent-for-all-hostnames
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://test.local.predator/
Origin: https://test.local.predator
DNT: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site

and

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://admin.asteroid.tl/media/images/elbow_sneeze.width-300.png. (Reason: CORS header ‘Access-Control-Allow-Origin’ does not match ‘https://haroman.tl’).

From a 3rd party web page's console:

// Example POST method implementation:
async function postData(url = '', data = {}) {
  // Default options are marked with *
  const response = await fetch(url, {
    method: 'POST', // *GET, POST, PUT, DELETE, etc.
    mode: 'cors', // no-cors, *cors, same-origin
    cache: 'no-cache', // *default, no-cache, reload, force-cache, only-if-cached
    credentials: 'same-origin', // include, *same-origin, omit
    headers: {
      'Content-Type': 'application/json'
      // 'Content-Type': 'application/x-www-form-urlencoded',
    },
    redirect: 'follow', // manual, *follow, error
    referrerPolicy: 'no-referrer', // no-referrer, *no-referrer-when-downgrade, origin, origin-when-cross-origin, same-origin, strict-origin, strict-origin-when-cross-origin, unsafe-url
    body: JSON.stringify(data) // body data type must match "Content-Type" header
  });
  return response.json(); // parses JSON response into native JavaScript objects
}

postData('https://example.com/answer', { answer: 42 })
  .then(data => {
    console.log(data); // JSON data parsed by `data.json()` call
  });

getting OPTIONS request first:

OPTIONS /answer HTTP/2
Host: example.com
User-Agent: useragent-for-all-hostnames
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type
Origin: https://test.local.predator
DNT: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
Cache-Control: max-age=0

and

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://example.com/answer. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://example.com/answer. (Reason: CORS request did not succeed).

So this seems to be fixed or irreproducible on desktop as proposed in comment 9.

Flags: needinfo?(honzab.moz) → needinfo?(wicher)

Wicher Minnaard

Reporter

Comment 11

•

4 years ago

I spent some time trying to isolate this bug. It's.... complicated. Bear with me ;-)

I managed to reproduce it, with many Firefoxes and many Geckoviews, on a bunch of sites that use a serviceworker (in this case, all based on Workbox, thought that is likely not relevant)*.
Yet when I globally disable dom.serviceWorkers.enabled through about.config, the bug goes away.
Also, with those exact same sites, Chromium (Chrome) never perform the spurious OPTIONS requests. It's unlikely that these OPTION requests are instigated explicitly at the SW code's behest, because my tracing of nsCORSListenerProxy::CheckPreflightNeeded() shows (see comment 5) that Firefox itself makes a wrong decision, classifying the user-agent header as cors-unsafe — ~~when that is simply never the case, the page JS can't set that header, it's directly against the spec~~. Or so I had understood at the time I ran in to this back in April. One month ago the MDN page on forbidden header names was updated, and now mentions:

Note: The User-Agent header is no longer forbidden, as per spec — see forbidden header name list (this was implemented in Firefox 43) — it can now be set in a Fetch Headers object, or via XHR setRequestHeader(). However, Chrome will silently drop the header from Fetch requests (see Chromium bug 571722).

So, that sheds some more light on it. It's not a misclassification. The UA header is no longer a forbidden header name, and thus, since it's settable from page JS AND not on the CORS request header whitelist, a preflight is warranted if we follow the letter of the law. Chrome doesn't, and drops it from any fetch(), while Firefox keeps it and indeed, doing a fetch() with a UA header set, classifies it as CORS-unsafe and thus necessitates a preflight.

That's useful info but that doesn't solve the problem (which is one of surprise, and of where the web extensions hook in).
Here's my current suspicion:

It seems that for a plain fetch() done from the page (or dev console), a request that from the point of view of the page should be cors-simple is classified correctly as cors-simple, no preflight needed. Indicating that the extension-supplied UA header is injected AFTER firefox has applied the CORS classification of simple vs preflight-needed.
For a request that is intercepted by the serviceworker (and then resolved from cache, or re-scheduled to launch over the network), it looks like the UA-header is injected before the SW intercepts it, or at least before Firefox applies a CORS classification (of simple vs preflight-needed). Firefox then sees a new request scheduled by the SW, with UA header set, goes "ok the SW set a UA, well, it's their party, they know what they're doing, so they get to keep the pieces, I'm preflighting it", and promptly preflights it.

Item 2 needs some further exploration. If the SW sees the extension-injected header at the point of interception, at least it can remove it to make sure that requests don't get preflighted when the developers don't expect it (and when the resource backend doesn't support OPTIONS). But that would go against the intentions of the user, who wanted to override the useragent with an extension. That is not an ideal situation! I'll have a look to verify if at any point the SW can see the extension-injected UA header.

A narrowed down minimal test case would be handy, which I think would need to involve

a serviceworker that MITMs a cross-origin request that is CORS-safe (a simple GET with only CORS-safelisted request headers would do)
a UA set through Firefox' extension mechanism
a way to see the spurious OPTION requests - they don't show up in the developer network console. Server logging would help, as would serverside forbidding of the OPTIONS method on the requested resource (as then the spurious CORS preflight fails, which will result in an obvious error in the developer JS console when performing the fetch().)

*) example instance, which I didn't set up for test case purposes; so no guarantees, don't wait three months to have a play with it because its setup may have changed by then: haroman.tl, go there and try a fetch: await fetch('https://admin.haroman.tl/media/images/elbow_sneeze.width-300.png', {"mode": "cors", "credentials": "omit"}) and if you have a custom UA set through a browser extension such as https://addons.mozilla.org/en-US/firefox/addon/user-agent-string-switcher/ you should be able to observe the OPTIONS preflight request being made (and failing, as the server doesn't support OPTIONS on that resource), and unsetting the custom user agent fixes that.

Flags: needinfo?(wicher)

Honza Bambas (:mayhemer)

Updated

•

4 years ago

Flags: needinfo?(honzab.moz)

Honza Bambas (:mayhemer)

Updated

•

4 years ago

Flags: needinfo?(honzab.moz)

kkapsner

Comment 12

•

3 years ago

There is a general problem with CORS when a web extension modifies standard headers. E.g. https://github.com/kkapsner/CanvasBlocker/issues/612 describes a real world problem. I could also see the problem when the referer header is changed.

Should I create a new bug with the details for the problem with web extension (I should be able to create a minimal web extension to show the problem) since this bug has the product wrong?

Flags: needinfo?(honzab.moz)

BugBot [:suhaib / :marco/ :calixte]

Comment 13

•

2 years ago

Redirect a needinfo that is pending on an inactive user to the triage owner.
:amoya, since the bug has recent activity, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(honzab.moz) → needinfo?(amoya)

Chris Peterson [:cpeterson]

Updated

•

2 years ago

Severity: normal → N/A

Flags: needinfo?(amoya)

Chris Peterson [:cpeterson]

Updated

•

2 years ago

Severity: N/A → S4

Bugzilla

CORS preflight triggered for User-Agent header when GeckoSessionSettings.setUserAgentOverride() is used for setting a custom user agent

Categories

(GeckoView :: General, defect, P5)

Tracking

(Not tracked)

People

(Reporter: wicher, Unassigned)

References

Details

Crash Data

Security

(public)

User Story

Description

Comment 1

Comment 2

Comment 3

Updated

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Updated

Updated

Comment 12

Comment 13

Updated

Updated