Open Bug 1623503 Opened 5 years ago Updated 5 years ago

Response payload is base64 encoded

Categories

(DevTools :: Netmonitor, defect, P3)

75 Branch
defect

Tracking

(Not tracked)

People

(Reporter: pkukielka, Unassigned)

References

(Blocks 1 open bug)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0

Steps to reproduce:

request:
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0
Accept: application/json, text/plain, /, application/msgpack
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded, application/msgpack
X-Requested-With: XMLHttpRequest
Content-Length: 203
Origin: https://example.com:8080
Connection: keep-alive
Referer: https://example.com:8080/api
Cookie:
Pragma: no-cache
Cache-Control: no-cache
TE: Trailers

response:
HTTP/2 200 OK
server: nginx
date: Wed, 18 Mar 2020 12:14:45 GMT
content-type: application/msgpack
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate
pragma: no-cache
access-control-allow-credentials: true
access-control-allow-methods: post, get
access-control-allow-origin: https://example.com:8080
access-control-allow-headers: content-type, x-requested-with
strict-transport-security: max-age=15768000
X-Firefox-Spdy: h2

im seding this response body (its php msgpack_pack(['data' => ['me' => null]]);):
¤data¢meÀ

Actual results:

response payload is base64 encoded in dev tools ie:
gaRkYXRhgaJtZcA=

note:
fetch with 'response.arraybuffer()' gives correct data

Expected results:

one of:

  1. explain behavior at https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor/request_details
  2. add notice to dev tools that what i see is not what server send
  3. dont encode but it might be some security setting

additional note:
doesnt happen in chrome

Moving to the right component so the development team can further analyze this.

Component: Untriaged → Netmonitor
Product: Firefox → DevTools

Thanks for the report!

Can you please provide a simple page (a test case) we could use to reproduce the issue on our machines?
That would already be a great help!

Honza

Flags: needinfo?(pkukielka)

link:

https://mozbug1623503.000webhostapp.com/index.php

src:

<?php 
    $headers = getallheaders();
	if (isset($headers['accept']) && $headers['accept'] === 'application/msgpack') {
		header('content-type: application/msgpack');
		
		echo 'meh';
		die;
?>

<?php } else { ?>
	<html>
		<body>
			<button id="bug" onclick="send()">Click me</button>
			
			<script>
				function send() {
					async function get() {
						const response = await fetch('https://mozbug1623503.000webhostapp.com/index.php', {
							method: 'POST',
							mode: 'cors',
							cache: 'no-cache', 
							credentials: 'same-origin',
							headers: {
							  'Content-Type': 'application/msgpack',
							  'accept': 'application/msgpack',
							},
							redirect: 'manual',
							referrerPolicy: 'no-referrer',
							body: 'test'
						});
						
						return await response.arrayBuffer();
					};
					
					get().then((res) => { 
						const arr = new Uint8Array(res)
						console.log(arr);
						
						const str = arr.reduce((acc, item) => acc += String.fromCharCode(item), '' )
						console.log(str);
					});
				}
			</script>
		</body>
	</html>
<?php } ?>

simplified to just echo 'meh' gives
{"Response Payload":{"EDITOR_CONFIG":{"text":"bWVo","mode":"application/msgpack"}}}

also not sure if its clear but using "Developer Edition" of FF

Flags: needinfo?(pkukielka)

Thanks for the test case!

I uploaded that online here:
http://www.janodvarko.cz/tests/bugzilla/1623503/index.php

But I am getting TypeError: NetworkError when attempting to fetch resource. when clicking the button.

What's wrong there?

Honza

Flags: needinfo?(pkukielka)

hello,

not sure it kind of works for me (i get normal render as with GET but its probably your server config [probably nothing set in getallheaders function]), thats why to be sure ive uploaded it to some free hosting with link

ive chaned php version to match yours (php 5.6) and updated code to not use getallheaders):

<?php 
	if (!empty(file_get_contents('php://input'))) {
		header('content-type: application/msgpack');
		
		echo 'meh';
		die;
?>

<?php } else { ?>
	<html>
		<body>
			<button id="bug" onclick="send()">Click me</button>
			
			<script>
				function send() {
					async function get() {
						const response = await fetch('./index.php', {
							method: 'POST',
							mode: 'cors',
							cache: 'no-cache', 
							credentials: 'same-origin',
							headers: {
							  'Content-Type': 'application/msgpack',
							  'accept': 'application/msgpack',
							},
							redirect: 'manual',
							referrerPolicy: 'no-referrer',
							body: 'test'
						});
						
						return await response.arrayBuffer();
					};
					
					get().then((res) => { 
						const arr = new Uint8Array(res)
						console.log(arr);
						
						const str = arr.reduce((acc, item) => acc += String.fromCharCode(item), '' )
						console.log(str);
					});
				}
			</script>
		</body>
	</html>
<?php } ?>

link is still the same
https://mozbug1623503.000webhostapp.com/index.php
sorry for ads on it :/

Flags: needinfo?(pkukielka)

The priority flag is not set for this bug.
:Honza, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(odvarko)

Thanks, I can reproduce the problem now!

STRs:

  1. Open DevTools and select the Network panel
  2. Load http://www.janodvarko.cz/tests/bugzilla/1623503/index.php
  3. Click the button on the page
  4. Select the request and check out the Response panel
  5. The response is bWVo (based64 for meh)

decodeURIComponent(atob("bWVo")) => meh

Comments:

application/msgpack is not in the list.

  • The Backend is setting a flag when encoding the response response.encoding = "base64"; but, it isn't checked out on the front end when rendering the response text in the Response panel.

https://searchfox.org/mozilla-central/rev/b7f3977978922d44c7d92ae01c0d4cc2baca7bc2/devtools/client/netmonitor/src/components/previews/SourcePreview.js#201

Honza

Status: UNCONFIRMED → NEW
Has STR: --- → yes
Ever confirmed: true
Flags: needinfo?(odvarko)
Summary: response payload is base64 encoded → Response payload is base64 encoded

Honza, should we pivot this bug into supporting msgpack formatting for responses; or is this affected other mime-types as well?

Flags: needinfo?(odvarko)

So, I am seeing several issues here:

  1. The way we are detecting textual response is weak - we are using fixed list of known textual mime types. It would be safer to check whether the response itself is binary or text. See bug 752288 (detect binary responses)

  2. This bug isn't only about application/msgpack it's about any binary response that is base64 encoded (to be sent to the client over RDP) but, not decoded when displayed to the client. The client is not checking the base64 field (set on the backend when encoding happens). Properly show binary response to the client is blocked by bug 1555641 (binary viewer)

  3. It would be great to parse application/msgpack responses and show formatted view (JSON tree) to the user. Just like we do for WS frames. I am unsure how we want to show binary fields within the application/msgpack JSON structure (there might be some, right?). Perhaps keep them base64 decoded?

So, yes we can use this bug to properly parse application/msgpack if it helps.

Honza

Flags: needinfo?(odvarko)

Related, I saw in user feedback a case where utf8 encoded responses got base64-encoded – presumably also because of missing mime types. So solving this for beyond msgpack would be good to keep in mind.

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is P3 (Backlog,) indicating it has been triaged, the bug's Severity is being updated to S3 (normal.)

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.