1567211 - Downloading a file results in a "could not be saved, because the source file could not be read"

Status: RESOLVED INVALID

(Core :: Networking: HTTP, defect)

Description

[Michael Craft]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36

Steps to reproduce:

I can replicate this bug at will using the following steps:

1. Went to our website and attempted to Review a report
2. After a while, since the report is around 14MB in size (I know this because I used IE11 to successfully download the file), we get the dialog box for the name of the report with the options Open With or Save File.
3. If you wait just a few seconds you get a new dialog box that opens up saying "C:\Users\<Username>\AppData\Local\Temp\<filename>.csv.part could not be saved, because the source file could not be read. Try again later, or contact the server administrator." with an OK button.
4. If you press the okay button the download dialog box and the new dialog box are closed and the file is not downloaded. NOTE: You do temporarily have a 0KB <filename>.csv.part file in the Temp directory, until you click the OK button on the dialog box

If you do not wait the few seconds in 3) and click on the Open with or Save File option then you will get a 0KB file with the name of the report that is empty.

I have tested this on a smaller file size and can replicate the issue. I have also done a log for the process which I am attaching to the ticket. There are other additional logs that I can provide upon request.

Since this is our personal webpage and contains HIPAA information I cannot share the webpage or do a testcase for the issue unfortunately. I have tested this using the 69 Branch and 70 Branch versions of Firefox and can reproduce the issue on both of those versions as well. I downloaded a 66.0.5 version of Firefox and was NOT able to replicate the issue. So it looks like the issue was introduced with the 68 branch since it was working just a week ago.

Actual results:

The file does not download

Expected results:

The file downloads correctly.

Comment 1

[Michael Craft]

Attachment: Link to Log file I created for Bug 1567211

Comment 2

[Luciana queirolo]

Hi,

I was not able to reproduce the issue on Nightly 70.0a1 (2019-07-23) and Release.
Also, Could you please try to see if it's reproducible on safe mode and on a new profile on Nightly? here is the link for download https://www.mozilla.org/en-US/firefox/nightly/all/

Will set the product to Toolkit and component to "Downloads API", feel free to change it if this is not correct.

Thanks!

Comment 3

[Michael Craft]

I upgraded to the (2019-07-23) version of Nightly and I can still replicate the issue. I also started Firefox in safe mode and continue to have the issue.

I am actually not using a profile for Nightly and its not using my one from the normal production version of Firefox I use. Do you want me to actually create a profile for the Nightly version?

Comment 4

[Michael Craft]

I disabled all Add-Ons as well and when I did that I got a different result. Instead of getting the "C:\Users\<Username>\AppData\Local\Temp\<filename>.csv.part could not be saved, because the source file could not be read. Try again later, or contact the server administrator." message I instead got the pop-up window to open the file in Excel. When I did that it still failed, but in the Downloads manager I saw the file listed with a Retry option. When I did the retry I did get a file but it was not one with the data I should be expecting. I am uploading the file I got to this ticket.

Comment 5

[Michael Craft]

Attachment: Billing_Review_CSV.csv

Comment 6

[Michael Craft]

Correction, If I gave it long enough with the Add-Ons disabled I will get the "C:\Users\<Username>\AppData\Local\Temp\<filename>.csv.part could not be saved, because the source file could not be read. Try again later, or contact the server administrator." message, it just seems to take a bit longer.

Comment 7

[Michael Craft]

Here is the different behavior if I disable Add-Ons

1. When I disabled the Add-Ons I get different behavior from Firefox. (Correction - I still get the "could not be saved..." message it just seems like it takes longer).
2. Instead of getting the "could not be saved, because the source file could not be read" message I get the pop-up to Open in Excel or Save the file.
   a) If I choose the Open in Excel the file does not open in Excel.
   1) I go to the FireFox Download manager (Ctrl-J) and can see the file listed (before there would be no file listed in the Download manager).
   2) I have the option to retry the file.
   3) When I click on that I do get an Excel file but it contains none of the data that you would expect in the report.
   b) If I choose the Save option
   1) Then I get a dialog box to save the file
   2) I chose the folder I want to save the file to.
   3) When I go to look at the file it is a 0 byte file and is blank when I open it.
   4) I also see a file named Billing_Review_CSV.csv.part in the same directory.
   a) If I change the file name to be just Billing_Review_CSV.csv then I actually have data from the report show up

Comment 8

[:Gijs (he/him)]

Thanks for your patience with all the questions so far.

Just to check the socket takes a very long time to get headers from the server (3 minutes, in the log, if I'm reading it right). With the smaller files that you tried, is that time shorter, and does the same issue happen? (that is, can we exclude the problem being that something (firefox, server, firewall, wifi, ...) kills off the socket / http connection?)

I think the following line in the log points to the problem:

2019-07-18 13:38:51.509000 UTC - [Parent 31928: Socket Thread]: V/nsHttp Partial transfer, incomplete HTTP response received: broken chunk

The server is using chunked encoding, and somehow we are not happy with what it's sending. AFAICT this is coming from https://searchfox.org/mozilla-central/rev/40e889be8ff926e32f7567957f4c316f14f6fbef/netwerk/protocol/http/nsHttpTransaction.cpp#1132 .

Based on this being new in 68, I suspect bug 1531344 is at fault. If I'm right, setting "network.http.enforce-framing.strict_chunked_encoding" to false in about:config might resolve the issue for you (I mean, there'd still be a bug somewhere, but you would be able to resume using your application). You might need to restart the browser for that pref change to take effect. Can you confirm if this is the case?

Dragana, can you take a look (and/or comment on what else would be needed here to do more in depth diagnosis of whether Firefox is misreading chunks or whether the server is sending bogus things or... ) ?

Comment 9

[Michael Craft]

:Gijs - Per your request I turned the "network.http.enforce-framing.strict_chunked_encoding" to false and retried my download. I did not receive the "C:\Users\<Username>\AppData\Local\Temp\<filename>.csv.part could not be saved, because the source file could not be read. Try again later, or contact the server administrator." message I was receiving. When I chose the Open in Excel the file opened with no issues what so ever. I then did the Save File option and again the file was downloaded with no issues. I then set that setting back to true and BAM the issue was right back. I re-changed it to false and the issue went away again.

This gives me some solid information that I can take to the developers/server admins of our application to see if they can look into what we might be able to do to help resolve this without having to make our customers go in and modify their Firefox configurations. Is there any documentation that I can get that shows exactly what the feature does? It would be helpful to have that going back to our developers/server admins.

Comment 10

[Michael Craft]

Just as a side note. We were also having an issue with PDF files not displaying and getting an error message with them about Opening in a Different Viewer. Changing the "network.http.enforce-framing.strict_chunked_encoding" to false has also resolved that issue as well. This also just started occurring with the release of Firefox 68.

Comment 11

[:Gijs (he/him)]

(In reply to Michael Craft from comment #9)

Good to hear the pref helped!

This gives me some solid information that I can take to the developers/server admins of our application to see if they can look into what we might be able to do to help resolve this without having to make our customers go in and modify their Firefox configurations. Is there any documentation that I can get that shows exactly what the feature does? It would be helpful to have that going back to our developers/server admins.

I'm not an expert in our networking stack, unfortunately, so all I can say is that, in the default configuration, Firefox 68 is less tolerant to broken http chunking. Flipping the pref returns it to the previous, less spec-compliant/strict but therefore more tolerant behaviour. The previous less strict implementation broke some things when using YouTube, which is why the change was made. Maybe Bryce can provide some more details, perhaps your developers would be able to make more sense of some of the context in bug 1531344 than I can.

From some of the comments on bug 1531344 and others, it seems that it's possible (perhaps via configuration options or updates?) to make the server behave correctly, which should fix this problem. The comments also lead me to believe a similar issue would affect Chrome, if you used it for the same task. If that's not the case, I suspect that'd be interesting to our network engineers.

Comment 12

[Michael Craft]

We actually don't recommend the use of Chrome with our application. Only Firefox and IE11, which interesting enough is also having an issue but it automatically gave us a retry option which then downloaded the file, unlike the True setting you had me change which just outright failed the download with no recovery/restart options.

Comment 13

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Dragana is the expert here, and will hopefully correct me if I'm wildly off base.

The regressing bug here changed how chunked transfer encoding is treated. Loose explanation of chunked encoding: in this mode the server will inform the browser that it plans to send a chunk of a certain size, then send through that chunk. This can be done a number of times until a final, terminating chunk is sent and the browser can assemble the sent data.

If the server says that it's going to send a chunk of a certain size, say 18 bytes, but then only sends 4 bytes, this is problematic. Similarly, if the server fails to send the terminating chunk. In some cases servers do this and expect the browser to roll with it and assemble the chunks that have been sent. However, in other cases these are legitimate errors the browser needs to handle and re-request that data from the server. If we behave correctly (per specifications) we break interaction with servers that rely on Firefox ignoring incorrect chunked encoding. If we are tolerant of incorrect chunked encoding, we risk breaking sites that rely on the correct behaviour.

Historically Firefox was tolerant of incorrect chunked encoding because a number of servers were doing it incorrectly and handling it properly would break various sites. However, in doing so we introduced incompatibility with other sites. A notable case of this is Youtube, which relies on us handling chunking correctly. Because of this, in bug 1531344 we moved towards stricter handling of chunked encoding.

I believe most other browsers have moved towards similar handling. Michael, do you see this fail in other browsers?

As noted in comment 11, the ideal fix here would be to update or reconfigure the server so it's handling chunked encoding to spec.

For those interested, see bug 1088850 and bug 237623 for history around this.

Comment 14

[Michael Craft]

:bryce For IE11 we do see it fail, but it has a retry option that lets us/our clients get the file eventually.

Comment 15

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Michael, are you in control of the server hosting the site, or are able to contact the host? Ensuring the servers involved are handling chunked encoding correctly is the best resolution I see here. I appreciate that it's frustrating to have this functionality no longer work, but browsers maintaining compatibility with this server bug is going to decrease, in my opinion.

Comment 16

[Michael Craft]

I personally do not have control of the server, I have escalated an internal issue within our company to try and address the issue on our servers but honestly don't know if that will be done or not. :(

At least you have given me a work around we can have our customers do if our management decides not to make changes to our servers to handle chunked data correctly.

Thanks all for your help, it has been GREATLY appreciated.

Comment 17

[Michael Craft]

Hello, my company is pushing back on me, do you have the Data Specifications that you are using to make/add the change in how you handle chuncked data? Where/Who are you getting them from and can I get a link of a document that shows that so I can then take it back to my company?

Comment 18

[Michael Craft]

From the other ticket I think the specifications you referenced are part of the HTTP 1.1 Protocol? Is that correct?

Comment 19

[Dragana Damjanovic [:dragana]]

(In reply to Michael Craft from comment #17)

Hello, my company is pushing back on me, do you have the Data Specifications that you are using to make/add the change in how you handle chuncked data? Where/Who are you getting them from and can I get a link of a document that shows that so I can then take it back to my company?

the rfc you are looking for is:
https://tools.ietf.org/html/rfc7230#section-4.1

Your server is not returning zero-length chunk at the end. The last chunk must be zero-length chunk.

Comment 20

[Ryan VanderMeulen [:RyanVM]]

AFAIK, we haven't seen any other regression reports come in yet for this change. Un-tracking for 69. Not sure this bug should even stay open if it's a server configuration issue?

Comment 21

[Honza Bambas (:mayhemer)]

Server issue.