1342659 - download UTF8 fails on ISO storage

Status: RESOLVED FIXED

(Toolkit :: Downloads API, defect, P5)

Description

[plop]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20170103200923

Steps to reproduce:

Try to save a web page, or try to download a file whihc name contains special characters (not ALPHA-DIGIT), like punctuation or letters with accents.


Actual results:

Final destination file, and part file created fine. Data are grabbed over network and put in the part file. When network transfert is completed, the download manager says the download failed. Data is kept in the .part file, final destination file remains empty.


Expected results:

I migrated to FF50 recently. Before, I was using FF17.  I have always been using ISO storage (since 1998), and download always went fine in firefox. My X is also using ISO locales (Enlightenment can stand UTF, but I intensively use Eterms, and Eterm port to accept UTF is heavily broken, so I never could find any better term accepting UTF, so, I have to stick to ISO). So, using FF17 last month, things were fine: data downloaded to .part file, and when completed, move file to the finale good name.

If, at download start, I clean the name from any char that could produce ambiguity between ISO and UTF (removing all accents and punctuation), then it goes all fine.

This is a regression bug compared to FF17.

Comment 1

[Loic]

Any example of file or page to test?

Comment 2

[plop]

https://www.youtube.com/watch?v=lZQqr6Lwjhg
hit ^s ... default saved as

-rw-r--r-- 1 dhp    dhp            0 2017-02-27 19:27 Degré 0 de l'informatique - 01 - e-penser numérique - YouTube.htm
-rw-r--r-- 1 dhp    dhp            0 2017-02-27 19:28 Degre 0 de l informatique - 01 - e-penser numérique - YouTube.htm
-rw-r--r-- 1 dhp    dhp            0 2017-02-27 19:27 Degre 0 de l'informatique - 01 - e-penser numérique - YouTube.htm
drwxr-xr-x 4 dhp    dhp         4096 2017-02-27 19:28 Degre 0 de l informatique - 01 - e-penser numérique - YouTube_files
drwxr-xr-x 4 dhp    dhp         4096 2017-02-27 19:27 Degre 0 de l'informatique - 01 - e-penser numérique - YouTube_files
drwxr-xr-x 2 dhp    dhp         4096 2017-02-27 19:27 Degré 0 de l'informatique - 01 - e-penser numérique - YouTube_files
-rw-r--r-- 1 dhp    dhp       886148 2017-02-27 19:28 Degre 0 de l informatique - 01 - e-penser numérique - YouTube.htm
-rw-r--r-- 1 dhp    dhp       879145 2017-02-27 19:27 Degre 0 de l'informatique - 01 - e-penser numérique - YouTube.htm
-rw-r--r-- 1 dhp    dhp       879412 2017-02-27 19:27 Degré 0 de l'informatique - 01 - e-penser numérique - YouTube.htm

(I tried to also change the e, and remove the single quote).

Now, fix the target name to Grrr ... and you get

drwxr-xr-x 4 dhp    dhp         4096 2017-02-27 19:28 grrrr_files
-rw-r--r-- 1 dhp    dhp       866645 2017-02-27 19:28 grrrr.htm

http://picpaste.com/2017-02-27-193041_4864x2304_scrot-zknhRHM7.png
link should be valid a few months.

Rox shows recent files as bold. Normal files as black; red is for invalid UTF. After renaming files, it can show UTF names as black, even when they contain accents and quotes. What matters here is that things work the first shot with grrrr, while for other files, you have a black empty file, and data in the red file. Thant's a quick test. I will try to make a second test with a different kind of case (with .part involved).

Comment 3

[plop]

Illustration file: download link.
http://dl.free.fr/cS3jyW7aJ
link valid only for 30d from now. This is the HTML file from previous tests. I don't know any other way to store it online a reliable way for longer time.

NB: when I download the file, it's automaticly saved in my download folder. I have set FF this way. Things happen a different way when I click on "save link target as", which opens a window letting me edit the name (and fix the bad chars).

Result in my desktop: screenshot.
http://picpaste.com/2017-02-27-210826_4864x2304_scrot-IvXLRytV.png
should be hosted at least a year. Hopefully forever. In upper part, the Firefox download manager in failed state. In lower part, RoxFiler showing the .part file sticking after end of download, containing data, and empty target file, both with the right name (the hosting server free.fr probably removed spaces from the name). They are red because Rox think they are bad UTF8.

With Firefox 17, the resulting file would be a single file with target name (without .part), in red (I am not reporting this bug against BAD UTF8 name when most Internet is now UTF8, and I am still using ISO Xorg and ISO storage - I am reporting a REGRESSION BUG on how Firefox handles file names). FF17 would have created a red .part file, renamed as red file without part; renaming was going ok. Problem here is that FF50 can create files, but not move them.

Comment 4

[Masatoshi Kimura [:emk]]

Could you find a regression range using mozregression?
https://mozilla.github.io/mozregression/

Comment 5

[plop]

It will take time.

Comment 6

[plop]

I asked version by date 2017-02-28; after full download:

2017-03-03T13:18:09: INFO : Running mozilla-central build for 2017-02-28
2017-03-03T13:18:09: ERROR : Unable to install '/home/dhp/.mozilla/mozregression/persist/2017-02-28--mozilla-central--firefox-54.0a1.en-US.linux-x86_64.tar.bz2'
2017-03-03T13:18:09: INFO : Stopped

No reason given. The only Google answer on this error is https://bugzilla.mozilla.org/show_bug.cgi?id=1248138 , and I am not in the case of network interruption or disk full. 

Then tried version by version number: 47, giving about the same result:

2017-03-03T13:23:15: INFO : Running mozilla-central build for 2016-03-07
2017-03-03T13:23:15: ERROR : Unable to install '/home/dhp/.mozilla/mozregression/persist/2016-03-07--mozilla-central--firefox-48.0a1.en-US.linux-x86_64.tar.bz2'
2017-03-03T13:23:15: INFO : Stopped

Can not go further.

Comment 7

[Loic]

It's an issue with your machine probably, did you check the folder where the build is downloaded has the correct rights?

Comment 8

[plop]

I have done manual extraction and execution of what MozRegression downloaded:

cd /home/dhp/.mozilla/mozregression/persist
f=2011-04-12--mozilla-central--firefox-4.2a1pre.en-US.linux-x86_64.tar.bz2
rm -rf firefox ; tar -xjf $f ; ./firefox/firefox --ProfileManager

http://dl.free.fr/cS3jyW7aJ

*** All following tests are done with a fresh user, and download the file to /home/user/Downloads, which is ZFS.

(said to be release 5)
2011-04-12--mozilla-central--firefox-4.2a1pre.en-US.linux-x86_64.tar.bz2
App roughly starts, goes a bit after the profile manager, then crashes (and offers to restart or report issue)

2011-11-08--mozilla-central--firefox-10.0a1.en-US.linux-x86_64.tar.bz2
same as 5 (exection fails after profile manager)

15:
2012-06-05--mozilla-central--firefox-16.0a1.en-US.linux-x86_64.tar.bz2
same as 5 (exection fails after profile manager)

17:
2012-08-27--mozilla-central--firefox-17.0a1.en-US.linux-x86_64.tar.bz2
Open in Firefox: ABSOLUTELY AMAZING: file is downloaded correctly in /tmp, file is reported in download manager to be downloaded correctly, and, opens it in ... guess who ... really ... INTERNET EXPLORER !!! really ? I was 100% certain my wine was so broken no app could ever work again; unlike all other firefox versions, it's not trying to open the page inside itself, but instead, will make a systemcall to open it via the default browser (r47 tries to open it in a new tab). And guess what, not only IE was successfully launched (no clue how), but it could open the file Z:\tmp\Degré0del'informatique-01-e-pensernumérique-YouTube.htm ... That's the most unexpected thing of the year. In a completely broken linux box, IE working better than FF. It's IE7.0, I have no clue how it was installed; I just know I have installed wine twice. I also have to mention that, when I click on hyper links in OpenOffice and XPDF, they are open in Firefox (when using FF 5, 17, 50 or 52). Man, I am so amazed, just this is worth spending the day on mozregression :=) NB: for mozregression i have created fresh moz users; but they are running under the classic Linux user cession, so, they should use the same systemcall when searching for how to execute the .html MIME type ...
Save file to disk: download just fine, like when I was using versions 2 and 3.

2012-10-08--mozilla-central--firefox-18.0a1.en-US.linux-x86_64.tar.bz2
open in FF: download fine, can't find the file
Save to disk: works fine (like 17)

19
2012-11-19--mozilla-central--firefox-19.0a1.en-US.linux-x86_64.tar.bz2
all same as 18
save file to disk: said to be ok in in-browser pop-up, and, correct size in download manager. Correctly stored on disk.
(after getting download manager refusing to show any file in version 20, version 19 can show new downloads: what broke the profile for DM in 20 in the user profile is NOT breaking user profile for 19)

20:
2013-01-07--mozilla-central--firefox-20.0a1.en-US.linux-x86_64.tar.bz2
Open in Firefox: download goes fine, file is stored in /tmp, the Firefox tab claims it can't find the file. When exiting firefox, the file is removed from /tmp (it claimed it could not view it in the browser, but it could still delete it)
Save file to disk: reported to be fine in the in browser pop-up (file name and size), unknown size in the download manager, correctly dstored on disk.
=> here we lost good behaviour of download manager, maybe broken profile due to trying all versions on the same profile

23:
2013-05-13--mozilla-central--firefox-23.0a1.en-US.linux-x86_64.tar.bz2
works fine like 17-19 (download manager broken: does not show any file at all)

25:
2013-08-05--mozilla-central--firefox-25.0a1.en-US.linux-x86_64.tar.bz2
all works fine (same as 17,18,19) (download manager not broken yet, did that test before 23)
second try: idem.

26:
2013-09-16--mozilla-central--firefox-26.0a1.en-US.linux-x86_64.tar.bz2
(download manager broken (not showing any file)
popup reporting good download; producing two files (ISO file containing data, empty UTF file)
=> THIS IS DIFFERENT FROM 25 *AND* FROM 27 ...
Second try (kill PID, re-extract all): failure in pop-up, correct single file (UTF8 containing data without .part)
The two tries gave different pop-ups: the OK button did not have the same shape. This seems to depend how fast i start the download after starting FF.
Third: try: faile in pop-up, single empty UTF file ...

27:
2013-10-28--mozilla-central--firefox-27.0a1.en-US.linux-x86_64.tar.bz2
broken behaviour, see 30 below.
second try: idem

30:
2014-03-17--mozilla-central--firefox-31.0a1.en-US.linux-x86_64.tar.bz2
save to disk: reported to fail in pop and manager, stored on disk like i explaind for 50 (data are stored in the .part file; file with good name without .part is 0B size; both UTF8 names)

# where does it work again between 30 and 40 ??????????? later

35:
2014-09-02--mozilla-central--firefox-35.0a1.en-US.linux-x86_64.tar.bz2
same as 5

40: launched it many times, git different behaviour after restarting the process:
2015-05-11--mozilla-central--firefox-41.0a1.en-US.linux-x86_64.tar.bz2
save to disk: success in pop-up, success in manager, disk contains two files: one with data under ISO name, empty file under UTF name (none is called .part)
save to disk: fail in manager, two files named UTF8: .part contains data, the name without .part is empty (as for initial report)
save to disk: success in popup and manager, produces two files, ISO name contains data, UTF8 is empty
can't exec: XPCOMGlueLoad error for file /mnt/Home_2014/Home_2014/dhp/.mozilla/mozregression/persist/firefox/libxul.so:
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /mnt/Home_2014/Home_2014/dhp/.mozilla/mozregression/persist/firefox/libxul.so) Couldn't load XPCOM.
After XPCOM error, "rm -rf firefox ; tar -xjf $f ; ./firefox/firefox --ProfileManager" allows to run it fine.

41-47 ?

44:
2015-10-29--mozilla-central--firefox-45.0a1.en-US.linux-x86_64.tar.bz2
reported ok in popup and manager, ISO file perfectly ok, UTF8 file empty. (none is called .part)

45:
2015-12-14--mozilla-central--firefox-46.0a1.en-US.linux-x86_64.tar.bz2
same as 40,44

46:
2016-01-25--mozilla-central--firefox-47.0a1.en-US.linux-x86_64.tar.bz2
same as 40-45

47:
2016-03-07--mozilla-central--firefox-48.0a1.en-US.linux-x86_64.tar.bz2
same as 40-45

48:
2016-04-18--mozilla-central--firefox-48.0a1.en-US.linux-x86_64.tar.bz2
same as 40-45

2017-02-28--mozilla-central--firefox-54.0a1.en-US.linux-x86_64.tar.bz2
Can't execute it:
XPCOMGlueLoad error for file /mnt/Home_2014/Home_2014/dhp/.mozilla/mozregression/persist/firefox/libxul.so:
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.4/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /mnt/Home_2014/Home_2014/dhp/.mozilla/mozregression/persist/firefox/libxul.so)
Couldn't load XPCOM.


*** storing download on /mnt/tmp which is ext2 ...

17,25: reported OK in pop-up, and manager, and stored correctly (under UTF8 name)

not going to test 26 ... 

27:
save file to disk question: behaves like 30-50 if I click on OK immediately (two UTF); produces ISO+UTF if I wait for end of download before click.

30, 40, 48:
same behaviour as described in the first message of this bug report against 50: reported failure by pop-up and manager, leaving two UTF8 files, one .part with data, and an empty one without .part in the name (I have probably clicked ok quickly, as fast as possible)

40: after restarting FF, download reported to be fine in pop-up and manager, if file download is initiated before firefox compleeted startup, and click ok (small save confirmation pop-up, which does not ask to confirm the name) quickly; produces ISO+UTF8 files. Second download (under same process , 1mn later), long wait before click OK (download completely done in BG) reported ok, produces two files (ISO+UTF). None is called .part.
Restarting process (remove folder, re extract): idem.
... seems to also depend how much time download takes (and if I have other apps using the connexion).

52: creates two UTF files when auto-saving file to disk without any confirmation pop-up. With a "save file to" ( big pop-up asking me to confirm the fill name of file), I got UTF+ISO. 

***

So, for version 52, it depends if save without confirmation, or, get the large pop-up asking me to confirm the full name. For versions 27 to 40, auto-save still shows a small confirmation pop-up, and behaviour seems to depend if I click on OK before, or after download is completed in /tmp (ext2).

Exhausted ...

Comment 9

[Masatoshi Kimura [:emk]]

I guess this is because nsIFile can deal with native charset (ISO-8859-* on reporter's system) while OS.File only supports UTF-8 path. FileUtils and BackgroundFileSaver still use nsIFile.

Comment 10

[:Paolo Amadini]

This general issue with file name encoding when using OS.File on Linux is likely to affect more than just downloads. For the moment I don't think there is anyone available to investigate this issue, on the other hand if I understand correctly it may be quite limited in scope to certain machines running Linux.

Comment 11

[Masatoshi Kimura [:emk]]

nsIFile is no longer inconsistent with OS.File since bug 960957 dropped support for non-UTF-8 filename encodings on Linux.