1680139 - Filename containig "。" when downloaded could turn into "%u3002"

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P2)

Description

[kzyswd]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

Steps to reproduce:

Download a file that contains "。" in filename.

Actual results:

Depending on the website, the character "。" in filename of downloaded file could turn into "%u3002".

Expected results:

Character "。" in filename stays as "。".

Comment 1

[kzyswd]

Test files (file could expire, please let me know and I'll try to re-upload)
Not affected: https://gofile.io/d/1ksLLf
Affected: https://volafile.org/r/1hhg8xmnc

Characters with UTF-16 %u3001 (、) and %u3003 (〃) were not affected. Only %u3002 (。) for some reason.

Comment 2

[Alice0775 White]

#1 regression window("、。〃" -> "%u3001%u3002%u3003"):
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=992784f79f96&tochange=f884c46fa07d

#2 partially progression("、。〃" -> "、%u3002〃"):
https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=febe82aa4ed2&tochange=265e01c7ff55

Comment 3

[Masatoshi Kimura [:emk]]

Bug 1101625 partially fixed the problem. The actual regressor is bug 415491.

Comment 4

[Zibi Braniecki [:zbraniecki][:gandalf]]

Masatoshi, Makoto - do you plan to take this? It's P2 without an assignee, so I'd like to either downgrade it or assign. :)

Comment 5

[Masatoshi Kimura [:emk]]

I'm fine with taking this, but what is the desirable behavior? Should we just stop escaping characters in UnEscapeURIForUI and WONTFIXing bug 415491?

Comment 6

[Henri Sivonen (:hsivonen)]

Zibi asked me to comment on this but: Why do we use the escaping format %uHHHH for anyhing at all? Or is that escape coming from the site? (The test case has expired.)

Comment 7

[Masatoshi Kimura [:emk]]

That's bug 1248812.

Comment 8

[kzyswd]

Reuploaded test files just in case
Not affected: https://gofile.io/d/oMT5cG
Affected: https://volafile.org/r/1hhg8xmnc

Comment 9

[Masatoshi Kimura [:emk]]

Not affected: https://gofile.io/d/oMT5cG

If the Content-Disposition: header gives a filename, this bug does not affect the filename.

Affected: https://volafile.org/r/1hhg8xmnc

Otherwise, filename would be derived from the URL and this bug affect the filename.

IMO it does not make sense to escape the downloaded file names given that we do not escape filenames from the Content-Disposition: header and it is not considered as a vulnerability. We should stop calling unEscapeURIForUI to derive filenames from the URL instead of tweaking unEscapeURIForUI.

Taking the bug.

Comment 10

[kzyswd]

Another char with similar problem.
‐
(hyphen %u2010)

Comment 11

[Zibi Braniecki [:zbraniecki][:gandalf]]

:emk - any progress on this bug? Wanted to check if the NI on :m_kato is still needed.

Comment 12

[Masatoshi Kimura [:emk]]

(In reply to Masatoshi Kimura [:emk] from comment #9)

If the Content-Disposition: header gives a filename, this bug does not affect the filename.

Correction: If the Content-Disposition: header gives a filename with raw utf-8 characters (such as gofile.io), this bug does not affect the filename. If the Content-Disposition: header gives a filename with the rfc 6266 format, this bug affects the filename.

Comment 13

[Masatoshi Kimura [:emk]]

Attachment: Bug 1680139 - Add the ability for unEscapeURIForUI to disable re-escaping IDN blocklisted characters. r?#necko-reviewers

Comment 14

[Masatoshi Kimura [:emk]]

Attachment: Bug 1680139 - Stop re-escaping IDN blocklisted characters in download file names. r?mak

Depends on D134401

Comment 15

[Masatoshi Kimura [:emk]]

Attachment: Bug 1680139 - Stop re-escaping IDN blocklisted characters in downloads.download. r?robwu

Depends on D134402

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4242ce68dcc9
Add the ability for unEscapeURIForUI to disable re-escaping IDN blocklisted characters. r=m_kato
https://hg.mozilla.org/integration/autoland/rev/a10592d98dcd
Stop re-escaping IDN blocklisted characters in download file names. r=necko-reviewers,dragana
https://hg.mozilla.org/integration/autoland/rev/4bab917cdac4
Stop re-escaping IDN blocklisted characters in downloads.download. r=robwu

Comment 17

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/4242ce68dcc9
https://hg.mozilla.org/mozilla-central/rev/a10592d98dcd
https://hg.mozilla.org/mozilla-central/rev/4bab917cdac4

Comment 18

[Masatoshi Kimura [:emk]]

The problem was not fixed somehow. Reopening.

Comment 19

[Masatoshi Kimura [:emk]]

Attachment: Bug 1680139 - Stop re-escaping IDN blocklisted characters in uriloader/exthandler/. r?mtigley

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fbeb948af56d
Stop re-escaping IDN blocklisted characters in uriloader/exthandler/. r=mtigley

Comment 21

[Narcis Beleuzu [:NarcisB]]

Backed out for bc failures browser_download_open_with_internal_handler.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/daf2f92f1a17c468659020e8167a68628c1446d5
Log link: https://treeherder.mozilla.org/logviewer?job_id=362690401&repo=autoland&lineNumber=3365

Comment 22

[Masatoshi Kimura [:emk]]

Some other tests in the same directory assume that the download list is empty at first. I'll add a cleanup function.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/18ad4f8ecac7
Stop re-escaping IDN blocklisted characters in uriloader/exthandler/. r=mtigley

Comment 24

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/18ad4f8ecac7