1456120 - Avoid bot-detection when downloading files from unicode.org

Status: RESOLVED WONTFIX

(Core :: JavaScript Engine, enhancement, P3)

Description

[André Bargull [:anba]]

unicode.org seems to have a very aggressive bot-detection installed. For example when I was running make_unicode.py to download four files from unicode.org, the fourth file was never downloaded (the download simply got stuck) and my IP address was apparently blacklisted (rerunning make_unicode.py didn't download any file and I couldn't even load unicode.org over a browser). Only after reconnecting my internet and getting a new IP from my ISP, I was able to connect to unicode.org again.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

Steven - is it possible to relax the policy to at least allow complete download to happen for us without workarounds?

Comment 2

[André Bargull [:anba]]

FWIW I've already written a patch to simply download UCD.zip instead of the four individual files we need and to set a "User-agent" header for the request. With that workaround I was able to download from unicode.org multiple times without having connection issues.

Comment 3

[André Bargull [:anba]]

Attachment: bug1456120.patch

This patch adds a "User-agent" header (copied from [1]) when downloading from unicode.org and also reduces the number of downloaded files to just downloading the bundled zip-file. These changes avoided to get black-listed by unicode.org when repeatedly downloading files, at least when I tested it locally. I've also updated the download links to use "https" instead of "http".

[1] https://searchfox.org/mozilla-central/rev/fcd5cb3515d0e06592bba42e378f1b9518497e3d/js/src/builtin/intl/make_intl_data.py#1055-1057

Comment 4

[Steven R. Loomis]

Hi. First of all, what are you downloading? You should NOT EVER be auto downloading from unicode.org. Download the file manually and check it into the repository.

You can also try my experimental mirror at https://sf.net/p/unicode-org (but do checksum comparisons)

Let's discuss this separately.

Comment 5

[Steven R. Loomis]

To follow up - Unicode data changes exactly yearly, it's not supposed to be re downloaded. That's why we put the aggressive anti bot stuff in. We also get build farms pulling down the same .xml over and over.

Comment 6

[André Bargull [:anba]]

The python script is only executed once when a new Unicode version is released and it is always started by hand and never automatically by some build steps. It downloads CaseFolding.txt, DerivedCoreProperties.txt, SpecialCasing.txt, and UnicodeData.txt and checks them into the Firefox source repo.
I've only rerun the python script now because I'm in the process of updating several scripts to be compatible with Python 3 (bug 1456118) and I wanted to test that the updated script runs properly in Python 3. :-)

Comment 7

[Steve Fink [:sfink] [:s:]]

Comment on attachment 8970278 [details] [diff] [review]
bug1456120.patch

Review of attachment 8970278 [details] [diff] [review]:
-----------------------------------------------------------------

r=me for the code, though given the discussion here it seems like it might be a good idea to have another argument --urlpattern, defaulting to "https://unicode.org/Public/{version}/ucd/UCD.zip" or perhaps "https://unicode.org/Public/{version}/ucd/{filename}". But I don't understand the need for the UNIDATA thing; it seems like --version=latest would work just as well.

You could use the mirror with --version=10.0.0 --urlpattern="https://sourceforge.net/projects/unicode-org/files/{version}/ucd/UCD.zip/download". I imagine there's a better url with https://sf.net/p/unicode-org but I don't know how those patterns map.

I don't really want to be the one deciding the policy question of whether it's ok to be spoofing user agent strings like this. So I'll request an additional review from gandalf, since he will at least have a better idea of who needs to be ok with it.

I suppose we could always default to not mucking with the user-agent, but have a --randomize option or something for the case that anba ran into.

Comment 8

[André Bargull [:anba]]

(In reply to Steve Fink [:sfink] [:s:] (PTO June 31) from comment #7)
> r=me for the code, though given the discussion here it seems like it might
> be a good idea to have another argument --urlpattern, defaulting to
> "https://unicode.org/Public/{version}/ucd/UCD.zip" or perhaps
> "https://unicode.org/Public/{version}/ucd/{filename}".

In the past make_unicode_data.py allowed a lot more arguments, but we've reduced it to the bare minimum for usability reasons. (In the optimal case this script is run exactly once per year and having to remember a bunch of options for a script which is run only very rarely didn't seem wise.)


> But I don't understand the need for the UNIDATA thing; it seems like --version=latest
> would work just as well.

I think I (?) have only named it UNIDATA to match the name used on unicode.org.


> You could use the mirror with --version=10.0.0
> --urlpattern="https://sourceforge.net/projects/unicode-org/files/{version}/
> ucd/UCD.zip/download". I imagine there's a better url with
> https://sf.net/p/unicode-org but I don't know how those patterns map.

But then we'd need to rely on sourceforge not mucking with the files, not that that's ever happened [1]. And checksums for the files hosted on unicode.org aren't yet available [2,3]. 

[1] https://en.wikipedia.org/wiki/SourceForge#Project_hijackings_and_bundled_malware
[2] https://unicode.org/pipermail/unicode/2017-October/005802.html
[3] https://www.unicode.org/L2/L2017/17406-ucd-signatures.pdf


> I don't really want to be the one deciding the policy question of whether it's ok to be spoofing user agent strings like this.

We may be able to get away with not spoofing the user-agent if we only download UCD.zip, because it seems like it is possible to download up to three files before being banned....

Comment 9

[Steve Fink [:sfink] [:s:]]

(In reply to André Bargull [:anba] from comment #8)
> (In reply to Steve Fink [:sfink] [:s:] (PTO June 31) from comment #7)
> > r=me for the code, though given the discussion here it seems like it might
> > be a good idea to have another argument --urlpattern, defaulting to
> > "https://unicode.org/Public/{version}/ucd/UCD.zip" or perhaps
> > "https://unicode.org/Public/{version}/ucd/{filename}".
> 
> In the past make_unicode_data.py allowed a lot more arguments, but we've
> reduced it to the bare minimum for usability reasons. (In the optimal case
> this script is run exactly once per year and having to remember a bunch of
> options for a script which is run only very rarely didn't seem wise.)

I was thinking of it as an option that you'd just never use, but whose default value was clear documentation on the path structured at unicode.org so that it would be straightforward to diagnose and fix future problems. But yeah, it's not really necessary.

> > But I don't understand the need for the UNIDATA thing; it seems like --version=latest
> > would work just as well.
> 
> I think I (?) have only named it UNIDATA to match the name used on
> unicode.org.

unicode.org does have the UNIDATA shortcut, but I doubt it gives you anything beyond the 'latest' shortcut, and I wasn't sure why you wanted to have knowledge of that particular shortcut embedded in the code. But maybe that's the "normal" way of downloading stuff that people are familiar with? Anyway, it doesn't really matter to me.

> > You could use the mirror with --version=10.0.0
> > --urlpattern="https://sourceforge.net/projects/unicode-org/files/{version}/
> > ucd/UCD.zip/download". I imagine there's a better url with
> > https://sf.net/p/unicode-org but I don't know how those patterns map.
> 
> But then we'd need to rely on sourceforge not mucking with the files, not
> that that's ever happened [1]. And checksums for the files hosted on
> unicode.org aren't yet available [2,3]. 

This was for a hypothetical use case where you were working on something and got blocked, and wanted a temporary workaround. But it sounds like this just isn't necessary once you have the randomization option.

> > I don't really want to be the one deciding the policy question of whether it's ok to be spoofing user agent strings like this.
> 
> We may be able to get away with not spoofing the user-agent if we only
> download UCD.zip, because it seems like it is possible to download up to
> three files before being banned....

Yeah, I don't know what the blocking keys off of. I intentionally tried to trigger it through my home DSL and failed. I downloaded UCD.zip half a dozen times, and a ReadMe.txt over 100 times. No blocking.

Anyway, I'm fine with the spoofing implementation. It just worries me to have something like that in our code base, obviously for the purpose of subverting a protection mechanism, especially if it's enabled by default. But there is a legitimate reason for it too, so it seems like a policy question to me.

Comment 10

[Steven R. Loomis]

Does this help? https://unicode.org/~srloomis/SHASUM512-unicodezip.txt.asc (signed by me)

Can you make the default at least:
 - use sourceforge
 - don't falsify the UA

Our blocking is another process, and @sfink if you want to be on the email discussion of the blocking I can add you.

Comment 11

[Steve Fink [:sfink] [:s:]]

(In reply to Steven R. Loomis from comment #10)
> Our blocking is another process, and @sfink if you want to be on the email
> discussion of the blocking I can add you.

I'm torn between curiosity and wanting to run away. :-) Sure, add [e-mail] to the list.

Though currently I'm thinking that given that this is only manually run in the first place, and the blocking has only shown up to be a problem during development, a --user-agent option would be good enough for the rare case of getting blocked. (Or --user-agent-token if random user agents won't work. Or --randomize would be most convenient, if upstream is ok with that option even existing.)

Comment 12

[Steven R. Loomis]

(In reply to Steve Fink [:sfink] [:s:] (PTO June 31) from comment #11)
> (In reply to Steven R. Loomis from comment #10)
> > Our blocking is another process, and @sfink if you want to be on the email
> > discussion of the blocking I can add you.
> 
> I'm torn between curiosity and wanting to run away. :-) Sure, add
> [e-mail] to the list.

Run away! Run away!

I'll add you.

> Though currently I'm thinking that given that this is only manually run in
> the first place, and the blocking has only shown up to be a problem during
> development, a --user-agent option would be good enough for the rare case of
> getting blocked. (Or --user-agent-token if random user agents won't work. Or
> --randomize would be most convenient, if upstream is ok with that option
> even existing.)

I'd prefer it didn't exist. But I think it's fine to have it there.

Please do try the SHASUM file above and see if it helps with making use of the sourceforge file. That would move things forward.

Comment 13

[Zibi Braniecki [:zbraniecki][:gandalf]]

Anba - Do you plan to update the patch to Steven's feedback?

Comment 14

[André Bargull [:anba]]

No, not right now, because I feel investing the extra time needed to learn how to properly download and verify files from sourceforge isn't worth it given that the Unicode update only happens once a year. Resolving as WONTFIX unless someone else likes to step up and implement the necessary changes.

Comment 15

[Zibi Braniecki [:zbraniecki][:gandalf]]

Comment on attachment 8970278 [details] [diff] [review]
bug1456120.patch

Ok. But I don't think I should be on a blocking path if you'll want to land it :)