Closed Bug 842501 Opened 11 years ago Closed 10 years ago

Occasionally redirected to a wrong version of Firefox when attempt to download.

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: leonhardeuler1707, Unassigned)

Details

(Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/94] ) 

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0
Build ID: 20130201190337

Steps to reproduce:

Tried to download US-English version of Firefox, for Windows, while in China.


Actual results:

Got the Chinese version instead, but after multiple attempts, sometimes I get the English and sometimes the Chinese.

I initially suspected foul play (maybe the Chinese government redirects the download link to a version containing spyware), but it seems that this is just the ordinary "zh-CN" version of Firefox.

Here is an example of a "botched" download where I got the Chinese version. I used wget on Linux so I could see where it's redirecting me, but also on a different Windows machine I got the same outcome.

me@ssssssss:~$ wget -O Firefox-en-US-wget01 "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/win32/en-US/Firefox Setup 18.0.2.exe"
--2013-02-19 17:58:36--  http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/win32/en-US/Firefox%20Setup%2018.0.2.exe
Resolving download.cdn.mozilla.net (download.cdn.mozilla.net)... 69.192.4.40, 69.192.4.64
Connecting to download.cdn.mozilla.net (download.cdn.mozilla.net)|69.192.4.40|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://218.249.165.37/download/36910231/50732121/3/exe/228/178/1360086432740_178/Firefox%20Setup%2018.0.2.exe [following]
--2013-02-19 17:58:37--  http://218.249.165.37/download/36910231/50732121/3/exe/228/178/1360086432740_178/Firefox%20Setup%2018.0.2.exe
Connecting to 218.249.165.37:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20192296 (19M) [application/octet-stream]
Saving to: `Firefox-en-US-wget01'

100%[==================================================================================================================>] 20,192,296  8.38M/s   in 2.3s    

2013-02-19 17:58:39 (8.38 MB/s) - `Firefox-en-US-wget01' saved [20192296/20192296]

The MD5 of this file is:
aa1631403f50fb656740235dffaad9a4
which exactly corresponds to the "zh-CN" version of Firefox (which I downloaded through a VPN server in London to make sure I'm not redirected anywhere in China).


As soon as this download finished I tried again from the same link:

me@ssssssss:~$ wget -O Firefox-en-US-wget02 "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/win32/en-US/Firefox Setup 18.0.2.exe"
--2013-02-19 17:59:03--  http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/win32/en-US/Firefox%20Setup%2018.0.2.exe
Resolving download.cdn.mozilla.net (download.cdn.mozilla.net)... 69.192.4.64, 69.192.4.40
Connecting to download.cdn.mozilla.net (download.cdn.mozilla.net)|69.192.4.64|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20300952 (19M) [application/octet-stream]
Saving to: `Firefox-en-US-wget02'

100%[==================================================================================================================>] 20,300,952  27.7K/s   in 7m 12s  

2013-02-19 18:06:15 (45.8 KB/s) - `Firefox-en-US-wget02' saved [20300952/20300952]

The MD5 of this file is:
181ac7211db54e793845e8c7216c1a8b
which is the correct version as it corresponds to the English version downloaded through VPN.


Expected results:

Should have gotten the English version.
I can't find a suitable component for the CDN
Assignee: nobody → server-ops
Component: Untriaged → Server Operations
Product: Firefox → mozilla.org
QA Contact: shyam
Version: 18 Branch → other
Assignee: server-ops → server-ops-webops
Component: Server Operations → Server Operations: Web Operations
QA Contact: shyam → nmaul
Umm... on that first request, there's a 302 redirect. That is unexpected. Once you get connected to Akamai (or Edgecast) there should be no redirection. That is *not* being served by us. Your wget shows a connection to Akamai (expected), but then a redirect away from Akamai to. 218.249.165.37 is not an Akamai IP, nor a Mozilla one. Something intercepted your download request and sent you somewhere else.

We don't have anything like this configured in either Mozilla infrastructure nor Akamai, as far as I am aware of (and I'd be the guy to know if we did).

Great Firewall issue? I can't think of anything better. CC'ing some folks who might have better insights.
By the way, thank you very much for the detailed bug report. Without the wget output and md5sum checking, this would have been virtually impossible to track down. Now we have a fighting chance. :)
Hi Jake, Thanks for checking it out. Indeed, a Great Firewall issue was my initial suspicion. There is a precedent: whenever you try to access skype.com from China, there is a thinly veiled DNS spoofing that sends you to a "government approved" version of the software. At first I was sure that similarly, you get a spyware-ridden version of Firefox (which is a security risk to all users in China basically), but the MD5 matches the regular Chinese version, from Mozilla, which I truly hope has no Chinese spyware...
We'll take a look at this and try to figure out why this is occurring.
(In reply to Joe Stevensen [:joes] from comment #5)
> We'll take a look at this and try to figure out why this is occurring.

Can we raise the urgency a bit on this investigation? I'm concerned that if we're having issues with downloads on CDN, we could similarly be having issues with updates (thus leaving users behind).
All the evidence here points to a China ISP (or the government) forcibly redirecting users improperly. I have been unable to replicate from nodes in our China datacenter, which leads me to believe it's either intermittent / inconsistent, or ISP-specific.

I don't have any contacts to tap for this. We need someone who can communicate with this ISP and explain the problem, get an explanation, and lobby for a fix. I am stalled here... nothing more I can do at the moment.



At this point we have no reason to suspect this affects anyone outside of China, and possibly only a single ISP within China. Additionally, it would not affect anyone using the zh-CN locale, as they'd still be getting the right file(s) anyway.


One thing that might (might) help mitigate this sort of issue is downloading installers over HTTPS. Without a lot more insight into how this is being done, I can't say if it would have any effect or not. This amounts to just a checkbox in the Bouncer configuration (one per bouncer product), once someone is prepared to sign off on it. At this point we can ask for fresh wget output and if the problem still exists.
per channel mtg:


(In reply to Alex Keybl [:akeybl] from comment #6)
> (In reply to Joe Stevensen [:joes] from comment #5)
> > We'll take a look at this and try to figure out why this is occurring.
> 
> Can we raise the urgency a bit on this investigation? I'm concerned that if
> we're having issues with downloads on CDN, we could similarly be having
> issues with updates (thus leaving users behind).


(In reply to Jake Maul [:jakem] from comment #7)
> All the evidence here points to a China ISP (or the government) forcibly
> redirecting users improperly. I have been unable to replicate from nodes in
> our China datacenter, which leads me to believe it's either intermittent /
> inconsistent, or ISP-specific.
> 
> I don't have any contacts to tap for this. We need someone who can
> communicate with this ISP and explain the problem, get an explanation, and
> lobby for a fix. I am stalled here... nothing more I can do at the moment.

jakem: nice debugging data, thanks.

lsblakk will reach out to other people inside China to see if they can repro.

joes: any news? 

leonhardeuler1707: any info on what ISP was used at the time this was noticed?




> At this point we have no reason to suspect this affects anyone outside of
> China, and possibly only a single ISP within China. Additionally, it would
> not affect anyone using the zh-CN locale, as they'd still be getting the
> right file(s) anyway.
> 
> 
> One thing that might (might) help mitigate this sort of issue is downloading
> installers over HTTPS. Without a lot more insight into how this is being
> done, I can't say if it would have any effect or not. This amounts to just a
> checkbox in the Bouncer configuration (one per bouncer product), once
> someone is prepared to sign off on it. At this point we can ask for fresh
> wget output and if the problem still exists.

we *think* that requiring https (instead of http) would have a non-trivial impact to CDN and ftp. Based on other investigations this may not be needed. However, just in case, can someone confirm, before we think further on the https idea?
I agree with comment 2 and comment 7 from :jakem.
I made the same checks and had the same conclusions.

One additional recommendation, when checking the file hashes, also check the sha512sum, not only the md5sum. Both checksums must be correct. It is possible to forge the md5 checksums on large files. It is much more unlikely to forge the sha512 checksum, and even harder to forge both properly (as of today, this is considered impossible)

You can verify the correctness of the checksums by looking at http://download-origin.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/MD5SUMS and http://download-origin.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/SHA512SUMS  after validating them with GnuPG. The key fingerprint is 9D03 193D 6BDC 541B D796  C4E4 7F4D 6645 1EBC AB3A.
The problem with version 18.0.2 remains as I reported it; I attach below new results for version 19.0, which are slightly different but equally disturbing. The downloads I reported in my initial bug report are on "server 1" as detailed below, within Peking University campus, I also tried a different machine in a different location in Beijing.

My current theory is that it must be an experiment... It's been rumored that the Great Firewall people are doing active research. I got a version identical to zh-CN (and now it appears, the correct English version but from an illegitimate download site), because they didn't actually have a spyware version of Firefox yet, but just wanted to see that if in principle they could redirect the download on the flight without anyone noticing. Their mistake was that they also redirected the English version to the Chinese file on their server, in version 18.0.2.

I suggest making it easier to find the MD5 and SHA512 on the Firefox download page, and maybe hint to users in China that doing a checksum before installing is a good idea...

Here is the most recent tests I did:
****** Begin test on server 1 ******
(used for the initial test for version 18.0.2 where I got the zh-CN instead of en-US)
IP:  159.226.XXX.XXX
ISP: ???
Location: Peking University Campus
************************************
me@xxxxxxxx:~/tmp$ wget -O Firefox-en-US-wget02 "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/win32/en-US/Firefox Setup 19.0.exe"
--2013-03-03 06:39:23--  http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/18.0.2/win32/en-US/Firefox%20Setup%2019.0.exe
Resolving download.cdn.mozilla.net (download.cdn.mozilla.net)... 69.192.4.160, 69.192.4.113
Connecting to download.cdn.mozilla.net (download.cdn.mozilla.net)|69.192.4.160|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://124.254.47.50/download/37639866/52416097/3/exe/254/197/1361245151998_709/Firefox%20Setup%2019.0.exe [following]
--2013-03-03 06:39:23--  http://124.254.47.50/download/37639866/52416097/3/exe/254/197/1361245151998_709/Firefox%20Setup%2019.0.exe
Connecting to 124.254.47.50:80... failed: Connection timed out.
Retrying.

--2013-03-03 06:40:27--  (try: 2)  http://124.254.47.50/download/37639866/52416097/3/exe/254/197/1361245151998_709/Firefox%20Setup%2019.0.exe
Connecting to 124.254.47.50:80... failed: Connection timed out.
Retrying.

[...]

--2013-03-03 06:47:13--  (try: 8)  http://124.254.47.50/download/37639866/52416097/3/exe/254/197/1361245151998_709/Firefox%20Setup%2019.0.exe
Connecting to 124.254.47.50:80... ^C

******* End test on server 1 *******
RESULT: A redirect was attempted but the download actually failed.


****** Begin test on server 2 ******
IP:  159.226.XXX.XXX
ISP: China Science And Technology Network
Location: About 7 km away from server 1
************************************

me@yyyyy:~$ wget -O FILE "http://download.mozilla.org/?lang=en-US&product=firefox-19.0&os=win"                                                                                                                                              
--2013-03-03 06:42:18--  http://download.mozilla.org/?lang=en-US&product=firefox-19.0&os=win                                                                                                                                                
Resolving download.mozilla.org... 2620:101:8008:5::2:a, 63.245.217.36, 63.245.217.39                                                                                                                                                        
Connecting to download.mozilla.org|2620:101:8008:5::2:a|:80... connected.                                                                                                                                                                   
HTTP request sent, awaiting response... 302 Found                                                                                                                                                                                           
Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/19.0/win32/en-US/Firefox%20Setup%2019.0.exe [following]                                                                                                          
--2013-03-03 06:42:19--  http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/19.0/win32/en-US/Firefox%20Setup%2019.0.exe                                                                                                       
Resolving download.cdn.mozilla.net... 68.232.45.253                                                                                                                                                                                         
Connecting to download.cdn.mozilla.net|68.232.45.253|:80... connected.                                                                                                                                                                      
HTTP request sent, awaiting response... 302 Found                                                                                                                                                                                           
Location: http://103.2.211.229:80/videoplayer/Firefox%20Setup%2019.0.exe?ich_u_r_i=a26a028379d7534021bfb7d93db70911&ich_s_t_a_r_t=0&ich_e_n_d=0&ich_k_e_y=1345038902752263402481&ich_t_y_p_e=1&ich_d_i_s_k_i_d=4&ich_u_n_i_t=1 [following]  
--2013-03-03 06:42:19--  http://103.2.211.229/videoplayer/Firefox%20Setup%2019.0.exe?ich_u_r_i=a26a028379d7534021bfb7d93db70911&ich_s_t_a_r_t=0&ich_e_n_d=0&ich_k_e_y=1345038902752263402481&ich_t_y_p_e=1&ich_d_i_s_k_i_d=4&ich_u_n_i_t=1
Connecting to 103.2.211.229:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20564496 (20M) [application/octet-stream]
Saving to: `FILE'

100%[===================================================================================================================================================================================================>] 20,564,496   835K/s   in 26s    

2013-03-03 06:42:45 (785 KB/s) - `FILE' saved [20564496/20564496]

me@yyyyy:~$ md5sum FILE
794f7257513725d69c0b4fdbc5160d34  FILE
me@yyyyy:~$ sha512sum FILE
d1484885451fccfa3555abfd4f8d910c6621d2f7962f46e6e7c03dd392daa5480a82f7d9beeefedda1793bc06aa59c15f13cf3f53ccae444793d55651c1fe32f  FILE
******* End test on server 2 *******
RESULT: There is a strange redirect but md5 and sha512 match the appropriate English version.
Sorry, a correction: the IP range of server 1 is 162.105.XXX.XXX, and not as I wrote above (which is the correct range of server 2)
I believe it is very hard to make the people check the checksums or signatures, even if we were to advertise them better.
One thing that could help technical users, is to sign the binaries instead of signing the hashes. Technically, both are fine, but it's a lot easier to verify the signed binary directly.

I think hosting downloads on HTTPS would have a better impact, and I suppose that we should consider that if we notice that downloads get redirected with added malware.

I checked the IPs you provided, but when coming from my source IP it return 403s (probably due to source ip, or time of access. It doesn't look like to be because of the parameters in the URL)

A search for those parameters show a few download services from China, most of which are for pirated games. Some point to Vietnamese servers as well.
It's also sometimes mirroring stuff like windows updates:
http://answers.microsoft.com/en-us/windows/forum/windows_7-windows_install/i-cant-download-windows61-kb976932-x64/0578797f-84e9-4f30-bc27-b77e196e24a0
http://social.msdn.microsoft.com/Forums/en-US/netfxsetup/thread/f1910344-5046-46ac-96f1-479c4519da94

I would also recommend to check your computer for any possible malware that could also create those redirections.
In response to comment 12:
- Thanks for that info!
- I agree that HTTPS is better.
- I used three computers with three different operating systems and in two IP ranges (currently, all I have access to). I think malware on all three is unlikely.
Note that at the moment, you can use https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/latest/ for a secure download (make sure https is used with a valid certificate), or the stub installer on Windows, or check the authenticode on Windows.
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/94]
Given the age of this bug, and the fact that all downloads are now done over https://, I am going to resolve this.
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.