920460 - can't download Firefox on needed language using wget from http://www.mozilla.org/

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[sunweb]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0 (Beta/Release)
Build ID: 20130911160237

Steps to reproduce:

wget https://download.mozilla.org/?product=firefox-24.0&os=win&lang=ru

Since Windows can't update some of the software i made a script that does the job for everything needed. Firefox has two ways of downloading but this one is much better, well was. It stopped working once Firefox 24 released. Worked perfectly before. And downloading through Firefox through the same page(www.mozilla.org/en-US/firefox/all/) had the same bug, but they fixed it, now only wget bug here.

path=/dev/shm/win-updates
if [ ! -d $path ]; then
  mkdir -m0700 $path
fi
cd $path
dname=ff.exe
wget $proxy -E -H -k -K -p -nd http://www.mozilla.org/en-US/firefox/all/
link=`grep -i -R "os=win" index.html | grep -i -R "lang=ru" | uniq -i | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -c 35- | sed 's/[" title="Download for Windows">Download</a></td>]*$//'`
wget -O $dname $link


Actual results:

Downloads English version. No matter what language you want, you still get English.


Expected results:

It should've download Russian Firefox.

Comment 1

[Loic]

Does it work from the FTP?
http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/24.0/

Comment 2

[sunweb]

Yes, ofcourse. I have a script that does it through ftp just fine but i was just wondering why can't i download with wget from this exact url anymore. wget wasn't updated, nothing on my pc was, so it have something to do with how you setup something on your side. 
It might not seem a big deal and maybe i should address wget devs as well but after you fixed dowloading through browser i thought its easier for firefox site devs to look at it.

Comment 3

[Loic]

Yes, I will triage it. Maybe the website has changed its policies.

Comment 4

[Chris More [:cmore]]

download.mozilla.org 302 redirect to CDNs that host the download and this is different than before. Is it possible that your wget command is just not following the redirect?

If you do a curl -I on that URL you will get a link to the CDN like:

http://download-installer.cdn.mozilla.net/pub/mozilla.org/firefox/releases/24.0/win32/en-US/Firefox%20Setup%2024.0.exe

You could have the script to a curl -I to get the URL and then do a wget on the cdn link to get the URL and that should be pretty easy to put together with shell.

Would that work?

Comment 6

[sunweb]

(In reply to Chris More [:cmore] from comment #4)
If i will get curl -I i will get the very same link on english version. So no.

Yeah, i gave up and am using ftp link. Here's the script if someone will be interested:

#!/bin/sh

dirL=/dev/shm/install
dirM=//10.0.1.201/Install
password="password"

if [ ! -d $dirL ]; then
  mkdir $dirL
fi
echo $password | sudo -S mount -t cifs -o username=login,password=password,file_mode=0777,dir_mode=0777,_netdev $dirM $dirL

path=/dev/shm/win-updates
pathC=$dirL/updates/compare
pathU=$dirL/updates/install

if [ ! -d $path ]; then
  mkdir -m0700 $path
else
  rm $path/*
fi

cd $path
#rm $path/*


############
# proxy
############
proxy_choice=0
login="login"
pass="password"

if [ $proxy_choice -eq 1 ]; then
  export http_proxy="http://server:port"
  proxy="--proxy-user=$login --proxy-password=$pass"
else
  proxy=""
fi
############
# compare_n_download
############
compare_n_download()
{
nW=$(wget $proxy --spider $link 2>&1 | grep Length | awk '{print $2}')
for fname in $dname
do
  # file size
  nC=$(stat -c %s $pathC/$fname)
  if [ $nW -ne $nC ]; then
    wget $proxy -O $dname $link
  fi
  rm $path/*.html
done
}
############
# compare_n_pick
############
compare_n_pick()
{
for fname in $dname
do
  if [ -e "$dname" ]; then
    # file size
    nN=$(stat -c %s $path/$fname)
    nC=$(stat -c %s $pathC/$fname)
    if [ $nN -ne $nC ]; then
      rsync $path/$fname $pathC
      cp $pathC/$fname $pathU
    fi
    rm $path/*
  fi
done
}
############
# download
############

############
flashplayer()
{
dname=fp.exe
link="http://download.macromedia.com/get/flashplayer/current/licensing/win/install_flash_player_11_plugin.exe"
compare_n_download
compare_n_pick
}
############
java()
{
dname=j.exe
wget $proxy -E -H -k -K -p -nd --accept=html http://www.java.com/en/download/manual.jsp
link=`grep -i -R "Download Java software for Windows Offline" manual.jsp.html | uniq -i | sed 's/^[ \t]*//;s/[ \t]*$//' | cut -c 61- | sed 's/">\+$//'`
compare_n_download
compare_n_pick
}
############
firefox()
{
dname=ff.exe
wget $proxy -E -H -k -K -p -nd --accept=html http://download.cdn.mozilla.net/pub/firefox/releases/latest/win32/ru/
link=`grep -i "fox" index.html | grep -v "tub" | uniq -i | sed 's/^[ \t]*//;s/[ \t]*$//' `
link=${link##<*http}
link=${link%%'">'*}
link="http$link"
compare_n_download
compare_n_pick
}
############
libreoffice()
{
dname=lo.msi
wget $proxy -E -H -k -K -p -nd --accept=html http://download.documentfoundation.org/libreoffice/stable/
vnumber=`grep -i [0-9].[0-9].[4-6] index.html | sed 's/^[ \t]*//;s/[ \t]*$//' `
vnumber=${vnumber##<*'stable/'[0-9].[0-9].[4-6]'/">'}
vnumber=${vnumber%%'/</a>'*}
vnumber=$vnumber
link="http://download.documentfoundation.org/libreoffice/stable/"$vnumber"/win/x86/LibreOffice_"$vnumber"_Win_x86.msi"
compare_n_download
compare_n_pick
}
############
# update
############
flashplayer
java
firefox
libreoffice

echo $password | sudo -S umount $dirL

Comment 7

[Chris More [:cmore]]

Ok, I had the same problem and I just got it to work. 

The following wget returns en-US build:

wget http://download.mozilla.org/?product=firefox-latest&os=win&lang=ru

the following returns an ru build:

http://download.mozilla.org/?product=firefox-latest\&os=win\&lang=ru

If you escape the & with \, this should work. Can you try this change in your script and see if it works?

Without the escaped & characters, wget will stop at ?product=firefox-latest and not include the OS or Lang. Download.mozilla.org will then 302 redirect to include those and I believe that is why your script is getting en-US only.

Can you try this?

Comment 8

[sunweb]

(In reply to Chris More [:cmore] from comment #7)
Yeah, it did work. But why? Could you explain why it works this way, please?

Comment 9

[sunweb]

What i meant is why did it work before but then stopped. Anyway thank you alot, it simplifies operations.

Comment 10

[Paul [:pmac] McLanahan]

I think the issue is that you're grepping through HTML to find the URL, and that HTML will change from time to time. This is an issue all HTML page scraping scripts have. What I'd recommend is to get the latest version of Firefox at that same URL[0], but using the 'data-latest-firefox="25.0"' attribute on the <html> emement. This should stay there and not change based on the look of the page. Once you have that, just construct the download URL yourself. This should also not change:

https://download.mozilla.org/?product=firefox-25.0&os=win&lang=ru

Just replace "25.0" with whatever you get out of the 'data-latest-firefox' attribute and it should work reliably. This will be the full installer download. If you want the stub installer (the smaller program that will itself download and install Firefox more reliably), you can just set the "product" query parameter on that URL to "firefox-latest" like :cmore did in comment #7. That will always work and doesn't require you to go search for the latest version number.

Hope this helps.

PS - I'm resolving this as "invalid", not because your concern is "invalid", but because this isn't an issue with the website. If you have further questions or issues with this please feel free to continue the conversation. 

[0] https://www.mozilla.org/en-US/firefox/all/

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Slight correction:

current full installer: wget 'https://download.mozilla.org/?product=firefox-latest&os=win&lang=ru'
current stub installer: wget 'https://download.mozilla.org/?product=firefox-stub&os=win&lang=ru'

Plus whatever -O strategy you use to handle the version being in the file delivered.

Comment 12

[Paul [:pmac] McLanahan]

Ah! My fault. :nthomas is right. The 'firefox-latest' product will always work, but only for 'os=win'. Or at least it used to be windows only. If a windows installer is what you want, you can just use the above URL and it should "Just Work".

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

So, sorry to be 'that guy' again, but firefox-latest will work for all OS now. Bug 863381 just got deployed today.

Comment 14

[Paul [:pmac] McLanahan]

:nthomas no that's great! Now bedrock (mozilla.org) can just point users to that and no longer have to keep track of version numbers for most download buttons. Thanks for the updates.