665559 - Verify integrity of binaries served through ftp

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[Axel Hecht [:Pike]]

All l10n windows nightlies on central are busted failing to unpack the en-US zip.

Comment 1

[Armen [:armenzg]]

This seems to be fixed for today's nightly but I wanted to have a look and see what happened.

If we look at the output of a locale we can see that there is some type of corruption [1].

If we look at what win32-ix-slave31 downloaded VS what I can download today [2] I can see there is a sha512 discrepancy [3].

What I don't understand is what happened that *all* of the win32 slaves that took a win32 l10n repack downloaded a corrupted win32 zip file except for few instances. ftp was probably serving a corrupted file during a short interval of time [4].

The right fix is to upload the en-US file and downloaded it immediately to verify the checksums. This should be done before triggering unit tests and L10n repacks since it has bit us before.

This doesn't happen enough but it is important to get it done.

######### [1] #############
find /e/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../dist/l10n-stage/firefox -maxdepth 1 -print0 | xargs -0 rm -f -r
e:/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/config/nsinstall.exe -D ../../dist/l10n-stage
cd ../../dist/l10n-stage && \
	  unzip /e/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../dist/firefox-7.0a1.en-US.win32.zip && (cd firefox && d:/mozilla-build/python25/python2.5.exe /e/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../config/optimizejars.py --deoptimize /e/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../jarlog//en-US ./ ./ && unzip -o omni.jar && rm -f components/binary.manifest && sed -e 's/^#binary-component/binary-component/' components/components.manifest > components.manifest && mv components.manifest components)
Archive:  e:/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../dist/firefox-7.0a1.en-US.win32.zip
find: /e/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../dist/l10n-stage/firefox: No such file or directory
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in e:/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../dist/firefox-7.0a1.en-US.win32.zip,
        and cannot find e:/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../dist/firefox-7.0a1.en-US.win32.zip.zip, period.
make: *** [/e/builds/moz2_slave/cen-w32-l10n-ntly/build/mozilla-central/browser/locales/../../dist/l10n-stage/firefox] Error 9
program finished with exit code 2

######### [2] #############
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2011-06-20-12-mozilla-central/firefox-7.0a1.en-US.win32.zip

######### [3] #############
# From win32-ix-slave31
E:\builds\moz2_slave\cen-w32-l10n-ntly\build\mozilla-central\dist>D:\mozilla-build\svn-win32-1.6.3\bin\openssl.exe dgst -sha512 firefox-7.0a1.en-
US.win32.zip
SHA512(firefox-7.0a1.en-US.win32.zip)= 6d5f3a1dcdeb0a5a8956f5e5714d45fa0eac33093bf23c210468764c71a9598530d4c1a14e22dc3f85c1af193a8504af2b180a1b4a
9e79f90dc58f6e12e030bf

# From checksums file
511dd741374dc1d1ee7aa2ec1af2ff415a5a05493a0cd8d0922706abcd1bd7bbc0e44869c571f67c90690d0b3a709abdeaba326d5c17fa56a65d1194b30ec388 sha512 18468288 firefox-7.0a1.en-US.win32.zip

# From a random windows machine
$ openssl dgst -sha512 ~/Desktop/temp/firefox-7.0a1.en-US.win32.zip
SHA512(c:/Documents and Settings/Administrator/Desktop/temp/firefox-7.0a1.en-US.
win32.zip)= 511dd741374dc1d1ee7aa2ec1af2ff415a5a05493a0cd8d0922706abcd1bd7bbc0e4
4869c571f67c90690d0b3a709abdeaba326d5c17fa56a65d1194b30ec388

######### [4] #############
It seems that the runs that went green took so long to get to the step that by then the ftp issue was gone. Times are start-end. There were only 8 green runs (I just took note of 4 of them) out of 80+ jobs:
Bad run  - 7:30-7:49
Good run - 7:23-8:37
Good run - 7:24-8:25
Good run - 7:23-8:21
Good run - 7:26-8:34

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

ftp.m.o has recently changed to having a pool of machines behind a load balancer (bug 662968), and those machines have been having some reliability issues. CC'ing dumitru.

Comment 3

[Chris Cooper [:coop] (he/him)]

This particular failure mode has been fixed for a while. Current issues are with hg rather than ftp (bug 734141).