632627 - 3.6.14build2 EUBallot build internal DLLs/EXEs don't match non-EUBallot versions

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

Attachment: verify-signatures-build2.log

For 3.6.14 build2 I had to trigger the EUBallot separately.
verify-signature failed.

I am not sure if this would be reached on the next release since I did not follow an standard order of operations but I will file it to determine if it requires further investigation.

When we reach "Verifying checksums of MARs and installers match" [1] we create a list of every unsigned file and then create a dictionary with the locale as the key.

The problem comes that we have partner-repacks and win32-EUBallot showing up for some locales with that logic.

Then we do a comparison that fails:
> if not sums_are_equal(packages[mar[0]], [packages[s] for s in packages.keys()]):

When I look at that function it seems that it disregards partner-repacks because they are not signed [3], and therefore, they don't take part of the comparison.

I get quite lost on the logic but I believe that this verification is not doing what it is supposed to when there are EUBallots.
I bet I am wrong here and there please feel free to correct me.

This can be reproduced with:
> ./verify-signature.py --product firefox unsigned-build2 signed-build2

>Comparing packages for locale: el
> - unsigned-build2/unsigned/partner-repacks/google/win32/el/Firefox Setup 3.6.14.exe
> - unsigned-build2/unsigned/update/win32/el/firefox-3.6.14.complete.mar
> - unsigned-build2/unsigned/win32/el/Firefox Setup 3.6.14.exe
> - unsigned-build2/unsigned/win32-EUBallot/el/Firefox Setup 3.6.14.exe
>helper.exe differs!
>Error: MARs and installer contents do not match

This does not happen when there is only partner repacks:
>Comparing packages for locale: ar
> - unsigned-build2/unsigned/update/win32/ar/firefox-3.6.14.complete.mar 
> - unsigned-build2/unsigned/partner-repacks/google/win32/ar/Firefox Setup 3.6.14.exe 
> - unsigned-build2/unsigned/win32/ar/Firefox Setup 3.6.14.exe


[1] http://mxr.mozilla.org/build/source/tools/release/signing/verify-signature.py#114
[2] http://mxr.mozilla.org/build/source/tools/release/signing/verify-signature.py#246
[3] http://mxr.mozilla.org/build/source/tools/release/signing/signing.py#158

Comment 1

[Aki Sasaki [:aki] (he/him) (UTC-7)]

Possible dup of bug 632412.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> Possible dup of bug 632412.

Not a dupe.

This bug is tracking the acute issue of 3.6.14build2 EUBallot builds being invalid. I filed bug 632793 on the root issue.

Comment 3

[[e-mail] (:bhearsum)]

Fixed these with the following:
cd ~/signing-work/firefox-3.6.14
rm -rf cache
# Create a new directory to plop signed ballot builds in, just in case we screw something up
rsync -av --exclude=win32-EUBallot signed-build2/ signed-build2-euballot-attempt2/
# Populate the cache by running sign-release.py w/ --keep-cache and --first-locale against the original directory. This causes it to unpack the first locale's build, cache it, and then exit because there's nothing to sign
for l in `ls unsigned-build2/unsigned/win32-EUBallot`; do ./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2-euballot-attempt2 -p --first-locale ${l%\/} --keep-cache unsigned-build2/; done
# Now, run sign-release.py on the directory with only the EUBallot builds in it. This will have a 100% hit rate, because all of the files from these locales are in the cache.
./sign-release.py --keydir=${KEYDIR} -j4 --product Firefox -o signed-build2-euballot-attempt2 -p --keep-cache unsigned-build2-euballot-fix/
# Generate detached sigs for them
cd signed-build2-euballot-attempt2
../signing/sign-files .
# Run verify-signatures to ensure everything is OK
./verify-signature.py --product firefox unsigned-build2 signed-build2-euballot-attempt2 > verify-signatures-build2-euballot.fix.log
# Upload!
rsync -e "ssh -i /home/cltsign/.ssh/ffxbld_dsa" -av signed-build2-euballot-attempt2/ [e-mail]:/home/ftp/pub/firefox/nightly/3.6.14-candidates/build2/

Also added these in the 3.6.14 build notes.