948543 - win32 update verify issues

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Aki Sasaki (not active)]

We hit this in 27.0b1, on all update verify builders for windows.

http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win32_update_verify_2%2F6/builds/4

06:06:57 (20.75 MB/s) - `Firefox Setup 27.0b1.exe' saved [24468432/24468432]

SOURCE DIRECTORY ../../update
DESTINATION DIRECTORY .
failed: 19
calling QuitProgressUI
FAIL: update status was not succeeded: failed: 19
FAIL: check_updates returned failure for WINNT_x86-msvc downloads/Firefox Setup 26.0b8.exe vs. downloads/Firefox Setup 27.0b1.exe: 1
Using  https://aus3.mozilla.org/update/1/Firefox/26.0/20131125215016/WINNT_x86-msvc/ak/betatest/update.xml?force=1
ERROR: Error verifying signature.
ERROR: Not all signatures were verified.

[10:45]	<bhearsum|buildduty>	i don't see anything obviously wrong about the certificates or fingerprints
[10:48]	<bhearsum|buildduty>	do we know that it's all previous versions failing, or just some?
[10:48]	<aki>	i don't know that
[10:48]	<jhopkins>	i didn't verify that all previous versions failed, only that multiple did
[10:51]	<bhearsum|buildduty>	FAIL: check_updates returned failure for WINNT_x86-msvc downloads/Firefox Setup 26.0b10.exe vs. downloads/Firefox Setup 27.0b1.exe: 1
[10:51]	<bhearsum|buildduty>	so...probably no passes
[10:51]	<bhearsum|buildduty>	i can verify this mar's signature by hand no problem...
[10:51]	<bhearsum|buildduty>	i'm going to ask bbondy for some help
[10:54]	<bhearsum|buildduty>	it looks like we may have only signed these mars with one of the certs, not both....

Comment 1

[[e-mail] (:bhearsum)]

Tracked this down with Brian Bondy's help. It turns out that the new rev2 Windows build machines get the Mozilla Maintenance service registry keys installed. Due to a horrible hack we have in our code, that forces updater.exe to use the self signed MAR certs to verify any MAR it encounters. This explains why QA tests passed and ours failed.

I think someone should do a manual test of any old beta -> 27.0b1 on some other machine to verify this. After that we should be safe to ship.

To fix this for the future we need to remove the Mozilla Maintenance service reg keys from the rev2 build machines. I think this is just an adjustment of the GPO policies. Note that we _only_ want to remove from the build machines, we still need them on the test machines. Once that's done we can rerun an update verify chunk to verify that things are OK.

Comment 2

[Aki Sasaki (not active)]

[12:45]	<tracy>	aki: sorry that took so long. had some trouble with file permissions on my Win 7 vm. eventually just pushed a few update requests through our automation. Those all passed.

Comment 3

[Tracy Walker [:tracy]]

figured out the MS bs permissions and was able to manually run and successfully update from a 26b to 27b1 on the betatest channel on a Win7 vm

Comment 4

[[e-mail] (:bhearsum)]

This is all fixed up now AFAIK.