bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

win32 update verify issues

RESOLVED FIXED

Status

Release Engineering
Release Automation: Other
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: aki, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
We hit this in 27.0b1, on all update verify builders for windows.

http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/release-mozilla-beta-win32_update_verify_2%2F6/builds/4

06:06:57 (20.75 MB/s) - `Firefox Setup 27.0b1.exe' saved [24468432/24468432]

SOURCE DIRECTORY ../../update
DESTINATION DIRECTORY .
failed: 19
calling QuitProgressUI
FAIL: update status was not succeeded: failed: 19
FAIL: check_updates returned failure for WINNT_x86-msvc downloads/Firefox Setup 26.0b8.exe vs. downloads/Firefox Setup 27.0b1.exe: 1
Using  https://aus3.mozilla.org/update/1/Firefox/26.0/20131125215016/WINNT_x86-msvc/ak/betatest/update.xml?force=1
ERROR: Error verifying signature.
ERROR: Not all signatures were verified.

[10:45]	<bhearsum|buildduty>	i don't see anything obviously wrong about the certificates or fingerprints
[10:48]	<bhearsum|buildduty>	do we know that it's all previous versions failing, or just some?
[10:48]	<aki>	i don't know that
[10:48]	<jhopkins>	i didn't verify that all previous versions failed, only that multiple did
[10:51]	<bhearsum|buildduty>	FAIL: check_updates returned failure for WINNT_x86-msvc downloads/Firefox Setup 26.0b10.exe vs. downloads/Firefox Setup 27.0b1.exe: 1
[10:51]	<bhearsum|buildduty>	so...probably no passes
[10:51]	<bhearsum|buildduty>	i can verify this mar's signature by hand no problem...
[10:51]	<bhearsum|buildduty>	i'm going to ask bbondy for some help
[10:54]	<bhearsum|buildduty>	it looks like we may have only signed these mars with one of the certs, not both....
Tracked this down with Brian Bondy's help. It turns out that the new rev2 Windows build machines get the Mozilla Maintenance service registry keys installed. Due to a horrible hack we have in our code, that forces updater.exe to use the self signed MAR certs to verify any MAR it encounters. This explains why QA tests passed and ours failed.

I think someone should do a manual test of any old beta -> 27.0b1 on some other machine to verify this. After that we should be safe to ship.

To fix this for the future we need to remove the Mozilla Maintenance service reg keys from the rev2 build machines. I think this is just an adjustment of the GPO policies. Note that we _only_ want to remove from the build machines, we still need them on the test machines. Once that's done we can rerun an update verify chunk to verify that things are OK.
(Reporter)

Updated

5 years ago
Depends on: 948573
(Reporter)

Comment 2

5 years ago
[12:45]	<tracy>	aki: sorry that took so long. had some trouble with file permissions on my Win 7 vm. eventually just pushed a few update requests through our automation. Those all passed.
figured out the MS bs permissions and was able to manually run and successfully update from a 26b to 27b1 on the betatest channel on a Win7 vm
This is all fixed up now AFAIK.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.