Closed Bug 711210 Opened 13 years ago Closed 10 years ago

sign 64-bit windows builds

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

x86_64
Windows Server 2008

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: bhearsum)

References

Details

(Whiteboard: [signing])

see bug 669277

signing the exes seems to work, and some tools think they're signed, but they're not runnable everywhere, and some other tools think internal exe's aren't signed.
Was bug 669277 transient? Can we close this, or is signing really broken here?
OS: All → Windows Server 2008
Priority: -- → P3
Hardware: All → x86_64
Whiteboard: [signing]
Signing is broken. I've turned it off win64 everywhere except elm, which is why the win64 builds on m-c started working again.
Blocks: 730824
No longer blocks: 730824
Blocks: 797032
Blocks: 715876
Blocks: 880004
How do we do the signing, using what tools?
signing is done on linux signing servers, with mono's version of signcode.
fwiw at my old company I had a script that ran on Windows that signed 32-bit and 64-bit binaries, I think with signtool.exe and we had never run into any problems.
Found in triage.
Component: Release Engineering → Release Engineering: Platform Support
QA Contact: coop
Product: mozilla.org → Release Engineering
Any time for this bug and bug 730824 this quarter?
A quick google search reveals:

http://blog.ej-technologies.com/2009/04/signing-launchers-and-installers.html

where they mentioned they modified signcode.exe to sign 64-bit binaries.  I can't find their source anywhere; we could reach out to them and ask for the patch (using their binary is probably a bad idea :).

I also see mention of osslsigncode:

http://sourceforge.net/projects/osslsigncode/files/

which seems to be a better and more supported option; looks to be flag-compatible, so may be a drop in replacement that would just work.  (Or we could use it just for win64 builds for now, but seems simplest to just use it for everything.)
I just tried osslsigncode, and it looks very promising! Looks like it can sign win32 files and win64 files successfully:
http://people.mozilla.org/~catlee/test-win32.exe
http://people.mozilla.org/~catlee/test-win64.exe

For comparison, signcode still breaks win64:
http://people.mozilla.org/~catlee/test-win64-signcode.exe

We'll need to integrate this into signing server and do more thorough testing.
Great :) 
I'm looking forward to that landing for x64 builds so I can do some maintenance service x64 work
Just followed up with catlee; he has not yet had a chance to look at this.
Just pinging this -- with testing happening on real hardware now, this is now blocking maintenance service/updater work, which is one of the last remaining bits.
vlad: I am looking into this now.
Assignee: nobody → jhopkins
Depends on: 974042
osslsigncode doesn't like being given a password on stdin inside popen, so I'm modifying the code to use pexpect.
Need to compare signing with the RPM-built binary vs. the binary I used for initial testing.  The former is failing to sign exe's for some reason.

Note: I'm on buildduty this week so progress may be slower.
We have deemed this to be a lower priority so I have not spent more time on it yet.
Assignee: jhopkins → george.miroshnykov
With john departing and this host still "on" in AWS, whats our next step here?
Flags: needinfo?(catlee)
No user activity since jhopkins has departed.

This host has been stopped until we know what we are going to do with it.
Flags: needinfo?(catlee)
Blocks: 1033159
Blocks: 1035562
Assignee: gmiroshnykov → nobody
I'll be looking into this very soon.
Assignee: nobody → bhearsum
Based on my read of this bug it sounds like osslsigncode is the best path forward unless we find a reason that it won't work. I'm going to continue on that assumption and work on getting the signing servers to support it. I intend to create a new signing format that we'll only use for 64-bit Windows builds (for now). This will make it possible to push to try to test patches, and also eliminate any risk to the 32-bit builds (because they'll continue using Mono's signcode).

I'm going to move most of the this work to other bugs and use this as a tracking bug.
Depends on: 1075705
Depends on: 1075709
Depends on: 1075712
Depends on: 1075723
Deploying this is going to be trickier than I thought. The signing server changes from bug 1075709 can land at any point once they're tested and reviewed. However, bug 1075723 needs to be on mozilla-central (hopefully merged to date and oak too) before bug 1075712 can land. Once everything is landed pushes to try that are based off of the latest mozilla-central push should get their 64-bit Windows builds signed.
I landed all of my patches to get things signed. It signed most things fine, but it's trying to sign the .zip package for some reason. I found this in the signing server logs:
Usage: signscript.py [options] format inputfile outputfile inputfilename

signscript.py: error: Invalid file for signing: firefox-35.0a1.en-US.win64-x86_64.zip


This is strange, and I don't think it happens on 32-bit Windows builds. I'm looking into it.
(In reply to Ben Hearsum [:bhearsum] from comment #22)
> I landed all of my patches to get things signed. It signed most things fine,
> but it's trying to sign the .zip package for some reason. I found this in
> the signing server logs:
> Usage: signscript.py [options] format inputfile outputfile inputfilename
> 
> signscript.py: error: Invalid file for signing:
> firefox-35.0a1.en-US.win64-x86_64.zip
> 
> 
> This is strange, and I don't think it happens on 32-bit Windows builds. I'm
> looking into it.

Turns out that this was because of an incomplete patch in bug 1075723. There's a follow-up in that bug that fixed it.

I _think_ this is all done now - I see the latest CI build on mozilla-central signing properly. I'm going to leave this open until tomorrow morning, when we have a successful nightly (which will be signed with a proper cert, instead of our self signed one like the CI builds). I'll have a quick look and make sure they're signed as far as Windows is concerned, but I won't be doing any further verification.
I had a look at the latest Nightly build and it looks signed. As far as I know, we're done here!
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Summary: signing win64 builds is busted → sign 64-bit windows builds
Depends on: 1121286
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.