Closed Bug 1417497 Opened 7 years ago Closed 7 years ago

Sign the Windows code coverage build

Categories

(Testing :: Code Coverage, enhancement)

enhancement
Not set
normal

Tracking

(firefox60 fixed)

RESOLVED FIXED
mozilla60
Tracking Status
firefox60 --- fixed

People

(Reporter: marco, Assigned: marco)

References

Details

Attachments

(2 files, 1 obsolete file)

This is needed for some xpcshell tests.

The build definition is being added in bug 1417436, but signing is not done yet (https://bugzilla.mozilla.org/page.cgi?id=splinter.html&bug=1417436&attachment=8928507).

Aki, can you help?
Flags: needinfo?(aki)
I think you need to add the appropriate labels (e.g. win64-ccov/debug ?) to this tuple: https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/loader/build_signing.py#l11
Flags: needinfo?(aki)
Attached patch Patch (obsolete) — Splinter Review
This patch enables signing for the Windows coverage build.

The build signing task fails though, as the server doesn't accept the xul.dll file as it's too big:
> 2017-11-16 12:14:48,583 - signingscript.sign - INFO - sign_file(): signing /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll with sha2signcode...
> 2017-11-16 12:14:48,583 - signingscript.sign - INFO - Certificate types: ['project:releng:signing:cert:dep-signing']
> 2017-11-16 12:14:48,583 - signingscript.utils - INFO - Running "/builds/scriptworker/bin/signtool -v -n /builds/scriptworker/work/nonce -t /builds/scriptworker/work/token -c /builds/scriptworker/lib/python3.5/site-packages/signingscript/data/host.cert -H signing4.srv.releng.scl3.mozilla.com:9110 -H signing5.srv.releng.scl3.mozilla.com:9110 -H signing6.srv.releng.scl3.mozilla.com:9110 -f sha2signcode -o /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll"
> 2017-11-16 12:14:48,586 - signingscript.utils - INFO - COMMAND OUTPUT: 
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,836 - in /
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - doing sha2signcode signing
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - possible hosts are ['https://signing4.srv.releng.scl3.mozilla.com:9110', 'https://signing6.srv.releng.scl3.mozilla.com:9110', 'https://signing5.srv.releng.scl3.mozilla.com:9110']
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll
> 2017-11-16 12:14:48,838 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - checking /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll for signature...
> 2017-11-16 12:14:49,711 - signingscript.utils - INFO - 2017-11-16 12:14:49,711 - 42972962ad4f1818bc268d91a2c7945564154680: processing /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll on https://signing4.srv.releng.scl3.mozilla.com:9110
> 2017-11-16 12:14:49,711 - signingscript.utils - INFO - 2017-11-16 12:14:49,711 - 42972962ad4f1818bc268d91a2c7945564154680: GET https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode/42972962ad4f1818bc268d91a2c7945564154680
> 2017-11-16 12:14:49,714 - signingscript.utils - INFO - 2017-11-16 12:14:49,714 - Starting new HTTPS connection (1): signing4.srv.releng.scl3.mozilla.com
> 2017-11-16 12:14:50,082 - signingscript.utils - INFO - 2017-11-16 12:14:50,082 - https://signing4.srv.releng.scl3.mozilla.com:9110 "GET /sign/sha2signcode/42972962ad4f1818bc268d91a2c7945564154680 HTTP/1.1" 404 None
> 2017-11-16 12:14:50,193 - signingscript.utils - INFO - 2017-11-16 12:14:50,193 - 42972962ad4f1818bc268d91a2c7945564154680: uploading for signing
> 2017-11-16 12:14:52,417 - signingscript.utils - INFO - 2017-11-16 12:14:52,417 - Starting new HTTPS connection (1): signing4.srv.releng.scl3.mozilla.com
> 2017-11-16 12:16:11,383 - signingscript.utils - INFO - 2017-11-16 12:16:11,382 - https://signing4.srv.releng.scl3.mozilla.com:9110 "POST /sign/sha2signcode HTTP/1.1" 400 0
> 2017-11-16 12:16:11,389 - signingscript.utils - INFO - 2017-11-16 12:16:11,384 - 42972962ad4f1818bc268d91a2c7945564154680: error uploading file for signing: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode
> 2017-11-16 12:16:11,389 - signingscript.utils - INFO - Traceback (most recent call last):
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/signtool/signing/client.py", line 86, in remote_signfile
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     r.raise_for_status()
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/requests/models.py", line 937, in raise_for_status
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     raise HTTPError(http_error_msg, response=self)
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode/42972962ad4f1818bc268d91a2c7945564154680
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - 
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - During handling of the above exception, another exception occurred:
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - 
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - Traceback (most recent call last):
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/signtool/signing/client.py", line 137, in remote_signfile
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     r.raise_for_status()
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/requests/models.py", line 937, in raise_for_status
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     raise HTTPError(http_error_msg, response=self)
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - requests.exceptions.HTTPError: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode

Here's the try build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f036c0c2e7fdc0fb58de296649a845bd4ef0ce71&selectedJob=145294162.
Flags: needinfo?(aki)
We have a max_filesize_sha2signcode = 157286400, or ~150 mb.
We have a ccov xul.dll that's 480mb.

We've adjusted the max filesize before, but I'm not sure we've ever increased it more than 3x the current limit. Asking around about this limit.

In the meantime, it may be worth asking if it's possible to set a flag in xpcshell to not require the signature.
Flags: needinfo?(aki)
(In reply to Aki Sasaki [:aki] from comment #3)
> We have a max_filesize_sha2signcode = 157286400, or ~150 mb.
> We have a ccov xul.dll that's 480mb.
> 
> We've adjusted the max filesize before, but I'm not sure we've ever
> increased it more than 3x the current limit. Asking around about this limit.
> 
> In the meantime, it may be worth asking if it's possible to set a flag in
> xpcshell to not require the signature.

I think we can, but we would need to disable a few tests, which means not collecting coverage for them.
We are trying to get a coverage build that is as close as possible to a normal debug build, as otherwise users of the coverage data will be confused.
Ben, Chris, what's the main reason we're setting max_filesize ? Security? Making sure we don't run out of disk?
Flags: needinfo?(catlee)
Flags: needinfo?(bhearsum)
Compressed as ZIP, it is ~150 MB.
(In reply to Aki Sasaki [:aki] (back nov27) from comment #5)
> Ben, Chris, what's the main reason we're setting max_filesize ? Security?
> Making sure we don't run out of disk?

I don't have a clear memory on this, but my suspicion is security.
Flags: needinfo?(bhearsum)
Can we lift this limit? Or upload compressed files?
I spoke with catlee on IRC. He has concerns increasing the file limit would slow things down.

The Windows coverage build only runs on mozilla-central pushes (and on try only when explicitly requested), so the overhead might be negligible compared to the other builds which are done on every commit on inbound, try, autoland.

Catlee, does this resolve your concern? Otherwise, can somebody look into uploading compressed files or help me do it?
Aki, Ben, Chris, can you help me figure out the next step here?
Flags: needinfo?(bhearsum)
Flags: needinfo?(aki)
(In reply to Marco Castelluccio [:marco] from comment #9)
> I spoke with catlee on IRC. He has concerns increasing the file limit would
> slow things down.
> 
> The Windows coverage build only runs on mozilla-central pushes (and on try
> only when explicitly requested), so the overhead might be negligible
> compared to the other builds which are done on every commit on inbound, try,
> autoland.
> 
> Catlee, does this resolve your concern? Otherwise, can somebody look into
> uploading compressed files or help me do it?

The limit is for all windows files on all branches, so raising the limit would allow for these files to grow significantly without error.

Someone could potentially allow for signing locally with a dev/testing cert.. the downstream tests would have to accept that testing cert. If we have to keep the filesize limit on the signing server for security reasons, that may be our next most likely solution.
Flags: needinfo?(aki)
Flags: needinfo?(bhearsum)
If we can figure out a way to restrict this size increase to the dep signing servers, that would be best.
Flags: needinfo?(catlee)
Attachment #8947200 - Flags: review?(aki) → review+
Attachment #8947200 - Flags: checked-in+
Blocks: 1434782
Thanks :catlee!
Attachment #8928981 - Attachment is obsolete: true
Attachment #8947306 - Flags: review?(aki)
Attachment #8947306 - Flags: review?(aki) → review+
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b713577bd83
Enable signing for the Windows code coverage build. r=aki
https://hg.mozilla.org/mozilla-central/rev/5b713577bd83
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Assignee: nobody → mcastelluccio
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: