1417497 - Sign the Windows code coverage build

Status: RESOLVED FIXED

(Testing :: Code Coverage, enhancement)

Description

[Marco Castelluccio [:marco]]

This is needed for some xpcshell tests.

The build definition is being added in bug 1417436, but signing is not done yet (https://bugzilla.mozilla.org/page.cgi?id=splinter.html&bug=1417436&attachment=8928507).

Aki, can you help?

Comment 1

[Aki Sasaki (not active)]

I think you need to add the appropriate labels (e.g. win64-ccov/debug ?) to this tuple: https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/loader/build_signing.py#l11

Comment 2

[Marco Castelluccio [:marco]]

Attachment: Patch

This patch enables signing for the Windows coverage build.

The build signing task fails though, as the server doesn't accept the xul.dll file as it's too big:
> 2017-11-16 12:14:48,583 - signingscript.sign - INFO - sign_file(): signing /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll with sha2signcode...
> 2017-11-16 12:14:48,583 - signingscript.sign - INFO - Certificate types: ['project:releng:signing:cert:dep-signing']
> 2017-11-16 12:14:48,583 - signingscript.utils - INFO - Running "/builds/scriptworker/bin/signtool -v -n /builds/scriptworker/work/nonce -t /builds/scriptworker/work/token -c /builds/scriptworker/lib/python3.5/site-packages/signingscript/data/host.cert -H signing4.srv.releng.scl3.mozilla.com:9110 -H signing5.srv.releng.scl3.mozilla.com:9110 -H signing6.srv.releng.scl3.mozilla.com:9110 -f sha2signcode -o /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll"
> 2017-11-16 12:14:48,586 - signingscript.utils - INFO - COMMAND OUTPUT: 
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,836 - in /
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - doing sha2signcode signing
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - possible hosts are ['https://signing4.srv.releng.scl3.mozilla.com:9110', 'https://signing6.srv.releng.scl3.mozilla.com:9110', 'https://signing5.srv.releng.scl3.mozilla.com:9110']
> 2017-11-16 12:14:48,837 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll
> 2017-11-16 12:14:48,838 - signingscript.utils - INFO - 2017-11-16 12:14:48,837 - checking /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll for signature...
> 2017-11-16 12:14:49,711 - signingscript.utils - INFO - 2017-11-16 12:14:49,711 - 42972962ad4f1818bc268d91a2c7945564154680: processing /builds/scriptworker/work/zipelqhj91g/firefox/xul.dll on https://signing4.srv.releng.scl3.mozilla.com:9110
> 2017-11-16 12:14:49,711 - signingscript.utils - INFO - 2017-11-16 12:14:49,711 - 42972962ad4f1818bc268d91a2c7945564154680: GET https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode/42972962ad4f1818bc268d91a2c7945564154680
> 2017-11-16 12:14:49,714 - signingscript.utils - INFO - 2017-11-16 12:14:49,714 - Starting new HTTPS connection (1): signing4.srv.releng.scl3.mozilla.com
> 2017-11-16 12:14:50,082 - signingscript.utils - INFO - 2017-11-16 12:14:50,082 - https://signing4.srv.releng.scl3.mozilla.com:9110 "GET /sign/sha2signcode/42972962ad4f1818bc268d91a2c7945564154680 HTTP/1.1" 404 None
> 2017-11-16 12:14:50,193 - signingscript.utils - INFO - 2017-11-16 12:14:50,193 - 42972962ad4f1818bc268d91a2c7945564154680: uploading for signing
> 2017-11-16 12:14:52,417 - signingscript.utils - INFO - 2017-11-16 12:14:52,417 - Starting new HTTPS connection (1): signing4.srv.releng.scl3.mozilla.com
> 2017-11-16 12:16:11,383 - signingscript.utils - INFO - 2017-11-16 12:16:11,382 - https://signing4.srv.releng.scl3.mozilla.com:9110 "POST /sign/sha2signcode HTTP/1.1" 400 0
> 2017-11-16 12:16:11,389 - signingscript.utils - INFO - 2017-11-16 12:16:11,384 - 42972962ad4f1818bc268d91a2c7945564154680: error uploading file for signing: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode
> 2017-11-16 12:16:11,389 - signingscript.utils - INFO - Traceback (most recent call last):
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/signtool/signing/client.py", line 86, in remote_signfile
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     r.raise_for_status()
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/requests/models.py", line 937, in raise_for_status
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     raise HTTPError(http_error_msg, response=self)
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode/42972962ad4f1818bc268d91a2c7945564154680
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - 
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - During handling of the above exception, another exception occurred:
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - 
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - Traceback (most recent call last):
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/signtool/signing/client.py", line 137, in remote_signfile
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     r.raise_for_status()
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.5/site-packages/requests/models.py", line 937, in raise_for_status
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO -     raise HTTPError(http_error_msg, response=self)
> 2017-11-16 12:16:11,390 - signingscript.utils - INFO - requests.exceptions.HTTPError: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9110/sign/sha2signcode

Here's the try build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f036c0c2e7fdc0fb58de296649a845bd4ef0ce71&selectedJob=145294162.

Comment 3

[Aki Sasaki (not active)]

We have a max_filesize_sha2signcode = 157286400, or ~150 mb.
We have a ccov xul.dll that's 480mb.

We've adjusted the max filesize before, but I'm not sure we've ever increased it more than 3x the current limit. Asking around about this limit.

In the meantime, it may be worth asking if it's possible to set a flag in xpcshell to not require the signature.

Comment 4

[Marco Castelluccio [:marco]]

(In reply to Aki Sasaki [:aki] from comment #3)
> We have a max_filesize_sha2signcode = 157286400, or ~150 mb.
> We have a ccov xul.dll that's 480mb.
> 
> We've adjusted the max filesize before, but I'm not sure we've ever
> increased it more than 3x the current limit. Asking around about this limit.
> 
> In the meantime, it may be worth asking if it's possible to set a flag in
> xpcshell to not require the signature.

I think we can, but we would need to disable a few tests, which means not collecting coverage for them.
We are trying to get a coverage build that is as close as possible to a normal debug build, as otherwise users of the coverage data will be confused.

Comment 5

[Aki Sasaki (not active)]

Ben, Chris, what's the main reason we're setting max_filesize ? Security? Making sure we don't run out of disk?

Comment 6

[Marco Castelluccio [:marco]]

Compressed as ZIP, it is ~150 MB.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] (back nov27) from comment #5)
> Ben, Chris, what's the main reason we're setting max_filesize ? Security?
> Making sure we don't run out of disk?

I don't have a clear memory on this, but my suspicion is security.

Comment 8

[Marco Castelluccio [:marco]]

Can we lift this limit? Or upload compressed files?

Comment 9

[Marco Castelluccio [:marco]]

I spoke with catlee on IRC. He has concerns increasing the file limit would slow things down.

The Windows coverage build only runs on mozilla-central pushes (and on try only when explicitly requested), so the overhead might be negligible compared to the other builds which are done on every commit on inbound, try, autoland.

Catlee, does this resolve your concern? Otherwise, can somebody look into uploading compressed files or help me do it?

Comment 10

[Marco Castelluccio [:marco]]

Aki, Ben, Chris, can you help me figure out the next step here?

Comment 11

[Aki Sasaki (not active)]

(In reply to Marco Castelluccio [:marco] from comment #9)
> I spoke with catlee on IRC. He has concerns increasing the file limit would
> slow things down.
> 
> The Windows coverage build only runs on mozilla-central pushes (and on try
> only when explicitly requested), so the overhead might be negligible
> compared to the other builds which are done on every commit on inbound, try,
> autoland.
> 
> Catlee, does this resolve your concern? Otherwise, can somebody look into
> uploading compressed files or help me do it?

The limit is for all windows files on all branches, so raising the limit would allow for these files to grow significantly without error.

Someone could potentially allow for signing locally with a dev/testing cert.. the downstream tests would have to accept that testing cert. If we have to keep the filesize limit on the signing server for security reasons, that may be our next most likely solution.

Comment 12

[Chris AtLee [:catlee]]

If we can figure out a way to restrict this size increase to the dep signing servers, that would be best.

Comment 13

[Chris AtLee [:catlee]]

Attachment: Allow signing large files on dep signing instances

Comment 14

[Marco Castelluccio [:marco]]

Attachment: Enable signing for the Windows code coverage build

Thanks :catlee!

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5b713577bd83
Enable signing for the Windows code coverage build. r=aki

Comment 16

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/5b713577bd83