Closed Bug 1477434 Opened 6 years ago Closed 6 years ago

Client Error: File too large when signing ASan Nightly reporter for Windows

Categories

(Release Engineering :: Release Automation: Signing, defect)

Product:
Release Engineering
Component:
Release Automation: Signing
Platform:
x86_64
Windows
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: decoder, Unassigned)

References

(Blocks 1 open bug)

Details

Since bug 1473259 has landed, we have win64-asan-reporter builds and updates. However, the NS job keeps throwing an exception. I looked into it and saw this:


2018-07-20 23:57:36,038 - signingscript.utils - INFO - 2018-07-20 23:57:36,038 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: uploading for signing
2018-07-20 23:57:37,417 - signingscript.utils - INFO - 2018-07-20 23:57:37,417 - Starting new HTTPS connection (1): signing4.srv.releng.scl3.mozilla.com:9100
2018-07-20 23:58:35,293 - signingscript.utils - INFO - 2018-07-20 23:58:35,292 - https://signing4.srv.releng.scl3.mozilla.com:9100 "POST /sign/sha2signcode HTTP/1.1" 400 0
2018-07-20 23:58:35,294 - signingscript.utils - INFO - 2018-07-20 23:58:35,294 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: error uploading file for signing: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode
2018-07-20 23:58:35,294 - signingscript.utils - INFO - Traceback (most recent call last):
2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.py", line 86, in remote_signfile
2018-07-20 23:58:35,295 - signingscript.utils - INFO -     r.raise_for_status()
2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line 939, in raise_for_status
2018-07-20 23:58:35,295 - signingscript.utils - INFO -     raise HTTPError(http_error_msg, response=self)
2018-07-20 23:58:35,295 - signingscript.utils - INFO - requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode/4a5eebc5f40da52d86218ee619c829a6fe4bbe58
2018-07-20 23:58:35,295 - signingscript.utils - INFO - 
2018-07-20 23:58:35,295 - signingscript.utils - INFO - During handling of the above exception, another exception occurred:
2018-07-20 23:58:35,295 - signingscript.utils - INFO - 
2018-07-20 23:58:35,295 - signingscript.utils - INFO - Traceback (most recent call last):
2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.py", line 137, in remote_signfile
2018-07-20 23:58:35,295 - signingscript.utils - INFO -     r.raise_for_status()
2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line 939, in raise_for_status
2018-07-20 23:58:35,295 - signingscript.utils - INFO -     raise HTTPError(http_error_msg, response=self)
2018-07-20 23:58:35,295 - signingscript.utils - INFO - requests.exceptions.HTTPError: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode
2018-07-20 23:58:36,296 - signingscript.utils - INFO - 2018-07-20 23:58:36,295 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: giving up after 5 tries
2018-07-20 23:58:36,313 - signingscript.utils - INFO - 2018-07-20 23:58:36,313 - Failed to sign /builds/scriptworker/work/zip6fgql6os/firefox/xul.dll with sha2signcode
2018-07-20 23:58:36,337 - signingscript.utils - INFO - exitcode 1

(taken from example log here: https://tools.taskcluster.net/groups/IwbRaz3XRumH5OWT-l41mw/tasks/RlpTq2VjRpiAKHtMirTBuw/runs/1/logs/public%2Flogs%2Flive_backing.log)


Looks to me as if the size for signing is simply exceeded (and I guess on try, no actual signing is done). ASan builds are larger and my guess is we added an exception for linux64 but not for win64.


:catlee, :bhearsum, can any of you help with this?
Flags: needinfo?(catlee)
Flags: needinfo?(bhearsum)
(In reply to Christian Holler (:decoder) from comment #0)
> Since bug 1473259 has landed, we have win64-asan-reporter builds and
> updates. However, the NS job keeps throwing an exception. I looked into it
> and saw this:
> 
> 
> 2018-07-20 23:57:36,038 - signingscript.utils - INFO - 2018-07-20
> 23:57:36,038 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: uploading for
> signing
> 2018-07-20 23:57:37,417 - signingscript.utils - INFO - 2018-07-20
> 23:57:37,417 - Starting new HTTPS connection (1):
> signing4.srv.releng.scl3.mozilla.com:9100
> 2018-07-20 23:58:35,293 - signingscript.utils - INFO - 2018-07-20
> 23:58:35,292 - https://signing4.srv.releng.scl3.mozilla.com:9100 "POST
> /sign/sha2signcode HTTP/1.1" 400 0
> 2018-07-20 23:58:35,294 - signingscript.utils - INFO - 2018-07-20
> 23:58:35,294 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: error uploading
> file for signing: 400 Client Error: File too large for url:
> https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode
> 2018-07-20 23:58:35,294 - signingscript.utils - INFO - Traceback (most
> recent call last):
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File
> "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.
> py", line 86, in remote_signfile
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -    
> r.raise_for_status()
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File
> "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line
> 939, in raise_for_status
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -     raise
> HTTPError(http_error_msg, response=self)
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -
> requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
> https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode/
> 4a5eebc5f40da52d86218ee619c829a6fe4bbe58
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO - 
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO - During handling of
> the above exception, another exception occurred:
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO - 
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO - Traceback (most
> recent call last):
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File
> "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.
> py", line 137, in remote_signfile
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -    
> r.raise_for_status()
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -   File
> "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line
> 939, in raise_for_status
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -     raise
> HTTPError(http_error_msg, response=self)
> 2018-07-20 23:58:35,295 - signingscript.utils - INFO -
> requests.exceptions.HTTPError: 400 Client Error: File too large for url:
> https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode
> 2018-07-20 23:58:36,296 - signingscript.utils - INFO - 2018-07-20
> 23:58:36,295 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: giving up after 5
> tries
> 2018-07-20 23:58:36,313 - signingscript.utils - INFO - 2018-07-20
> 23:58:36,313 - Failed to sign
> /builds/scriptworker/work/zip6fgql6os/firefox/xul.dll with sha2signcode
> 2018-07-20 23:58:36,337 - signingscript.utils - INFO - exitcode 1
> 
> (taken from example log here:
> https://tools.taskcluster.net/groups/IwbRaz3XRumH5OWT-l41mw/tasks/
> RlpTq2VjRpiAKHtMirTBuw/runs/1/logs/public%2Flogs%2Flive_backing.log)
> 
> 
> Looks to me as if the size for signing is simply exceeded (and I guess on
> try, no actual signing is done). ASan builds are larger and my guess is we
> added an exception for linux64 but not for win64.
> 
> 
> :catlee, :bhearsum, can any of you help with this?

It looks like it's failing on xul.dll, which means it alone is already larger than 150MB. When we increased the limit for Linux, that was for the MARs, and we raised it to 200MB (https://bugzilla.mozilla.org/show_bug.cgi?id=1438967). How big should we expect xul.dll to be? And the entire installer? (They both share the same size limit for...reasons.)

IIRC, we try to keep the size limits low enough to avoid certain attacks. If the Windows ASan builds are that much larger we may want to consider a separate signing format to avoid raising the limit for other Windows builds.

Do you have any thoughts, catlee?
Flags: needinfo?(bhearsum) → needinfo?(choller)
target.installer.exe is currently 162 MB large while xul.dll has 325 MB.

The task I took the info from is here: https://tools.taskcluster.net/groups/Dng1X5-kQfqbB_rsdjNc6w/tasks/H7dTd7ooQEWUU7byQxDt4g/runs/0/artifacts
Flags: needinfo?(choller) → needinfo?(bhearsum)
We ended up bumping the max size, should be fixed for the next set of nightlies.
Flags: needinfo?(catlee)
Flags: needinfo?(bhearsum)
(In reply to Ben Hearsum (:bhearsum) from comment #4)
> We ended up bumping the max size, should be fixed for the next set of
> nightlies.

And that next set of nightlies just made it to the fx-sig-verify check ;) (Installer only at this time.)

What value did you end up using?
Flags: needinfo?(bhearsum)
This was fixed and Nightlies are now produced correctly, thanks!
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.