Closed Bug 900546 Opened 11 years ago Closed 11 years ago

signing server client doesn't always shuffle servers on retry

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: rail)

Details

Attachments

(1 file, 1 obsolete file)

Eg in https://tbpl.mozilla.org/php/getParsedLog.php?id=26009554&full=1&branch=ux

  MOZ_SIGN_CMD=python /builds/slave/ux-osx64-000000000000000000000/tools/release/signing/signtool.py --cachedir /builds/slave/ux-osx64-000000000000000000000/signing_cache -t /builds/slave/ux-osx64-000000000000000000000/token -n /builds/slave/ux-osx64-000000000000000000000/nonce -c /builds/slave/ux-osx64-000000000000000000000/tools/release/signing/host.cert -H mac-signing1.srv.releng.scl3.mozilla.com:9110 -H mac-signing2.srv.releng.scl3.mozilla.com:9110 -H mac-signing3.build.scl1.mozilla.com:9110 -H mac-signing4.build.scl1.mozilla.com:9110

cd ../../dist && cd ./universal/firefox && python /builds/slave/ux-osx64-000000000000000000000/tools/release/signing/signtool.py --cachedir /builds/slave/ux-osx64-000000000000000000000/signing_cache -t /builds/slave/ux-osx64-000000000000000000000/token -n /builds/slave/ux-osx64-000000000000000000000/nonce -c /builds/slave/ux-osx64-000000000000000000000/tools/release/signing/host.cert -H mac-signing1.srv.releng.scl3.mozilla.com:9110 -H mac-signing2.srv.releng.scl3.mozilla.com:9110 -H mac-signing3.build.scl1.mozilla.com:9110 -H mac-signing4.build.scl1.mozilla.com:9110 -f dmg   FirefoxUX.app && rm FirefoxUX.app/Contents/CodeResources && cp FirefoxUX.app/Contents/_CodeSignature/CodeResources FirefoxUX.app/Contents && cd /builds/slave/ux-osx64-000000000000000000000/build/obj-firefox/i386/browser/installer/../../dist && /builds/slave/ux-osx64-000000000000000000000/build/build/package/mac_osx/pkg-dmg --source "universal/firefox" --target "firefox-25.0a1.en-US.mac.dmg" --volname "UX"  --copy "branding/dsstore:/.DS_Store" --mkdir /.background --copy "branding/background.png:/.background" --icon "branding/disk.icns" --symlink "/Applications:/ "
2013-07-31 22:40:05,959 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:40:07,164 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:40:17,731 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:40:18,733 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:40:18,760 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:40:29,009 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:40:30,010 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:40:30,035 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:40:40,289 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:40:41,290 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:40:41,317 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:40:51,431 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:40:52,432 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:40:52,457 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:02,957 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:41:03,958 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:03,985 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:14,045 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:41:15,046 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:15,225 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:25,344 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:41:26,345 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:26,375 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:36,590 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:41:37,591 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:37,617 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:47,645 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:41:48,646 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:48,674 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:41:58,823 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:41:59,825 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:41:59,850 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:42:09,911 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:42:10,912 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:42:10,940 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:42:21,112 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:42:22,113 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:42:22,140 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:42:32,221 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:42:33,222 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:42:33,249 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:42:43,311 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:42:44,312 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:42:44,342 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:42:54,378 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:42:55,380 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:42:55,557 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:43:05,727 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:43:06,728 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:43:06,752 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:43:16,856 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:43:17,858 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:43:17,883 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:43:28,244 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:43:29,246 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:43:29,273 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:43:39,488 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:43:40,490 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: processing FirefoxUX.app.tar.gz on https://mac-signing2.srv.releng.scl3.mozilla.com:9110
2013-07-31 22:43:40,515 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: uploading for signing
2013-07-31 22:43:50,661 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: error uploading file for signing: File too large
2013-07-31 22:43:51,662 - 0e5932e084241987ba0b74a1b4a3e948e0a76ca5: giving up after 20 tries
Attached patch shuffle them (obsolete) — Splinter Review
Assignee: nobody → rail
Status: NEW → ASSIGNED
Attachment #784505 - Flags: review?(bhearsum)
Comment on attachment 784505 [details] [diff] [review]
shuffle them

Review of attachment 784505 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think this will work...get_signing_cmd is only run at reconfig time AFAICT. I'm pretty sure the signing script itself needs to be updated for this.
Attachment #784505 - Flags: review?(bhearsum) → review-
(In reply to Rail Aliiev [:rail] from comment #3)
> Hmm, we already shuffle there:
> http://hg.mozilla.org/build/tools/file/ec64cdbee5b9/release/signing/signtool.
> py#l152

Yeah, that doesn't re-shuffle after an error though. I think that needs to happen inside of remote_signfile.
Product: mozilla.org → Release Engineering
Attached patch pop/appendSplinter Review
It turns out that we don't rotate signing server if urllib2.HTTPError.code is not 202.
Attachment #784505 - Attachment is obsolete: true
Attachment #799880 - Flags: review?(catlee)
Attachment #799880 - Flags: review?(catlee) → review+
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: