Closed Bug 833997 Opened 10 years ago Closed 10 years ago

android repacks on AWS machines sometimes (always?) fail to upload

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: hwine, Assigned: mozilla)

References

Details

Attachments

(2 files, 1 obsolete file)

del MOZ_SIGN_CMD from upload_env
1.04 KB, patch
Details | Diff | Splinter Review
remove -f jar from upload_env
1.01 KB, patch
bhearsum
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review 
all 6 repack uploads failed after signing. We manually uploaded to stage & moved to candidates.

release-mozilla-beta-android_repack_*/6 all failed at step 12 (run_script) with a last stdio lines of:

11:33:49     INFO - #####
11:33:49     INFO - ##### Running upload-repacks step.
11:33:49     INFO - #####
11:33:49     INFO - Getting output from command: ['mock_mozilla', '-r', 'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/rel-m-beta-andrd-rpk-6/build/mozilla-beta/obj-l10n/mobile/android/locales', '--unpriv', '--shell', u'/usr/bin/env "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-firefox PATH=/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing3.srv.releng.scl3.mozilla.com:9120 SHLVL=1 UPLOAD_HOST=stage.mozilla.org G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ UPLOAD_USER=ffxbld MOZILLA_REV=FENNEC_19_0b3_RELEASE JAVA_HOME=/tools/jdk6 MOZ_PKG_VERSION=19.0b3 HG_SHARE_BASE_DIR=/builds/hg-shared SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com CCACHE_DIR=/builds/ccache CLOBBERER_URL=http://clobberer.pvt.build.mozilla.org/always_clobber.php BUILDBOTCUSTOM=http://hg.mozilla.org/build/buildbotcustom SHIP_LICENSED_FONTS=1 UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa TINDERBOX_OUTPUT=1 COMM_REV=FENNEC_19_0b3_RELEASE "MOZ_SIGN_CMD=python /builds/slave/rel-m-beta-andrd-rpk-6/build/tools/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-andrd-rpk-6/build/signing_cache -t /builds/slave/rel-m-beta-andrd-rpk-6/token -n /builds/slave/rel-m-beta-andrd-rpk-6/nonce -c /builds/slave/rel-m-beta-andrd-rpk-6/build/tools/release/signing/host.cert -f jar -H signing3.srv.releng.scl3.mozilla.com:9120" LC_ALL=C UPLOAD_TO_TEMP=1 _=/tools/buildbot/bin/python BUILDBOT_CONFIGS=http://hg.mozilla.org/build/buildbot-configs MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=beta HOSTNAME=bld-linux64-ec2-309.build.aws-us-east-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/rel-m-beta-andrd-rpk-6 PROPERTIES_FILE=/builds/slave/rel-m-beta-andrd-rpk-6/buildprops.json "POST_UPLOAD_CMD=post_upload.py -p mobile -n 1 -v 19.0b3 --builddir android/sl --release-to-mobile-candidates-dir --nightly-dir=candidates" MOZ_CRASHREPORTER_NO_REPORT=1 CCACHE_COMPRESS=1 make upload AB_CD=sl 2>&1'] in /builds/slave/rel-m-beta-andrd-rpk-6/build/mozilla-beta/obj-l10n/mobile/android/locales
11:33:49     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/rel-m-beta-andrd-rpk-6/build/mozilla-beta/obj-l10n/mobile/android/locales --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-firefox PATH=/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing3.srv.releng.scl3.mozilla.com:9120 SHLVL=1 UPLOAD_HOST=stage.mozilla.org G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ UPLOAD_USER=ffxbld MOZILLA_REV=FENNEC_19_0b3_RELEASE JAVA_HOME=/tools/jdk6 MOZ_PKG_VERSION=19.0b3 HG_SHARE_BASE_DIR=/builds/hg-shared SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com CCACHE_DIR=/builds/ccache CLOBBERER_URL=http://clobberer.pvt.build.mozilla.org/always_clobber.php BUILDBOTCUSTOM=http://hg.mozilla.org/build/buildbotcustom SHIP_LICENSED_FONTS=1 UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa TINDERBOX_OUTPUT=1 COMM_REV=FENNEC_19_0b3_RELEASE \"MOZ_SIGN_CMD=python /builds/slave/rel-m-beta-andrd-rpk-6/build/tools/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-andrd-rpk-6/build/signing_cache -t /builds/slave/rel-m-beta-andrd-rpk-6/token -n /builds/slave/rel-m-beta-andrd-rpk-6/nonce -c /builds/slave/rel-m-beta-andrd-rpk-6/build/tools/release/signing/host.cert -f jar -H signing3.srv.releng.scl3.mozilla.com:9120\" LC_ALL=C UPLOAD_TO_TEMP=1 _=/tools/buildbot/bin/python BUILDBOT_CONFIGS=http://hg.mozilla.org/build/buildbot-configs MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=beta HOSTNAME=bld-linux64-ec2-309.build.aws-us-east-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/rel-m-beta-andrd-rpk-6 PROPERTIES_FILE=/builds/slave/rel-m-beta-andrd-rpk-6/buildprops.json \"POST_UPLOAD_CMD=post_upload.py -p mobile -n 1 -v 19.0b3 --builddir android/sl --release-to-mobile-candidates-dir --nightly-dir=candidates\" MOZ_CRASHREPORTER_NO_REPORT=1 CCACHE_COMPRESS=1 make upload AB_CD=sl 2>&1"

command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1941.698539
I see a nightly failing with the same thing, so this is probably fallout from bug 705807 :(.

https://tbpl.mozilla.org/php/getParsedLog.php?id=19055414&tree=Firefox&full=1
Component: Release Engineering: Automation (Release Automation) → Release Engineering: Automation (General)
QA Contact: bhearsum → catlee
Err, might not be fallout from 705807, just related to AWS.
Summary: apk uploads from AWS failed for 19.0b3 → android repacks on AWS machines sometimes (always?) fail to upload
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-l10n/?C=M;O=A

From output on the 21st, I see some good repacks (with mozpass.py defined), on AWS.
I also see a broken one without mozpass.py defined (must have been post-reconfig / landing).

I don't know why uploads would break, but now I can focus debugging efforts.
Yay run_command()'s copy/paste output!!

I ran a repack on bld-linux64-ec2-305 and killed it when it hung on make upload.

Then I ran the make upload command by hand, and it kept trying to send the checksum file(s) to the signing server.

Then I removed the MOZ_SIGN_CMD from the command line, and it worked!

GOOD

mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/m-aurora-andrd-l10n-4/build/mozilla-aurora/obj-l10n/mobile/android/locales --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-firefox SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing3.srv.releng.scl3.mozilla.com:9110 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ UPLOAD_USER=ffxbld JAVA_HOME=/tools/jdk6 HG_SHARE_BASE_DIR=/builds/hg-shared SYMBOL_SERVER_HOST=dev-stage01.srv.releng.scl3.mozilla.com CCACHE_DIR=/builds/ccache UPLOAD_HOST=stage.mozilla.org SHIP_LICENSED_FONTS=1 PATH=/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa TINDERBOX_OUTPUT=1 LC_ALL=C UPLOAD_TO_TEMP=1 _=/tools/buildbot/bin/python MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=aurora HOSTNAME=bld-linux64-ec2-305.build.aws-us-east-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-aurora-andrd-l10n-4 PROPERTIES_FILE=/builds/slave/m-aurora-andrd-l10n-4/buildprops.json \"POST_UPLOAD_CMD=post_upload.py -b mozilla-aurora-android-l10n -p mobile -i 20130123042017 --release-to-latest --release-to-dated\" MOZ_CRASHREPORTER_NO_REPORT=1 CCACHE_COMPRESS=1 make upload AB_CD=nl 2>&1"
mkdir -p `dirname "../../../dist//fennec-20.0a2.nl.android-arm.checksums"`
CHECKSUM FILE START
4ba67b947fa775ec78955826450ac79e1ea4f9a998c6f1e4a9747457bf0f3eb5c8032f049df5027346633c973a531241ce9eb85e82c068e98551d58b838383a8 sha512 23240005 fennec-20.0a2.nl.android-arm.apk
f2b2c0df7632df9f1fad99653d0e97ec md5 23240005 fennec-20.0a2.nl.android-arm.apk
e59ad8287929c232d7010400bfd76487cfbfaa9a sha1 23240005 fennec-20.0a2.nl.android-arm.apk
1c4e0038a4c5b3c9b6c0914f2d61ce90add50e5d1aa4478f2c69185ec40f8cfb0ad179dd8a2cc190dc11f99380328508b26824113d3f49a607142f78b87d0de3 sha512 23209123 gecko-unsigned-unaligned.apk
49ee1c9e61dae919ce62f7f8f67dc890 md5 23209123 gecko-unsigned-unaligned.apk
2405de74880ce22bf0e91f374b1409d6a7d41338 sha1 23209123 gecko-unsigned-unaligned.apk
CHECKSUM FILE END
/builds/slave/m-aurora-andrd-l10n-4/build/mozilla-aurora/obj-l10n/_virtualenv/bin/python /builds/slave/m-aurora-andrd-l10n-4/build/mozilla-aurora/build/upload.py --base-path ../../../dist \
                "../../../dist/fennec-20.0a2.nl.android-arm.apk"            ../../../dist/gecko-unsigned-unaligned.apk \
                "../../../dist//fennec-20.0a2.nl.android-arm.checksums"
Uploading /builds/slave/m-aurora-andrd-l10n-4/build/mozilla-aurora/obj-l10n/dist/fennec-20.0a2.nl.android-arm.apk
fennec-20.0a2.nl.android-arm.apk                                                                                                                                                                            100%   22MB   3.7MB/s   00:06    
Uploading /builds/slave/m-aurora-andrd-l10n-4/build/mozilla-aurora/obj-l10n/dist/gecko-unsigned-unaligned.apk
gecko-unsigned-unaligned.apk                                                                                                                                                                                100%   22MB   3.7MB/s   00:06    
Uploading /builds/slave/m-aurora-andrd-l10n-4/build/mozilla-aurora/obj-l10n/dist/fennec-20.0a2.nl.android-arm.checksums
fennec-20.0a2.nl.android-arm.checksums                                                                                                                                                                      100%  678     0.7KB/s   00:00    
Running post-upload command: post_upload.py -b mozilla-aurora-android-l10n -p mobile -i 20130123042017 --release-to-latest --release-to-dated
sys.argv: ['/usr/local/bin/post_upload.py', '-b', 'mozilla-aurora-android-l10n', '-p', 'mobile', '-i', '20130123042017', '--release-to-latest', '--release-to-dated', '/tmp/tmp.aEyQ4jSgwN/', '/tmp/tmp.aEyQ4jSgwN//fennec-20.0a2.nl.android-arm.apk', '/tmp/tmp.aEyQ4jSgwN//gecko-unsigned-unaligned.apk', '/tmp/tmp.aEyQ4jSgwN//fennec-20.0a2.nl.android-arm.checksums']
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2013/01/2013-01-23-04-20-17-mozilla-aurora-android-l10n/fennec-20.0a2.nl.android-arm.apk
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2013/01/2013-01-23-04-20-17-mozilla-aurora-android-l10n/gecko-unsigned-unaligned.apk
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2013/01/2013-01-23-04-20-17-mozilla-aurora-android-l10n/fennec-20.0a2.nl.android-arm.checksums
Upload complete


BROKEN

mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/m-aurora-andrd-l10n-4/build/mozilla-aurora/obj-l10n/mobile/android/locales --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld MOZ_OBJDIR=obj-firefox SYMBOL_SERVER_USER=ffxbld DISPLAY=:2 CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing3.srv.releng.scl3.mozilla.com:9110 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_mob/ UPLOAD_USER=ffxbld JAVA_HOME=/tools/jdk6 HG_SHARE_BASE_DIR=/builds/hg-shared SYMBOL_SERVER_HOST=dev-stage01.srv.releng.scl3.mozilla.com CCACHE_DIR=/builds/ccache UPLOAD_HOST=stage.mozilla.org SHIP_LICENSED_FONTS=1 PATH=/tools/jdk6/bin:/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/ UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa TINDERBOX_OUTPUT=1 \"MOZ_SIGN_CMD=python /builds/slave/m-aurora-andrd-l10n-4/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-aurora-andrd-l10n-4/build/signing_cache -t /builds/slave/m-aurora-andrd-l10n-4/token -n /builds/slave/m-aurora-andrd-l10n-4/nonce -c /builds/slave/m-aurora-andrd-l10n-4/build/tools/release/signing/host.cert -f jar -H signing3.srv.releng.scl3.mozilla.com:9110\" LC_ALL=C UPLOAD_TO_TEMP=1 _=/tools/buildbot/bin/python MAIL=/var/spool/mail/cltbld MOZ_UPDATE_CHANNEL=aurora HOSTNAME=bld-linux64-ec2-305.build.aws-us-east-1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/ffxbld_dsa HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/m-aurora-andrd-l10n-4 PROPERTIES_FILE=/builds/slave/m-aurora-andrd-l10n-4/buildprops.json \"POST_UPLOAD_CMD=post_upload.py -b mozilla-aurora-android-l10n -p mobile -i 20130123042017 --release-to-latest --release-to-dated\" MOZ_CRASHREPORTER_NO_REPORT=1 CCACHE_COMPRESS=1 make upload AB_CD=nl 2>&1"
mkdir -p `dirname "../../../dist//fennec-20.0a2.nl.android-arm.checksums"`
CHECKSUM FILE START
4ba67b947fa775ec78955826450ac79e1ea4f9a998c6f1e4a9747457bf0f3eb5c8032f049df5027346633c973a531241ce9eb85e82c068e98551d58b838383a8 sha512 23240005 fennec-20.0a2.nl.android-arm.apk
f2b2c0df7632df9f1fad99653d0e97ec md5 23240005 fennec-20.0a2.nl.android-arm.apk
e59ad8287929c232d7010400bfd76487cfbfaa9a sha1 23240005 fennec-20.0a2.nl.android-arm.apk
1c4e0038a4c5b3c9b6c0914f2d61ce90add50e5d1aa4478f2c69185ec40f8cfb0ad179dd8a2cc190dc11f99380328508b26824113d3f49a607142f78b87d0de3 sha512 23209123 gecko-unsigned-unaligned.apk
49ee1c9e61dae919ce62f7f8f67dc890 md5 23209123 gecko-unsigned-unaligned.apk
2405de74880ce22bf0e91f374b1409d6a7d41338 sha1 23209123 gecko-unsigned-unaligned.apk
CHECKSUM FILE END
python /builds/slave/m-aurora-andrd-l10n-4/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-aurora-andrd-l10n-4/build/signing_cache -t /builds/slave/m-aurora-andrd-l10n-4/token -n /builds/slave/m-aurora-andrd-l10n-4/nonce -c /builds/slave/m-aurora-andrd-l10n-4/build/tools/release/signing/host.cert -f jar -H signing3.srv.releng.scl3.mozilla.com:9110 -f gpg "../../../dist//fennec-20.0a2.nl.android-arm.checksums"
2013-01-23 21:51:53,329 - e328e6e83c10d91bec9e385437c118415a395e18: processing ../../../dist//fennec-20.0a2.nl.android-arm.checksums on https://signing3.srv.releng.scl3.mozilla.com:9110
2013-01-23 21:51:54,777 - e328e6e83c10d91bec9e385437c118415a395e18: processing ../../../dist//fennec-20.0a2.nl.android-arm.checksums on https://signing3.srv.releng.scl3.mozilla.com:9110

...
Attached patch fix android l10n (obsolete) — — Splinter Review 
Assignee: nobody → aki

        Attachment #705702 -
        Flags: review?(bhearsum)

    
    

    
  
Comment on attachment 705702 [details] [diff] [review]
fix android l10n

I think I actually have to del upload_env['MOZ_SIGN_CMD']... checking.

        Attachment #705702 -
        Attachment is obsolete: true

        Attachment #705702 -
        Flags: review?(bhearsum)

    
    

    
  


  

        Attachment #705717 -
        Flags: review?(bhearsum)

    
    

    
  
Nice detective work. I just looked over at the signing logs and found this:
2013-01-23 18:38:52,581 - INFO - Signing fennec-20.0a2.nl.android-arm.checksums (jar - e328e6e83c10d91bec9e385437c118415a395e18)
2013-01-23 18:38:52,835 - INFO - 5086: run_signscript: Failed with rc 1; retrying in a bit
2013-01-23 18:38:54,031 - INFO - 10.134.53.61 GET /sign/jar/e328e6e83c10d91bec9e385437c118415a395e18
2013-01-23 18:38:58,088 - INFO - 5089: run_signscript: Failed with rc 1; retrying in a bit
2013-01-23 18:39:03,343 - INFO - 5096: run_signscript: Failed with rc 1; retrying in a bit
2013-01-23 18:39:08,603 - INFO - 5114: run_signscript: Failed with rc 1; retrying in a bit
2013-01-23 18:39:13,859 - INFO - 5117: run_signscript: Failed with rc 1; retrying in a bit
2013-01-23 18:39:13,859 - WARNING - run_signscript: Exceeded maximum number of retries; exiting
2013-01-23 18:39:13,860 - WARNING - Signing failed fennec-20.0a2.nl.android-arm.checksums (jar - e328e6e83c10d91bec9e385437c118415a395e18)
2013-01-23 18:39:13,860 - WARNING - Signing log: 2013-01-23 18:38:52,633 - unzip output: Archive:  /home/cltsign/instances/dep-key-signing-server-2/signed-files/jar/e328e6e83c10d91bec9e385437c118415a395e18.tmp
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /home/cltsign/instances/dep-key-signing-server-2/signed-files/jar/e328e6e83c10d91bec9e385437c118415a395e18.tmp or
        /home/cltsign/instances/dep-key-signing-server-2/signed-files/jar/e328e6e83c10d91bec9e385437c118415a395e18.tmp.zip, and cannot find /home/cltsign/instances/dep-key-signing-server-2/signed-files/jar/e328e6e83c10d91bec9e385437c118415a395e18.tmp.ZIP, period.


And looking at MOZ_SIGN_CMD, I'm not surprised:
python /builds/slave/m-aurora-andrd-l10n-4/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-aurora-andrd-l10n-4/build/signing_cache -t /builds/slave/m-aurora-andrd-l10n-4/token -n /builds/slave/m-aurora-andrd-l10n-4/nonce -c /builds/slave/m-aurora-andrd-l10n-4/build/tools/release/signing/host.cert -f jar -H signing3.srv.releng.scl3.mozilla.com:9110 -f gpg "../../../dist//fennec-20.0a2.nl.android-arm.checksums"

We have a "-f jar" in there that doesn't make any sense. It sounds like the root problem here is a bug in the build system code. Maybe we should fix that instead so we can have signed checksums files?

    
    

    
  
So, the checksum sign command gets set here:
https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#732

Which means that MOZ_SIGN_CMD already has "-f jar" in it. This is different than all of the other things we do with the signing server, where the build system plugs in the formats at the last second. I see that we already have "-f jar" being appended to it here: https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/packager.mk#250

And indeed, the full JARSIGNER command ends up being:
python /builds/slave/rel-m-beta-andrd-rpk-6/build/tools/release/signing/signtool.py --cachedir /builds/slave/rel-m-beta-andrd-rpk-6/build/signing_cache -t /builds/slave/rel-m-beta-andrd-rpk-6/token -n /builds/slave/rel-m-beta-andrd-rpk-6/nonce -c /builds/slave/rel-m-beta-andrd-rpk-6/build/tools/release/signing/host.cert -f jar -H signing3.srv.releng.scl3.mozilla.com:9120 -f jar /builds/slave/rel-m-beta-andrd-rpk-6/build/mozilla-beta/obj-l10n/mobile/android/locales/../../../dist/gecko.apk

Note the two "-f jar". So, we should just remove the "-f jar" that mozharness sets. Then we'll get signed checksums!

        Attachment #705717 -
        Flags: review?(bhearsum)

    
    

    
  
Hm.
I don't think we can [easily] remove the -f jar, since query_moz_sign_cmd() will insert a |-f gpg| by default:

http://hg.mozilla.org/build/mozharness/file/7b46580edbc7/mozharness/mozilla/signing.py#l113

However, I can certainly switch the upload_env to use the default |-f gpg|...

    
    

    
  


  

        Attachment #705927 -
        Flags: review?(bhearsum)

    
    

    
  
(In reply to Aki Sasaki [:aki] from comment #10)
> Hm.
> I don't think we can [easily] remove the -f jar, since query_moz_sign_cmd()
> will insert a |-f gpg| by default:
> 
> http://hg.mozilla.org/build/mozharness/file/7b46580edbc7/mozharness/mozilla/
> signing.py#l113
> 
> However, I can certainly switch the upload_env to use the default |-f gpg|...

Shouldn't mozharness be letting the build system deal with formats? Either way, you could make formats optional and iterate over them if they exist.

    
    

    
  
(In reply to Ben Hearsum [:bhearsum] from comment #12)
> Shouldn't mozharness be letting the build system deal with formats? Either
> way, you could make formats optional and iterate over them if they exist.

I don't know what that would break.

    
    

    
  
Happy to let you test all signed mozharness things though.

    
    

    
  
Comment on attachment 705927 [details] [diff] [review]
remove -f jar from upload_env

Review of attachment 705927 [details] [diff] [review]:
-----------------------------------------------------------------

No wonder we have so much tech debt.

        Attachment #705927 -
        Flags: review?(bhearsum) → review+
Product: mozilla.org → Release Engineering
Component: General Automation → General

          You need to log in
          before you can comment on or make changes to this bug.