666760 - android sdk 'helpfully' generates a debug.keystore that expires after one year, which we don't want on CI builds.

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

We use this key for opt and debug builds.  It has expired.

make[6]: Entering directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox/build/mobile/sutagent/android'
/tools/android-sdk/platforms/android-8/tools/aapt package -f -M /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/AndroidManifest.xml -I /tools/android-sdk/platforms/android-8/android.jar  -S res -F sutAgentAndroid.ap_
/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox/config/nsinstall -D classes
"/tools/jdk6/bin/javac" -target 1.5 -classpath /tools/android-sdk/platforms/android-8/android.jar:/builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/network-libs/commons-net-2.0.jar -bootclasspath /tools/android-sdk/platforms/android-8/android.jar -encoding ascii -g  -d classes  /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/AlertLooperThread.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/ASMozStub.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/CmdWorkerThread.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/DataWorkerThread.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/DoAlert.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/DoCommand.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/NtpMessage.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/Power.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/RedirOutputThread.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/RunCmdThread.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/RunDataThread.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/SUTAgentAndroid.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/SUTStartupIntentReceiver.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/WifiConfiguration.java /builds/slave/cen-lnx-andrd-dbg/build/build/mobile/sutagent/android/R.java
/tools/android-sdk/platforms/android-8/tools/dx --dex --output=classes.dex classes
/tools/android-sdk/platforms/android-8/../../tools/apkbuilder sutAgentAndroid-unsigned-unaligned.apk -v  -z sutAgentAndroid.ap_ -f classes.dex
Using keystore: /home/cltbld/.android/debug.keystore
com.android.apkbuilder.ApkBuilder$ApkCreationException: Debug Certificate expired on 6/23/11 1:20 PM
NEXT ERROR make[6]: *** [sutAgentAndroid-unsigned-unaligned.apk] Error 1
make[6]: *** Deleting file `sutAgentAndroid-unsigned-unaligned.apk'
make[6]: Leaving directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox/build/mobile/sutagent/android'
NEXT ERROR make[5]: *** [tools] Error 2
make[5]: Leaving directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox/build'
NEXT ERROR make[4]: *** [tools_tier_base] Error 2
make[4]: Leaving directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox'
NEXT ERROR make[3]: *** [tier_base] Error 2
make[3]: Leaving directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox'
make[1]: *** [realbuild] Error 2
make[1]: Leaving directory `/builds/slave/cen-lnx-andrd-dbg/build'
make: *** [build] Error 2

Comment 1

[Daniel Holbert [:dholbert]]

(In reply to comment #0)
> We use this key for opt and debug builds.  It has expired.
> com.android.apkbuilder.ApkBuilder$ApkCreationException: Debug Certificate
> expired on 6/23/11 1:20 PM

FWIW, we had an opt build start at 1:44 PM and succeed:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1308861846.1308862463.4987.gz
so opt builds might not be hurt by this (but I don't know why)

Comment 2

[Marco Bonardo [:mak]]

mozilla-inbound has an Android opt failing on the same issue.
Thus since we don't have tests on Android I'm closing the trees.

Comment 3

[Marco Bonardo [:mak]]

I closed both m-c and m-i.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

i am trying to reproduce this on the command line so I can try a couple things to try and fix this

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

I deleted the ~/.android/debug.keystore file and re-ran |make -f client.mk build| on the offending slave and got

make[6]: Entering directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox/build/mobile/sutagent/android'
/tools/android-sdk/platforms/android-8/../../tools/apkbuilder sutAgentAndroid-unsigned-unaligned.apk -v  -z sutAgentAndroid.ap_ -f classes.dex
Using keystore: /home/cltbld/.android/debug.keystore
/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox/build/mobile/sutagent/android/classes.dex => classes.dex
cp  sutAgentAndroid-unsigned-unaligned.apk sutAgentAndroid-unaligned.apk
/tools/android-sdk/platforms/android-8/../../tools/zipalign -f -v 4 sutAgentAndroid-unaligned.apk sutAgentAndroid.apk
Verifying alignment of sutAgentAndroid.apk (4)...
      60 res/drawable/ateamlogo.png (OK)
     836 res/drawable/ic_stat_first.png (OK)
    2376 res/drawable/ic_stat_neterror.png (OK)
    3068 res/drawable/ic_stat_second.png (OK)
    4244 res/drawable/ic_stat_warning.png (OK)
    4992 res/drawable/icon.png (OK)
    7241 res/layout/main.xml (OK - compressed)
    7671 AndroidManifest.xml (OK - compressed)
    9296 resources.arsc (OK)
   11373 classes.dex (OK - compressed)
   68264 META-INF/MANIFEST.MF (OK - compressed)
   68755 META-INF/CERT.SF (OK - compressed)
   69278 META-INF/CERT.RSA (OK - compressed)
Verification succesful
make[6]: Leaving directory `/builds/slave/cen-lnx-andrd-dbg/build/obj-firefox/build/mobile/sutagent/android'

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

it seems that debug.keystore is generated if not present, with a 1 year validity period, on each build.  This means that this particular slave happened to run its first android debug build one year ago.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

http://code.google.com/p/android/issues/detail?colspec=ID%20Type%20Status%20Owner%20Summary%20Stars&id=15370

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

I can confirm that each slave has a different date of generation to the debug.keystore file, likely when it first did an android build

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

Alright, we have deleted all debug.keystore files.  Back to a 'normal' bug, leaving open for work to prevent this from happening again.

Options to deal with this in future that I can see:

1) delete with puppet on boot
2) add autoconf+makefile goo to delete this file
3) rebuild sdk with https://review.source.android.com/#change,22128

option 1 is probably the best, and what i'll be doing.  Leaving this bug open for this work.

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: delete the debug keystore on boot

Since we know that we reboot each machine at least once a year, this should fix the problem.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 541537 [details] [diff] [review]
delete the debug keystore on boot

http://hg.mozilla.org/build/puppet-manifests/rev/807451aa6b4b

In production