1131611 - delete NDK and SDK packages from all repos

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

These have been available since 2012, but it seems they shouldn't be.  They're in both the mockbuild repos and the puppetagain repos.  But they are used by neither -- we get the latest NDKs and SDKs from tooltool.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, but these are still used in the tst-emulator64 and tst-linux64 platforms:

/var/log/httpd/access_log-20150215:10.134.49.4 - - [13/Feb/2015:01:38:05 -0800] "GET /repos/apt/releng/pool/main/a/android-sdk18/android-sdk18_0.r18moz1-0moz1_amd64.deb HTTP/1.1" 200 193302528 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10)"
/var/log/httpd/access_log-20150215:10.134.48.124 - - [13/Feb/2015:02:09:07 -0800] "GET /repos/apt/releng/pool/main/a/android-sdk18/android-sdk18_0.r18moz1-0moz1_amd64.deb HTTP/1.1" 200 193302528 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10)"

dmitchell@admin1a ~ $ host 10.134.48.124
124.48.134.10.in-addr.arpa domain name pointer tst-emulator64-ec2-golden.test.releng.use1.mozilla.com.
dmitchell@admin1a ~ $ host 10.134.49.4
4.49.134.10.in-addr.arpa domain name pointer tst-linux64-ec2-golden.test.releng.use1.mozilla.com.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Chris, since you managed to figure out that we don't need the RPMs anymore for builders, what do you think about testers?

We *can* move these to a private repo, but among other things that may mean that builds of our test environment fail (/cc mrrrgn).  Which would mean that they're currently not re-distributable, too.

Comment 3

[Chris AtLee [:catlee]]

I'm not sure - perhaps nalexander has an idea?

Comment 4

[Nick Alexander :nalexander [he/him]]

(In reply to Chris AtLee [:catlee] from comment #3)
> I'm not sure - perhaps nalexander has an idea?

I don't have any deep knowledge, but based on http://mxr.mozilla.org/build/search?string=android-sdk18&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=build it looks like we're expecting ADB from androids-skd18.  It was my understanding that we don't actually *use* ADB -- instead, we use SUTagent -- but someone with more depth will need to address that.  gbrown seems a likely candidate, I'll redirect.

Comment 5

[Geoff Brown [:gbrown]]

The piece of the puzzle that I know is that the Android emulator test suites ("Android 2.3 API9 opt" and "Android 4.2 x86 opt") need to get the emulator and adb from the sdk. (They use sut for most device communication, but also use adb for a couple of things.) 

android_emulator_unittest.py assumes that "emulator" is in its PATH, for example, at http://hg.mozilla.org/build/mozharness/annotate/4f00febdee3a/scripts/android_emulator_unittest.py#l229. It sets its PATH to include /tools/android-sdk18/tools:/tools/android-sdk18/platform-tools at http://hg.mozilla.org/build/mozharness/annotate/4f00febdee3a/configs/android/androidarm.py#l24 and afaik always picks up the emulator and adb from there.

I'm not entirely clear on how /tools/android-sdk18 is populated.

Looking ahead a little bit, there are hopes/plans to start running a new Android emulator test suite (bug 1133833). Those tests will also need an emulator and adb and will use android_emulator_unittest.py again. The emulator in tools/android-sdk18 is too old for the new tests -- we need the new one from the r24 android sdk. I was tentatively thinking of leveraging http://hg.mozilla.org/build/mozharness/annotate/4f00febdee3a/mozharness/mozilla/testing/unittest.py#l217 to get the new sdk from tooltool, but am open to suggestions.

(To make this slightly more interesting, the new, r24 adb does not work as well as the old one in tools/android-sdk18, so it would be best to use the emulator from r24, but adb from r18!)

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

If such things can be installed via tooltool, that's ideal.  I'll block this bug on bug 1133833, then.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I verified that these packages don't seem to be hit on the mockbuild repos, either.

I think we can proceed to delete all android-[ns]dk RPMs; it's just the deb's that are still in active use.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I made a copy in the `private/` directory, which is what it says on the tin.

[[e-mail] repos]# ls -al /data/repos/private/android-internal/
total 1375312
drwxr-xr-x 2 root       root            4096 Feb 25 12:53 .
drwxr-xr-x 9 puppetsync puppetsync      4096 Feb 25 12:54 ..
-rw-r--r-- 1 puppetsync puppetsync  49163962 Jul 17  2012 android-ndk5-r5c-0moz3.i686.rpm
-rw-r--r-- 1 puppetsync puppetsync  94724465 Jul 17  2012 android-ndk7-r7b-0moz2.i686.rpm
-rw-r--r-- 1 puppetsync puppetsync 164965784 Dec 21  2012 android-ndk8-r8c-0moz1.i686.rpm
-rw-r--r-- 1 puppetsync puppetsync 179743122 Dec 21  2012 android-ndk8-r8c-0moz1.src.rpm
-rw-r--r-- 1 puppetsync puppetsync 403617333 Jul 17  2012 android-sdk15-r15-0moz1.i686.rpm
-rw-r--r-- 1 puppetsync puppetsync 516083559 Aug 17  2012 android-sdk16-r16-0moz2.i686.rpm

and deleted them from the mockbuild repos, too:

[[e-mail]3 mockbuild-repos]# find /mnt/netapp/relengweb/mockbuild-repos/ -name 'android-?dk*'
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/i386/android-ndk5-r5c-0moz3.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/i386/android-ndk7-r7b-0moz2.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/i386/android-ndk8-r8c-0moz1.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/i386/android-ndk8-r8c-0moz1.src.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/i386/android-sdk15-r15-0moz1.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/i386/android-sdk16-r16-0moz2.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/x86_64/android-ndk5-r5c-0moz3.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/x86_64/android-ndk7-r7b-0moz2.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/x86_64/android-ndk8-r8c-0moz1.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/x86_64/android-ndk8-r8c-0moz1.src.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/x86_64/android-sdk15-r15-0moz1.i686.rpm
/mnt/netapp/relengweb/mockbuild-repos/releng/public/CentOS/6/x86_64/android-sdk16-r16-0moz2.i686.rpm

[[e-mail]3 mockbuild-repos]# ./update
Synchronizing to S3 -- this is pretty slow
INFO: Compiling list of local files...
INFO: Retrieving list of remote files for s3://mozilla-releng-mock-centos-us-east-1/ ...
INFO: Found 19449 local files, 19461 remote files
INFO: Applying --exclude/--include
INFO: Verifying attributes...
INFO: Summary: 0 local files to upload, 12 remote files to delete
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/i386/android-ndk5-r5c-0moz3.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/i386/android-ndk7-r7b-0moz2.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/i386/android-ndk8-r8c-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/i386/android-ndk8-r8c-0moz1.src.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/i386/android-sdk15-r15-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/i386/android-sdk16-r16-0moz2.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/android-ndk5-r5c-0moz3.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/android-ndk7-r7b-0moz2.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/android-ndk8-r8c-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/android-ndk8-r8c-0moz1.src.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/android-sdk15-r15-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-east-1/releng/public/CentOS/6/x86_64/android-sdk16-r16-0moz2.i686.rpm'
INFO: Done. Uploaded 0 bytes in 0.0 seconds, 0.00 B/s
INFO: Compiling list of local files...
INFO: Retrieving list of remote files for s3://mozilla-releng-mock-centos-us-west-2/ ...
INFO: Found 19449 local files, 19461 remote files
INFO: Applying --exclude/--include
INFO: Verifying attributes...
INFO: Summary: 0 local files to upload, 12 remote files to delete
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/i386/android-ndk5-r5c-0moz3.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/i386/android-ndk7-r7b-0moz2.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/i386/android-ndk8-r8c-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/i386/android-ndk8-r8c-0moz1.src.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/i386/android-sdk15-r15-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/i386/android-sdk16-r16-0moz2.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/x86_64/android-ndk5-r5c-0moz3.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/x86_64/android-ndk7-r7b-0moz2.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/x86_64/android-ndk8-r8c-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/x86_64/android-ndk8-r8c-0moz1.src.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/x86_64/android-sdk15-r15-0moz1.i686.rpm'
deleted: 's3://mozilla-releng-mock-centos-us-west-2/releng/public/CentOS/6/x86_64/android-sdk16-r16-0moz2.i686.rpm'
INFO: Done. Uploaded 0 bytes in 0.0 seconds, 0.00 B/s

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Geoff Brown [:gbrown] from bug 1137743 comment #10)
> https://hg.mozilla.org/build/mozharness/rev/e170cecfdee2, landed in bug
> 1062365, has the changes to pull the sdk from tooltool. We just need to get
> that in production now...

Great!  How many stations does that need to ride the train through before we can adjust the system image to no longer contain the package?  I assume try jobs based on old revisions will be out of luck, but c'est la vie..

Comment 10

[Geoff Brown [:gbrown]]

I was thinking we could push to aurora, beta and up fairly soon rather than waiting for the trains, but the try issue is troubling.

:armenzg -- Any thoughts? This seems like a disadvantage of pinning.

Comment 11

[Armen [:armenzg]]

(In reply to Geoff Brown [:gbrown] from comment #10)
> I was thinking we could push to aurora, beta and up fairly soon rather than
> waiting for the trains, but the try issue is troubling.
> 
> :armenzg -- Any thoughts? This seems like a disadvantage of pinning.

It is the disadvantage of pinning. Limiting damage.

I think we should make a push to try from an old head and see where we stand.
We could file a bug which treeherder would match it to so developers can find out what they need to make their old try push to work.

I'm lookint at:
https://hg.mozilla.org/build/mozharness/rev/e170cecfdee2
Why didn't those manifests go to testing/config/tooltool-manifests/*/releng.manifest?

Comment 12

[Geoff Brown [:gbrown]]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #11)
> I think we should make a push to try from an old head and see where we stand.
> We could file a bug which treeherder would match it to so developers can
> find out what they need to make their old try push to work.

Thanks - that's a good idea.

> I'm lookint at:
> https://hg.mozilla.org/build/mozharness/rev/e170cecfdee2
> Why didn't those manifests go to
> testing/config/tooltool-manifests/*/releng.manifest?

I wanted to leverage the existing mechanism at https://hg.mozilla.org/build/mozharness/rev/e170cecfdee2#l4.33.

Comment 13

[Geoff Brown [:gbrown]]

The mozharness change has now been uplifted to beta: https://bugzilla.mozilla.org/show_bug.cgi?id=1062365#c38.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

I'll admit that I'm vague on train timetables, but isn't the next release an ESR?  If so, it might be good to jump the trains and get the MH change into that release so that we're not bothered by test failures for the lifetime of the ESR.

Comment 15

[Geoff Brown [:gbrown]]

On irc, word is that the next esr is 38 (today's aurora). Also, there are no plans for an android esr.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

I'm still vague on the plans here -- waiting to ride all of the trains?  Or can we push to the remaining branches now that it seems stable?

Comment 17

[Geoff Brown [:gbrown]]

I checked test logs for Android 2.3 Opt and Android 4.2 x86 on mozilla-aurora, mozilla-beta, and mozilla-release and verified that my changes are in use in all those cases: aurora, beta and release all pull the sdk from tooltool now. I *think* that means you are clear to proceed with deletion.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Great!  I'll start by removing the code to install the packages.  If that doesn't break, then I'll move the packages aside and see if *that* breaks.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

I just discovered that this is installed on signing servers, too.  I'm not sure it's *used* on signing servers, though.  I'll work around that temporarily by installing from a private repository, but if possible I'd like to get rid of the package altogether.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1131611-ubuntu-sdk18.patch

This just addresses the Ubuntu installs.  We won't see much effect from this until the nightly AMI build; onsite ubuntu hosts will still have the package installed.  Callek, can you let the sheriffs know that this is a likely cause if all of the EC2 emulator jobs start failing the morning after I land this?

I'll send up another patch shortly to allow installation of sdk16 on signing servers (which is currently broken for new signing servers).

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1131611-signing.patch

tested successfully on signing4

Comment 22

[Justin Wood (:Callek)]

Comment on attachment 8581827 [details] [diff] [review]
bug1131611-ubuntu-sdk18.patch

Review of attachment 8581827 [details] [diff] [review]:
-----------------------------------------------------------------

I have an internal gut feeling that stuff will blow up, though I can't pinpoint why that feeling exists.
So if you're willing to be on the hook to rollback the AMI if need be, the code looks good.

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8581827 [details] [diff] [review]
bug1131611-ubuntu-sdk18.patch

remote:   https://hg.mozilla.org/build/puppet/rev/f14dc1356da1
remote:   https://hg.mozilla.org/build/puppet/rev/0fdaa817c1e8

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8581835 [details] [diff] [review]
bug1131611-signing.patch

remote:   https://hg.mozilla.org/build/puppet/rev/9c86dba6aa37
remote:   https://hg.mozilla.org/build/puppet/rev/0fdaa817c1e8

Comment 25

[Dustin J. Mitchell [:dustin] (he/him)]

Remaining:
 * ensure this deploys correctly (considering a manual regen today)
 * remove debs from repos
 * try uninstalling from a signing server and see if things break (signs point to "no")

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Dustin J. Mitchell [:dustin] from comment #25)
> Remaining:
>  * ensure this deploys correctly (considering a manual regen today)

Looks good

>  * remove debs from repos

[[e-mail] apt]# pwd
/data/repos/apt
[[e-mail] apt]# tar -zcf ../private/android-apt.tar.gz releng/pool/main/a/android-sdk18
[[e-mail] apt]# rm -rf releng/pool/main/a/android-sdk18

../private is, as the name suggests, not publicly visible.  So at this point, we're no longer making these packages available to the public anywhere that I'm aware of.

I left the packages in the apt indexes, mostly because the script to re-index doesn't run on CentOS and the repo is deprecated, anyway (another project the bugfairies are busily at work on).

>  * try uninstalling from a signing server and see if things break (signs
> point to "no")

Coming up next.

Comment 27

[Dustin J. Mitchell [:dustin] (he/him)]

15:07 < dustin> coop|buildduty: for bug 1131611 expectatoins point to not needing the sdk -- I'd like to uninstall it on signing4 and leave puppet disabled (otherwise it will reinstall the sdk) and see if android signings fail
15:07 < dustin> rolling back is as simple as running puppet
15:07 < dustin> should I do that today or hold off?

Comment 28

[Dustin J. Mitchell [:dustin] (he/him)]

removed on signing4 (per irc OK)

Comment 29

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1131611-remove-on-signing.patch

Note that this RPM has no dependencies on other packages, so I don't think there's anything to do to deploy this beyond a manual 'yum remove':

[[e-mail] android-sdk]# rpm -qRp android-sdk16-r16-0moz2.i686.rpm 
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1

(those are just dependencies on rpmlib features)

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

OK, just cleanup remains.  I'll stash the packages in ~root just in case we find a burning need for them (I doubt they're reproducible anymore).

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

[[e-mail] ~]# ls android-*
android-apt.tar.gz  android-sdk-yum.tar.gz

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

See bug 1149580 for the fallout from running releng-update.sh in comment 26.