933918 - Replace sdk tools directory on Android x86 emulator test slaves

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Geoff Brown [:gbrown] (pto Apr 11-18)]

In bug 917562, a bug was identified in the Android x86 emulator, and an emulator patch was submitted for qemu-devel: http://lists.gnu.org/archive/html/qemu-devel/2013-09/msg04162.html. 

To benefit from this change in the near term, we need to update the x86 emulator binaries used for Android x86 emulator tests.

The emulator is currently started from /tools/android-sdk18/tools on the test machines. As far as I know, /tools/android-sdk18 is a simple copy of the Android SDK, version 18.

http://people.mozilla.org/~gbrown/tools-tools_r22-patched.tar.gz is a new build of the Android SDK tools directory, with the emulator fix applied. We need to replace /tools/android-sdk18/tools (note -- only the tools subdirectory) with the contents of tools-tools_r22-patched.tar.gz.

Comment 1

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Notes on building tools-tools_r22-patched.tar.gz:

 - I used the instructions at http://source.android.com/source/building.html to build Android in a 64 bit Ubuntu 12.04 environment using tag "tools_r22":

repo init -u https://android.googlesource.com/platform/manifest -b tools_r22
repo sync
<apply patch from http://lists.gnu.org/archive/html/qemu-devel/2013-09/msg04162.html>
source build/envsetup.sh
lunch sdk-eng
make sdk

cd out/host/linux-x86/sdk/android-sdk_eng.mozdev_linux-x86/tools
tar cvf ~/tools-tools_r22-patched.tar *
gzip ~/tools-tools_r22-patched.tar


Output summary from lunch:
============================================
PLATFORM_VERSION_CODENAME=AOSP
PLATFORM_VERSION=4.2.2.2.2.2.2.2.2.2
TARGET_PRODUCT=sdk
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
TARGET_CPU_VARIANT=generic
HOST_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-3.5.0-23-generic-x86_64-with-Ubuntu-12.04-precise
HOST_BUILD_TYPE=release
BUILD_ID=OPENMASTER
OUT_DIR=out
============================================

mozdev@ubuntu:~/aosp$ cat sdk/files/tools_source.properties
Pkg.UserSrc=false
Pkg.Revision=22.0.5
Platform.MinPlatformToolsRev=18

mozdev@ubuntu:~/aosp$ out/host/linux-x86/sdk/android-sdk_eng.mozdev_linux-x86/tools/emulator -version
Android emulator version 22.0.5.0 (build_id OPENMASTER-eng.mozdev.20131031.112001)
 ...

Comment 2

[Armen [:armenzg]]

Noting that we only want to substitute the contents of /tools/android-sdk18/tools:

FTR, this is what we are running:
[[e-mail] ~]# /tools/android-sdk18/tools/emulator -version
Android emulator version 22.0.5.0 (build_id OPENMASTER-757759)

Comment 3

[Armen [:armenzg]]

dustin, Callek: where do I read on how to re-bundle the previous android-sdk18 with a newer version?
Do I have to create an rpm?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Are these different from AVDs (bug 913011)?

RPMs won't help much on Ubuntu!

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Which is to say, I don't know of a way, and you'll probably need to invent one.

Comment 6

[Justin Wood (:Callek)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #3)
> dustin, Callek: where do I read on how to re-bundle the previous
> android-sdk18 with a newer version?
> Do I have to create an rpm?

You'll need to rebuild based on the .deb info, currently at https://github.com/rail/android-sdk

I think it was rail who did it for me initially so I suspect he can achieve this for us this time around.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, cool - and hopefully get android-sdk-debian put into the puppet tree?

Comment 8

[Rail Aliiev [:rail]]

http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/pool/main/a/android-sdk18/android-sdk18_0.r18-0moz1.debian.tar.gz actually contains everything you need to build the package. https://github.com/rail/android-sdk is just more convenient (for me) way to play around with debianization when you want to upgrade.

Probably we should have a policy for Debian packages we build. Unfortunately unlike RPMs they don't use a singe file describing packaging and different packages may use different approaches to keep debianization separated from the source code...

Comment 9

[Armen [:armenzg]]

Attachment: deb generation for android sdk 18 - version moz1

In my next patch I will change this to generate the next version of the SDK we need to deploy.

I could generate it with this command:
~/bin/puppetagain-build-deb precise amd64 modules/packages/manifests/android-sdk18-debian

[[e-mail] android-sdk18-precise-amd64]$ ls -l
total 500076
-rw-r--r-- 1 armenzg armenzg      1163 Nov  7 08:46 android-sdk18_0.r18-0moz1.debian.tar.gz
-rw-r--r-- 1 armenzg armenzg       879 Nov  7 08:46 android-sdk18_0.r18-0moz1.dsc
-rw-r--r-- 1 armenzg armenzg      1566 Nov  7 08:47 android-sdk18_0.r18-0moz1_amd64.changes
-rw-r--r-- 1 armenzg armenzg 173646176 Nov  7 08:47 android-sdk18_0.r18-0moz1_amd64.deb
-rw-r--r-- 1 armenzg armenzg 338407646 Nov  7 08:43 android-sdk18_0.r18.orig.tar.bz2
[[e-mail] android-sdk18-precise-amd64]$ ls -l /data/repos/apt/releng/pool/main/a/android-sdk18/
total 669712
-rw-rw-r-- 1 500 500      1196 Aug 27 07:54 android-sdk18_0.r18-0moz1.debian.tar.gz
-rw-rw-r-- 1 500 500       879 Aug 27 07:54 android-sdk18_0.r18-0moz1.dsc
-rw-rw-r-- 1 500 500 173679916 Aug 27 07:54 android-sdk18_0.r18-0moz1_amd64.deb
-rw-rw-r-- 1 500 500 173679952 Aug 27 07:55 android-sdk18_0.r18-0moz1_i386.deb
-rw-rw-r-- 1 500 500 338407646 Aug 27 07:54 android-sdk18_0.r18.orig.tar.bz2

The debian.tar.gz are slightly different in size, however, when I untar the files all of them have the file size.

Comment 10

[Armen [:armenzg]]

Attachment: differences

This comes out of comparing what gbrown provides and the tools inside of the orig tar ball we have in the puppet repo.

diff -pU 8 -r rail/orig-source/amd64/sdk/tools gbrown > differences

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 828732 [details] [diff] [review]
differences

That can't be right - that's just a (very long) change to NOTICE.txt

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 828732 [details] [diff] [review]
differences

oh, sorry, bugzilla failed to show me the binary diffs.  So the important bit is

> Binary files rail/orig-source/amd64/sdk/tools/emulator-arm and gbrown/emulator-arm differ
> Binary files rail/orig-source/amd64/sdk/tools/emulator-mips and gbrown/emulator-mips differ
> Binary files rail/orig-source/amd64/sdk/tools/emulator-x86 and gbrown/emulator-x86 differ
> Binary files rail/orig-source/amd64/sdk/tools/emulator64-arm and gbrown/emulator64-arm differ
> Binary files rail/orig-source/amd64/sdk/tools/emulator64-mips and gbrown/emulator64-mips differ
> Binary files rail/orig-source/amd64/sdk/tools/emulator64-x86 and gbrown/emulator64-x86 differ
> Binary files rail/orig-source/amd64/sdk/tools/lib/emma_device.jar and gbrown/lib/emma_device.jar differ
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86/plugins: com.android.ide.eclipse.base_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86/plugins: com.android.ide.eclipse.base_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86/plugins: com.android.ide.eclipse.ddms_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86/plugins: com.android.ide.eclipse.ddms_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86/plugins: com.android.ide.eclipse.gldebugger_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86/plugins: com.android.ide.eclipse.gldebugger_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86/plugins: com.android.ide.eclipse.hierarchyviewer_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86/plugins: com.android.ide.eclipse.hierarchyviewer_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86/plugins: com.android.ide.eclipse.monitor_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86/plugins: com.android.ide.eclipse.monitor_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86/plugins: com.android.ide.eclipse.traceview_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86/plugins: com.android.ide.eclipse.traceview_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86_64/plugins: com.android.ide.eclipse.base_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86_64/plugins: com.android.ide.eclipse.base_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86_64/plugins: com.android.ide.eclipse.ddms_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86_64/plugins: com.android.ide.eclipse.ddms_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86_64/plugins: com.android.ide.eclipse.gldebugger_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86_64/plugins: com.android.ide.eclipse.gldebugger_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86_64/plugins: com.android.ide.eclipse.hierarchyviewer_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86_64/plugins: com.android.ide.eclipse.hierarchyviewer_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86_64/plugins: com.android.ide.eclipse.monitor_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86_64/plugins: com.android.ide.eclipse.monitor_22.0.5.201310311246.jar
> Only in rail/orig-source/amd64/sdk/tools/lib/monitor-x86_64/plugins: com.android.ide.eclipse.traceview_22.0.5.201307292138.jar
> Only in gbrown/lib/monitor-x86_64/plugins: com.android.ide.eclipse.traceview_22.0.5.201310311246.jar
> Binary files rail/orig-source/amd64/sdk/tools/support/annotations.jar and gbrown/support/annotations.jar differ

Comment 13

[Armen [:armenzg]]

This is a hacky way to generate the source file:
cp -r gbrown/* new_source/orig-source/amd64/sdk/tools/
tar zcvf android-sdk18_0.r18-0moz2.debian.tar.gz orig-source

rail, what do you think? what should the right way of generating this be?

gbrown, how did you generate the tools bundle? Not sure if I need to know.

Comment 14

[Armen [:armenzg]]

Comment on attachment 828714 [details] [diff] [review]
deb generation for android sdk 18 - version moz1

https://hg.mozilla.org/build/puppet/rev/9e4dde12cf9e

Comment 15

[Armen [:armenzg]]

gbrown, where the tools to replace amd64/sdk/tools or i386/sdk/tools?

Comment 16

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #13)
> gbrown, how did you generate the tools bundle? Not sure if I need to know.

I think all of my steps are shown in Comment 1.

Comment 17

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #15)
> gbrown, where the tools to replace amd64/sdk/tools or i386/sdk/tools?

I am not sure I understand. I thought there was only one sdk. These binaries (like those in the original sdk) are both 32- and 64-bit:

[[e-mail] myemu-tools_r22-patched]$ file emulator-x86
emulator-x86: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped
[[e-mail] myemu-tools_r22-patched]$ file emulator64-x86
emulator64-x86: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped

Comment 18

[Armen [:armenzg]]

I'm trying two things:
1) try to generate a new "orig" file by modifying "rules"
2) use puppetagain-build-deb to generate a .deb file

rail, what am I doing wrong to create a "orig" file by using the "rules" instructions?

On the other hand, I think I have managed to create a second version of the deb packages. See output at the end.
I will have to verify that it matches what we expect to be different.

########### 1 #############
root@ubuntu64packager1:/tmp/rail# sudo cowbuilder --login --basepath /var/cache/pbuilder/precise-amd64/ --bindmount /tmp
root@ubuntu64packager1:/tmp/rail# apt-get install mercurial build-essential devscripts wget debhelper libparse-debianchangelog-perl
root@ubuntu64packager1:/tmp/rail# mkdir /tmp/rail
root@ubuntu64packager1:/tmp/rail# cd /tmp/rail
root@ubuntu64packager1:/tmp/rail# wget http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/pool/main/a/android-sdk18/android-sdk18_0.r18-0moz1.debian.tar.gz
root@ubuntu64packager1:/tmp/rail# wget http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/pool/main/a/android-sdk18/android-sdk18_0.r18.orig.tar.bz2
root@ubuntu64packager1:/tmp/rail# tar zxvf android-sdk18_0.r18-0moz1.debian.tar.gz
root@ubuntu64packager1:/tmp/rail# debuild
 dpkg-buildpackage -rfakeroot -D -us -uc
dpkg-buildpackage: warning: using a gain-root-command while being root
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor): -Wl,-Bsymbolic-functions -Wl,-z,relro
dpkg-buildpackage: source package android-sdk18
dpkg-buildpackage: source version 0.r18-0moz1
dpkg-buildpackage: source changed by Rail Aliiev <[e-mail]>
 dpkg-source --before-build rail
dpkg-buildpackage: host architecture amd64
 fakeroot debian/rules clean
dh clean
   dh_testdir
   dh_auto_clean
   dh_clean
 dpkg-source -b rail
dpkg-source: info: using source format `3.0 (quilt)'
dpkg-source: info: building android-sdk18 using existing ./android-sdk18_0.r18.orig.tar.bz2
bunzip2: (stdin) is not a bzip2 file.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
dpkg-source: error: tar --no-same-owner --no-same-permissions -xf - gave error exit status 2
dpkg-buildpackage: error: dpkg-source -b rail gave error exit status 2
debuild: fatal error at line 1350:
dpkg-buildpackage -rfakeroot -D -us -uc failed

########### 2 #############
cp ~/create_source/new_source/android-sdk18_0.r18.orig.tar.bz2 ~/puppet/modules/packages/manifests
cd puppet
patch -p1 < ~/puppet_android_sdk.diff
chmod +x modules/pkgbuilder/files/puppetagain-build-deb
modules/pkgbuilder/files/puppetagain-build-deb precise amd64 modules/packages/manifests/android-sdk18-debian/
ls -l /tmp/android-sdk18-precise-amd64
total 169596
-rw-r--r-- 1 armenzg armenzg      1275 Nov  7 14:16 android-sdk18_0.r18-0moz2.debian.tar.gz
-rw-r--r-- 1 armenzg armenzg       879 Nov  7 14:16 android-sdk18_0.r18-0moz2.dsc
-rw-r--r-- 1 armenzg armenzg      1366 Nov  7 14:17 android-sdk18_0.r18-0moz2_amd64.changes
-rw-r--r-- 1 armenzg armenzg 173646326 Nov  7 14:17 android-sdk18_0.r18-0moz2_amd64.deb
ls -l /data/repos/apt/releng/pool/main/a/android-sdk18/
total 669712
-rw-rw-r-- 1 500 500      1196 Aug 27 07:54 android-sdk18_0.r18-0moz1.debian.tar.gz
-rw-rw-r-- 1 500 500       879 Aug 27 07:54 android-sdk18_0.r18-0moz1.dsc
-rw-rw-r-- 1 500 500 173679916 Aug 27 07:54 android-sdk18_0.r18-0moz1_amd64.deb
-rw-rw-r-- 1 500 500 173679952 Aug 27 07:55 android-sdk18_0.r18-0moz1_i386.deb
-rw-rw-r-- 1 500 500 338407646 Aug 27 07:54 android-sdk18_0.r18.orig.tar.bz2

Comment 19

[Armen [:armenzg]]

Attachment: change how we generate the source file for the deb generation of the patched version of the android sdk

How can I test that this works?

Comment 20

[Armen [:armenzg]]

Attachment: tools differences

This shows the differences in the tools directory after deploying gbrown's tools tar ball.

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 829347 [details] [diff] [review]
change how we generate the source file for the deb generation of the patched version of the android sdk

Review of attachment 829347 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/packages/manifests/android-sdk18-debian/changelog
@@ +1,4 @@
> +android-sdk18 (0.r18-0moz2) precise; urgency=low
> +
> +  * 0.r18 (stable) release
> +  * Fixes https://bugzilla.mozilla.org/show_bug.cgi?id=933918 

a nit, remove the trailing space here

::: modules/packages/manifests/android-sdk18-debian/rules
@@ +40,5 @@
> +tools_download:
> +	@echo "# Downloading..."
> +	wget $(TOOLS_URL)
> +
> +get-orig-source: adt-bundle-linux-x86-$(BUNDLE_VER).zip adt-bundle-linux-x86_64-$(BUNDLE_VER).ziptools_download

s/ziptools_download/zip tools_download/ here, but that would run the "tools_download" target every time, since it's not .PHONY

Can you try this patch on top of yours?
https://gist.github.com/rail/7373793

Comment 22

[Armen [:armenzg]]

Attachment: deb generation for android sdk 18 - version moz2

After dealing with the chicken and the egg, I made a chicken omelet. 

* This is how I created the source file with the latest _rules_ changes
sudo cowbuilder --login --basepath /var/cache/pbuilder/precise-amd64/ --bindmount /tmp
apt-get install mercurial build-essential devscripts wget debhelper libparse-debianchangelog-perl unzip
cd /tmp
hg clone http://hg.mozilla.org/build/puppet
cd puppet
patch -p1 < ../android_sdk_deb.diff
mkdir create_source_file && cd create_source_file
cp -r ../modules/packages/manifests/android-sdk18-debian debian
make -f debian/rules get-orig-source

* This is how I created the deb packages
cd ~/puppet
patch -p1 < ~/android_sdk_deb.diff
chmod +x modules/pkgbuilder/files/puppetagain-build-deb
cp /tmp/puppet/create_source_file/android-sdk18_0.r18.orig.tar.bz2 ~/puppet/modules/packages/manifests
modules/pkgbuilder/files/puppetagain-build-deb precise amd64 modules/packages/manifests/android-sdk18-debian
ls -lrt /tmp/android-sdk18-precise-amd64/
total 188788
-rw-r--r-- 1 1048 1048      1339 Nov  8 18:12 android-sdk18_0.r18-0moz2.debian.tar.gz
-rw-r--r-- 1 1048 1048       879 Nov  8 18:12 android-sdk18_0.r18-0moz2.dsc
-rw-r--r-- 1 1048 1048 193302436 Nov  8 18:13 android-sdk18_0.r18-0moz2_amd64.deb
-rw-r--r-- 1 1048 1048      1366 Nov  8 18:13 android-sdk18_0.r18-0moz2_amd64.changes
[[e-mail] puppet]$ ls -l /data/repos/apt/releng/pool/main/a/android-sdk18/
total 669712
-rw-rw-r-- 1 500 500      1196 Aug 27 07:54 android-sdk18_0.r18-0moz1.debian.tar.gz
-rw-rw-r-- 1 500 500       879 Aug 27 07:54 android-sdk18_0.r18-0moz1.dsc
-rw-rw-r-- 1 500 500 173679916 Aug 27 07:54 android-sdk18_0.r18-0moz1_amd64.deb
-rw-rw-r-- 1 500 500 173679952 Aug 27 07:55 android-sdk18_0.r18-0moz1_i386.deb
-rw-rw-r-- 1 500 500 338407646 Aug 27 07:54 android-sdk18_0.r18.orig.tar.bz2

Comment 23

[Armen [:armenzg]]

* To verify that I got things right I've done this:
ar vx /tmp/android-sdk18-precise-amd64/android-sdk18_0.r18-0moz2_amd64.deb
tar zxvf data.tar.gz
diff -pU 8 gbrown/ unpacked/tools/android-sdk18/tools              
Common subdirectories: gbrown/ant and unpacked/tools/android-sdk18/tools/ant
Common subdirectories: gbrown/apps and unpacked/tools/android-sdk18/tools/apps
Common subdirectories: gbrown/lib and unpacked/tools/android-sdk18/tools/lib
Common subdirectories: gbrown/proguard and unpacked/tools/android-sdk18/tools/proguard
Common subdirectories: gbrown/support and unpacked/tools/android-sdk18/tools/support
Common subdirectories: gbrown/systrace and unpacked/tools/android-sdk18/tools/systrace
Common subdirectories: gbrown/templates and unpacked/tools/android-sdk18/tools/templates

Comment 24

[Armen [:armenzg]]

Attachment: deb generation for android sdk 18 - version moz2

I've repeated the steps from comment 22 but we added a change that would allow us to generate a different "orig" file for this second version of the sdk.

-android-sdk18 (0.r18-0moz2) precise; urgency=low
+android-sdk18 (0.r18moz1-0moz1) precise; urgency=low

ls -l /tmp/android-sdk18-precise-amd64
total 373364
-rw-r--r-- 1 armenzg armenzg      1341 Nov  8 11:39 android-sdk18_0.r18moz1-0moz1.debian.tar.gz
-rw-r--r-- 1 armenzg armenzg       907 Nov  8 11:39 android-sdk18_0.r18moz1-0moz1.dsc
-rw-r--r-- 1 armenzg armenzg      1708 Nov  8 11:41 android-sdk18_0.r18moz1-0moz1_amd64.changes
-rw-r--r-- 1 armenzg armenzg 193302528 Nov  8 11:40 android-sdk18_0.r18moz1-0moz1_amd64.deb
-rw-r--r-- 1 armenzg armenzg 189000931 Nov  8 11:37 android-sdk18_0.r18moz1.orig.tar.bz2

The reason we're doing this is because when we try to deploy the newer deb artifacts, the naming of the orig file conflicts:
[[e-mail] apt]# reprepro -V --basedir . remove precise android-sdk18
removing 'android-sdk18' from 'precise|main|amd64'...
removing 'android-sdk18' from 'precise|main|i386'...
removing 'android-sdk18' from 'precise|main|source'...
Exporting indices...
 looking for changes in 'precise|main|amd64'...
  replacing './releng/dists/precise/main/binary-amd64/Packages' (uncompressed,bzip2ed)
 looking for changes in 'precise|main|i386'...
  replacing './releng/dists/precise/main/binary-i386/Packages' (uncompressed,bzip2ed)
 looking for changes in 'precise|main|source'...
  replacing './releng/dists/precise/main/source/Sources' (gzipped)
5 files lost their last reference.
(dumpunreferenced lists such files, use deleteunreferenced to delete them.)
[[e-mail] apt]# reprepro -V --basedir . removesrc precise android-sdk18
[[e-mail] apt]# reprepro -V --basedir . include precise /tmp/android-sdk18-precise-amd64/*.changes
File "pool/main/a/android-sdk18/android-sdk18_0.r18.orig.tar.bz2" is already registered with different checksums!
md5 expected: 6e3d770abbb75ee30d349d5a45993d0d, got: 97b636a76e3bc64e06a338f2d8ff64c1
sha1 expected: a350fdc902dbc48478f15be74cb1581214fd1683, got: a935d7137d36d1d872babaa7849ec9569d6406ea
sha256 expected: 8aeaf6c0967063fbab66a217b12b5e1f3b0571c0215fefbcd4fd0f3ce7acccc4, got: 93573fcadfde81a50b72edbecc1d2e014455a424a8434cfec86d2deeff433bc0
size expected: 338407646, got: 189000933
There have been errors!

Here's what I've run differently (notice a differently named _orig_ file was generated):
cp /tmp/puppet/create_source_file/android-sdk18_0.r18moz1.orig.tar.bz2 ~/puppet/modules/packages/manifests

Comment 25

[Armen [:armenzg]]

rail, can I proceed with these last three steps? :)
 rsync -av /data/repos/apt/releng/ releng-puppet2.srv.releng.use1.mozilla.com:/data/repos/apt/releng/
 rsync -av /data/repos/apt/db/ releng-puppet2.srv.releng.use1.mozilla.com:/data/repos/apt/db/
 puppetmaster-fixperms

The version seems that is pinned (ensure => '0.r18-0moz1';):
http://hg.mozilla.org/build/puppet/file/d19a008383a0/modules/packages/manifests/mozilla/android_sdk18.pp#l11

[[e-mail] apt]# reprepro -V --basedir . include precise /tmp/android-sdk18-precise-amd64/*.changes
Exporting indices...
 looking for changes in 'precise|main|amd64'...
  replacing './releng/dists/precise/main/binary-amd64/Packages' (uncompressed,bzip2ed)
 looking for changes in 'precise|main|i386'...
 looking for changes in 'precise|main|source'...
  replacing './releng/dists/precise/main/source/Sources' (gzipped)
4 files lost their last reference.
(dumpunreferenced lists such files, use deleteunreferenced to delete them.)
[[e-mail] apt]# ls -l /data/repos/apt/releng/pool/main/a/android-sdk18/
total 1043080
-rw-rw-r-- 1  500  500 173679916 Aug 27 07:54 android-sdk18_0.r18-0moz1_amd64.deb
-rw-rw-r-- 1  500  500      1196 Aug 27 07:54 android-sdk18_0.r18-0moz1.debian.tar.gz
-rw-rw-r-- 1  500  500       879 Aug 27 07:54 android-sdk18_0.r18-0moz1.dsc
-rw-rw-r-- 1  500  500 173679952 Aug 27 07:55 android-sdk18_0.r18-0moz1_i386.deb
-rw-r--r-- 1 root root 193302528 Nov  8 12:12 android-sdk18_0.r18moz1-0moz1_amd64.deb
-rw-r--r-- 1 root root      1341 Nov  8 12:12 android-sdk18_0.r18moz1-0moz1.debian.tar.gz
-rw-r--r-- 1 root root       907 Nov  8 12:12 android-sdk18_0.r18moz1-0moz1.dsc
-rw-r--r-- 1 root root 189000931 Nov  8 12:12 android-sdk18_0.r18moz1.orig.tar.bz2
-rw-rw-r-- 1  500  500 338407646 Aug 27 07:54 android-sdk18_0.r18.orig.tar.bz2

Comment 26

[Armen [:armenzg]]

Comment on attachment 829464 [details] [diff] [review]
deb generation for android sdk 18 - version moz2

https://hg.mozilla.org/build/puppet/rev/95a4fe91e31b

Comment 27

[Armen [:armenzg]]

Attachment: deploy the patched Android SDK + *only* deploy kvm and the sdk to linux64-bit in-house machines

Comment 28

[Armen [:armenzg]]

I don't understand why it can't find it even though I see it in here:
http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/pool/main/a/android-sdk18/

Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install android-sdk18=0.r18moz1-0moz1' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '0.r18moz1-0moz1' for 'android-sdk18' was not found

Error: /Stage[main]/Packages::Mozilla::Android_sdk18/Package[android-sdk18]/ensure: change from 0.r18-0moz1 to 0.r18moz1-0moz1 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install android-sdk18=0.r18moz1-0moz1' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Version '0.r18moz1-0moz1' for 'android-sdk18' was not found

Notice: Finished catalog run in 33.02 seconds

Comment 29

[Dustin J. Mitchell [:dustin] (he/him)]

You'll need to run apt-get update, and then in the final puppet patch, bump the counter in modules/packages/manifests/setup.pp to cause all ubuntu machines to run apt-get update.  That step should probably be added to the repo.  You'll see I forgot that in bug 931875 and had to land a bustage patch.

Comment 30

[Armen [:armenzg]]

dustin, I didn't change anything, however, this morning I managed to successfully sync the machine.
Am I goot to go without the setup.pp version bump?

[[e-mail] ~]# puppet agent --test --environment=armenzg --server=releng-puppet2.srv.releng.scl3.mozilla.com
...
Notice: /Stage[main]/Packages::Mozilla::Android_sdk18/Package[android-sdk18]/ensure: ensure changed '0.r18-0moz1' to '0.r18moz1-0moz1'
...
[[e-mail] ~]# sha1sum /tools/android-sdk18/tools/emulator-x86
35c4f51d03ef3858593dc1d7dc23bf6f922c026b  /tools/android-sdk18/tools/emulator-x86

[[e-mail] ~]$ sha1sum /tools/android-sdk18/tools/emulator-x86 
3f24aa1f41aedc9741cd3f3a4c04b5be542661d4  /tools/android-sdk18/tools/emulator-x86

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

They all sync every 24h anyway, so practically, you're good now -- but doing the bump anyway won't hurt and it's good practice.

Comment 32

[[e-mail] (:bhearsum)]

I merged this to puppet production.

Comment 33

[Armen [:armenzg]]

I will land the patch + bumping the counter tomorrow morning.

Comment 34

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 829505 [details] [diff] [review]
deploy the patched Android SDK + *only* deploy kvm and the sdk to linux64-bit in-house machines

This looks good, but please update the patch description to indicate both things it does.

Comment 35

[Armen [:armenzg]]

Comment on attachment 829505 [details] [diff] [review]
deploy the patched Android SDK + *only* deploy kvm and the sdk to linux64-bit in-house machines

Landed and merge to production:
http://hg.mozilla.org/build/puppet/rev/95d1e62b4a62

talos-linux64-ix should be getting updated as they reboot.

gbrown, feel free to re-trigger Cedar Android x86 test jobs at the end of the day.
If a slave takes a job and does not give you the expected results you can have a look at this and see when it was the last time it rebooted by looking at the time of its last job:
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slavetype.html?class=test&type=talos-linux64-ix

Comment 36

[Armen [:armenzg]]

Comment on attachment 829505 [details] [diff] [review]
deploy the patched Android SDK + *only* deploy kvm and the sdk to linux64-bit in-house machines

It is affecting Darwin machines.
I will have to rework the patch.

Comment 37

[Armen [:armenzg]]

Attachment: deploy the patched Android SDK + *only* deploy kvm and the sdk to linux64-bit in-house machines

This has been tested on a Darwin host, tst-linux{32,64} host and talos-linux{32,64} hosts.

Comment 38

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 830957 [details] [diff] [review]
deploy the patched Android SDK + *only* deploy kvm and the sdk to linux64-bit in-house machines

Review of attachment 830957 [details] [diff] [review]:
-----------------------------------------------------------------

Is it specifically the GPU that's relevant here, or are we selecting these hosts for other reasons?  I assume "yes", but if not then gpu.pp is the wrong place.

With the small suggested comment adjustment, this looks fine.

Note that the rollback failed somewhat because the older version of android_sdk18 is no longer available in the repo, since apt repos can't contain more than one version of a package (ugh, apt, u suck!).  So, please land in two patches - one to upgrade the SDK version (android_sdk18.pp and setup.pp), and one to change the deployment location (init.pp and gpu.pp).  Then if the latter patch fails, you don't need to (incompletely) roll back the former again.

::: modules/toplevel/manifests/slave/test/gpu.pp
@@ +10,5 @@
> +    case $::operatingsystem {
> +        Ubuntu: {
> +            case $::hardwaremodel {
> +                # We only run Android x86 test jobs on
> +                # the 64-bit gpu host machines

This isn't quite true anymore, though - we only run it on the 64-bit Ubuntu hosts that have a GPU.

Comment 39

[Armen [:armenzg]]

Attachment: eploy the patched Android SDK to linux64-bit in-house & ec2 machines

graydon has been making progress on running jobs on ec2.
Let's get the hosts ready for it.

Comment 40

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 831500 [details] [diff] [review]
eploy the patched Android SDK to linux64-bit in-house & ec2 machines

Review of attachment 831500 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/talos/manifests/init.pp
@@ +38,5 @@
>                  "x86_64": {
>                      include packages::cpu-checker
>                      include packages::qemu-kvm
>                      include packages::bridge-utils
> +                    include androidemulator

This reads as, "all talos systems with a 64-bit OS will install the android emulator" - including OS X and Windows.  I don't think that's correct.

Comment 41

[Armen [:armenzg]]

Comment on attachment 831500 [details] [diff] [review]
eploy the patched Android SDK to linux64-bit in-house & ec2 machines

That section falls under "Ubuntu":
http://hg.mozilla.org/build/puppet/file/default/modules/talos/manifests/init.pp#l16

Sorry, I should have used more lines of context!

Comment 42

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 831500 [details] [diff] [review]
eploy the patched Android SDK to linux64-bit in-house & ec2 machines

Well, don't I look silly!

Comment 43

[Armen [:armenzg]]

Comment on attachment 831500 [details] [diff] [review]
eploy the patched Android SDK to linux64-bit in-house & ec2 machines

Live:
http://hg.mozilla.org/build/puppet/rev/6530aed914a2

No issues on puppet-errors so far.

Comment 44

[Armen [:armenzg]]

This was a good bug for learning, that's for sure :)

No issues AFAICT.