1232468 - Android CI builds are busted

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

See https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=3f02f1340ad2

They fail to fetch files from tooltool:

ERROR - The following files failed: 'android-sdk-linux.tar.xz'
retry: Calling <function run_with_timeout at 0x7ff7f8b16c08> with args: (['/tools/tooltool.py', '--url', 'http://tooltool.pvt.build.mozilla.org/build', '--overwrite', '-m', 'mobile/android/config/tooltool-manifests/android/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #10

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

FTR, the manifest adjustment we've needed the last couple of cycles already landed at http://hg.mozilla.org/releases/mozilla-beta/rev/c34caf2ee6b4.

Comment 3

[Jordan Lund (:jlund)]

(In reply to Nick Thomas [:nthomas] from comment #2)
> FTR, the manifest adjustment we've needed the last couple of cycles already
> landed at http://hg.mozilla.org/releases/mozilla-beta/rev/c34caf2ee6b4.

hm, I thought so too. but then I saw https://treeherder.mozilla.org/logviewer.html#?job_id=683977&repo=mozilla-beta

It was one of the jobs that failed. I think the others passed because it found it on the slaves tooltool cache.

Comment 4

[Jordan Lund (:jlund)]

(In reply to Rail Aliiev [:rail] from comment #0)
> See
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=3f02f1340ad2
> 
> They fail to fetch files from tooltool:

I added android-sdk-linux.tar.xz to the old tooltool repo (relengwebadm.private.scl3:/mnt/netapp/relengweb/tooltool/pvt/build/sha512/) as buildbot is still using old tooltool url.

I've retriggered one of the jobs that failed in the above link (Android 2.3 API9 debug) and hopefully we don't hit the cache on those

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

The two 2.3 debug retriggers got past tooltool on the back of that. The 2.3 opt build (on 3f02f1340ad2) must've started after the copy, and later hit bug 1232466.

The Android 4.0 opt and 4.2 x86 builds hit several import errors:
/builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/com/adjust/sdk/AdjustFactory.java:5: error: package ch.boye.httpclientandroidlib.client does not exist
import ch.boye.httpclientandroidlib.client.HttpClient;
                                          ^
/builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/com/adjust/sdk/AdjustFactory.java:6: error: package ch.boye.httpclientandroidlib.impl.client does not exist
import ch.boye.httpclientandroidlib.impl.client.DefaultHttpClient;
....

eg https://treeherder.mozilla.org/logviewer.html#?job_id=683974&repo=mozilla-beta

The 4.0 debug build hit tooltool, but will probably hit the import error now that's resolved.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Update - Android 2.3 builds are busted with bug 1232466, while the 4.0 in the style of comment #5.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

nalaxendar, sorry to do this to you. We were building on aurora fine using mozharness, but beta is still using buildbot factories. Something about that is breaking the build with those import errors. Could you please give us some pointers on what might be wrong.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Current theory - mozharness (aurora and below) call 'mach build', which ends up in mach_commands and creates a bunch of symlinks, eg src/thirdparty/src/main/java --> mobile/android/thirdparty (see http://hg.mozilla.org/releases/mozilla-beta/file/3f02f1340ad2/mobile/android/mach_commands.py#l105). On beta the call is 'make -f client.mk build'.

Comment 9

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Thomas [:nthomas] from comment #7)
> nalaxendar, sorry to do this to you.

No trouble.

 We were building on aurora fine using
> mozharness, but beta is still using buildbot factories. Something about that
> is breaking the build with those import errors. Could you please give us
> some pointers on what might be wrong.

I'll dig in.

Comment 10

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Thomas [:nthomas] from comment #8)
> Current theory - mozharness (aurora and below) call 'mach build', which ends
> up in mach_commands and creates a bunch of symlinks, eg
> src/thirdparty/src/main/java --> mobile/android/thirdparty (see
> http://hg.mozilla.org/releases/mozilla-beta/file/3f02f1340ad2/mobile/android/
> mach_commands.py#l105). On beta the call is 'make -f client.mk build'.

This should only happen with an explicit |mach gradle-install| invocation, which is definitely not mainstream.

Comment 11

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Thomas [:nthomas] from comment #5)
> The two 2.3 debug retriggers got past tooltool on the back of that. The 2.3
> opt build (on 3f02f1340ad2) must've started after the copy, and later hit
> bug 1232466.

This seems like a legit issue that we'll need to patch on beta.  I commented on Bug 1232466.

> The Android 4.0 opt and 4.2 x86 builds hit several import errors:
> /builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/
> com/adjust/sdk/AdjustFactory.java:5: error: package
> ch.boye.httpclientandroidlib.client does not exist
> import ch.boye.httpclientandroidlib.client.HttpClient;
>                                           ^
> /builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/
> com/adjust/sdk/AdjustFactory.java:6: error: package
> ch.boye.httpclientandroidlib.impl.client does not exist
> import ch.boye.httpclientandroidlib.impl.client.DefaultHttpClient;
> ....
> 
> eg
> https://treeherder.mozilla.org/logviewer.html#?job_id=683974&repo=mozilla-
> beta
> 
> The 4.0 debug build hit tooltool, but will probably hit the import error now
> that's resolved.

This is a real issue which I expect to be fallout from Bug 1174244 in response to changes to the Android platform.  I'll dig into this now.

Comment 12

[Nick Alexander :nalexander [he/him]]

OK, I've pushed https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=10dcb76438d5 to try to address the 4.0 ch.boye issue.  I'm quite confident this will at least build, since I can reproduce and address the issue locally.  It's possible tests would fail but I doubt it.

I'll look at the 2.3 issue now.

Comment 13

[Nick Alexander :nalexander [he/him]]

https://hg.mozilla.org/integration/fx-team/rev/1801cee7c4eff941cc2e8e5ebc06d5c899274f04
Bug 1232468 - Add ch.boye dependency for Adjust SDK. r=me a=bustage

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/1801cee7c4ef