If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Android CI builds are busted

RESOLVED FIXED

Status

Release Engineering
General Automation
--
blocker
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rail, Assigned: jlund)

Tracking

(Depends on: 1 bug)

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox46 fixed)

Details

(Reporter)

Description

2 years ago
See https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=3f02f1340ad2

They fail to fetch files from tooltool:

ERROR - The following files failed: 'android-sdk-linux.tar.xz'
retry: Calling <function run_with_timeout at 0x7ff7f8b16c08> with args: (['/tools/tooltool.py', '--url', 'http://tooltool.pvt.build.mozilla.org/build', '--overwrite', '-m', 'mobile/android/config/tooltool-manifests/android/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #10
Duplicate of this bug: 1232478
(Assignee)

Updated

2 years ago
Assignee: nobody → jlund
FTR, the manifest adjustment we've needed the last couple of cycles already landed at http://hg.mozilla.org/releases/mozilla-beta/rev/c34caf2ee6b4.
(Assignee)

Comment 3

2 years ago

(In reply to Nick Thomas [:nthomas] from comment #2)
> FTR, the manifest adjustment we've needed the last couple of cycles already
> landed at http://hg.mozilla.org/releases/mozilla-beta/rev/c34caf2ee6b4.

hm, I thought so too. but then I saw https://treeherder.mozilla.org/logviewer.html#?job_id=683977&repo=mozilla-beta

It was one of the jobs that failed. I think the others passed because it found it on the slaves tooltool cache.
(Assignee)

Comment 4

2 years ago
(In reply to Rail Aliiev [:rail] from comment #0)
> See
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=3f02f1340ad2
> 
> They fail to fetch files from tooltool:

I added android-sdk-linux.tar.xz to the old tooltool repo (relengwebadm.private.scl3:/mnt/netapp/relengweb/tooltool/pvt/build/sha512/) as buildbot is still using old tooltool url.

I've retriggered one of the jobs that failed in the above link (Android 2.3 API9 debug) and hopefully we don't hit the cache on those
The two 2.3 debug retriggers got past tooltool on the back of that. The 2.3 opt build (on 3f02f1340ad2) must've started after the copy, and later hit bug 1232466.

The Android 4.0 opt and 4.2 x86 builds hit several import errors:
/builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/com/adjust/sdk/AdjustFactory.java:5: error: package ch.boye.httpclientandroidlib.client does not exist
import ch.boye.httpclientandroidlib.client.HttpClient;
                                          ^
/builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/com/adjust/sdk/AdjustFactory.java:6: error: package ch.boye.httpclientandroidlib.impl.client does not exist
import ch.boye.httpclientandroidlib.impl.client.DefaultHttpClient;
....

eg https://treeherder.mozilla.org/logviewer.html#?job_id=683974&repo=mozilla-beta

The 4.0 debug build hit tooltool, but will probably hit the import error now that's resolved.
Update - Android 2.3 builds are busted with bug 1232466, while the 4.0 in the style of comment #5.
Flags: needinfo?(nalexander)
nalaxendar, sorry to do this to you. We were building on aurora fine using mozharness, but beta is still using buildbot factories. Something about that is breaking the build with those import errors. Could you please give us some pointers on what might be wrong.
Current theory - mozharness (aurora and below) call 'mach build', which ends up in mach_commands and creates a bunch of symlinks, eg src/thirdparty/src/main/java --> mobile/android/thirdparty (see http://hg.mozilla.org/releases/mozilla-beta/file/3f02f1340ad2/mobile/android/mach_commands.py#l105). On beta the call is 'make -f client.mk build'.
(In reply to Nick Thomas [:nthomas] from comment #7)
> nalaxendar, sorry to do this to you.

No trouble.

 We were building on aurora fine using
> mozharness, but beta is still using buildbot factories. Something about that
> is breaking the build with those import errors. Could you please give us
> some pointers on what might be wrong.

I'll dig in.
Flags: needinfo?(nalexander)
(In reply to Nick Thomas [:nthomas] from comment #8)
> Current theory - mozharness (aurora and below) call 'mach build', which ends
> up in mach_commands and creates a bunch of symlinks, eg
> src/thirdparty/src/main/java --> mobile/android/thirdparty (see
> http://hg.mozilla.org/releases/mozilla-beta/file/3f02f1340ad2/mobile/android/
> mach_commands.py#l105). On beta the call is 'make -f client.mk build'.

This should only happen with an explicit |mach gradle-install| invocation, which is definitely not mainstream.
(In reply to Nick Thomas [:nthomas] from comment #5)
> The two 2.3 debug retriggers got past tooltool on the back of that. The 2.3
> opt build (on 3f02f1340ad2) must've started after the copy, and later hit
> bug 1232466.

This seems like a legit issue that we'll need to patch on beta.  I commented on Bug 1232466.

> The Android 4.0 opt and 4.2 x86 builds hit several import errors:
> /builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/
> com/adjust/sdk/AdjustFactory.java:5: error: package
> ch.boye.httpclientandroidlib.client does not exist
> import ch.boye.httpclientandroidlib.client.HttpClient;
>                                           ^
> /builds/slave/m-beta-and-x86-000000000000000/build/mobile/android/thirdparty/
> com/adjust/sdk/AdjustFactory.java:6: error: package
> ch.boye.httpclientandroidlib.impl.client does not exist
> import ch.boye.httpclientandroidlib.impl.client.DefaultHttpClient;
> ....
> 
> eg
> https://treeherder.mozilla.org/logviewer.html#?job_id=683974&repo=mozilla-
> beta
> 
> The 4.0 debug build hit tooltool, but will probably hit the import error now
> that's resolved.

This is a real issue which I expect to be fallout from Bug 1174244 in response to changes to the Android platform.  I'll dig into this now.
OK, I've pushed https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=10dcb76438d5 to try to address the 4.0 ch.boye issue.  I'm quite confident this will at least build, since I can reproduce and address the issue locally.  It's possible tests would fail but I doubt it.

I'll look at the 2.3 issue now.
Depends on: 1174244
Depends on: 1232798
https://hg.mozilla.org/integration/fx-team/rev/1801cee7c4eff941cc2e8e5ebc06d5c899274f04
Bug 1232468 - Add ch.boye dependency for Adjust SDK. r=me a=bustage

Comment 14

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/1801cee7c4ef
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox46: --- → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.