1204800 - Install tooltool packages for B2G Device Image builds

Status: RESOLVED DUPLICATE of bug 1282093

(Testing :: General, defect)

Description

[John Dai[:jdai]]

According to bug 1146713 comment 150, we should run install-packages.sh for b2g device image builds as we do for b2g desktop.

Comment 1

[John Dai[:jdai]]

Attachment: Add tooltool dependencies to phone builds.

Try link:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b63fcefdf2d1

Comment 2

[Wander Lairson Costa]

Comment on attachment 8661695 [details] [diff] [review]
Add tooltool dependencies to phone builds.

Review of attachment 8661695 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 3

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4717107189fe

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Backed out for device build bustage
https://hg.mozilla.org/integration/mozilla-inbound/rev/a4708ebe67c5

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://treeherder.mozilla.org/logviewer.html#?job_id=14263074&repo=mozilla-inbound

Comment 6

[John Dai[:jdai]]

I saw you have 3 commit in a run [1]. After I saw error message, I think the error is came from other patch, I will make sure my patch is works fine [2]. 

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4717107189fe
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1ffeab8b6c5

Comment 7

[John Dai[:jdai]]

All the phone build are pass. 

Try link:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1ffeab8b6c5

Comment 8

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e244ce56e745

Comment 9

[Carsten Book [:Tomcat]]

sorry had to back this out for build bustage like https://treeherder.mozilla.org/logviewer.html#?job_id=14418785&repo=mozilla-inbound

01:03 < nigelb|sheriffduty> those oranges are pretty consistent.
01:04 < nigelb|sheriffduty> test -n "$TOOLTOOL_MANIFEST"
01:04 < nigelb|sheriffduty> aha
01:04 < nigelb|sheriffduty> the $TOOLTOOL_MANIFEST is empty there.

Comment 10

[Pulsebot]

Backout:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4825787d4b8b

Comment 11

[John Dai[:jdai]]

(In reply to Carsten Book [:Tomcat] from comment #9)
> sorry had to back this out for build bustage like
> https://treeherder.mozilla.org/logviewer.html#?job_id=14418785&repo=mozilla-
> inbound
> 
> 01:03 < nigelb|sheriffduty> those oranges are pretty consistent.
> 01:04 < nigelb|sheriffduty> test -n "$TOOLTOOL_MANIFEST"
> 01:04 < nigelb|sheriffduty> aha
> 01:04 < nigelb|sheriffduty> the $TOOLTOOL_MANIFEST is empty there.

Hi :Tomcat,
Thank you catch this error. May I ask why only mozilla-inbound has Nexus 4 and Nexus 5-L phone build? Do we miss to add them in try build?

Comment 12

[John Dai[:jdai]]

Attachment: Add tooltool dependencies to phone builds. r=wcosta

Add tooltool dependencies in Nexus 4 and Nexus 5-L phone build.

Comment 13

[John Dai[:jdai]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ff7a5275a267

Comment 14

[John Dai[:jdai]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4f69d532995f

Comment 15

[John Dai[:jdai]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6959adf7a16a

Comment 16

[Carsten Book [:Tomcat]]

hey john, actually a good question. I don't know but maybe the guys in #b2g or #developers know how to run try runs on device builds

Comment 17

[John Dai[:jdai]]

Attachment: Add tooltool dependencies to phone builds. r=wcosta

Try link:
https://treeherder.allizom.org/#/jobs?repo=try&revision=b806d082c699

Comment 18

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5bf9fdd2d084

Comment 19

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I backed this out in https://hg.mozilla.org/integration/mozilla-inbound/rev/f2ae8ff2e038 for being the likely reason for the Flame-KK builds suddenly failing like:

https://treeherder.mozilla.org/logviewer.html#?job_id=14699383&repo=mozilla-inbound

Comment 20

[John Dai[:jdai]]

Attachment: Add tooltool dependencies to phone builds. r=wcosta

Remove backup-flame.tar.xz configuration to avoid extracting a file which does not exist.

Try link:
https://treeherder.allizom.org/#/jobs?repo=try&revision=429e45e73de8

Comment 21

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c37cbf6f6296

Comment 22

[Nigel Babu [:nigelb]]

Backed out for bustage in Nexus 4 KK builds in:

https://hg.mozilla.org/integration/mozilla-inbound/rev/14031459c9af

Bustage logs:

https://treeherder.mozilla.org/logviewer.html#?job_id=15026621&repo=mozilla-inbound

Comment 23

[John Dai[:jdai]]

(In reply to Nigel Babu [:nigelb] from comment #22)
> Backed out for bustage in Nexus 4 KK builds in:
> 
> https://hg.mozilla.org/integration/mozilla-inbound/rev/14031459c9af
> 
> Bustage logs:
> 
> https://treeherder.mozilla.org/logviewer.html#?job_id=15026621&repo=mozilla-
> inbound

I didn't notice that we add a Nexus 4 KK on TaskCluster(Bug 1206368) that's why it cause this backed out. I will fix in next patch.

Comment 24

[John Dai[:jdai]]

Attachment: Add tooltool dependencies to phone builds. r=wcosta

Add Nexus 4 KK configuration.

Try link:
https://treeherder.allizom.org/#/jobs?repo=try&revision=4bd9933b3254

Comment 25

[Carsten Book [:Tomcat]]

Hi, this failed to apply:

adding 1204800 to series file
renamed 1204800 -> bug_1204800_phone.patch
applying bug_1204800_phone.patch
patching file testing/taskcluster/scripts/phone-builder/pre-build.sh
Hunk #1 FAILED at 20
1 out of 1 hunks FAILED -- saving rejects to file testing/taskcluster/scripts/phone-builder/pre-build.sh.rej

can you take a look, thanks!

Comment 26

[John Dai[:jdai]]

Attachment: Add tooltool dependencies to phone builds. r=wcosta

rebase

Comment 27

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e58169d5f96a

Comment 28

[Carsten Book [:Tomcat]]

done :)

Comment 29

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Backed out alongside bug 1207548 for Flame-KK build failures:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b42ed97d98cc

https://treeherder.mozilla.org/logviewer.html#?job_id=15255656&repo=mozilla-inbound

Comment 30

[Carsten Book [:Tomcat]]

maybe we would need a full try before this gets checked in again

Comment 31

[John Dai[:jdai]]

I saw flame kitkat device image build (engineering) is a buildbot build and there has some conflict with TC build[1]. Do we really need this in mozilla-inbound? If we need to keep this, how to avoid the conflict?

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=e58169d5f96a&filter-searchStr=B2G%20Device%20Image%20opt%20Flame%20KitKat%20Device%20Image%20Flame%20KitKat%20Device%20Image%20Build%20%28Engineering%29%20Flame-KK%28Be%29

Comment 32

[John Dai[:jdai]]

From Comment #22 to Comment #27, I only modified Nexus 4 kk configuration and rebased. It should not have different situation which is Comment #22 Flame-KK engineering build success, but Comment #27 Flame-KK engineering build failed.

Comment 33

[John Dai[:jdai]]

How to use try testing Flame-KK engineering build on Treeherder[1]? Because I didn't find out any configuration in testcluster[2].

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=e58169d5f96a&filter-searchStr=B2G%20Device%20Image%20opt%20Flame%20KitKat%20Device%20Image%20Flame%20KitKat%20Device%20Image%20Build%20%28Engineering%29%20Flame-KK%28Be%29
[2] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/branches/base_jobs.yml?from=base_jobs.yml

Comment 34

[Wander Lairson Costa]

(In reply to John Dai[:johnz][:jdai] from comment #33)
> How to use try testing Flame-KK engineering build on Treeherder[1]? Because
> I didn't find out any configuration in testcluster[2].
> 
> [1]
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> inbound&revision=e58169d5f96a&filter-
> searchStr=B2G%20Device%20Image%20opt%20Flame%20KitKat%20Device%20Image%20Flam
> e%20KitKat%20Device%20Image%20Build%20%28Engineering%29%20Flame-KK%28Be%29
> [2]
> https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/
> branches/base_jobs.yml?from=base_jobs.yml

"try: -p flame-kk-eng -b o" should work

Comment 35

[John Dai[:jdai]]

(In reply to Wander Lairson Costa [:wcosta] from comment #34)
> (In reply to John Dai[:johnz][:jdai] from comment #33)
> > How to use try testing Flame-KK engineering build on Treeherder[1]? Because
> > I didn't find out any configuration in testcluster[2].
> > 
> > [1]
> > https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> > inbound&revision=e58169d5f96a&filter-
> > searchStr=B2G%20Device%20Image%20opt%20Flame%20KitKat%20Device%20Image%20Flam
> > e%20KitKat%20Device%20Image%20Build%20%28Engineering%29%20Flame-KK%28Be%29
> > [2]
> > https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/
> > branches/base_jobs.yml?from=base_jobs.yml
> 
> "try: -p flame-kk-eng -b o" should work

It shows [tc] b2g flame kk eng not flame kitkat device image build (engineering)[1].

Hi :jgriffin,
Do you know how to enable flame kitkat device image build (engineering)[2] on try? How can I know this build is passed? Thank you.

[1] https://treeherder.allizom.org/#/jobs?repo=try&revision=893cb67ce9f3
[2] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=e58169d5f96a&filter-searchStr=B2G%20Device%20Image%20opt%20Flame%20KitKat%20Device%20Image%20Flame%20KitKat%20Device%20Image%20Build%20%28Engineering%29%20Flame-KK%28Be%29

Comment 36

[Greg Arndt [:garndt]]

Passing in the flags that Wander explained before should work.  Here is a push I just did:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c3df8f9a9ab0

The Be symbol is for the flame-kk engineering build.

Comment 37

[Jonathan Griffin (:jgriffin)]

Looks like garndt answered, so clearing ni.

Comment 38

[Julien Wajsberg [:julienw]]

Hey John, do you have the necessary information to move forward ?

Comment 39

[John Dai[:jdai]]

(In reply to Julien Wajsberg [:julienw] from comment #38)
> Hey John, do you have the necessary information to move forward ?

I already have a build pass patch on try[1]. According to bug 1213798 comment 6, I know that even try build is passed, I can't guarantee it can pass on mozilla-inbound phone build. 

[1]https://treeherder.mozilla.org/#/jobs?repo=try&revision=a865e68b5bcd

Comment 40

[Julien Wajsberg [:julienw]]

Maybe land on b2g-inbound so that mozilla-inbound is still OK for gecko developers, even in case of failures ?

Comment 41

[John Dai[:jdai]]

(In reply to Julien Wajsberg [:julienw] from comment #40)
> Maybe land on b2g-inbound so that mozilla-inbound is still OK for gecko
> developers, even in case of failures ?

B2g-inbound also has flame kitkat device image build (engineering)[1].
[1]https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&filter-searchStr=Flame-KK

Comment 42

[Julien Wajsberg [:julienw]]

I mean, if it breaks and requires a backout, it doesn't impair other gecko developers...

Of course it's better to land on an inbound that has an image build so that we see breakage before it arrives in Central ;)

Or maybe I misunderstood you ?

So what's missing to move forward here ?

Comment 43

[John Dai[:jdai]]

(In reply to Wes Kocher (:KWierso) from comment #29)
> Backed out alongside bug 1207548 for Flame-KK build failures:
> https://hg.mozilla.org/integration/mozilla-inbound/rev/b42ed97d98cc
> 
> https://treeherder.mozilla.org/logviewer.html#?job_id=15255656&repo=mozilla-
> inbound

According to the fail log[1], I found that it is because I removed backup-flame.tar.xz in releng-flame-kk.tt file[2], and it cause builtbot flame kitkat(engineering) build failed[3]. If I add backup-flame.tar.xz back, TC will build fail[4]. Also, we already download the backup file in pre-build.sh[5], can we reuse this file in builtbot environment?


[1] http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-flame-kk_eng/1444137536/b2g_mozilla-inbound_flame-kk_eng_dep-bm91-build1-build1438.txt.gz
[2] https://hg.mozilla.org/integration/mozilla-inbound/diff/e58169d5f96a/b2g/config/flame-kk/releng-flame-kk.tt
[3] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=e58169d5f96a&filter-searchStr=B2G%20Device%20Image 
[4] https://treeherder.allizom.org/#/jobs?repo=try&revision=e684c7512e27
[5] https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/phone-builder/pre-build.sh?from=pre-build.sh#39

Comment 44

[Wander Lairson Costa]

(In reply to John Dai[:johnz][:jdai] from comment #43)
> (In reply to Wes Kocher (:KWierso) from comment #29)
> > Backed out alongside bug 1207548 for Flame-KK build failures:
> > https://hg.mozilla.org/integration/mozilla-inbound/rev/b42ed97d98cc
> > 
> > https://treeherder.mozilla.org/logviewer.html#?job_id=15255656&repo=mozilla-
> > inbound
> 
> According to the fail log[1], I found that it is because I removed
> backup-flame.tar.xz in releng-flame-kk.tt file[2], and it cause builtbot
> flame kitkat(engineering) build failed[3]. If I add backup-flame.tar.xz
> back, TC will build fail[4]. Also, we already download the backup file in
> pre-build.sh[5], can we reuse this file in builtbot environment?
> 
> 
> [1]
> http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-
> flame-kk_eng/1444137536/b2g_mozilla-inbound_flame-kk_eng_dep-bm91-build1-
> build1438.txt.gz
> [2]
> https://hg.mozilla.org/integration/mozilla-inbound/diff/e58169d5f96a/b2g/
> config/flame-kk/releng-flame-kk.tt
> [3]
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> inbound&revision=e58169d5f96a&filter-searchStr=B2G%20Device%20Image 
> [4] https://treeherder.allizom.org/#/jobs?repo=try&revision=e684c7512e27
> [5]
> https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/scripts/
> phone-builder/pre-build.sh?from=pre-build.sh#39

The question is why TC cannot download backup-flame.tar.xz. Aries is understandable, as it never ran on buildbot so probably there is no backup file on tooltool, but flame should work.

Comment 45

[Chris AtLee [:catlee]]

Yes, I think the TC build should be downloading backup-flame from tooltool as well. In particular, the tasks need the docker-worker:relengapi-proxy:tooltool.download.internal scope and the relengapiProxy feature. These are added by the decision task.

Comment 46

[John Dai[:jdai]]

Attachment: Add tooltool dependencies to phone builds. r=wcosta

In this patch, I add back backup-flame.tar.xz configuration, and provide docker-worker:relengapi-proxy:tooltool.download.internal scope for TC tasks.

Try result:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f59c289df2a0

Comment 47

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3cc789ec2bf0

Comment 48

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/3cc789ec2bf0

Comment 49

[Wander Lairson Costa]

I still see issues on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f3fdc799c749

Comment 50

[John Dai[:jdai]]

(In reply to Wander Lairson Costa [:wcosta] from comment #49)
> I still see issues on try:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=f3fdc799c749

Yes, you are right. Please help me back it out. Thanks.

Comment 51

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/35f73f6ea2a9

Comment 52

[John Dai[:jdai]]

No longer work on this.

Comment 53

[:gerard-majax]

I have been fixing something close to this in bug 1282093, can you tell me what would be missing to mark this bug as resolved or should I dupe it against bug 1282093 ?

At least in bug 1282093 I am able to pull NDK from tooltool, but I don't make use of the install-packages.sh script. Is this mandatory ? Now that tooltool fetch works with NDK (internal) should we also switch to tooltool from S3 to host the backup file of blobs for z3c device ?

Comment 54

[Wander Lairson Costa]

(In reply to Alexandre LISSY :gerard-majax from comment #53)
> I have been fixing something close to this in bug 1282093, can you tell me
> what would be missing to mark this bug as resolved or should I dupe it
> against bug 1282093 ?
> 

I am more inclined to dup this bug and solve it at bug 1282093.

> At least in bug 1282093 I am able to pull NDK from tooltool, but I don't
> make use of the install-packages.sh script. Is this mandatory ? Now that
> tooltool fetch works with NDK (internal) should we also switch to tooltool
> from S3 to host the backup file of blobs for z3c device ?

Moving S3 backups to tooltool would definitely be much appreciated (one less secret shipping in phone-builder).