840170 - b2g (on "master") fails to build for pandaboard with fresh gonk snapshot

Status: RESOLVED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Rail Aliiev [:rail]]

Attachment: b2g-snapshot.sh

In bug 837019 I tried to update the gonk snapshot for mozilla-central. ATM panda fails to build with the following error:

07:04:52    ERROR -  make: *** No rule to make target `out/target/product/panda/obj/lib/libexif.so', needed by `out/target/product/panda/obj/SHARED_LIBRARIES/camera.omap4_intermediates/LINKED/camera.omap4.so'.  Stop.

The corresponding m-c try changeset is https://hg.mozilla.org/try/rev/c27a3210771b
and tbpl is https://tbpl.mozilla.org/?tree=Try&rev=c27a3210771b.

The snapshot was created with BRANCH=master exported. I manually checked the status of B2G/.repo/manifests directory:

$ git branch -va
* default                 4b8f040 Merge pull request #47 from ting-yuan/prelink
  remotes/m/master        -> origin/master
  remotes/origin/master   4b8f040 Merge pull request #47 from ting-yuan/prelink
  remotes/origin/nightly  20e8dc1 Merge pull request #38 from armenzg/nightly
  remotes/origin/v1-train f53e105 freezing revsion attributes for the v1-train branch
  remotes/origin/v1.0.0   272a722 freezing revsion attributes for the v1.0.0 branch

jhford tried to reproduce this, but it worked fine to him (I don't have the details though).

Could anyone help debugging the problem? Attached is the full script we use to generate snapshots. It was ran as "export BRANCH=master && ./b2g-snapshot.sh pandaboard"

Comment 1

[Rail Aliiev [:rail]]

jhford, could you describe the steps you followed to build the snapshot?

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Rail Aliiev [:rail] from comment #1)
> jhford, could you describe the steps you followed to build the snapshot?

I didn't build a snapshot, rather I pulled down sources that would go into the snapshot and those weren't able to reproduce your error.

Comment 3

[Rail Aliiev [:rail]]

Attachment: releng-excludes

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to John Ford [:jhford] from comment #2)
> (In reply to Rail Aliiev [:rail] from comment #1)
> > jhford, could you describe the steps you followed to build the snapshot?
> 
> I didn't build a snapshot, rather I pulled down sources that would go into
> the snapshot and those weren't able to reproduce your error.

jhford: 
1) Can you please try using the attached b2g-snapshot.sh to generate a snapshot on your machine, and see if that reproduces the build failure? Or if you prefer, Rail could send you a copy of the snapshot he generated. 
2) Also, it would be helpful if you can send us exact steps you ran, and output log, so we can verify if you and rail are actually attempting to build with the same bits...

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John O'Duinn [:joduinn] from comment #4)
> jhford: 
> 1) Can you please try using the attached b2g-snapshot.sh to generate a
> snapshot on your machine, and see if that reproduces the build failure? Or
> if you prefer, Rail could send you a copy of the snapshot he generated.

These snapshots are not a supported build configuration.  Rail, please use the supported build procedure to see whether or not this issue is limited in scope to the snapshot system or if it's a real code issue.

The simple build can be done with:

git clone git://github.com/mozilla-b2g/B2G.git &&
cd B2G &&
BRANCH=master ./config.sh pandaboard &&
./build.sh

> 2) Also, it would be helpful if you can send us exact steps you ran, and
> output log, so we can verify if you and rail are actually attempting to
> build with the same bits...

I don't have that information and I don't normally keep input and output logs for most things.

Comment 6

[Rail Aliiev [:rail]]

(In reply to John Ford [:jhford] from comment #5)
> These snapshots are not a supported build configuration.

Could you elaborate on this? Do you mean that the way how we build b2g in releng infra is not supported? I'm a little bit confused with the "these snapshots" phrase...

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per today's b2g meeting: 

jhford and rail will meet tomorrow (wed, as soon as the v1.0.1 branching is completed) to debug. Vidyo would be most efficient, as the back/forth here already shows this topic is too tricky to debug by irc/bugmail.

Comment 8

[Rail Aliiev [:rail]]

btw, the script perfectly worked for v1.0.1 manifests: https://tbpl.mozilla.org/?tree=Try&rev=39d33cb8ac6c

Comment 9

[Rail Aliiev [:rail]]

v1.0.0 worked fine as well https://tbpl.mozilla.org/?tree=Try&rev=1f070a0c5a23

Comment 10

[Rail Aliiev [:rail]]

The updated v1-train snapshot worked fine as well: https://bugzilla.mozilla.org/show_bug.cgi?id=840863

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

Rail and I just met.

Rail will try doing a build on a slave using the supported, standard build config to rule out the possibility of the snapshotting process causing the error.  Rail is also going to try without using the download-panda archive to test that it is using the correct downloaded file.

I am going to try figure out where libexif.so comes from and figure out which module installs it into the target build root.

Rail,  you can shorten your test to "./build.sh camera.omap4", which tells the build system to only build the camera.omap4 module, which is where the dependency failure you're seeing.

To prevent this issue from recurring, we should really have integration testing for Gonk changes on a regular basis

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Rail Aliiev [:rail] from comment #9)
> v1.0.0 worked fine as well
> https://tbpl.mozilla.org/?tree=Try&rev=1f070a0c5a23

(In reply to Rail Aliiev [:rail] from comment #8)
> btw, the script perfectly worked for v1.0.1 manifests:
> https://tbpl.mozilla.org/?tree=Try&rev=39d33cb8ac6c

(In reply to Rail Aliiev [:rail] from comment #10)
> The updated v1-train snapshot worked fine as well:
> https://bugzilla.mozilla.org/show_bug.cgi?id=840863

All other branches working with refreshed snapshots; only "master" breaks, so modifying summary to match.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Rail, have you been able to reproduce using the supported build process?

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: pandaboard manifest with jhead

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

I found that the dependency that libexif.so comes from was removed from the manifests two weeks ago.  I have submitted a patch which adds jhead to the manifest.

Rail, please try to reproduce using:

curl -L "https://bugzilla.mozilla.org/attachment.cgi?id=714154" > manifest.xml &&
./config.sh pandaboard manifest.xml &&
rm -rf out/ &&
./build.sh -j1 camera.omap4

If that works, please run ./build.sh to see if you have any other errors.

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John Ford [:jhford] from comment #15)
> I found that the dependency that libexif.so comes from was removed from the
> manifests two weeks ago.  I have submitted a patch which adds jhead to the
> manifest.

This patch landed, please use the master branch.

Comment 17

[Rail Aliiev [:rail]]

I have just pushed https://tbpl.mozilla.org/?tree=Try&rev=58d29af4e127

If it fails, we can loan you a builder so you can debug the issue. I think, it will be just faster than if someone not that familiar with b2g build system would try to figure out what's the problem.

Comment 18

[Rail Aliiev [:rail]]

Panda now fails with a different error: see https://tbpl.mozilla.org/php/getParsedLog.php?id=19785832&tree=Try

15:01:38 ERROR - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/panda/obj/EXECUTABLES/sutagent_intermediates/src/Logger.o: in function Logger::Logger():external/negatus/src/Logger.cpp:18: error: undefined reference to 'Config::instance()'
15:01:38 ERROR - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/panda/obj/EXECUTABLES/sutagent_intermediates/src/SUTAgent.o: in function main:external/negatus/src/SUTAgent.cpp:184: error: undefined reference to 'Config::instance()'
15:01:38 ERROR - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/panda/obj/EXECUTABLES/sutagent_intermediates/src/SUTAgent.o: in function main:external/negatus/src/SUTAgent.cpp:184: error: undefined reference to 'Config::setTestRoot(std::basic_string<char, std::char_traits<char>, std::allocator<char> >)'
15:01:38 ERROR - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/panda/obj/EXECUTABLES/sutagent_intermediates/src/CommandEventHandler.o: in function CommandEventHandler::testroot(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&):external/negatus/src/CommandEventHandler.cpp:851: error: undefined reference to 'Config::instance()'
15:01:38 ERROR - prebuilt/linux-x86/toolchain/arm-linux-androideabi-4.4.x/bin/../lib/gcc/arm-linux-androideabi/4.4.3/../../../../arm-linux-androideabi/bin/ld: out/target/product/panda/obj/EXECUTABLES/sutagent_intermediates/src/CommandEventHandler.o: in function CommandEventHandler::rebt(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&):external/negatus/src/CommandEventHandler.cpp:675: error: undefined reference to 'Config::instance()'
15:02:05 ERROR - Return code: 2
15:02:05 FATAL - failed to build
15:02:05 FATAL - Exiting 2

Comment 19

[Rail Aliiev [:rail]]

Any ideas?

Comment 20

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Rail Aliiev [:rail] from comment #19)
> Any ideas?

Looks like it's a negatus (sut agent) build failure.  The A-Team knows this code best.  Try doing rm -rf $B2G_ROOT/external/negatus right before ./build.sh (or use a manifest that doesn't have negatus) and see if it compiles.  If it does, then the problem is limited to negatus.

To isolate to building just negatus, you can run "./build.sh -j1 sutagent".

Comment 21

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per mtg w/jhford:

1)negatus issues look resolved in bug#843221. No manifest updates needed.

2) rail, please try new builds on m-c and see if we hit further problems.

Comment 22

[Rail Aliiev [:rail]]

This worked fine with changes from bug 843221