966962 - Hamachi nightlies on b2g18 are burning with "Error: Couldn't find /builds/slave/b2g_m-b18_ham_ntly-00000000000/build/out/host/linux-x86/bin/fs_config"

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

Has been burning since the 31-Jan nightly. Only affects Hamachi nightlies. Everything else is building fine.

https://tbpl.mozilla.org/php/getParsedLog.php?id=33995818&tree=Mozilla-B2g18

b2g_mozilla-b2g18_hamachi_nightly on 2014-02-02 19:54:14 PST for push c5dcca9aeff8
slave: bld-linux64-ix-036

22:10:01     INFO -  Using yaffs2 filesystem
22:10:01     INFO -  No recovery, but not Ext FS
22:10:01     INFO -  Mounting /system from "system"
22:10:01     INFO -  Mounting /data   from "userdata"
22:10:01     INFO -  Generating FOTA update package
22:10:01     INFO -  mkdir -p `dirname out/target/product/hamachi/fota/partial/update.zip` || true
22:10:07     INFO -  Error: Couldn't find /builds/slave/b2g_m-b18_ham_ntly-00000000000/build/out/host/linux-x86/bin/fs_config
22:10:07     INFO -  make: *** [out/target/product/hamachi/fota/partial/update.zip] Error 1
22:10:07     INFO -  real	3m26.275s
22:10:07     INFO -  user	3m8.451s
22:10:07     INFO -  sys	0m29.800s
22:10:07     INFO -  
22:10:07     INFO -  > Build failed! <
22:10:07     INFO -  Build with |./build.sh -j1| for better messages
22:10:07     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
22:10:07    ERROR - Return code: 2
22:10:07    FATAL - failed to create complete update
22:10:07    FATAL - Running post_fatal callback...
22:10:07    FATAL - Exiting 2

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Probably a regression from bug 958202.

Comment 2

[Dave Hylands [:dhylands]]

Is this a case of the "toolchain" on the build servers needs to be updated?

fs_config is build-machine tool (from android) that we normally build when we do a build for a phone. But I seem to recall that a bunch of the android stuff has to be "installed" on the build machines or something? When I was first working on the AutoMounter a couple of years ago, I remember that a bunch of the header file that we used from android had to be installed as part of the toolchain on the build servers.

When we do a normal build, the built fs_config winds up here:
out/host/linux-x86/bin/fs_config

The source comes from here:
build/tools/fs_config

The Android.mk file from build/tools/fs_config has it marked as:
LOCAL_MODULE_TAGS := eng optional

However, we include it as a module to build in gonk-misc/b2g.mk

I've never quite understood why the builds we do on TBPL aren't the same as the builds regular developers do.

Comment 3

[Dave Hylands [:dhylands]]

Hmm.

This is b2g18 - maybe fs_config is just missing from the gonk-misc/b2g.mk ?

Comment 4

[Dave Hylands [:dhylands]]

Yeah. I took a look here:

https://github.com/mozilla-b2g/gonk-misc/blob/v1-train/b2g.mk

I think that the fix is to add fs_config to the list of PRODUCT_PACKAGES

It said b2g18 which I'm assuming means v1-train - <sarcasm>I just love it when all the branch names and stuff are consistent :( </sarcasm>

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=34061062&tree=Mozilla-B2g18

Comment 6

[Aki Sasaki (not active)]

:gsvelto, are we missing something in v1-train? Looks like comment 3 and comment 4 suggest that.

Comment 7

[Dave Hylands [:dhylands]]

Attachment: Add fs_config to the build so that user and user_debug variants can create FOTA images

I was able to reproduce the problem by building for hamachi and then trying:

./build.sh gecko-update-fota

After applying this patch, then the above command worked successfully.

However gonk-misc doesn't appear to have a v1-train branch, we'll either need to create one or verify that all of the rest of the changes don't break b2g18.

Comment 8

[Dave Hylands [:dhylands]]

https://github.com/mozilla-b2g/gonk-misc/commit/1e1c773c70afee9327486f8d63c1bc2e4eca299e

Comment 9

[Dave Hylands [:dhylands]]

About the v1-train thing. I was messed up and figured out what was wrong, so I got the commit merged into v1-train properly.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Confirmed that today's nightly ran OK :)