1095679 - Mulet builds on !trunk are broken from today's reconfig

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-b2g34_v2_1-linux64-mulet/1415387045/mozilla-b2g34_v2_1-linux64-mulet-bm73-build1-build13.txt.gz

b2g/dev/config/tooltool-manifests/linux64/releng.manifest doesn't exist, skipping...


Which leads to...

checking for gcc... /builds/slave/m-b34_21-l64-mulet-00000000000/build/gcc/bin/gcc
checking whether the C compiler (/builds/slave/m-b34_21-l64-mulet-00000000000/build/gcc/bin/gcc ) works... no
configure: error: installation or configuration problem: C compiler cannot create executables.
------ config.log ------
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
configure:1184: checking host system type
configure:1205: checking target system type
configure:1223: checking build system type
configure:1299: checking for gawk
configure:1384: checking for python2.7
configure:1494: checking Python environment is Mozilla virtualenv
configure:1715: checking for perl5
configure:1715: checking for perl
configure:2191: checking for objcopy
configure:3194: checking for gcc
configure:3307: checking whether the C compiler (/builds/slave/m-b34_21-l64-mulet-00000000000/build/gcc/bin/gcc ) works
configure:3323: /builds/slave/m-b34_21-l64-mulet-00000000000/build/gcc/bin/gcc -o conftest conftest.c 1>&5
/builds/slave/m-b34_21-l64-mulet-00000000000/build/configure: line 3322: /builds/slave/m-b34_21-l64-mulet-00000000000/build/gcc/bin/gcc: No such file or directory
configure: failed program was:
#line 3318 "configure"
#include "confdefs.h"
main(){return(0);}
configure: error: installation or configuration problem: C compiler cannot create executables.
*** Fix above errors and then restart with\
"make -f client.mk build"
make[2]: *** [configure] Error 1
make[2]: Leaving directory `/builds/slave/m-b34_21-l64-mulet-00000000000/build'
make[1]: *** [/builds/slave/m-b34_21-l64-mulet-00000000000/build/obj-firefox/Makefile] Error 2
make[1]: Leaving directory `/builds/slave/m-b34_21-l64-mulet-00000000000/build'
make: *** [build] Error 2

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Also affecting Gum (presumably any non-trunk trees)

Comment 2

[Michael Shal [:mshal]]

So I think we want a patch like this to use the older tooltool manifest for previous versions (untested):

 # Enable win32/macosx64 mulet in gecko 36+
 for name, branch in items_before(BRANCHES, 'gecko_version', 36):
+    if 'linux64-mulet' in branch['platforms']:
+        branch['platforms']['linux64-mulet']['tooltool_manifest_src'] = 'browser/config/tooltool-manifests/linux64/releng.manifest'
     if 'win32-mulet' in branch['platforms']:
         del branch['platforms']['win32-mulet']
     if 'macosx64-mulet' in branch['platforms']:
         del branch['platforms']['macosx64-mulet']

Though given that this logic is supposed to be disabling win32-mulet and macosx-64mulet for gecko_version < 36, I find it odd that we get win32 and macosx builds on Gum. (I was expecting either no win32/macosx builds, or for it to have m-c's tooltool manifests). According to http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/project_branches.py#227 gum should be gecko_version==35, but I guess this is different for b2g: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/b2g_project_branches.py#61

Anyway, I'm not sure what the correct fix here is. I suspect we'll want something like this with the branch info sorted out, and possibly with additional tests for osx/win32 with their previous tooltool manifest locations.

Comment 3

[Jan Keromnes [:janx]]

After reconfig, Mulet is supposed to use the new tooltool manifests at b2g/dev/config/tooltool-manifests/{win32,linux64,macosx64}/releng.manifest which were landed in bug 1011562 as https://hg.mozilla.org/mozilla-central/rev/8a53b8b76456

Maybe these manifests are only in trunk? I'm not sure why it's failing to find gcc though, that seems unrelated.

The buildbot patch that needed reconfig was a switch to the new manifests: https://bug1011562.bugzilla.mozilla.org/attachment.cgi?id=8517399, but just after reconfig I was able to push bug 1011562's last patch to try and it built successfully with the new tooltool manifests.

Comment 4

[Jan Keromnes [:janx]]

Also, it shouldn't affect Gum unless Gum uses Mulet configurations.

Comment 5

[Jan Keromnes [:janx]]

Disregard my previous comments. The reconfig made buildbots look for new tooltool manifests for Mulet (files b2g/dev/config/tooltool-manifests/{win32,linux64,macosx64}/releng.manifest) that were landed in bug 1011562 as https://hg.mozilla.org/mozilla-central/rev/8a53b8b76456.

My guess is that these manifests are not present in non-trunk trees, causing Mulet builds to fail.

If that's the case, I see two solutions for non-trunk trees:
- Patches https://bug1011562.bugzilla.mozilla.org/attachment.cgi?id=8518112 and https://bug1011562.bugzilla.mozilla.org/attachment.cgi?id=8518934 should be landed there, adding the right manifests.
- Or make non-trunk trees use the old browser manifests for Mulet (files browser/config/tooltool-manifests/{win32,linux64,macosx64}/releng.manifest), as Michael suggests in comment 2. Not sure how to "back out" the reconfig on non-trunk trees brought by buildbot patch https://bug1011562.bugzilla.mozilla.org/attachment.cgi?id=8517399

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Special-casing build configs sucks and backporting looks trivial. Let's go that way :)

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Only attachment 8518112 [details] [diff] [review] has landed on trunk so far, so I'm only uplifting that for now.

Comment 8

[Jan Keromnes [:janx]]

Sure! Then again I'm not even sure you want the commented code in mozconfig files (might cause conflicts), you just need the manifests. I shouldn't have done both changes as one patch.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Victory!

Comment 10

[Jan Keromnes [:janx]]

Yay! Ryan, thanks a lot for the heavy (up)lifting :)