Closed Bug 1221502 Opened 5 years ago Closed 5 years ago

mozbuild.mozconfig.MozconfigLoadException: Evaluation of your mozconfig exited with an error.

Categories

(MailNews Core :: Build Config, defect)

defect
Not set
blocker

Tracking

(thunderbird44 unaffected, thunderbird45+ fixed)

RESOLVED FIXED
Thunderbird 45.0
Tracking Status
thunderbird44 --- unaffected
thunderbird45 + fixed

People

(Reporter: aleth, Unassigned)

References

Details

(Keywords: intermittent-failure, regression)

Attachments

(6 files, 7 obsolete files)

334.17 KB, text/plain
Details
95.51 KB, text/plain
Details
1.45 KB, patch
ewong
: review+
Details | Diff | Splinter Review
2.01 KB, patch
jlund
: review+
rail
: feedback+
Details | Diff | Splinter Review
1.07 KB, patch
jlund
: review+
Details | Diff | Splinter Review
2.04 KB, patch
Details | Diff | Splinter Review
New Linux bustage, probably fallout from bug 1186748.
https://hg.mozilla.org/comm-central/rev/ec0e994b8733274aca3df7e5faa9752511d3792b
Bug 1221502 - Port Bug 1186748 - Now that all builds are pulling the Gtk+3 tooltool package, remove the Gtk+2 fallback in mozconfig.gtk. r=mshal rs,a=aleth bustage-fix
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 45.0
Doesn't look like that was enough to fix it :-(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
"$TOOLTOOL_DIR/gtk3 comes from tooltool, and must be included in the tooltool manifest"

Don't know where that is, but maybe that's not set?
(In reply to Magnus Melin from comment #3)
> "$TOOLTOOL_DIR/gtk3 comes from tooltool, and must be included in the
> tooltool manifest"
> 
> Don't know where that is, but maybe that's not set?

mail/config/tooltool-manifests matches mozilla/browser/config/tooltool-manifests.
Severity: major → blocker
Duplicate of this bug: 1221813
Attached file bustage log (obsolete) —
Attached file good log
Attached file bustage log
Attachment #8683584 - Attachment is obsolete: true
I am not sure if this is the same bug or not, but
I refreshed my comm-central source tree about a couple of dozen minutes ago, and 
when I run configure, I encountered the following error.

--- begin quote ---
...

intl/icu/target> config.status: creating samples/Makefile
intl/icu/target> config.status: creating samples/date/Makefile
intl/icu/target> config.status: creating samples/cal/Makefile
intl/icu/target> config.status: creating samples/layout/Makefile

Reticulating splines...
Traceback (most recent call last):
  File "./config.status", line 1066, in <module>
    config_status(**args)
  File "/NREF-COMM-CENTRAL/comm-central/mozilla/python/mozbuild/mozbuild/config_status.py", line 178, in config_status
    the_backend.consume(definitions)
  File "/NREF-COMM-CENTRAL/comm-central/mozilla/python/mozbuild/mozbuild/backend/base.py", line 118, in consume
    for obj in objs:
  File "/NREF-COMM-CENTRAL/comm-central/mozilla/python/mozbuild/mozbuild/frontend/emitter.py", line 162, in emit
    for out in output:
  File "/NREF-COMM-CENTRAL/comm-central/mozilla/python/mozbuild/mozbuild/frontend/reader.py", line 1059, in read_mozbuild
    raise bre
mozbuild.frontend.reader.BuildReaderError: 
==============================
ERROR PROCESSING MOZBUILD FILE
==============================

The error occurred while processing the following file:

    /NREF-COMM-CENTRAL/comm-central/mail/app/moz.build

The error was triggered on line 8 of this file:

    GENERATED_INCLUDES += [

The underlying problem is an attempt to read a reserved UPPERCASE variable that does not exist.

The variable read causing the error is:

    GENERATED_INCLUDES

Maybe you meant GENERATED_FILES or GENERATED_WEBIDL_FILES?

Please use

    LOCAL_INCLUDES += [ '!foo' ]

instead of

    GENERATED_INCLUDES += [ 'foo' ]

*** Fix above errors and then restart with\
               "make -f client.mk build"
/NREF-COMM-CENTRAL/comm-central/client.mk:361: recipe for target 'configure' failed

--- end quote ---

I have no idea where the error in the log in this bugzilla happened exactly.
I just wanted to see if this bug of mine may be related or not.
By changing GENERATED_INCLUDES to LOCAL_INCLUDES with "!" (I have no idea what it is),
the configure and compilation seems to proceed much further (compilation is still running).

TIA
Actually, I think there is a script error somewhere else, too.
I am compiling comm-central TB under linux 64-bit.

I applied the following patch to my  source tree that was refreshed in less than hour ago.
This was to fix the issue in my problem reported in my previous message.
(I am not entirely sure if this problem of mine is the same issue reported here, but
since the origin must be in scripts somewhere, I am reporting it here FYI.)
That got me to configure and proceed to compile.
Patch at the end.

However, I noticed that during configure I see many warning lines like the following:
--- only an excerpt:
Creating /NREF-COMM-CENTRAL/objdir-tb3/dist/private/dbm
cd src; make private_export
There are no private exports.
There are no private exports.
cd softoken; make private_export
../../coreconf/Linux.mk:172: !!! OS_TARGET is Android, disabling -Werror
cd legacydb; make private_export
../../coreconf/Linux.mk:172: !!! OS_TARGET is Android, disabling -Werror
There are no private exports.
../../coreconf/Linux.mk:172: !!! OS_TARGET is Android, disabling -Werror
cd base; make private_export
There are no private exports.
cd dev; make private_export
../../coreconf/Linux.mk:172: !!! OS_TARGET is Android, disabling -Werror
cd mangle; make private_export
There are no private exports.
../../coreconf/Linux.mk:172: !!! OS_TARGET is Android, disabling -Werror
There are no private exports.
---

Huh?

Looking at Linux.mk line, I think somehow the configure thinks I am on Android which is not the case. I am using the latest plain Debian.
Something is wrong somewhere.
Linux.mk:
--- begin quote ---
...
ifndef NSS_ENABLE_WERROR
  ifneq ($(OS_TARGET),Android)
    # Android lollipop generates the following warning:
    # error: call to 'sprintf' declared with attribute warning:
    #   sprintf is often misused; please use snprintf [-Werror]
    # So, just suppress -Werror entirely on Android
    NSS_ENABLE_WERROR = 0
    $(warning !!! OS_TARGET is Android, disabling -Werror)
  else
    ifeq ($(COMPILER_TAG),_clang)
...
--- end quote ---

This could be due to the tree movement from local server to Amazon S3 server (?)
Anyway, the tree seems to be in a strange state.

TIA

The patch to paste over the initial script issue:
I have no idea what "!file" means. I simply followed the printed messages.

diff --git a/mail/app/moz.build b/mail/app/moz.build
--- a/mail/app/moz.build
+++ b/mail/app/moz.build
@@ -1,17 +1,17 @@
 # vim: set filetype=python:
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 DIRS += ['profile']
 
-GENERATED_INCLUDES += [
-    '/build',
+LOCAL_INCLUDES += [
+    '!/build',
 ]
 
 GeckoProgram(CONFIG['MOZ_APP_NAME'], msvcrt='static')
 USE_LIBS += ['mozglue']
 SOURCES += ['nsMailApp.cpp']
 LOCAL_INCLUDES += [
     '/mozilla/toolkit/xre',
     '/mozilla/xpcom/base',
(In reply to ISHIKAWA, Chiaki from comment #9)
> mozbuild.frontend.reader.BuildReaderError: 
> ==============================
> ERROR PROCESSING MOZBUILD FILE
> ==============================
> 
> The error occurred while processing the following file:
> 
>     /NREF-COMM-CENTRAL/comm-central/mail/app/moz.build
> 
> The error was triggered on line 8 of this file:
> 
>     GENERATED_INCLUDES += [
> 
> The underlying problem is an attempt to read a reserved UPPERCASE variable
> that does not exist.
> 
> The variable read causing the error is:
> 
>     GENERATED_INCLUDES
> 
> Maybe you meant GENERATED_FILES or GENERATED_WEBIDL_FILES?
> 
> Please use
> 
>     LOCAL_INCLUDES += [ '!foo' ]
> 
> instead of
> 
>     GENERATED_INCLUDES += [ 'foo' ]

This seems unrelated. See bug 1222479.
(In reply to aleth [:aleth] from comment #11)
> (In reply to ISHIKAWA, Chiaki from comment #9)
> > mozbuild.frontend.reader.BuildReaderError: 
> > ==============================
> > ERROR PROCESSING MOZBUILD FILE
> > ==============================
> > 
    ...

> >     GENERATED_INCLUDES += [ 'foo' ]
> 
> This seems unrelated. See bug 1222479.

Thank you. This solved the particular issue. (I bit the bullet and refreshed the source again.)

However, I still see many (about a dozen) false identification of my OS (Debian GNU/Linux as Android)

> ./../coreconf/Linux.mk:172: !!! OS_TARGET is Android, disabling -Werror

Should I file a bug?

TIA
(In reply to ISHIKAWA, Chiaki from comment #12)
> (In reply to aleth [:aleth] from comment #11)
> > (In reply to ISHIKAWA, Chiaki from comment #9)
> However, I still see many (about a dozen) false identification of my OS
> (Debian GNU/Linux as Android)
> 
> > ./../coreconf/Linux.mk:172: !!! OS_TARGET is Android, disabling -Werror
> 
> Should I file a bug?

Yes, of course. (Even better if you fix it! ;)) You should also check if you see similar lines in a recent Linux build log on treeherder.
> Should I file a bug?

No, it's fixed by bug 1211568, and will be fixed when NSS 3.21 is pulled in m-c. It's also a harmless warning.
This is the *simplest* way of getting a build done.
Attachment #8684489 - Flags: review?(jlund)
(In reply to Edmund Wong (:ewong) from comment #15)
> Created attachment 8684489 [details] [diff] [review]
> [buildbotcustom] proposed patch (v1)
> 
> This is the *simplest* way of getting a build done.

Credit is to :glandium.
Product: Thunderbird → MailNews Core
Comment on attachment 8684489 [details] [diff] [review]
[tb-buildbotcustom] proposed patch (v1)

This is for Thunderbird's releng.
Attachment #8684489 - Attachment description: [buildbotcustom] proposed patch (v1) → [tb-buildbotcustom] proposed patch (v1)
Attachment #8684494 - Flags: review?(bugspam.Callek)
I didn't know TB had a forked buildbotcustom? :-/
(In reply to Mike Hommey [:glandium] from comment #14)
> > Should I file a bug?
> 
> No, it's fixed by bug 1211568, and will be fixed when NSS 3.21 is pulled in
> m-c. It's also a harmless warning.

Oops. Sorry, I didn't notice this post and so I posted a bugzilla entry.
Bug 1222657 - Strange warning message: OS_TARGET is Android, disabling -Werror
(In reply to aleth [:aleth] from comment #19)
> I didn't know TB had a forked buildbotcustom? :-/

No, it's SeaMonkey that has a branched buildbotcustom.
Comment on attachment 8684489 [details] [diff] [review]
[tb-buildbotcustom] proposed patch (v1)

Review of attachment 8684489 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +1347,5 @@
>                                    WithProperties('MOZ_BUILD_DATE=%(buildid:-)s')]
>  
>          if self.profiledBuild:
>              command.append('MOZ_PGO=1')
> +        self.env.update({'TOOLTOOL_DIR': WithProperties('%(basedir)s/build')}) 

this mutate self.env for the whole MercurialBuildFactory instance life yes?

we should probably use copy() or deepcopy() if we want to only set this for a certain build step.

also, what effect will this have for all builds outside of c-c?
Flags: needinfo?(ewong)
(In reply to Jordan Lund (:jlund) from comment #22)
> Comment on attachment 8684489 [details] [diff] [review]
> [tb-buildbotcustom] proposed patch (v1)

> also, what effect will this have for all builds outside of c-c?

I am referring to android and desktop firefox. particularly on release channels.
(In reply to Jordan Lund (:jlund) from comment #23)
> (In reply to Jordan Lund (:jlund) from comment #22)
> > Comment on attachment 8684489 [details] [diff] [review]
> > [tb-buildbotcustom] proposed patch (v1)
> 
> > also, what effect will this have for all builds outside of c-c?
> 
> I am referring to android and desktop firefox. particularly on release
> channels.

From my cursory look through the code, while mobile/desktop firefox go
through this code, mobile isn't affected since it doesn't touch 
mozconfig.gtk (which is where TOOLTOOL_DIR is used) so having TOOLTOOL_DIR
in the env "shouldn't" be a problem.  As for desktop firefox, I've also
looked at the code and the linux* builds does care about the TOOLTOOL_DIR;
but since the TOOLTOOL_DIR env points to ./build,  it shouldn't be a 
problem also as that's where the gtk/, gcc/ paths are found.

But my concern  here is whether TOOLTOOL_DIR and $topsrcdir from
the following line from mozconfig.gtk is actually the same:

http://mxr.mozilla.org/comm-central/source/mozilla/build/unix/mozconfig.gtk#5

  TOOLTOOL_DIR=${TOOLTOOL_DIR:-$topsrcdir}

But, yeah, re: deepcopying the self.env... that's a better idea.
Flags: needinfo?(ewong)
Attachment #8684489 - Attachment is obsolete: true
Attachment #8684489 - Flags: review?(jlund)
Attachment #8685736 - Flags: review?(jlund)
Comment on attachment 8685736 [details] [diff] [review]
[tb-buildbotcustom] proposed patch (v2)

Review of attachment 8685736 [details] [diff] [review]:
-----------------------------------------------------------------

we should test this out on beta as soon as it goes to production, preferably before we do our next desktop ff beta

I think topsrcdir should point to the same path

::: process/factory.py
@@ +1347,5 @@
>                                    WithProperties('MOZ_BUILD_DATE=%(buildid:-)s')]
>  
>          if self.profiledBuild:
>              command.append('MOZ_PGO=1')
> +        tmpEnv = copy.deepcopy(self.env)

maybe rename this var to something more descriptive like compile_env

@@ +1348,5 @@
>  
>          if self.profiledBuild:
>              command.append('MOZ_PGO=1')
> +        tmpEnv = copy.deepcopy(self.env)
> +        tmpEnv.update({'TOOLTOOL_DIR': WithProperties('%(basedir)s/build')}) 

nit: whitespace
Attachment #8685736 - Flags: review?(jlund) → review+
(In reply to Jordan Lund (:jlund) from comment #26)
> Comment on attachment 8685736 [details] [diff] [review]
> [tb-buildbotcustom] proposed patch (v2)
> 
> Review of attachment 8685736 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> we should test this out on beta as soon as it goes to production, preferably
> before we do our next desktop ff beta
> 
> I think topsrcdir should point to the same path
> 
> ::: process/factory.py
> @@ +1347,5 @@
> >                                    WithProperties('MOZ_BUILD_DATE=%(buildid:-)s')]
> >  
> >          if self.profiledBuild:
> >              command.append('MOZ_PGO=1')
> > +        tmpEnv = copy.deepcopy(self.env)
> 
> maybe rename this var to something more descriptive like compile_env

Fixed.

> 
> @@ +1348,5 @@
> >  
> >          if self.profiledBuild:
> >              command.append('MOZ_PGO=1')
> > +        tmpEnv = copy.deepcopy(self.env)
> > +        tmpEnv.update({'TOOLTOOL_DIR': WithProperties('%(basedir)s/build')}) 
> 
> nit: whitespace

Fixed.
Attachment #8685736 - Attachment is obsolete: true
Attachment #8686337 - Flags: review+
(In reply to Edmund Wong (:ewong) from comment #28)
> Created attachment 8686337 [details] [diff] [review]
> [tb-buildbotcustom] proposed patch (v3)

Check-in Needed on this patch.
Still seeing the same error, but now later in the build:

 TEST-UNEXPECTED-FAIL | /builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/config/tests/test_mozbuild_reading.py | line 76, test_orphan_file_patterns: Evaluation of your mozconfig exited with an error. This could be triggered
 by a command inside your mozconfig failing. Please change your mozconfig
 to not error and/or to catch errors in executed commands.
 ERROR: test_orphan_file_patterns (__main__.TestMozbuildReading)
 Traceback (most recent call last):
   File "/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/config/tests/test_mozbuild_reading.py", line 76, in test_orphan_file_patterns
     mb = MozbuildObject.from_environment(detect_virtualenv_mozinfo=False)
   File "/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/python/mozbuild/mozbuild/base.py", line 169, in from_environment
     config = loader.read_mozconfig(mozconfig, moz_build_app=current_project)
   File "/builds/slave/tb-c-cen-l64-00000000000000000/build/mozilla/python/mozbuild/mozbuild/mozconfig.py", line 243, in read_mozconfig
     raise MozconfigLoadException(path, MOZCONFIG_BAD_EXIT_CODE, lines)
 MozconfigLoadException: Evaluation of your mozconfig exited with an error. This could be triggered
 by a command inside your mozconfig failing. Please change your mozconfig
 to not error and/or to catch errors in executed commands.
Flags: needinfo?(ewong)
Flags: needinfo?(ewong)
Attachment #8686875 - Flags: review?(jlund)
it *helps* a lot if I 1) saved the changes.
Attachment #8686875 - Attachment is obsolete: true
Attachment #8686875 - Flags: review?(jlund)
Attachment #8686884 - Flags: feedback?(rail)
Comment on attachment 8686884 [details] [diff] [review]
[tb-buildbotcustom] proposed patch for fixing update-packaging

this might not even be the solution...  so obsoleting this patch.
Attachment #8686884 - Attachment is obsolete: true
Attachment #8686884 - Flags: feedback?(rail)
Attachment #8684494 - Attachment is obsolete: true
Attachment #8684494 - Flags: review?(bugspam.Callek)
Attachment #8686932 - Flags: review?(bugspam.Callek)
Comment on attachment 8686884 [details] [diff] [review]
[tb-buildbotcustom] proposed patch for fixing update-packaging

having just tested a patch and pushed to try-comm-central:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=a4ae72b0ec7c

it is another one of these topsrcdir == mozilla problem.
Attachment #8686884 - Attachment is obsolete: false
Attachment #8686884 - Flags: feedback?(rail)
Comment on attachment 8686884 [details] [diff] [review]
[tb-buildbotcustom] proposed patch for fixing update-packaging

from my test push, I got a bunch of lines in the output :

which ultimately ended in:
/builds/slave/tb-try-c-cen-lx-00000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29: /builds/slave/tb-try-c-cen-lx-00000000000000/build/mozilla/gtk3/setup.sh: No such file or directory

Same thing that happened with the compile step.
TBH, I have close to zero knowledge about mozbuild. It'd be better to ask someone who knows. hg blame FTW! :)
Comment on attachment 8686884 [details] [diff] [review]
[tb-buildbotcustom] proposed patch for fixing update-packaging

lgtm
Attachment #8686884 - Flags: feedback?(rail) → feedback+
Comment on attachment 8686884 [details] [diff] [review]
[tb-buildbotcustom] proposed patch for fixing update-packaging

:glandium,  this patch basically adds the TOOLTOOL_DIR env to the
updatePackaging step.  Looking at the orange results, it's my understanding
that the |make -C <path>/update-packaging ..|it processes the mozconfig.gtk file 
and encounters the same situation as the compile step.

Is this the right approach?
Flags: needinfo?(mh+mozilla)
Why not just set TOOLTOOL_DIR everywhere?
Flags: needinfo?(mh+mozilla)
(In reply to Mike Hommey [:glandium] from comment #43)
> Why not just set TOOLTOOL_DIR everywhere?

That's up to the reviewer if it is a good idea.  And that reminds me...
will ask for a review to get this thing fixed.
Comment on attachment 8686884 [details] [diff] [review]
[tb-buildbotcustom] proposed patch for fixing update-packaging

Here's another change.

Should I just add the TOOLTOOL_DIR env to MercurialBuildFactory env?  I
can't think of any reasons why we shouldn't since if it doesn't affect
that particular build, nothing happens;  if it does, it's pointing to
the right directory; unless a particular build expects TOOLTOOL_DIR to
point to something else. (tbh, i haven't checked b2g.)
Attachment #8686884 - Flags: review?(jlund)
Comment on attachment 8686884 [details] [diff] [review]
[tb-buildbotcustom] proposed patch for fixing update-packaging

Review of attachment 8686884 [details] [diff] [review]:
-----------------------------------------------------------------

feel free to add it everywhere.. what could go wrong? :)

seriously though, I think given that the compile step hasn't been unhappy (https://bug1221502.bmoattachments.org/attachment.cgi?id=8686337) our other steps should be fine to have this in the env.
Attachment #8686884 - Flags: review?(jlund) → review+
https://hg.mozilla.org/build/buildbotcustom/rev/cfa985f4d180b5fc70a21359fabc518982d83406
Bug 1221502 - Add TOOLTOOL_DIR to the make update-packaging env. r=jlund
Looks like this didn't work?
Flags: needinfo?(ewong)
(In reply to aleth [:aleth] from comment #49)
> Looks like this didn't work?

Actually, it did work.  It's just crashing out on the |make check| stage.
Flags: needinfo?(ewong)
This should be the last.  I just looked at the log and this is at the
end of the build.
Attachment #8689815 - Flags: review?(jlund)
Attachment #8689815 - Flags: review?(jlund) → review+
Attachment #8686932 - Attachment is obsolete: true
Attachment #8686932 - Flags: review?(bugspam.Callek)
Attachment #8690623 - Flags: review?(bugspam.Callek)
Attachment #8690623 - Attachment is obsolete: true
Attachment #8690623 - Flags: review?(bugspam.Callek)
Attachment #8691768 - Flags: review?(bugspam.Callek)
Comment on attachment 8691768 [details] [diff] [review]
[sm-custom] add TOOLTOOL-DIR to compile, check and configure steps

wrong patch
Attachment #8691768 - Attachment is obsolete: true
Attachment #8691768 - Flags: review?(bugspam.Callek)
Attachment #8690623 - Attachment is obsolete: false
Attachment #8690623 - Flags: review?(bugspam.Callek)
ewong: can this bug be resolved fixed or are you really still waiting on a review for SM here?
Flags: needinfo?(ewong)
(In reply to aleth [:aleth] from comment #57)
> ewong: can this bug be resolved fixed or are you really still waiting on a
> review for SM here?

Theoretically, yes I am.  In practice though, the patch has been manually
applied to the master (and not officially pushed to the seamonkey-production branch.)
Flags: needinfo?(ewong)
(In reply to Edmund Wong (:ewong) from comment #58)
> (In reply to aleth [:aleth] from comment #57)
> > ewong: can this bug be resolved fixed or are you really still waiting on a
> > review for SM here?
> 
> Theoretically, yes I am.  In practice though, the patch has been manually
> applied to the master (and not officially pushed to the seamonkey-production
> branch.)

Resolving it then :-)
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
Comment on attachment 8690623 [details] [diff] [review]
[sm-custom] add TOOLTOOL_DIR to compile & make check env (v3) [checked-in]

Pushed to buildbotcustom:
https://hg.mozilla.org/build/buildbotcustom/rev/dd8421732e92
Attachment #8690623 - Attachment description: [sm-custom] add TOOLTOOL_DIR to compile & make check env (v3) → [sm-custom] add TOOLTOOL_DIR to compile & make check env (v3) [checked-in]
(In reply to Edmund Wong (:ewong) from comment #60)
> Comment on attachment 8690623 [details] [diff] [review]
> [sm-custom] add TOOLTOOL_DIR to compile & make check env (v3) [checked-in]
> 
> Pushed to buildbotcustom:
> https://hg.mozilla.org/build/buildbotcustom/rev/dd8421732e92

wrong cset:

https://hg.mozilla.org/build/buildbotcustom/rev/49cc5c0acb8e
Attachment #8690623 - Flags: review?(bugspam.Callek)
You need to log in before you can comment on or make changes to this bug.