950864 - Build system should fail if Python in optimize mode

Status: NEW

(Firefox Build System :: General, defect, P3)

Description

[Nick Cameron [:nrc]]

Error messages (I get these for hours, I presume for every IDL file):

    SyntaxError: Can't build lexer
    f:\Firefox\obj-debug\dom\bindings\Makefile:71:0: command 'mozbuild.action.webidl
     main f:/Firefox/src/dom/bindings' failed, return code -127
    ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3447: No regular expression
    defined for rule 't_FLOATLITERAL'
    ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3452: No regular expression
    defined for rule 't_INTEGER'
    ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3465: No regular expression
    defined for rule 't_IDENTIFIER'
    ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3470: No regular expression
    defined for rule 't_STRING'
    ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3475: No regular expression
    defined for rule 't_WHITESPACE'
    ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3479: No regular expression
    defined for rule 't_ELLIPSIS'
    ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3484: No regular expression
    defined for rule 't_OTHER' 

Python stack trace:

    Traceback (most recent call last):
      File "f:\Firefox\src\build\pymake\pymake\process.py", line 397, in run
        rv = m.__dict__[self.method](self.argv)
      File "f:\Firefox\src\python\mozbuild\mozbuild\action\webidl.py", line 13, in m
    ain
        manager.generate_build_files()
      File "f:\Firefox\src\dom\bindings\mozwebidlcodegen\__init__.py", line 243, in
    generate_build_files
        self._parse_webidl()
      File "f:\Firefox\src\dom\bindings\mozwebidlcodegen\__init__.py", line 313, in
    _parse_webidl
        parser = WebIDL.Parser(self._cache_dir)
      File "f:\Firefox\src\dom\bindings\parser\WebIDL.py", line 4806, in __init__
        Tokenizer.__init__(self, outputdir, lexer)
      File "f:\Firefox\src\dom\bindings\parser\WebIDL.py", line 3568, in __init__
        reflags=re.DOTALL)
      File "f:\Firefox\src\other-licenses\ply\ply\lex.py", line 894, in lex
        raise SyntaxError("Can't build lexer")
    SyntaxError: Can't build lexer

Comment 1

[Nick Cameron [:nrc]]

I bisected this down to the patch 'Part 4: Rewrite WebIDL build system integration' from bug 928195.

Comment 2

[Gregory Szorc [:gps]]

I am not able to reproduce this failure and you are the only person I've heard of that is experiencing it. I suspect something about your config or build process is at fault.

Please provide the following information:

- The contents of your mozconfig and/or the output of |mach environment|.
- The command(s) used to build the tree.
- Whether this error is intermittent or not.
- Whether you experience this error with |mach build -j1|.

Comment 3

[Nick Cameron [:nrc]]

mozconfig:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../obj-debug
mk_add_options MOZ_MAKE_FLAGS="-j16"
ac_add_options --enable-debug
ac_add_options --enable-tests
ac_add_options --enable-dump-painting
ac_add_options --disable-optimize
mk_add_options AUTOCLOBBER=1
ac_add_options --disable-maintenance-service

I get the same errors with an opt build too.

Build command:

export MOZCONFIG=/f/firefox/mozconfig-debug
python -OO ./build/pymake/make.py -f client.mk -s

It is not intermittent, 100% reproducible.

I'll try a mach build later - using the build I have at the moment.

Comment 4

[Gregory Szorc [:gps]]

Please build with -j1 to rule out a race condition. You should only need to |mach build -j1 export| or |make.py -j1 export| from the objdir.

Comment 5

[Nick Cameron [:nrc]]

(In reply to Gregory Szorc [:gps] from comment #4)
> Please build with -j1 to rule out a race condition. You should only need to
> |mach build -j1 export| or |make.py -j1 export| from the objdir.

I get the same problems doing a j1 build with make. I get an error trying to run mach - ValueError: Expecting property name: line 1 column 2 (char 1) - I expect that is a separate error though. Perhaps I need to update mozbuild or something...

Comment 6

[Gregory Szorc [:gps]]

Please paste more context for that mach error.

I'm going to try harder to reproduce this with your mozconfig.

This /might/ be an issue resolving paths from your mozconfig.

Do the srcdir and objdir paths referenced in all error messages seem right?

Comment 7

[Gregory Szorc [:gps]]

I reproduced your mozconfig contents and apparently filesystem layout on a Windows machine and was unable to reproduce this error :/

Comment 8

[Nick Cameron [:nrc]]

Yeah, all the paths look OK. I tried to update my moz build (was 1.8, now 1.9), but I still get the same errors. Going to try mach again to get more details of the errors...

Comment 9

[Nick Cameron [:nrc]]

My mach error is:

Traceback (most recent call last):
  File "mach", line 55, in <module>
    main(sys.argv)
  File "mach", line 30, in main
    info = json.load(open(mozinfo_path))
  File "f:\mozilla-build\python\lib\json\__init__.py", line 290, in load
    **kw)
  File "f:\mozilla-build\python\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "f:\mozilla-build\python\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "f:\mozilla-build\python\lib\json\decoder.py", line 381, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 2 (char 1)

I get this running |./mach|, |./mach build|, or |./mach help|

My script:

export MOZCONFIG=/f/firefox/mozconfig-debug2
./mach build

And that mozconfig (I stripped it down from the last one):

#debug build
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../obj-debug
ac_add_options --enable-debug
ac_add_options --disable-optimize
mk_add_options AUTOCLOBBER=1
ac_add_options --disable-maintenance-service

Comment 10

[Nick Cameron [:nrc]]

Attachment: Cope with a bad mozinfo.json

The mach error turned out to be because there was a mozinfo.json in my src dir (no idea where it came from) and that file had incorrect syntax. Here is a small patch so mach doesn't crash when that happens.

I'll now try testing with a mach build.

Comment 11

[Nick Cameron [:nrc]]

Comment on attachment 8349626 [details] [diff] [review]
Cope with a bad mozinfo.json

Oh well, turns out we just crash later on with the same error in MozBuild and that there is not an obvious fix there.

Comment 12

[Nick Cameron [:nrc]]

After removing the bad json file and a whole load of other cruft from my src directory, mach builds now work and I don't get the codegen errors. Trying a make build now...

Comment 13

[Nick Cameron [:nrc]]

Nope, I still get the same errors using make :-( I'll grab a full log, see if there are any interesting warnings I've been missing.

Comment 14

[Nick Cameron [:nrc]]

Attachment: log of build

This is a non-clobber -j1 make build. I don't see anything interesting there.

Comment 15

[Gregory Szorc [:gps]]

Nick: Please apply the following patch and paste the error message that gets printed:

diff --git a/other-licenses/ply/ply/lex.py b/other-licenses/ply/ply/lex.py
--- a/other-licenses/ply/ply/lex.py
+++ b/other-licenses/ply/ply/lex.py
@@ -886,16 +886,17 @@ def lex(module=None,object=None,debug=0,
     else:
         ldict = get_caller_module_dict(2)

     # Collect parser information from the dictionary
     linfo = LexerReflect(ldict,log=errorlog,reflags=reflags)
     linfo.get_all()
     if not optimize:
         if linfo.validate_all():
+            print(linfo.error)
             raise SyntaxError("Can't build lexer")

     if optimize and lextab:
         try:
             lexobj.readtab(lextab,ldict)
             token = lexobj.token
             input = lexobj.input
             lexer = lexobj

Comment 16

[Gregory Szorc [:gps]]

BTW, the infinite loop issue is a bug in pymake and is being fixed in bug 952206. Expect your error output to be much sooner very shortly.

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8349626 [details] [diff] [review]
Cope with a bad mozinfo.json

Review of attachment 8349626 [details] [diff] [review]:
-----------------------------------------------------------------

Can you please file a new bug for this patch?

Comment 18

[Gregory Szorc [:gps]]

Here's another hunch: this might be due to some old .pyc gunk in your srcdir.

Can you try this on a fresh clone? Easier way to do that is to run |hg purge --all|. You may want to do |hg purge --all --print| first so you see what it will delete so it doesn't purge anything accidentally.

Alternatively, you can always |hg clone path/to/checkout new/path|, but that's more intensive. I recommend periodically purging unwanted cruft as part of your regular developer workflow.

Comment 19

[Nick Cameron [:nrc]]

(In reply to Gregory Szorc [:gps] from comment #15)
> Nick: Please apply the following patch and paste the error message that gets
> printed:
> 
> diff --git a/other-licenses/ply/ply/lex.py b/other-licenses/ply/ply/lex.py
> --- a/other-licenses/ply/ply/lex.py
> +++ b/other-licenses/ply/ply/lex.py
> @@ -886,16 +886,17 @@ def lex(module=None,object=None,debug=0,
>      else:
>          ldict = get_caller_module_dict(2)
> 
>      # Collect parser information from the dictionary
>      linfo = LexerReflect(ldict,log=errorlog,reflags=reflags)
>      linfo.get_all()
>      if not optimize:
>          if linfo.validate_all():
> +            print(linfo.error)
>              raise SyntaxError("Can't build lexer")
> 
>      if optimize and lextab:
>          try:
>              lexobj.readtab(lextab,ldict)
>              token = lexobj.token
>              input = lexobj.input
>              lexer = lexobj

I don't think this is doing anything (not sure what to expect). I get:

1
ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3451: No regular expression defined for rule 't_FLOATLITERAL'
ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3456: No regular expression defined for rule 't_INTEGER'
ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3469: No regular expression defined for rule 't_IDENTIFIER'
ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3474: No regular expression defined for rule 't_STRING'
ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3479: No regular expression defined for rule 't_WHITESPACE'
ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3483: No regular expression defined for rule 't_ELLIPSIS'
ERROR: f:\Firefox\src\dom\bindings\parser\WebIDL.py:3488: No regular expression defined for rule 't_OTHER'
Can't build lexer
Traceback (most recent call last):
  File "f:\Firefox\src\build\pymake\pymake\process.py", line 397, in run
    rv = m.__dict__[self.method](self.argv)
  File "f:\Firefox\src\python\mozbuild\mozbuild\action\webidl.py", line 13, in main
    manager.generate_build_files()
  File "f:\Firefox\src\dom\bindings\mozwebidlcodegen\__init__.py", line 243, in generate_build_files
    self._parse_webidl()
  File "f:\Firefox\src\dom\bindings\mozwebidlcodegen\__init__.py", line 313, in _parse_webidl
    parser = WebIDL.Parser(self._cache_dir)
  File "f:\Firefox\src\dom\bindings\parser\WebIDL.py", line 4810, in __init__
    Tokenizer.__init__(self, outputdir, lexer)
  File "f:\Firefox\src\dom\bindings\parser\WebIDL.py", line 3572, in __init__
    reflags=re.DOTALL)
  File "f:\Firefox\src\other-licenses\ply\ply\lex.py", line 895, in lex
    raise SyntaxError("Can't build lexer")
SyntaxError: Can't build lexer
f:\Firefox\obj-debug\dom\bindings\Makefile:72:0: command 'mozbuild.action.webidl main f:/Firefox/src/dom/bindings' failed, return code -127
Error remaking makefiles (ignored)

Comment 20

[Nick Cameron [:nrc]]

(In reply to Gregory Szorc [:gps] from comment #18)
> Here's another hunch: this might be due to some old .pyc gunk in your srcdir.
> 
> Can you try this on a fresh clone? Easier way to do that is to run |hg purge
> --all|. You may want to do |hg purge --all --print| first so you see what it
> will delete so it doesn't purge anything accidentally.
> 
> Alternatively, you can always |hg clone path/to/checkout new/path|, but
> that's more intensive. I recommend periodically purging unwanted cruft as
> part of your regular developer workflow.

I tried this (and there was a lot of cruft), but I still get the error.

Comment 21

[Gregory Szorc [:gps]]

Don't build with python -OO and the problem will go away.

That's because lex.py has this gem:

                if not f.__doc__:
                    self.log.error("%s:%d: No regular expression defined for rule '%s'",file,line,f.__name__)
                    self.error = 1
                    continue

python -OO optimizes out docstrings (.__doc__ attributes).

Comment 22

[Nick Cameron [:nrc]]

(In reply to Gregory Szorc [:gps] from comment #21)
> Don't build with python -OO and the problem will go away.
> 
> That's because lex.py has this gem:
> 
>                 if not f.__doc__:
>                     self.log.error("%s:%d: No regular expression defined for
> rule '%s'",file,line,f.__name__)
>                     self.error = 1
>                     continue
> 
> python -OO optimizes out docstrings (.__doc__ attributes).

Success \o/ Nice find. Thanks for helping me out here!

Comment 23

[Gregory Szorc [:gps]]

Changing bug summary.

Since the build system no longer works with python -OO, we should fail fast if that is detected.

Comment 24

[Boris Zbarsky [:bzbarsky]]

Yeah, the setup ply has where you specify the grammar via docstrings is ... special.

Comment 25

[Gregory Szorc [:gps]]

Attachment: Disallow running python -OO because it breaks the build

The mach_bootstrap.py change isn't strictly necessary. But it helps to
break the habit.

Comment 26

[Mike Hommey [:glandium]]

Comment on attachment 8350709 [details] [diff] [review]
Disallow running python -OO because it breaks the build

Review of attachment 8350709 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/mach_bootstrap.py
@@ +136,5 @@
> +    if sys.flags.optimize > 1:
> +        print('Running Python with -OO is not allowed because it introduces '
> +            'known failures in the build system.')
> +        sys.exit(1)
> +

Why here too?

Comment 27

[Gregory Szorc [:gps]]

Attachment: Disallow running python -OO because it breaks the build

Removed mach bootstrap change. It isn't strictly necessary. No sense to
make a blanket policy of it.

Comment 28

[Mike Hommey [:glandium]]

Comment on attachment 8358523 [details] [diff] [review]
Disallow running python -OO because it breaks the build

Review of attachment 8358523 [details] [diff] [review]:
-----------------------------------------------------------------

Thinking more about it, I think we should error out in scripts that use ply instead, because there are more ways than running pymake to have this breakage, like setting PYTHONOPTIMIZE=2 in the environment on a non-pymake build.