(CPU time waste) LALR parser table is regenerated unnecessarily by every invocation of top-level make

NEW
Unassigned

Status

Firefox Build System
General
6 years ago
3 months ago

People

(Reporter: ISHIKAWA, Chiaki, Unassigned)

Tracking

Trunk
x86_64
Linux

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
(I am using comm-central for building TB. This is about mozilla subdirectory there.)

I noticed a case of unnecessary waste of CPU time during repeated build for debugging.
At every invocation of top-level make by
make -f client.mk I see the following message in the output every time,
"Generating LALR tables".
That is, LALR parser table for idl parsing is re-generated every time
even if I have not touched any files related to IDL headers, etc.

E.g.: Excerpt from the console log:

ishikawa@debian-vm:/TB-NEW/TB-3HG/new-src$ ./do-make.sh
Sat Sep 15 02:49:34 JST 2012
Adding client.mk options from /home/ishikawa/TB-NEW/TB-3HG/new-src/mozconfig-tb3:
    MOZ_OBJDIR=$(TOPSRCDIR)/../objdir-tb3
    MOZ_CO_PROJECT=mail
    BUILD_OFFICIAL=1
    MOZILLA_OFFICIAL=1
    BUILD_MODULES=all
    MOZ_MAKE_FLAGS=-s
make -s -C /TB-NEW/TB-3HG/new-src/../objdir-tb3
PASS
PASS
Making all in include
Making all in testsuite
Making all in man
cd config; make BUILD_DEBUG=optimize export
cd ldap; make BUILD_DEBUG=optimize export
cd include; make BUILD_DEBUG=optimize export
cd libraries; make BUILD_DEBUG=optimize export
cd liblber; make BUILD_DEBUG=optimize export
cd libldif; make BUILD_DEBUG=optimize export
cd libiutil; make BUILD_DEBUG=optimize export
cd libldap; make BUILD_DEBUG=optimize export
cd libprldap; make BUILD_DEBUG=optimize export
Generating LALR tables        <======= THIS IS THE PROBLEM
cd include; make -j1 export
Creating /TB-NEW/TB-3HG/objdir-tb3/mozilla/security/build/../../dist/public/dbm
cd src; make -j1 export
cd include; make -j1 libs
cd src; make -j1 libs
    ... [omitted] ...

Since LALR parser table generation is very CPU intensive, and now 
python is used for LALR parser table generation if I am not mistaken, it takes many seconds.
(On my PC, I think it takes close to 30+ seconds.)

This hurts when I am trying to see the effect of a header change by trying to find next compilation errors (admittedly very brute-force approach) and the compilation error is noticed after 1 min 30 sec. real time elapsed time. Of the 90 seconds, 
30+ seconds seem to be wasted by unnecessary re-generation of LALR parsing tables.

The message comes from near line 3216 of mozilla/other-licenses/ply/ply/yacc.py:

    # Run the LRGeneratedTable on the grammar
    if debug:
        errorlog.debug("Generating %s tables", method)
   
This unnecessary re-generation was easily spotted after I put

   mk_add_options MOZ_MAKE_FLAGS="-s" 

in my mozconfig file to make the build log very terse.
After outputting "Generating LALR tables", nothing is shown on the console for about 30+ seconds :-(

Looking  at mozilla/ipc/ipdl/Makefile.in, I think the one rule abut export:: there
should properly set dependencies (against the generated files) so that
unnecessary expensive generation of LALR parser table is not attempted.
But I don't know the details.

TIA
Thanks for filing the bug, note that if it is in the mozilla/ directory, then please file it in the core product, as then it'll get the proper focus for being fixed. Thanks.
Component: Build Config → Build Config
Product: Thunderbird → Core

Comment 2

5 years ago
I think this is a duplicate of bug 629668. That one had a patch in 2011, but wasn't applied for breaking windows (bug 711549).

Updated

3 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.