1591229 - |mach ide eclipse| fails for C-C TB - "ImportError: No module named mozbuild.util"

ISHIKAWA, Chiaki

Reporter

Description

•

5 years ago

•

Edited

It seems that |mach ide eclipse| ought to start Eclipse IDE.

However, with C-C TB, and Eclipse under linux (the eclipse binary is in the PATH), the command fails with the following.

... setting up environment variables, etc. ...
++ /NREF-COMM-CENTRAL/mozilla/mach ide eclipse
kind=None
environ.get('TERM', 'unknown')=xterm-256color
environ.get('TERM', 'linux')=xterm-256color
 0:00.39 Clobber not needed.
 0:00.39 Adding make options from /NREF-COMM-CENTRAL/mozilla/comm/mozconfig-tb3
    AUTOCLOBBER=1
    MOZ_CO_PROJECT=mail
    BUILD_OFFICIAL=1
    MOZILLA_OFFICIAL=1
    BUILD_MODULES=all
    export RUSTC_WRAPPER=sccache
    export CARGO_INCREMENTAL=0
    MOZ_MAKE_FLAGS=-j6
    MOZ_OBJDIR=/KERNEL-SRC/moz-obj-dir/objdir-tb3
    OBJDIR=/KERNEL-SRC/moz-obj-dir/objdir-tb3
    FOUND_MOZCONFIG=/NREF-COMM-CENTRAL/mozilla/comm/mozconfig-tb3
    export FOUND_MOZCONFIG
 0:00.40 /usr/bin/make -f client.mk MOZ_PARALLEL_BUILD=6 -s
 0:00.53 Elapsed: 0.00s; From dist/public: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.53 Elapsed: 0.00s; From dist/private: Kept 0 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.54 Elapsed: 0.01s; From dist/xpi-stage: Kept 53 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.73 Elapsed: 0.18s; From _tests: Kept 1580 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.79 Elapsed: 0.25s; From dist/bin: Kept 2831 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.94 Elapsed: 0.41s; From dist/include: Kept 6317 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.96 ./buildid.h.stub
 0:01.06 ./source-repo.h.stub
 0:01.20 build/application.ini.stub
 0:01.33 build/application.ini.h.stub
 0:01.45 toolkit/library/rust/force-cargo-library-build
 0:01.45 testing/geckodriver/force-cargo-program-build
 0:01.51     Blocking waiting for file lock on package cache
 0:01.94     Blocking waiting for file lock on package cache
 0:02.32     Finished dev [optimized + debuginfo] target(s) in 0.87s
 0:02.34     Finished dev [optimized + debuginfo] target(s) in 0.88s
 0:03.65 comm/mail/app
 0:04.81 comm/mail/app/thunderbird
 0:16.88 platform.ini updated.
 0:18.31 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/manifestparser
 0:18.62   Requirement already satisfied (use --upgrade to upgrade): manifestparser==1.2 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/manifestparser in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:18.62 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozfile
 0:18.81   Requirement already satisfied (use --upgrade to upgrade): mozfile==2.1.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozfile in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:18.81 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozinfo
 0:19.02   Requirement already satisfied (use --upgrade to upgrade): mozinfo==1.1.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozinfo in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:19.02 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozlog
 0:19.23   Requirement already satisfied (use --upgrade to upgrade): mozlog==4.2.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozlog in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:19.24 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprofile
 0:19.45   Requirement already satisfied (use --upgrade to upgrade): mozprofile==2.3.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprofile in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:19.45 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozcrash
 0:19.66   Requirement already satisfied (use --upgrade to upgrade): mozcrash==1.1.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozcrash in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:19.66 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/moznetwork
 0:19.86   Requirement already satisfied (use --upgrade to upgrade): moznetwork==0.27 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/moznetwork in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:19.86 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprocess
 0:20.08   Requirement already satisfied (use --upgrade to upgrade): mozprocess==1.0.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozprocess in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:20.08 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozdevice
 0:20.27   Requirement already satisfied (use --upgrade to upgrade): mozdevice==3.0.5 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozdevice in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:20.27 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozrunner
 0:20.50   Requirement already satisfied (use --upgrade to upgrade): mozrunner==7.6.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/testing/mozbase/mozrunner in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:20.52 Processing /NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozterm
 0:20.72   Requirement already satisfied (use --upgrade to upgrade): mozterm==1.0.0 from file:///NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozterm in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:20.72 Processing ./jsbridge
 0:20.93   Requirement already satisfied (use --upgrade to upgrade): jsbridge-thunderbird==2.5.2 from file:///NEW-SSD/moz-obj-dir/objdir-tb3/_tests/mozmill/resources/jsbridge in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:20.93 Processing ./mozmill
 0:21.14   Requirement already satisfied (use --upgrade to upgrade): mozmill-thunderbird==1.7.1 from file:///NEW-SSD/moz-obj-dir/objdir-tb3/_tests/mozmill/resources/mozmill in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages
 0:21.14 Requirement already satisfied: six>=1.10.0 in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages (from manifestparser==1.2)
 0:21.15 Requirement already satisfied: blessings>=1.3 in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages (from mozlog==4.2.0)
 0:21.15 Requirement already satisfied: mozterm in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages (from mozlog==4.2.0)
 0:21.24 You are using pip version 9.0.3, however version 19.3.1 is available.
 0:21.24 You should consider upgrading via the 'pip install --upgrade pip' command.
 0:21.29 Python: 2.7.17 (default, Oct 19 2019, 23:36:22)
 0:21.30 [GCC 9.2.1 20191008]
 0:21.33 Packaging specialpowers@mozilla.org.xpi...
 0:21.45 Packaging quitter@mozilla.org.xpi...
 0:21.59 0 compiler warnings present.
 0:21.61 Overall system resources - Wall time: 21s; CPU: 29%; Read bytes: 0; Write bytes: 7393280; Read time: 0; Write time: 1699
 0:21.63 ccache (direct) hit rate: 0.0%; (preprocessed) hit rate: 0.0%; miss rate: 100.0%
To view resource usage of the build, run |mach resource-usage|.
 0:21.63 Your build was successful!
To take your build for a test drive, run: |mach run|
 0:21.63 /KERNEL-SRC/moz-obj-dir/objdir-tb3/_virtualenvs/init/bin/python /KERNEL-SRC/moz-obj-dir/objdir-tb3/config.status --backend=CppEclipse
Traceback (most recent call last):
  File "/KERNEL-SRC/moz-obj-dir/objdir-tb3/config.status", line 1365, in <module>
    from mozbuild.util import patch_main
ImportError: No module named mozbuild.util

real	0m21.801s
user	0m28.425s
sys	0m8.505s

Looks to me some kind of version/module issues in the _virtualenvs?

ISHIKAWA, Chiaki

Reporter

Comment 1

•

5 years ago

•

Edited

/KERNEL-SRC/moz-obj-dir/objdir-tb3/config.status", line 1365 is as follows.

 1355	    'XP_UNIX': '1',
 1356	    '_REENTRANT': '1',
 1357	    'commreltopsrcdir': 'comm',
 1358	    'commtopobjdir': '/NEW-SSD/moz-obj-dir/objdir-tb3/comm',
 1359	    'commtopsrcdir': '/NEW-SSD/NREF-COMM-CENTRAL/mozilla/comm',
 1360	    'mozreltopsrcdir': '.',
 1361	    'moztopsrcdir': '/NEW-SSD/NREF-COMM-CENTRAL/mozilla',
 1362	}
 1363	__all__ = ['topobjdir', 'topsrcdir', 'defines', 'non_global_defines', 'substs', 'mozconfig']
 1364	if __name__ == '__main__':
 1365	    from mozbuild.util import patch_main
 1366	    patch_main()
 1367	    from mozbuild.config_status import config_status
 1368	    args = dict([(name, globals()[name]) for name in __all__])
 1369	    config_status(**args)

Rob Lemley [:rjl]

Comment 2

•

5 years ago

Hi,

|mach ide eclipse| worked without any issues for me following the documentation on MDN.

My first thought, perhaps you missed the |mach build| step prior to running |mach ide eclipse|?

Usually when I see Python import errors like this, there is something else in Python's sys.path that is interfering.
Try this:

➜ ./mach python                                                                                              11s 
Python 2.7.17rc1 (default, Oct 10 2019, 10:26:01) 
[GCC 9.2.1 20191008] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mozbuild
>>> mozbuild.__path__
['/home/rob/moz/unified/python/mozbuild/mozbuild']

With your setup, the last line should be: ['/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozbuild/mozbuild']

It's not clear from the your output what your current directory is when you run |mach ide eclipse|. The sys.path that the Python virtualenv is set up with will include your current directory. Is there a 'mozbuild' directory there perhaps?

ISHIKAWA, Chiaki

Reporter

Comment 3

•

5 years ago

(In reply to Rob Lemley [:rjl] from comment #2)

Hi,

|mach ide eclipse| worked without any issues for me following the documentation on MDN.

My first thought, perhaps you missed the |mach build| step prior to running |mach ide eclipse|?

Usually when I see Python import errors like this, there is something else in Python's sys.path that is interfering.
Try this:
➜ ./mach python                                                                                              11s 
Python 2.7.17rc1 (default, Oct 10 2019, 10:26:01) 
[GCC 9.2.1 20191008] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mozbuild
>>> mozbuild.__path__
['/home/rob/moz/unified/python/mozbuild/mozbuild']
With your setup, the last line should be: ['/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozbuild/mozbuild']

It's not clear from the your output what your current directory is when you run |mach ide eclipse|. The sys.path that the Python virtualenv is set up with will include your current directory. Is there a 'mozbuild' directory there perhaps?

Thank you for your tips.

Well, I have run |mach build| step. That is part of C-C TB build step and I have successfully executed that many times.
This is the first time I have tried to run Eclipse with C-C tree.

Now that said, I think I have a directory position/search issue.

++ /NREF-COMM-CENTRAL/mozilla/mach python
kind=None
environ.get('TERM', 'unknown')=xterm-256color
environ.get('TERM', 'linux')=xterm-256color
Python 2.7.17 (default, Oct 19 2019, 23:36:22) 
[GCC 9.2.1 20191008] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mozbuild
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named mozbuild

So something is wrong.

You see I am specifying |mach| with the absolute path.

lrwxrwxrwx 1 root root 26 10月 10 17:31 /NREF-COMM-CENTRAL -> /NEW-SSD/NREF-COMM-CENTRAL/
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$ ls -ld /NEW-SSD/NREF-COMM-CENTRAL/mozilla/
drwxr-xr-x 60 ishikawa ishikawa 4096 10月 25 16:40 /NEW-SSD/NREF-COMM-CENTRAL/mozilla//
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla$

/NREF-COMM-CENTRAL is symlinked to /NEW-SSD/NREF-COMM-CENTRAL in my environment.
/NEW-SSD/NREF-COMM-CENTRAL/mozilla/ is the M-C tree.
There mozilla/comm is the subdirectory to hold C-C tree.

In the source tree and In the build (MOZOBJ) directories, I see

/NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozbuild/mozbuild
/NEW-SSD/moz-obj-dir/objdir-tb3/_tests/python/python/mozbuild/mozbuild

These look like as follows by checking |ls| command:

ls /NEW-SSD/NREF-COMM-CENTRAL/mozilla/python/mozbuild/mozbuild
./			 build_commands.pyc	     jar.pyc		schedules.pyc
../			 chunkify.py		     mach_commands.py	shellutil.py
__init__.py		 chunkify.pyc		     mach_commands.pyc	shellutil.pyc
__init__.pyc		 code-analysis/		     makeutil.py	sphinx.py
__pycache__/		 codecoverage/		     makeutil.pyc	telemetry.py
action/			 compilation/		     moz_yaml.py	telemetry.pyc
analyze/		 config_status.py	     mozconfig.py	test/
android_version_code.py  config_status.pyc	     mozconfig.pyc	testing.py
artifact_builds.py	 configure/		     mozconfig_loader*	testing.pyc
artifact_builds.pyc	 controller/		     mozinfo.py*	util.py
artifact_cache.py	 doctor.py		     mozinfo.pyc	util.pyc
artifact_cache.pyc	 dotproperties.py	     nodeutil.py	vendor_aom.py
artifact_commands.py	 export_telemetry_schema.py  nodeutil.pyc	vendor_dav1d.py
artifact_commands.pyc	 faster_daemon.py	     preprocessor.py	vendor_manifest.py
artifacts.py		 frontend/		     preprocessor.pyc	vendor_python.py
artifacts.pyc		 gen_test_backend.py	     pythonutil.py	vendor_rust.py
backend/		 generated_sources.py	     pythonutil.pyc	virtualenv.py
base.py			 gn_processor.py	     repackaging/	virtualenv.pyc
base.pyc		 html_build_viewer.py	     resources/
build_commands.py	 jar.py			     schedules.py
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ ls /NEW-SSD/moz-obj-dir/objdir-tb3/_tests/moz
mozbase/ mozmill/ 
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ ls /NEW-SSD/moz-obj-dir/objdir-tb3/_tests/python/python/mozbuild/
./  ../  dumbmake/  mozbuild/  mozpack/
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ ls /NEW-SSD/moz-obj-dir/objdir-tb3/_tests/python/python/mozbuild/mozbuild/
./  ../  test/
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ ls /NEW-SSD/moz-obj-dir/objdir-tb3/_tests/python/python/mozbuild/mozbuild/test
./  ../  python.ini@  python2.ini@
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$ 
ishikawa@ip030:/NREF-COMM-CENTRAL/mozilla/comm$

BUT, I am not even so sure if these are the path(s) seen by |make mozmill| test suite. It seems to invoke python in virtualenvironment and so I am a bit loss here.

BTW, Are you building/debugging C-C TB? C-C setup for both build time AND runtime seems to be pretty complex regarding python usage :-(

Thank you again.

Rob Lemley [:rjl]

Comment 4

•

5 years ago

Yes, I am working with a C-C tree updated in the last 24 hours.

The contents of your paths look about right. I can't say for sure with mozmill, I don't run those a whole lot locally. Pretty much anything "mach" does is going to run in a virtualenv. It should be setting things up pretty much by itself though and "just work". Mozmill I believe sets up its own virtualenv separate from the one for building.

What environment variables are you setting for |mach build| and |mach clang-format|? PYTHONPATH and PYTHONHOME in particular will wreak havoc with the virtualenvs.

ISHIKAWA, Chiaki

Reporter

Comment 5

•

5 years ago

•

Edited

(In reply to Rob Lemley [:rjl] from comment #4)

Yes, I am working with a C-C tree updated in the last 24 hours.

Great to hear that.

The contents of your paths look about right. I can't say for sure with mozmill, I don't run those a whole lot locally. Pretty much anything "mach" does is going to run in a virtualenv. It should be setting things up pretty much by itself though and "just work". Mozmill I believe sets up its own virtualenv separate from the one for building.

As far as |mozmill| goes, I don't really need to run it from within Eclipse.

My main aim was to use call graph generator within Eclipse to learn the static structure of the code.

What environment variables are you setting for |mach build| and |mach clang-format|? PYTHONPATH and PYTHONHOME in particular will wreak havoc with the virtualenvs.

You may not believe this, I checked and none of PYTHONPATH or PHTHONHOME was set up by my own script.
Here is the gist of environmental variables that are set in my do-make-tb.sh script that calls |mach| near the end.

STARTING_SEC=$(date +"%s")
# I have used preloaded module for some hacky I/O error
# checks for some bugs. Currently not used.
LD_PRELOAD=
env LANG=C LC_TIME=C date

# Now this may have something to do with python.
# I think it was to avoid automatic update of mozmill or some such
# package in the virtualenv some  years ago,
# but not sure if it is wise to keep it now.
export VIRTUALENV_NO_DOWNLOAD=1

export TMP=/COMM-CENTRAL/TMP-DIR
export TMPDIR=/COMM-CENTRAL/TMP-DIR
export LANG=C.UTF-8
export LC_ALL=C.UTF-8
export LC_TIME=C
export LC_DATE=C
export LLVM_CONFIG=/usr/bin/llvm-config-4.0
export CCACHE_DIR=/FF-NEW/cache-dir
export SCCACHE_DIR=/KERNEL-SRC/sccache-dir
export SCCACHE_CACHE_SIZE=10G 
export CCACHE_LOGFILE=/FF-NEW/cache-dir/logfile
export CCACHE_HASHDIR=
export CCACHE_LIMIT_MULTIPLE=0.99
export CCACHE_COMPRESS=true
export PATH=~/.mozbuild/sccache:~/.cargo/bin:/usr/local/bin:$PATH
export MOZCONFIG=/NREF-COMM-CENTRAL/mozilla/comm/mozconfig-tb3
export CFLAGS=... I omit the gory detail here.
export CXXFLAGS=... ditto
export CC="/usr/bin/gcc-9"
export CXX="/usr/bin/g++-9"
PATH=~/.mozbuild/clang/bin/:$PATH
MACH=/NREF-COMM-CENTRAL/mozilla/mach

        then I invoke |mach| with the passed argument.
        So here it is called, e.g,
        |mach clang-format ...|
        |mach build|
        |mach ide eclipse|

ENDING_SEC=$(date +"%s")

The only thing that may remotely have something to do is
export VIRTUALENV_NO_DOWNLOAD=1

Here is one progress: I am not sure if I am doing things right or not.
You mentioned that

The sys.path that the Python virtualenv is set up with will include your current directory. Is there a 'mozbuild' directory there perhaps?

So I tried to see what happens if I invoke my script beneath
/NEW-SSD/moz-obj-dir/objdir-tb3/_tests/ directory.
There is a |modules| directory beneath it.
(Recall there is mozbuild directory below python subdirectory there.)
NEW-SSD/moz-obj-dir/objdir-tb3/_tests/python/python/mozbuild/mozbuild

Surprize discovery:

The invocation of |bash /NREF-COMM-CENTRAL/mozilla/do-make-tb.sh ide eclipse| under /NEW-SSD/moz-obj-dir/objdir-tb3/_tests/ directory.
somehow began compiling the whole source tree again (!?)
I am using |ccache| and |sccache| and so the cached objects seemed to have been used (I see direct hits in the ccache log), so it was relatively fast.

And then Ecplise started and began automatically indexing C/C++ source files (I installed C/C++ version of Eclipse from Eclipse foundation).
This indexing took more than half an hour (it showed progress) on my Ryzen 1700 CPU PC. [But the indexing was a single-thread operation and so multi cores would not help that much. Higher CPU frequency would help.)

Here is the tail end of the console log of the terminal window where the script was invoked (you will notice I put the script job to the background while I checked other things.):

    .... omitted ...
 3:26.09 Requirement already satisfied: mozterm in /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/lib/python2.7/site-packages (from mozlog==4.2.0)
 3:26.53 You are using pip version 9.0.3, however version 19.3.1 is available.
 3:26.53 You should consider upgrading via the 'pip install --upgrade pip' command.
 3:26.57 Python: 2.7.17 (default, Oct 19 2019, 23:36:22)
 3:26.57 [GCC 9.2.1 20191008]
 3:26.60 Packaging specialpowers@mozilla.org.xpi...
 3:26.71 Packaging quitter@mozilla.org.xpi...
 3:26.85 0 compiler warnings present.
 3:26.92 Overall system resources - Wall time: 206s; CPU: 48%; Read bytes: 4325154816; Write bytes: 9922179072; Read time: 277384; Write time: 167780
 3:26.92 Swap in/out (MB): 0/143
 3:27.02 ccache (direct) hit rate: 96.7%; (preprocessed) hit rate: 0.2%; miss rate: 3.1%
To view resource usage of the build, run |mach resource-usage|.
 3:27.02 Your build was successful!
To take your build for a test drive, run: |mach run|
 3:27.02 /NEW-SSD/moz-obj-dir/objdir-tb3/_virtualenvs/init/bin/python /NEW-SSD/moz-obj-dir/objdir-tb3/config.status --backend=CppEclipse
Reticulating splines...
 0:02.78 File already read. Skipping: /NEW-SSD/NREF-COMM-CENTRAL/mozilla/gfx/angle/targets/angle_common/moz.build
Create.
Opening 'Gecko'.
Warning: Nashorn engine is planned to be removed from a future JDK release
Saving workspace.
Finished reading 1851 moz.build files in 4.36s
Read 67 gyp files in parallel contributing 0.00s to total wall time
Processed into 10770 build config descriptors in 3.30s
CppEclipse backend executed in 31.32s
Generated Cpp Eclipse workspace in "/NEW-SSD/NREF-COMM-CENTRAL/eclipse_objdir-tb3".
If missing, import the project using File > Import > General > Existing Project into workspace

Run with: eclipse -data /NEW-SSD/NREF-COMM-CENTRAL/eclipse_objdir-tb3

Total wall time: 39.58s; CPU time: 7.22s; Efficiency: 18%; Untracked: 0.59s
Warning: Nashorn engine is planned to be removed from a future JDK release
log4j:WARN No appenders could be found for logger (com.spotify.docker.client.DockerConfigReader).
log4j:WARN Please initialize the log4j system properly.
^Z
[1]+  Stopped                 bash /NREF-COMM-CENTRAL/mozilla/do-make-tb.sh ide eclipse
ishikawa@ip030:/NEW-SSD/moz-obj-dir/objdir-tb3/_tests$ bg

(And then the C/C++ source files indexing ran for about half an hour or so.)

I am not sure if the particular invocation of |mach ide emacs| under $(MOZOBJ)/_tests is the right thing or not.
But I think I am moving in the right direction somehow.
I will try to see if I can use the features of Eclipse under this environment.
But I think I need to set the log setup first so that errors/warnings are properly recorded.
There were warnings regarding this immediately after the eclipse window appeared on the screen.

Oh I should mention that installing Eclipse today (October 2019) under Debian GNU/Linux requires special caution.
https://groups.google.com/d/msg/mozilla.dev.apps.thunderbird/musukBzrGZY/XRvFaOmYEAAJ

I am trying Eclipse.

Installing Eclipse under Debian GNU/Linux needs one blackmagic.
Without the tips in the following post, installation won't proceed at all (!)

https://superuser.com/questions/1455397/debian-9-org-eclipse-swt-swterror-no-more-handles-immediately-after-opening-ec

It seems to have something to do with multi-octet unicode input system such as for Japanese.
(I have no idea how that can affect Eclipse installer.)

But ten, unfortunately, |mach ide eclipse| fails with a missing module or something.



See https://bugzilla.mozilla.org/show_bug.cgi?id=1591229

Stay tuned...

Chiaki

So Rob, thanks to your observation, I am moving ahead slowly. I am hoping to get the call graph listing very soon now :-)
Maybe too optimistic, but now that |make ide eclipse| starts, things are moving up (!)

Thank you again.

ISHIKAWA, Chiaki

Reporter

Comment 6

•

5 years ago

•

Edited

Attached image Eclipse screen that shows errors and warnings. — Details

Dear Bob,

I thought I was making a progress.
Yes, of a sort.

As I looked at the Eclipse window carefully, I noticed errors and warnings.
The number may change due to my local patches.

C-C with my local patches builds locally and tyrserver can build it. So I don't believe there are real errors.

Do you see the similar errors/warnings in your Eclipse window?

PS: This is the Eclipse screen when I launched it under _test directory.
Do you "build" C-C TB inside Eclipse?
My immediate goal is to invoke call graph feature.

ISHIKAWA, Chiaki

Reporter

Updated

•

5 years ago

Flags: needinfo?(rob)

ISHIKAWA, Chiaki

Reporter

Comment 7

•

5 years ago

•

Edited

Attached image Another error: maybe include is not handled properly? — Details

Another error that makes me think that I am not invoking eclipse correctly is that
the particular error.
That some methods are not defined is simply incorrect.
I suspect that some include files are not handled properly?

BTW, this is under Debian GNU/Linux.

Rob Lemley [:rjl]

Comment 8

•

5 years ago

I do not actually use Eclipse. I work mostly with the Taskcluser and build code which is written predominantly Python, so I use Pycharm. That doesn't help you though.

I let |mach ide eclipse| do it's build and whatever else until it launched the GUI though. I'm not seeing errors like you have, just some warnings about suggested parenthesis. It looks like maybe that indexing process didn't work right for you? I really don't know unfortunately.

If you adjust your mozconfig to build Firefox instead, do the includes work? If that's broken as well I would talk to the Firefox folks.

Flags: needinfo?(rob)

ISHIKAWA, Chiaki

Reporter

Comment 9

•

5 years ago

Thank you for your comment.

If the said command |mach ide ecplise| works right in the
GUI without seeing the errors I mention, I must have made a mistake somewhere.
Maybe the indexing did not work right ?
(I did see the warnings about suggested parenthesis.)

The reason I say is that I kept a few old trees under the name comm-old, comm-pending, comm-....
It was necessary to transition from an old PC or after a major source tree disruption happened, etc.
Initially, I noticed eclipse indexed all of the files. I realized it when I look for the declaration of
a certain function, and Eclipse asked me which definition in which file (!!).

Although, I have no idea how to clear the existing index. [Maybe there is a command for that.],
I will do the re-indexing, and then if that does not work, will try the firefox build.

IF that does not work, I will ask firefox people.

Thank you again for your comment.
Just knowing that someone has C-C TB built under eclipse is a great encouragement.

Rob Lemley [:rjl]

Comment 10

•

5 years ago

I did run an actual build in Eclipse after finishing my comment last night, and it did run all the way through taking just over 30 minutes. Zero errors reported by mach. I don't have indexing or content assist going though.

ISHIKAWA, Chiaki

Reporter

Comment 11

•

5 years ago

(In reply to Rob Lemley [:rjl] from comment #10)

I did run an actual build in Eclipse after finishing my comment last night, and it did run all the way through taking just over 30 minutes. Zero errors reported by mach. I don't have indexing or content assist going though.

Wow, great.

I am not sure what is wrong with my setup. Maybe out-of-tree object directory may be the root cause...

Here is what I did after I wrote my previous post.
Even running eclipse against M-C FF showed "errors".

I did the following and it seemed to work (!)

Since I was not sure what the re-indexing command is, I simply deleted the whole
project tree: it was

/NEW-SSD/NREF-COMM-CENTRAL/eclipse_objdir-tb3

The directory was shown in the message that is shown during the start up of eclipse.

...
Generated Cpp Eclipse workspace in "/NEW-SSD/NREF-COMM-CENTRAL/eclipse_objdir-tb3".
If missing, import the project using File > Import > General > Existing Project into workspace

Run with: eclipse -data /NEW-SSD/NREF-COMM-CENTRAL/eclipse_objdir-tb3
...

I made sure that I removed the duplicate comm-* directories.

On the next |/NREF-COMM-CENTRAL/mozilla/do-make-tb.sh ide eclipse|, |eclipse| started fine and began reindexing (!).

Once it was over, everything seems to be fine. I don't see the error lines.
Well, come to think of it, I don't see the warning lines (?) <- this may be visible in different file?
Unfortunately, I was not quit correct. After playing with Eclipse for about 10 minutes, now I see errors.
I think the indexing or parsing may have progressed asynchronously...

So I switched the approach. Now I tried to see if |mach ide eclipse| works for Firefox build.
I tweaked my old script to build FF so that it works in my current setup.
I first made sure that it could build FF. (Invoking |mach build|).
Then I ran it to see |mach ide eclipse| worked or not.
Well, eclipse started and began indexing files.
Eclipse started.
However, again, several minutes of browsing the source files, I see "errors".

However, they may not be real errors, but run-time(?) type resolution which is not quite possible at compile time(?) or
Eclipse's static type resolution derivation is not as strong as that of C++ compiler (?) Ugh.

I am attaching a screenshot to see what I mean. There are, for example, different(?) sqlite3_stmt data types in multiple files.
Eclipse seems unable to figure out statically which data types are used on the line.

I am not familiar with Eclipse at all. I am a newbie. I wonder if this is all right for Eclipse.
I will study a bit more to see if this is the ordinary state.

OTOH, I notice that I invoked my script to build FF and invoke eclipse using mozilla directory as the current directory: it worked, sort of.
I wonder why my script to build TB did not work as expected ...

I will investigate a bit more.

ISHIKAWA, Chiaki

Reporter

Comment 12

•

5 years ago

Attached image Eclipse complaining that it could not resolve sqlite3_stmt data types and asks a user to choose which declaration to examine. — Details

Here is the screenshot of the example of sqlite3_stmt in my post.
This is not exactly an error, but limit of static type analysis (?).

ISHIKAWA, Chiaki

Reporter

Comment 13

•

5 years ago

Aha, I have a feeling that I am using "parallelized silent build" and this may interfere with the full functioning of Eclipse, esp. code assistance.
Stay tuned.
(It turns out object directory needs to be out side of the source tree. So my set up is correct in that regard.)

ISHIKAWA, Chiaki

Reporter

Comment 14

•

5 years ago

I think my problem is related to unresolved include file issues.
See the attached screen.

I will re-read MDN eclipse document very carefully to see what I can do.
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Eclipse/Eclipse_CDT

Unfortunately, the document seems quite outdated :-(
I mean description below "Setup time" seems out of sync pretty much now that |mach ide eclipse| takes care of the initial workspace configuration and Eclipse menu entries seem to have changed quite a bit. Ugh...

Anyway, I will report back when I make progress.

ISHIKAWA, Chiaki

Reporter

Comment 15

•

5 years ago

Attached image I have unresolved inclusion file issues as mentioned in mozilla eclipse document page. — Details

As you can see, I have the unresolved include file issues as documented in mozilla eclipse document.
(Maybe because I run parallelized silent build.)
I will see if I can fix this by installing the fake build .py file that seems to print out the
necessary command line info for Eclipse to capture.

From the mozilla eclipse web page:

Below we will configure Eclipse's build step so that you use it only occasionally to manually trigger a special "build" (actually a fast script that fakes a build) purely for the purposes of setting/updating the compiler options that Eclipse associates with each source file.

Either I will have to live with the unparallelized verbose build OR
be done with the fast script that fakes build.

Stay tuned.

ISHIKAWA, Chiaki

Reporter

Comment 16

•

5 years ago

•

Edited

Well, fake_build.py did not help.

I could, however, eliminate the unresolved header files in the following manner.

I have to run my script that invokes |mach| eventually UNDER MOZ_OBJDIR directory still.
It is defined in MOZCONFIG as mk_add_options MOZ_OBJDIR=/KERNEL-SRC/moz-obj-dir/objdir-tb3
[I have tried a few variations including defining PYTHONPATH to see if I can invoke the command elsewhere, but to no avail. I always get the mozbuild.util missing error.]
/KERNEL-SRC/moz-obj-dir is a symlink to /NEW-SSD/moz-obj-dir on my PC, and so there may be some mixed up description in
my shell script and MOZCONFIG.
Come to think of it, the symlink acrobatics may have something to do with the failure to invoke |mach ide eclipse|, but
if so, it is not obvious.
I modified my script to invoke |mach| to run |mach build -v| (note "-v", that is the key ) immediately before
invoking |mach ide eclipse|.

This seems to be the trick.

The automatic invocation of |mach| when I say |mach ide eclipse| seems to invoke |mach build| WITHOUT "-v" and thus the problem of unresolved header files.
(Well, my bash script has so many arcane bits that handled many toolchain issues before and not pleasant to read, but I will upload this script
and my mozconfig for other readers to reference just in case they have similar issues.

Oh, don't forget to run the black magic in comment 5 when you install Eclipse if you are a Debian user.

Right now, eclipse session seems to work and I can get call hierarchy rather handy (!)

Thank you again for your tips, Rob.

ISHIKAWA, Chiaki

Reporter

Comment 17

•

5 years ago

•

Edited

Attached file do-make-tb.sh (should now be called do-mach-tb.sh, I suppose) — Details

This is my script to invoke |mach|.

It has been called do-make-tb.sh: it used to be |make| was used to build TB.

This script is placed under
/NREF-COMM-CENTRAL/mozilla
my M-C source tree root.

I invoke the script as in:

./do-make-tb.sh clobber
./do-make-tb.sh configure
./do-make-tb.sh build (actually without any argument, |mach build| is invoked.)

./do-make-tb.sh ide eclipse

Now the last command invokes |mach build -v| (with -v) immediately before invoking |mach ide eclipse|.

ISHIKAWA, Chiaki

Reporter

Comment 18

•

5 years ago

Attached file mozconfig-tb3: my mozconfig — Details

This config file also has many crufts that dealt with debugging issues over the years and may not be pleasant to read.

This is placed in
/NREF-COMM-CENTRAL/mozilla/comm
and referenced in do-make-tb.sh as
export MOZCONFIG=/NREF-COMM-CENTRAL/mozilla/comm/mozconfig-tb3

ISHIKAWA, Chiaki

Reporter

Updated

•

5 years ago

Attachment #9105701 - Attachment mime type: application/x-shellscript → application/text

ISHIKAWA, Chiaki

Reporter

Updated

•

5 years ago

Attachment #9105701 - Attachment mime type: application/text → text/plain

ISHIKAWA, Chiaki

Reporter

Updated

•

5 years ago

Attachment #9105702 - Attachment mime type: application/octet-stream → text/plain

ISHIKAWA, Chiaki

Reporter

Comment 19

•

5 years ago

(In reply to ISHIKAWA, Chiaki from comment #17)

Created attachment 9105701 [details]
do-make-tb.sh (should now be called do-mach-tb.sh, I suppose)

...

./do-make-tb.sh ide eclipse

Now the last command invokes |mach build -v| (with -v) immediately before invoking |mach ide eclipse|.

Come to think of it, |mach build -v| before invoking eclipse is required ONLY ONCE to register the include directories in eclipse database.
So once it is called, it can be commented out, I think.

BMO Automation

Updated

•

2 years ago

Severity: normal → S3

Eclipse screen that shows errors and warnings. 5 years ago ISHIKAWA, Chiaki 398.66 KB, image/png		Details
Another error: maybe include is not handled properly? 5 years ago ISHIKAWA, Chiaki 419.75 KB, image/png		Details
Eclipse complaining that it could not resolve sqlite3_stmt data types and asks a user to choose which declaration to examine. 5 years ago ISHIKAWA, Chiaki 545.97 KB, image/png		Details
I have unresolved inclusion file issues as mentioned in mozilla eclipse document page. 5 years ago ISHIKAWA, Chiaki 451.21 KB, image/png		Details
do-make-tb.sh (should now be called do-mach-tb.sh, I suppose) 5 years ago ISHIKAWA, Chiaki 9.26 KB, text/plain		Details
mozconfig-tb3: my mozconfig 5 years ago ISHIKAWA, Chiaki 9.45 KB, text/plain		Details