683660 - xpccheck.py should be (mostly?) upstreamed to ManifestDestiny

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Jeff Hammel]

http://mxr.mozilla.org/mozilla-central/source/build/xpccheck.py

srcManifestParser, getIniTests, verifyDirectory, and other utility functions should be suitably generalized and upstreamed to manifestparser.py (e.g. no hard-coded "test_*"

the __main__ section would be more difficult, because of the hardcoded path:

http://mxr.mozilla.org/mozilla-central/source/build/xpccheck.py#138

Comment 1

[Jeff Hammel]

This should also have tests with it

Comment 2

[Jeff Hammel]

So the steps are:
1. assess what functionality is in xpccheck.py that should be ported
2. assess where to put it in manifestparser.py
3. write tests!
4. rewrite (really, delete most of) xpccheck.py to use the manifestparser.py methods; maybe it can even be eliminated (ideally)

Comment 3

[Jeff Hammel]

A few notes up front:

- verifyDirectory and verifyIniFile are to be ported; I intend to have these bound methods on ManifestParser

- xpcshell has a flat directory; we'll want recursion (I think?)

- getIniTests is unneeded

Comment 4

[ISHIKAWA, Chiaki]

(In reply to Jeff Hammel [:jhammel] from comment #3)
> A few notes up front:
> 
> - verifyDirectory and verifyIniFile are to be ported; I intend to have these
> bound methods on ManifestParser
> 
> - xpcshell has a flat directory; we'll want recursion (I think?)
> 
> - getIniTests is unneeded


Hi, I am not sure if the build/configure issue (of comm-central source tree)
I experienced is related to this entry : I searched for "xpccheck.py" in bugzilla.

I fetched TB source tree (comm-central) and then during configure/build I hit upon this missing xpccheck.py problem. [This is after I clobbered the object directory, actually removed the old object directory and re-created one.]
The version of comm-central:
ishikawa@debian-vm:/REF-COMM-CENTRAL/comm-central$ hg identify
c29c9f0c7405 tip
ishikawa@debian-vm:/REF-COMM-CENTRAL/comm-central$ cd mozilla/
ishikawa@debian-vm:/REF-COMM-CENTRAL/comm-central/mozilla$ hg identify
32c3ccd8946c tip


Excerpt from the end of the failure log.

---
	...

BUILDSTATUS TIERDIR_FINISH ../ldap/xpcom
BUILDSTATUS TIERDIR_START ../db
BUILDSTATUS TIERDIR_FINISH ../db
BUILDSTATUS TIERDIR_START ../mailnews
/TEST-MAIL-DIR/objdir-tb3/mozilla/_virtualenv/bin/python: can't open file '/REF-COMM-CENTRAL/comm-central/mozilla/build/xpccheck.py': [Errno 2] No such file or directory
make[7]: *** [libs] Error 2
make[6]: *** [libs] Error 2
make[5]: *** [addrbook_libs] Error 2
make[4]: *** [libs_tier_platform] Error 2
make[3]: *** [tier_platform] Error 2
make[2]: *** [default] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2

real	1m13.776s
user	0m39.450s
sys	0m12.405s

---

  [ ... ]
/TEST-MAIL-DIR/objdir-tb3      is my MOZ_OBJDIR.
/REF-COMM-CENTRAL/comm-central is my MOZ_SRCDIR.

After searching for xpccheck.py in my local comm-central tree, I found the following.


$  find . \( -name ".hg" -prune \) -o \( -type f -print0 \) | xargs -0 egrep xpccheck.py
./mozilla/python/mozbuild/mozbuild/action/xpccheck.py:Usage: xpccheck.py <directory> [<directory> ...]
./mozilla/python/mozbuild/mozbuild/action/xpccheck.py:    print >>sys.stderr, "Usage: xpccheck.py <topsrcdir> <directory> [<directory> ...]"
./config/rules.mk:	$(PYTHON) $(MOZILLA_DIR)/build/xpccheck.py \
ishikawa@debian-vm:/REF-COMM-CENTRAL/comm-central$ 


==== So my bandage fix:

Copying ./mozilla/python/mozbuild/mozbuild/action/xpccheck.py to 
        ./mozilla/build/xpccheck.py

After the copying, the configuration/compilation is proceeding much further
and is proceeding still. (I don't know if I would hit another issue, though.)

I understand the build system is going through a drastic change.
Should I file a new bugzilla entry for this?

TIA

Comment 5

[Philip Chee]

ISHIKAWA-san I too have this error. Someone pointed out Bug 891474 as the proximal cause.

Comment 6

[ISHIKAWA, Chiaki]

(In reply to Philip Chee from comment #5)
> ISHIKAWA-san I too have this error. Someone pointed out Bug 891474 as the
> proximal cause.

Thank you. I refreshed my source tree about 10 minutes ago, and the latest fix in Bug 891474 seems to have cured the issue.

I hope the recent surgery on build system is worth it in terms of long-term maintenance, etc. We have to live through some rough ride along the way, I suppose.

Thank you again!

Comment 7

[Jeff Hammel]

for posterity: comment 4 through comment 6 are unrelated to this bug

Comment 8

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #0)
> http://mxr.mozilla.org/mozilla-central/source/build/xpccheck.py

Moved to 

http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/action/xpccheck.py

Comment 9

[Jeff Hammel]

Attachment: bug-683660.diff

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 782611 [details] [diff] [review]
bug-683660.diff

Review of attachment 782611 [details] [diff] [review]:
-----------------------------------------------------------------

the tests are great to have!  This is something that might need tweaking as we put it to a more wide variety of use cases, but as it stands it will serve the normal use cases we have needed in the past.

Comment 11

[Jeff Hammel]

pushed: https://github.com/mozilla/mozbase/commit/b630a9049d1047710e3f6660698c9cf86bde8364

Indeed, I can think of several ways this could be more flexible going forward.  The most general case would be passing in a callback that would take the file path and return whether or not it should be counted or not.