The default bug view has changed. See this FAQ.

PGO builds with pymake are broken because pymake chokes on a builtin with shell metacharacters

RESOLVED FIXED

Status

()

Core
Build Config
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: RyanVM, Assigned: khuey)

Tracking

unspecified
x86
Windows 7
Points:
---
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
libs_tier_nspr
c:\mozbuild\mozilla-central\nsprpub\config\rules.mk:384:0:native command 'pymake.builtins rm -f *.pgd *.gcda': shell metacharacter '*' in command line
c:\mozbuild\mozilla-central\nsprpub\config\rules.mk:189:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py -C config export' failed, return code 2
<all>: Found error
<all>: Found error
c:\mozbuild\mozilla-central\objdir-fx\config\nspr\Makefile:64:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py -C ../../nsprpub PROFILE_USE_CFLAGS="-GL -wd4624 -wd4952"' failed, return code 2
c:\mozbuild\mozilla-central\config\rules.mk:710:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py -C config/nspr libs' failed, return code 2
c:\mozbuild\mozilla-central\config\rules.mk:721:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py libs_tier_nspr' failed, return code 2
c:\mozbuild\mozilla-central\config\rules.mk:671:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py  tier_nspr' failed, return code 2
c:\mozbuild\mozilla-central\client.mk:349:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py -s -j 4 -C c:/mozbuild/mozilla-central/objdir-fx' failed, return code 2
c:\mozbuild\mozilla-central\client.mk:212:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py -f c:/mozbuild/mozilla-central/client.mk realbuild MOZ_PROFILE_GENERATE=1' failed, return code 2
c:\mozbuild\mozilla-central\client.mk:174:0: command 'c:/mozbuild/python/python.exe c:/mozbuild/mozilla-central/build/pymake/pymake/../make.py -f c:/mozbuild/mozilla-central/client.mk profiledbuild' failed, return code 2
This is a pymake bug.  We should be falling back to the shell rm here.
Assignee: wtc → nobody
Component: NSPR → Build Config
Product: NSPR → Core
QA Contact: nspr → build-config
Version: 4.8.9 → unspecified
Hrmph. We should just support shell globbing in pymake native commands.
Assignee: nobody → khuey
Committed a test http://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/rev/eb164dd01eb2
Created attachment 554925 [details] [diff] [review]
Implement shell globbing for pymake
Attachment #554925 - Flags: review?(ted.mielczarek)
Summary: Bug 665978 broke PGO pymake builds → PGO builds with pymake are broken because pymake chokes on a builtin with shell metacharacters
Comment on attachment 554925 [details] [diff] [review]
Implement shell globbing for pymake

Review of attachment 554925 [details] [diff] [review]:
-----------------------------------------------------------------

::: pymake/process.py
@@ +41,5 @@
> +    """
> +    globbedargs = []
> +    for arg in args:
> +        if _needsglob.search(arg):
> +            globbedargs.extend(glob.glob(os.path.join(cwd, arg)))

It seems unfortunate to run this regex for every single argument. Can you either a) run it once over all args or b) find out if glob is smart enough to not iterate over the entire directory unless there are wildcard characters?

::: tests/native-command-shell-glob.mk
@@ +8,4 @@
>  	$(RM) shell-glob-test/*.txt
> +	$(RM) shell-glob-test/?.foo
> +	rmdir shell-glob-test
> +	@echo TEST-PASS

I was confused as to why this was patching an existing test. You landed the test in the pymake repo first!
Attachment #554925 - Flags: review?(ted.mielczarek) → review+
http://hg.python.org/cpython/file/498b03a55297/Lib/glob.py#l24
So the problem with globbing everything is that things that don't exist will vanish from the command line.  Do you really think the regex matching will be slow enough for this to matter?
Ah, that's a good point, you're right. Too bad that both your code and glob will regex check each argument. :-/
http://hg.mozilla.org/users/bsmedberg_mozilla.com/pymake/rev/04fee3d7bdec
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Flags: in-testsuite+
The pymake change in this bug broke builds on OS X and potentially other platforms. See bug 683686.
You need to log in before you can comment on or make changes to this bug.