Closed Bug 729752 Opened 12 years ago Closed 9 years ago

Old .pyc files in srcdir shouldn't break the build

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1188224

People

(Reporter: jruderman, Unassigned)

References

Details

Attachments

(2 files)

patch, remove .pyc/.pyo from the .hgignore file
473 bytes, patch
ted
: review-
Details | Diff | Splinter Review
build fix for python cache files
6.32 KB, patch
Details | Diff | Splinter Review 
make[8]: Entering directory `/Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl'
make[8]: *** No rule to make target `../../../xpcom/idl-parser/xpidllex.py', needed by `libs'.  Stop.

Workaround: rm $srcdir/xpcom/idl-parser/*.pyc (see bug 723861)

Why does the existence of obsolete *.pyc files break the build? Please make configure warn or delete them, or make |make| not fall over, or something.
See Also: → 686233
See Also: → 685957
You can add linux to it. Cannot get firefox to be built with the same error on my archlinux.
OS: Mac OS X → All
Hardware: x86_64 → All
(In reply to Jesse Ruderman from comment #0)

> Why does the existence of obsolete *.pyc files break the build? Please make
> configure warn or delete them, or make |make| not fall over, or something.

Yes, it'd be nice to fix this. To be specific, though, I don't think configure should delete them - running configure should not modify the source tree in any way, IMO. And a configure warning would be too easy to overlook.

The correct fix, I think, is to make sure that at |make| time they don't distract |make| (or header.py?) from doing the right thing. But I haven't dug around to understand what all this is really about.
The same goes for Seamomkey as well:
(Linux Ubuntu 11.10, gcc 4.6.1)

rm $srcdir/mozilla/xpcom/idl-parser/*.pyc is fixing this bug.
Ping. Any |make| gurus out there?
I don't think we want make automatically changing what's in your srcdir.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
FWIW, pyc files are Python build artifacts and should never[tm] matter, so as a Python dev I would not be upset if a build script rm-ed pyc files inside my build dir.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #5)
> I don't think we want make automatically changing what's in your srcdir.

As stated in the description, make should not fall over like this.  Deleting contents of the src dir doesn't sound like the solution but a solution is needed.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
I think the proper solution would be something along the lines of bug 661908. Having .py files scattered about like we do is just silly and nobody denies that.
AFAIK, the problem from comment 0 occurred because we used to have these (generated) .py files checked-in to the repo, and we moved to having them generated in the objdir. While it's unfortunate that this broke stuff, it's also not a really common scenario.
(In reply to Ted Mielczarek [:ted] (away until ~March 7) from comment #9)
> AFAIK, the problem from comment 0 occurred because we used to have these
> (generated) .py files checked-in to the repo, and we moved to having them
> generated in the objdir. While it's unfortunate that this broke stuff, it's
> also not a really common scenario.

Then shouldn't the .pyc files be obsoleted?  I seem to hit this when updating local clones that I haven't updated for a couple months.  Damn annoying...
When building on OSX 10.5, I seem to hit this fairly often.  Not sure if this is related but it seems to be correlated with seeing this hg message:

/builds/mozcentral$ hg update
note: possible conflict - build/unix/check_debug_ranges.py was renamed multiple times to:
 build/autoconf/check_debug_ranges.py
 js/src/build/autoconf/check_debug_ranges.py
922 files updated, 0 files merged, 26 files removed, 0 files unresolved

Building on OSX 10.7 I see the same message from hg but I don't run into the problem with .pyc files causing the build to fail.
That's unrelated, FWIW.
Many engineer hours have been wasted hitting this bug. Perhaps, the fix is obvious for the engineers who have knowledge in the compilation process, but for many engineers, it is completely unintuitive what to do once they hit this bug. First, they might try to rm -rf the object directory and do a clean rebuild (as I tried to do), but that doesn't work, so they might resort to doing a clean clone, which takes hours.

Could we get this fixed to avoid more hours of developers' time wasted?
If .pyc/.pyo files are now build into the obj-dir, then it seems like they shouldn't be included in .hgignore.
Attachment #606104 - Flags: review?(ted.mielczarek)
Comment on attachment 606104 [details] [diff] [review]
patch, remove .pyc/.pyo from the .hgignore file

This patch doesn't look correct to me because .pyc files for non-generated .py files will still be generated in the srcdir.

Do you still hit this bug after changeset https://hg.mozilla.org/mozilla-central/rev/b6627f28b7ec ?
Comment on attachment 606104 [details] [diff] [review]
patch, remove .pyc/.pyo from the .hgignore file

Review of attachment 606104 [details] [diff] [review]:
-----------------------------------------------------------------

They're not, in the general case, only in that one particular case. We still have lots of .py files in the srcdir that wind up with .pyc files next to them.

Someone mentioned on IRC that Python 3.2 has an option to specify an alternate directory for .pyc files, so maybe at some point in the future we can use that.
Attachment #606104 - Flags: review?(ted.mielczarek) → review-
(In reply to Ted Mielczarek [:ted] from comment #16)
> Someone mentioned on IRC that Python 3.2 has an option to specify an
> alternate directory for .pyc files, so maybe at some point in the future we
> can use that.

Alternatively, we could use the -B option, available since python 2.6. which may or may not make things slightly slower.
Lately, I'm having this issue on the second pass of my pgo builds when I delete all *.pyc files in my tree prior to starting the build.
(In reply to Ryan VanderMeulen from comment #19)
> Lately, I'm having this issue on the second pass of my pgo builds when I
> delete all *.pyc files in my tree prior to starting the build.

I'm using pymake and have the same issue right after its supposed to be generating PGO'd code (a bit after the test window closes).

No rule to make target '../../../xpcom/idl-parser/xpidllex.py' needed by ['<comm and-line>', '../../../xpcom/idl-parser/xpidllex.py']
I remove all the *.pyc files in xpcom/idl-parser.  I remove xpidllex.py and
xpidlyacc.py.  (hg status gives me ?)

Now I get this:

make[9]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpt/tools'
make[8]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpt/tools'
make[7]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpt'
make -C xpidl export
make[7]: Entering directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
c:/mozstuff/mozprgs/sm/objdir-sm/mozilla/config/nsinstall.exe -D ../../../dist/sdk/bin/ply
c:/mozstuff/mozprgs/sm/objdir-sm/mozilla/config/nsinstall.exe -m 755 /c/mozstuff/mozprgs/sm/mozilla/other-licenses/ply/ply/__init__.py /c/mozstuff/mozprgs/sm/mozilla/other-licenses/ply/ply/lex.py /c/mozstuff/mozprgs/sm/mozilla/other-licenses/ply/ply/yacc.py ../../../dist/sdk/bin/ply
make[8]: Entering directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
make[8]: *** No rule to make target `../../../xpcom/idl-parser/xpidllex.py', needed by `libs'.  Stop.
make[8]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
make[7]: *** [export] Error 2
make[7]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib/xpidl'
make[6]: *** [export] Error 2
make[6]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom/typelib'
make[5]: *** [export] Error 2
make[5]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla/xpcom'
make[4]: *** [export_tier_platform] Error 2
make[4]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla'
make[3]: *** [tier_platform] Error 2
make[3]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla'
make[2]: *** [default] Error 2
make[2]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm/mozilla'
make[1]: *** [default] Error 2
make[1]: Leaving directory `/c/mozstuff/mozprgs/sm/objdir-sm'
make: *** [build] Error 2
I think we should do the -B option.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #22)
> I think we should do the -B option.

Sounds right. Needs an assignee.
It needs performance testing too.
This problem occurs in the firefox-13 tarball, and there are no pyc files in the source. This was discussed, and a patch found, on mozilla.dev.apps.firefox.

https://groups.google.com/forum/#!topic/mozilla.dev.apps.firefox/o9FXYPnZu5A

This patch is from that thread. I didn't create it, but I've tested it and it works for me.
(In reply to Robert from comment #25)
> Created attachment 635321 [details] [diff] [review]
> build fix for python cache files
> 
> This problem occurs in the firefox-13 tarball, and there are no pyc files in
> the source. This was discussed, and a patch found, on
> mozilla.dev.apps.firefox.
> 
> https://groups.google.com/forum/#!topic/mozilla.dev.apps.firefox/o9FXYPnZu5A
> 
> This patch is from that thread. I didn't create it, but I've tested it and
> it works for me.

Bug 754625 fixed that particular problem.
I just encountered this build error on Windows. How can I circumvent it?
From irc:
[18:59]	<glandium>	mw22: find -name \*.pyc | xargs rm
(In reply to Martijn Wargers [:mw22] (QA - IRC nick: mw22) from comment #28)
> From irc:
> [18:59]	<glandium>	mw22: find -name \*.pyc | xargs rm

That'll barf if you have whitespace anywhere in the results -- you can just use

find -name \*.pyc -delete
I just ran into this. Please fix!
Is this dup bug 1188224?
Flags: needinfo?(mh+mozilla)
Status: REOPENED → RESOLVED
Closed: 12 years ago → 9 years ago
Flags: needinfo?(mh+mozilla)
Resolution: --- → DUPLICATE
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: