525047 - 'make clean' deletes automationutils.py

Status: RESOLVED FIXED

Description

[Enno]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.21) Gecko/20090410 SUSE/1.1.16-1.1 SeaMonkey/1.1.16
Build Identifier: SEAMONKEY_2_0rc2_RELEASE

'make clean' deletes mozilla/build/automationutils.py which is a proper and necessary source file and not eg. generated from automationutils.py.in 

Should not happen

Reproducible: Always

Comment 1

[R P Mozley]

I've run into this problem also with SeaMonkey 2.0 release on Mac OS X

Comment 2

[Ted Mielczarek [:ted.mielczarek]]

I assume you're doing a srcdir build?

Comment 3

[[e-mail]]

build/automation-build.mk copies automation.py to the objdir (why?) and adds it to the garbage. This is of course unhelpful for a srcdir build ;-)

Comment 4

[[e-mail]]

Created attachment 409058 [details] [diff] [review]
Possible patch

Comment 5

[Ted Mielczarek [:ted.mielczarek]]

(In reply to comment #3)
> build/automation-build.mk copies automation.py to the objdir (why?) and adds it
> to the garbage. This is of course unhelpful for a srcdir build ;-)

Because PYTHONPATH mangling can be a bit of a pain, and our preprocessing of Python files doesn't make it any easier. Ideally we wouldn't preprocess any of our Python, and we could stop copying these files around.

Comment 6

[Enno]

(In reply to comment #2)
> I assume you're doing a srcdir build?

Yes, if that means not using objdir

Comment 7

[Enno]

I know little about the python preprocessing but could a solution be introducing a automationutils.py.in with nill reprocessing?

Comment 8

[Ted Mielczarek [:ted.mielczarek]]

No. I don't want that.

Comment 9

[[e-mail]]

(In reply to comment #5)
> (In reply to comment #3)
> > build/automation-build.mk copies automation.py to the objdir (why?) and adds it
> > to the garbage. This is of course unhelpful for a srcdir build ;-)
> Because PYTHONPATH mangling can be a bit of a pain, and our preprocessing of
> Python files doesn't make it any easier. Ideally we wouldn't preprocess any of
> our Python, and we could stop copying these files around.
How does automation-utils.py relate to the preprocessing of automation.py?

Comment 10

[Ted Mielczarek [:ted.mielczarek]]

Comment on attachment 409058 [details] [diff] [review]
Possible patch

I'm pretty sure this will break Mochitest/reftest. When running them in-tree, they rely on having automationutils.py copied to the local directory, then copied again to $objdir/_test/whatever.

Comment 11

[[e-mail]]

(In reply to comment #10)
> (From update of attachment 409058 [details] [diff] [review])
> I'm pretty sure this will break Mochitest/reftest. When running them in-tree,
> they rely on having automationutils.py copied to the local directory, then
> copied again to $objdir/_test/whatever.
Oh I see, I need to change the references from $(CURDIR)/automationutils.py to $(topsrcdir)/build/automationutils.py, to avoid the unnecessary copy.

Comment 12

[[e-mail]]

Created attachment 412563 [details] [diff] [review]
Possible patch

Comment 13

[Ted Mielczarek [:ted.mielczarek]]

Comment on attachment 412563 [details] [diff] [review]
Possible patch

Ok, seems reasonable.

Comment 14

[[e-mail]]

Pushed changeset dbdbe3ad0234 to mozilla-central.

Comment 15

[:Gavin Sharp [email: [e-mail]]]

Backed out, since this caused clobbered leak test machines to fail with:

ImportError: No module named automationutils

e.g.: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1260932775.1260937785.2108.gz&fulltext=1

https://hg.mozilla.org/mozilla-central/rev/4bbe26190c7d

Comment 16

[[e-mail]]

(In reply to comment #15)
> Backed out, since this caused clobbered leak test machines to fail with:
> 
> ImportError: No module named automationutils
> 
> e.g.:
> http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1260932775.1260937785.2108.gz&fulltext=1

Weird. That log shows automationutils.py being installed into _leaktest:

/builds/moz2_slave/mozilla-central-linux-debug/build/obj-firefox/config/nsinstall -R automation.py /builds/moz2_slave/mozilla-central-linux-debug/build/build/automationutils.py leaktest.py /builds/moz2_slave/mozilla-central-linux-debug/build/build/bloatcycle.html /builds/moz2_slave/mozilla-central-linux-debug/build/build/pgo/server-locations.txt ../_leaktest

And python leaktest.py -- -register certainly succeeds locally...

Comment 17

[[e-mail]]

My bad, this works on Windows because it copies instead of symlinks...

Comment 18

[[e-mail]]

(In reply to comment #11)
> Oh I see, I need to change the references from $(CURDIR)/automationutils.py to
> $(topsrcdir)/build/automationutils.py, to avoid the unnecessary copy.
And I need to not forgot to prefix $(topsrcdir)/build/ !

Comment 19

[[e-mail]]

And I need to remember not to typo the checkin comment again too...

Pushed changeset 5a2692b1abee to mozilla-central.

Comment 20

[Serge Gautherie (:sgautherie)]

(In reply to comment #19)
> Pushed changeset 5a2692b1abee to mozilla-central.

http://hg.mozilla.org/mozilla-central/rev/d86a2cf35ca8
backout

Comment 21

[[e-mail]]

(In reply to comment #18)
> (In reply to comment #11)
> > Oh I see, I need to change the references from $(CURDIR)/automationutils.py to
> > $(topsrcdir)/build/automationutils.py, to avoid the unnecessary copy.
> And I need to not forgot to prefix $(topsrcdir)/build/ !
That's not it, since if you're building from $(CURDIR)/build/ then $(topsrcdir)/build/ is in your $(VPATH)...

Comment 22

[[e-mail]]

Created attachment 418843 [details] [diff] [review]
Stupid Python

Comment 23

[[e-mail]]

Created attachment 418907 [details] [diff] [review]
Stupid Python

Comment 24

[Serge Gautherie (:sgautherie)]

(In reply to comment #22)

Ftr,

> Created an attachment (id=418843) [details]

This one uses multiple |NSDISTMODE = copy|.

(In reply to comment #23)
> Created an attachment (id=418907) [details]

This one uses |sys.path.insert(0, "");|

Comment 25

[Ted Mielczarek [:ted.mielczarek]]

Comment on attachment 418843 [details] [diff] [review]
Stupid Python

Yeah, I'd rather keep the symlinks, they make development easier.

Comment 26

[Ted Mielczarek [:ted.mielczarek]]

Comment on attachment 418907 [details] [diff] [review]
Stupid Python

>diff --git a/build/automation.py.in b/build/automation.py.in
>--- a/build/automation.py.in
>+++ b/build/automation.py.in
>@@ -45,17 +45,17 @@ import re
> import re
> import select
> import shutil
> import signal
> import subprocess
> import sys
> import threading
> import tempfile
>-
>+sys.path.insert(0, "");

What does this wind up doing, adding the script's directory to sys.path? Could we make this clearer using os.path.dirname(__file__) or something? (I tested, and dirname(__file__) returns the containing dir even if __file__ is a symlink.)

Comment 27

[[e-mail]]

Created attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]

We already have a variable with the appropriate path in question :-)

Comment 28

[Ted Mielczarek [:ted.mielczarek]]

Comment on attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]

Yeah, that's the ticket.

For the record, when Neil says "stupid Python" he means "when Python adds the script directory to sys.path, if the script is a symlink, it actually adds the directory of the original file".

Comment 29

[[e-mail]]

Pushed changeset fa0a5a38b7f4 to mozilla-central.

(In reply to comment #28)
> For the record, when Neil says "stupid Python" he means "when Python adds the
> script directory to sys.path, if the script is a symlink, it actually adds the
> directory of the original file".
Which IMHO is still a stupid thing to do...

Comment 30

[Ted Mielczarek [:ted.mielczarek]]

Not disagreeing, just clarifying! (since it wasn't mentioned anywhere else on the bug)

Comment 31

[Serge Gautherie (:sgautherie)]

Comment on attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]


"approval1.9.2.2=?":
Zero risk, build config only.

Comment 32

[Daniel Veditz [:dveditz] PTO until August]

Comment on attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]

Approved for 1.9.2.2, a=dveditz for release-drivers

Comment 33

[Serge Gautherie (:sgautherie)]

Comment on attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]


"approval1.9.1.9=?":
Zero risk, build config only.

Comment 34

[Serge Gautherie (:sgautherie)]

Comment on attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]


http://hg.mozilla.org/releases/mozilla-1.9.2/rev/08f2d33c06f6

Comment 35

[Daniel Veditz [:dveditz] PTO until August]

Comment on attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]

Approved for 1.9.1.9, a=dveditz for release-drivers

Comment 36

[Serge Gautherie (:sgautherie)]

Comment on attachment 421035 [details] [diff] [review]
Even better
[Checkin: Comment 29 & 34 & 36]


http://hg.mozilla.org/releases/mozilla-1.9.1/rev/4fdc73ee8729

Comment 37

[Marco Zehe (:MarcoZ)]

Verified that patches pushed in comment#34 and comment#36 landed.