788842 - Mirror moztest to m-c, and post to pypi, for bug 783713

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Jonathan Griffin (:jgriffin)]

In order to land bug 783713, we'll need to post a moztest package to pypi, and mirror it to m-c.  I can do both of these things, unless someone else would prefer to do it.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Mirror moztest to m-c,

This patch mirrors moztest to m-c

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: Mirror moztest to m-c,

Oops, last patch unintentially picked up a mozcrash change

Comment 3

[Jeff Hammel]

Comment on attachment 658701 [details] [diff] [review]
Mirror moztest to m-c,

Our current policy is to mirror the whole of mozbase as described in https://wiki.mozilla.org/Auto-tools/Projects/MozBase#Mirroring . I can help with this, or do this as desired.  While there has been some discussion about changing this procedure, it remains currently "the thing to do".

Also, I *think* that other packages need to be bumped and released to pypi too.  hard to say without seeing the diff.

Comment 4

[Jonathan Griffin (:jgriffin)]

Attachment: mirror mozbase to m-c

Thanks for the link Jeff!  Here's the resulting patch.

Comment 5

[Jonathan Griffin (:jgriffin)]

This patch does bump the versions of manifestparser, mozprocess, and mozrunner, as well as include the new modules mozcrash and moztest.

Comment 6

[Jeff Hammel]

So looking very casually, it looks like pypi needs version bumps for at least:

moztest 0.1 : pypi version None
mozcrash 0.1 : pypi version None
mozrunner 5.11 : pypi version 5.10
mozprocess 0.6 : pypi version 0.5

Possibly more.  I'll look at the patch and see if I missed any.

Comment 7

[Jeff Hammel]

Comment on attachment 658963 [details] [diff] [review]
mirror mozbase to m-c

I am somewhat concerned that the 'abc' module is 2.6+ and I'm not sure what we require atm.  Not sure if its worth a separate bug or not.

I believe my list is probably "good enough".  If we want to be more thorough, we can bump all of the software that has had setup.py touched.

Comment 8

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #6)
> So looking very casually, it looks like pypi needs version bumps for at
> least:
> 
> moztest 0.1 : pypi version None
> mozcrash 0.1 : pypi version None
> mozrunner 5.11 : pypi version 5.10
> mozprocess 0.6 : pypi version 0.5
> 
> Possibly more.  I'll look at the patch and see if I missed any.

I've version-bumped and tagged mozrunner and mozprocess and released to pypi.  moztest and mozcrash will also need this treatment on release to m-c.

A real disadvantage of our versioning approach, ABICT, is that one cannot release to m-c and tag at that exact changeset.  In other words, we should probably tag these first and then release to m-c.  In other other words, we should come up with a strategy that, ideally, we could have exact versions living in m-c.  I'm open to ideas.

Comment 9

[Jonathan Griffin (:jgriffin)]

Attachment: mirror mozbase to m-c, v0.2

New version with the moztest changes.  I'll push to try as well.

Comment 10

[Jonathan Griffin (:jgriffin)]

pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=64d8fd8049d0

Comment 11

[Jonathan Griffin (:jgriffin)]

Attachment: mirror mozbase to m-c, v0.3

Very minor update to pick up https://github.com/mozilla/mozbase/commit/a5f76f09bf367df9da277e8c9c0ad02ff457c8f8

Comment 12

[Jonathan Griffin (:jgriffin)]

Attachment: mirror mozbase to m-c, v0.4

Pick up one more moztest change:  https://github.com/mozilla/mozbase/commit/30a2b278430e0ab9946e0e93197b3f8ded8c634a

This is the last one, I swear!

Comment 13

[Jeff Hammel]

Comment on attachment 659300 [details] [diff] [review]
mirror mozbase to m-c, v0.4

So this doesn't delete the old testing files in manifestdestiny since they were changed to unittest vs doctest.  This is actually my fault, or rather the fault of my script.

Comment 14

[Jonathan Griffin (:jgriffin)]

Attachment: mirror mozbase to m-c, v0.5

The raw patch created by mc-diff.py nukes testing/mozbase/Makefile.in.  I don't think that's what we want, so I stripped that change out of this patch.  Other than that, I *think* it's doing the right thing.

Comment 15

[Mozilla RelEng Bot]

Try run for 64d8fd8049d0 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=64d8fd8049d0
Results (out of 189 total builds):
    success: 170
    warnings: 19
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-64d8fd8049d0

Comment 16

[Jeff Hammel]

(In reply to Jonathan Griffin (:jgriffin) from comment #14)
> Created attachment 659331 [details] [diff] [review]
> mirror mozbase to m-c, v0.5
> 
> The raw patch created by mc-diff.py nukes testing/mozbase/Makefile.in.  I
> don't think that's what we want, so I stripped that change out of this
> patch.  Other than that, I *think* it's doing the right thing.

A ha!  I knew there was a reason that I didn't just delete the damn thing to begin with. :( I'll fix the script.  Good catch!

Comment 17

[Jeff Hammel]

I believe i fixed the script.  Now on to the review

Comment 18

[Jeff Hammel]

Comment on attachment 659331 [details] [diff] [review]
mirror mozbase to m-c, v0.5

r+ ; the try run is a little scary, but ABICT not this patch's fault

Comment 19

[Jonathan Griffin (:jgriffin)]

Thanks Jeff.

https://hg.mozilla.org/integration/mozilla-inbound/rev/69243e829931

Comment 20

[Jonathan Griffin (:jgriffin)]

backed out due to a strange build error on OSX10.7 which didn't appear on try:

https://hg.mozilla.org/integration/mozilla-inbound/rev/b2ee42b9b749

The error is:

if test -d obj-firefox/i386/dist/test-package-stage -a                 \
                -d obj-firefox/x86_64/dist/test-package-stage; then              \
           cp obj-firefox/i386/dist/test-package-stage/mochitest/automation.py \
             obj-firefox/x86_64/dist/test-package-stage/mochitest/;              \
           cp obj-firefox/i386/dist/test-package-stage/reftest/automation.py   \
             obj-firefox/x86_64/dist/test-package-stage/reftest/;                \
           /builds/slave/m-in-osx64/build/build/macosx/universal/unify                 \
             --unify-with-sort "\.manifest$" \
             --unify-with-sort "all-test-dirs\.list$"               \
             obj-firefox/i386/dist/test-package-stage                          \
             obj-firefox/x86_64/dist/test-package-stage                          \
             obj-firefox/i386/dist/universal/test-package-stage; fi
/builds/slave/m-in-osx64/build/build/macosx/universal/unify: copyIfIdentical: files differ:
  obj-firefox/i386/dist/test-package-stage/mozbase/manifestdestiny/manifestparser/manifestparser.pyc,
  obj-firefox/x86_64/dist/test-package-stage/mozbase/manifestdestiny/manifestparser/manifestparser.pyc
make[2]: *** [postflight_all] Error 1
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2

I'll file a separate bug to figure out what's going on here.

Comment 21

[Jonathan Griffin (:jgriffin)]

Landed again as https://hg.mozilla.org/integration/mozilla-inbound/rev/34b091c8b0c8, after clobbering the slaves seems to have resolved this build problem.

The weirdness about having .pyc files in test-package-stage still exists.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/34b091c8b0c8

Comment 23

[Justin Wood (:Callek)]

This is busting SeaMonkey on win32 (including clobber). Is this a "Python version is out of date" or is there a different issue here?

e:/builds/slave/comm-cen-trunk-w32-ntly/build/objdir/mozilla/_virtualenv/Scripts/python.exe /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/client/marionette/tests/print-manifest-dirs.py \
          /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla \
          /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/client/marionette/tests/unit-tests.ini \
          | (cd /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla && xargs tar -chf -) \
          | (cd ./dist/test-package-stage/marionette/tests && tar -xf -)
make[1]: Leaving directory `/e/builds/slave/comm-cen-trunk-w32-ntly/build/objdir/mozilla'
Traceback (most recent call last):
  File "e:/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/client/marionette/tests/print-manifest-dirs.py", line 37, in <module>
    print_test_dirs(sys.argv[1], sys.argv[2])
  File "e:/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/client/marionette/tests/print-manifest-dirs.py", line 24, in print_test_dirs
    manifest.read(manifest_file)
  File "e:\builds\slave\comm-cen-trunk-w32-ntly\build\mozilla\testing\mozbase\manifestdestiny\manifestparser\manifestparser.py", line 473, in read
    self._read(here, filename, defaults)
  File "e:\builds\slave\comm-cen-trunk-w32-ntly\build\mozilla\testing\mozbase\manifestdestiny\manifestparser\manifestparser.py", line 429, in _read
    self._read(root, include_file, include_defaults)
  File "e:\builds\slave\comm-cen-trunk-w32-ntly\build\mozilla\testing\mozbase\manifestdestiny\manifestparser\manifestparser.py", line 444, in _read
    relpath = os.path.relpath(path, self.rootdir)
AttributeError: 'module' object has no attribute 'relpath'
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
tar: This does not look like a tar archive
tar: Error exit delayed from previous errors
make[1]: *** [stage-marionette] Error 2
make: *** [package-tests] Error 2
program finished with exit code 2

Comment 24

[Jeff Hammel]

(In reply to Justin Wood (:Callek) from comment #23)
> This is busting SeaMonkey on win32 (including clobber). Is this a "Python
> version is out of date" or is there a different issue here?
> 
> e:/builds/slave/comm-cen-trunk-w32-ntly/build/objdir/mozilla/_virtualenv/
> Scripts/python.exe
> /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/
> client/marionette/tests/print-manifest-dirs.py \
>           /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla \
>          
> /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/
> client/marionette/tests/unit-tests.ini \
>           | (cd /e/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla &&
> xargs tar -chf -) \
>           | (cd ./dist/test-package-stage/marionette/tests && tar -xf -)
> make[1]: Leaving directory
> `/e/builds/slave/comm-cen-trunk-w32-ntly/build/objdir/mozilla'
> Traceback (most recent call last):
>   File
> "e:/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/
> client/marionette/tests/print-manifest-dirs.py", line 37, in <module>
>     print_test_dirs(sys.argv[1], sys.argv[2])
>   File
> "e:/builds/slave/comm-cen-trunk-w32-ntly/build/mozilla/testing/marionette/
> client/marionette/tests/print-manifest-dirs.py", line 24, in print_test_dirs
>     manifest.read(manifest_file)
>   File
> "e:\builds\slave\comm-cen-trunk-w32-
> ntly\build\mozilla\testing\mozbase\manifestdestiny\manifestparser\manifestpar
> ser.py", line 473, in read
>     self._read(here, filename, defaults)
>   File
> "e:\builds\slave\comm-cen-trunk-w32-
> ntly\build\mozilla\testing\mozbase\manifestdestiny\manifestparser\manifestpar
> ser.py", line 429, in _read
>     self._read(root, include_file, include_defaults)
>   File
> "e:\builds\slave\comm-cen-trunk-w32-
> ntly\build\mozilla\testing\mozbase\manifestdestiny\manifestparser\manifestpar
> ser.py", line 444, in _read
>     relpath = os.path.relpath(path, self.rootdir)
> AttributeError: 'module' object has no attribute 'relpath'
> tar: Cowardly refusing to create an empty archive
> Try `tar --help' or `tar --usage' for more information.
> tar: This does not look like a tar archive
> tar: Error exit delayed from previous errors
> make[1]: *** [stage-marionette] Error 2
> make: *** [package-tests] Error 2
> program finished with exit code 2

Beh, this is my mistake.  os.path.relpath is 2.6+.  We only require 2.5 so I've backported the function, but stupidly refused to call it. See bug 789961 . I'll fix that and mirror...again.

Comment 25

[Jeff Hammel]

This will be mirrored to m-c with bug 790018