774817 - peptest busted

Status: RESOLVED FIXED

(Testing Graveyard :: Peptest, defect)

Description

[Aki Sasaki (not active)]

Windows:

03:47:49     INFO - Getting output from command: ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall', 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip', '--destination', 'c:\\talos-slave\\test\\build\\application']
03:47:50    ERROR - Errors received:
03:47:50     INFO - Reading from file tmpfile_stderr
03:47:50    ERROR -  Traceback (most recent call last):
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Scripts\mozinstall-script.py", line 9, in <module>
03:47:50    ERROR -      load_entry_point('mozInstall==1.1', 'console_scripts', 'mozinstall')()
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Lib\site-packages\mozinstall\mozinstall.py", line 355, in install_cli
03:47:50    ERROR -      install_path = install(src, options.dest)
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Lib\site-packages\mozinstall\mozinstall.py", line 115, in install
03:47:50    ERROR -      install_dir = _extract(src, dest)[0]
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Lib\site-packages\mozinstall\mozinstall.py", line 228, in _extract
03:47:50    ERROR -      filename = os.path.realpath(os.path.join(dest, name))
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\lib\ntpath.py", line 90, in join
03:47:50    ERROR -      assert len(path) > 0
03:47:50    ERROR -  mozinstall.mozinstall.InstallError: Failed to install "c:\talos-slave\test\build\firefox-17.0a1.en-US.win32.zip"
03:47:50    ERROR - Return code: 1

I currently suspect the new mozinstall here; Siddarth landed that on July 3.
http://hg.mozilla.org/mozilla-central/rev/a9fda5347abd
This is also breaking Jordan's desktop unit test installs in mozharness.


Snow Leopard:

03:23:11  WARNING -  PEP WARNING    | MOZMILL fail {"message":"[JavaScript Warning: \"Error: Failed to preserve wrapper of wraPEP TEST-START test_openWindow.js
03:23:11     INFO -  Exception in thread Thread-2:
03:23:11    ERROR -  Traceback (most recent call last):
03:23:11     INFO -    File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 522, in __bootstrap_inner
03:23:11     INFO -      self.run()
03:23:11     INFO -    File "/Users/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/mozrunner/runner.py", line 255, in run
03:23:11     INFO -      self.ph.waitForFinish()
03:23:11     INFO -    File "/Users/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/mozprocess/processhandler.py", line 626, in waitForFinish
03:23:11     INFO -      self.processOutputLine(line.rstrip())
03:23:11     INFO -    File "/Users/cltbld/talos-slave/test/build/tests/peptest/peptest/pepprocess.py", line 98, in processOutputLine
03:23:11     INFO -      ' | unresponsive time: ' + tokens[3].rstrip() + ' ms')
03:23:11    ERROR -  TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
03:23:11    ERROR -  PEP ERROR      | TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
03:23:11    ERROR - Return code: 2
03:23:11    ERROR - /Users/cltbld/talos-slave/test/build/venv/bin/python exited with return code 2: harness failure
03:23:11    ERROR - # TBPL FAILURE #

Comment 1

[Jordan Lund (:jlund)]

the error seems to be from the _extract method when the src is a zip, the OS is windows, and we can not use bundle.extractall (since we are using python 2.5): https://github.com/mozilla/mozbase/blob/a48971b7cd835835efaeef88e3a7c8171b926174/mozinstall/mozinstall/mozinstall.py#L138

we are calling mozinstall with: ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall', 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip', '--destination', 'c:\\talos-slave\\test\\build\\application']

where the src=firefox-17.0a1.en-US.win32.zip and dest=c:\\talos-slave\\test\\build\\application  both exist

this seems to copy one file from src zip to dest and then fall over on github URL above

Comment 2

[Jeff Hammel]

(In reply to Jordan Lund (:jlund) from comment #1)
> the error seems to be from the _extract method when the src is a zip, the OS
> is windows, and we can not use bundle.extractall (since we are using python
> 2.5):
> https://github.com/mozilla/mozbase/blob/
> a48971b7cd835835efaeef88e3a7c8171b926174/mozinstall/mozinstall/mozinstall.
> py#L138
> 
> we are calling mozinstall with:
> ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall',
> 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip',
> '--destination', 'c:\\talos-slave\\test\\build\\application']
> 
> where the src=firefox-17.0a1.en-US.win32.zip and
> dest=c:\\talos-slave\\test\\build\\application  both exist
> 
> this seems to copy one file from src zip to dest and then fall over on
> github URL above

Ah this is a stupid stupid error of someone that wrote a code path that they didn't test and overwrote the dest variable because they were greedy.  Boo.

Comment 3

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #2)
> (In reply to Jordan Lund (:jlund) from comment #1)
> > the error seems to be from the _extract method when the src is a zip, the OS
> > is windows, and we can not use bundle.extractall (since we are using python
> > 2.5):
> > https://github.com/mozilla/mozbase/blob/
> > a48971b7cd835835efaeef88e3a7c8171b926174/mozinstall/mozinstall/mozinstall.
> > py#L138
> > 
> > we are calling mozinstall with:
> > ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall',
> > 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip',
> > '--destination', 'c:\\talos-slave\\test\\build\\application']
> > 
> > where the src=firefox-17.0a1.en-US.win32.zip and
> > dest=c:\\talos-slave\\test\\build\\application  both exist
> > 
> > this seems to copy one file from src zip to dest and then fall over on
> > github URL above
> 
> Ah this is a stupid stupid error of someone that wrote a code path that they
> didn't test and overwrote the dest variable because they were greedy.  Boo.

Filed: https://bugzilla.mozilla.org/show_bug.cgi?id=774875

Comment 4

[Jeff Hammel]

incidentally, if peptest were unhidden then the try run wouldn't have been green and the offending m-c changeset wouldn't have gotten pushed

Comment 5

[Aki Sasaki (not active)]

I think I hid them at request, and don't mind if they're unhidden once green.

Comment 6

[Jordan Lund (:jlund)]

mozilla-central/mozilla-inbound/try peptests for Linux are now visible

Comment 7

[Aki Sasaki (not active)]

Attachment: may fix snow leopard

Comment 8

[Andrew Halberstadt [:ahal]]

I think the snow leopard issue is probably something that needs to be fixed in the peptest harness.

I'm not 100% sure, but this might actually break several other harnesses as they use a line of None to signal the test has stopped running.

I'll take a look into it, but it seems that maybe someone changed the EventTracer output to be a different format which is throwing off the log parsing in peptest: https://github.com/mozilla/peptest/blob/655e97306abc8d14fa5d3252e681a66ae71cfeaf/peptest/pepprocess.py#L98

Comment 9

[Jeff Hammel]

I agree with Andrew that I would be worried about the fix proposed.  That said, I'm not sure why we're getting None where we are (though I don't know peptest code).  This may be a mozprocess bug. FWIW, I hope its only a peptest bug ;)

Has anyone been able to reproduce this outside of automation?  Is this an OSX only issue?

Comment 10

[Andrew Halberstadt [:ahal]]

Unfortunately I can't seem to reproduce the Mac issue on my OSX 10.7 machine :(

Comment 11

[Andrew Halberstadt [:ahal]]

I'm going to check in a patch to peptest that uses string substitutions instead so hopefully it will fail a little bit better.

Comment 12

[Andrew Halberstadt [:ahal]]

The fix in bug 774875 was merged into m-c on Friday, yet windows builds are still failing with: >PEP ERROR      | OSError: IO Completion Port failed to signal process shutdown

There was a recent change to mozprocess to fix a regression caused by Flash, this might have something to do with it. Investigating further.

Comment 13

[Andrew Halberstadt [:ahal]]

With the landing of bug 778133 it looks like peptest is now green again on all platforms: https://tbpl.mozilla.org/?noignore=1&rev=89dcadd42ec4

Aki: Would it be possible to unhide it on Windows and OSX as well so stuff like this is less likely to happen in the future?

Comment 14

[Aki Sasaki (not active)]

Unhidden.
Thanks for greening those up!

Comment 15

[Ed Morley [:emorley]]

(In reply to Aki Sasaki [:aki] from comment #14)
> Unhidden.
> Thanks for greening those up!

Re hiding the OSX 10.5 peptest, since all 10.5 builds are busted (they should be switched off already, but bug 773120 didn't stick the first time).

Please can you ping me on IRC or CC me on any hides/unhides just so I can keep track of things like this (there are often other trees that need keeping on sync etc) :-)

Comment 16

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #15)
> Re hiding the OSX 10.5 peptest, since all 10.5 builds are busted (they
> should be switched off already, but bug 773120 didn't stick the first time).

Also spotted WINNT 6.1 x64 peptest was shown on mozilla-central - have rehidden since we show builds only for windows x64 (given our current Windows x64 stance & the unreliability of suites on that platform on tinderbox).

Comment 17

[Aki Sasaki (not active)]

Sure, sorry about that.

I think we're done here; time to resolve this bug?

Comment 18

[Ed Morley [:emorley]]

Np :-)

Yup think all done; resolving.