Closed Bug 774817 Opened 8 years ago Closed 8 years ago

peptest busted

Categories

(Testing Graveyard :: Peptest, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: aki, Unassigned)

References

Details

Attachments

(1 file)

Windows:

03:47:49     INFO - Getting output from command: ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall', 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip', '--destination', 'c:\\talos-slave\\test\\build\\application']
03:47:50    ERROR - Errors received:
03:47:50     INFO - Reading from file tmpfile_stderr
03:47:50    ERROR -  Traceback (most recent call last):
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Scripts\mozinstall-script.py", line 9, in <module>
03:47:50    ERROR -      load_entry_point('mozInstall==1.1', 'console_scripts', 'mozinstall')()
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Lib\site-packages\mozinstall\mozinstall.py", line 355, in install_cli
03:47:50    ERROR -      install_path = install(src, options.dest)
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Lib\site-packages\mozinstall\mozinstall.py", line 115, in install
03:47:50    ERROR -      install_dir = _extract(src, dest)[0]
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\Lib\site-packages\mozinstall\mozinstall.py", line 228, in _extract
03:47:50    ERROR -      filename = os.path.realpath(os.path.join(dest, name))
03:47:50    ERROR -    File "c:\talos-slave\test\build\venv\lib\ntpath.py", line 90, in join
03:47:50    ERROR -      assert len(path) > 0
03:47:50    ERROR -  mozinstall.mozinstall.InstallError: Failed to install "c:\talos-slave\test\build\firefox-17.0a1.en-US.win32.zip"
03:47:50    ERROR - Return code: 1

I currently suspect the new mozinstall here; Siddarth landed that on July 3.
http://hg.mozilla.org/mozilla-central/rev/a9fda5347abd
This is also breaking Jordan's desktop unit test installs in mozharness.


Snow Leopard:

03:23:11  WARNING -  PEP WARNING    | MOZMILL fail {"message":"[JavaScript Warning: \"Error: Failed to preserve wrapper of wraPEP TEST-START test_openWindow.js
03:23:11     INFO -  Exception in thread Thread-2:
03:23:11    ERROR -  Traceback (most recent call last):
03:23:11     INFO -    File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 522, in __bootstrap_inner
03:23:11     INFO -      self.run()
03:23:11     INFO -    File "/Users/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/mozrunner/runner.py", line 255, in run
03:23:11     INFO -      self.ph.waitForFinish()
03:23:11     INFO -    File "/Users/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/mozprocess/processhandler.py", line 626, in waitForFinish
03:23:11     INFO -      self.processOutputLine(line.rstrip())
03:23:11     INFO -    File "/Users/cltbld/talos-slave/test/build/tests/peptest/peptest/pepprocess.py", line 98, in processOutputLine
03:23:11     INFO -      ' | unresponsive time: ' + tokens[3].rstrip() + ' ms')
03:23:11    ERROR -  TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
03:23:11    ERROR -  PEP ERROR      | TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
03:23:11    ERROR - Return code: 2
03:23:11    ERROR - /Users/cltbld/talos-slave/test/build/venv/bin/python exited with return code 2: harness failure
03:23:11    ERROR - # TBPL FAILURE #
the error seems to be from the _extract method when the src is a zip, the OS is windows, and we can not use bundle.extractall (since we are using python 2.5): https://github.com/mozilla/mozbase/blob/a48971b7cd835835efaeef88e3a7c8171b926174/mozinstall/mozinstall/mozinstall.py#L138

we are calling mozinstall with: ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall', 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip', '--destination', 'c:\\talos-slave\\test\\build\\application']

where the src=firefox-17.0a1.en-US.win32.zip and dest=c:\\talos-slave\\test\\build\\application  both exist

this seems to copy one file from src zip to dest and then fall over on github URL above
(In reply to Jordan Lund (:jlund) from comment #1)
> the error seems to be from the _extract method when the src is a zip, the OS
> is windows, and we can not use bundle.extractall (since we are using python
> 2.5):
> https://github.com/mozilla/mozbase/blob/
> a48971b7cd835835efaeef88e3a7c8171b926174/mozinstall/mozinstall/mozinstall.
> py#L138
> 
> we are calling mozinstall with:
> ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall',
> 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip',
> '--destination', 'c:\\talos-slave\\test\\build\\application']
> 
> where the src=firefox-17.0a1.en-US.win32.zip and
> dest=c:\\talos-slave\\test\\build\\application  both exist
> 
> this seems to copy one file from src zip to dest and then fall over on
> github URL above

Ah this is a stupid stupid error of someone that wrote a code path that they didn't test and overwrote the dest variable because they were greedy.  Boo.
(In reply to Jeff Hammel [:jhammel] from comment #2)
> (In reply to Jordan Lund (:jlund) from comment #1)
> > the error seems to be from the _extract method when the src is a zip, the OS
> > is windows, and we can not use bundle.extractall (since we are using python
> > 2.5):
> > https://github.com/mozilla/mozbase/blob/
> > a48971b7cd835835efaeef88e3a7c8171b926174/mozinstall/mozinstall/mozinstall.
> > py#L138
> > 
> > we are calling mozinstall with:
> > ['c:\\talos-slave\\test\\build\\venv\\Scripts\\mozinstall',
> > 'c:\\talos-slave\\test\\build\\firefox-17.0a1.en-US.win32.zip',
> > '--destination', 'c:\\talos-slave\\test\\build\\application']
> > 
> > where the src=firefox-17.0a1.en-US.win32.zip and
> > dest=c:\\talos-slave\\test\\build\\application  both exist
> > 
> > this seems to copy one file from src zip to dest and then fall over on
> > github URL above
> 
> Ah this is a stupid stupid error of someone that wrote a code path that they
> didn't test and overwrote the dest variable because they were greedy.  Boo.

Filed: https://bugzilla.mozilla.org/show_bug.cgi?id=774875
Depends on: 774875
No longer depends on: 774875
incidentally, if peptest were unhidden then the try run wouldn't have been green and the offending m-c changeset wouldn't have gotten pushed
Depends on: 774875
I think I hid them at request, and don't mind if they're unhidden once green.
Depends on: 775127
mozilla-central/mozilla-inbound/try peptests for Linux are now visible
Attachment #643690 - Flags: review?(jhammel)
I think the snow leopard issue is probably something that needs to be fixed in the peptest harness.

I'm not 100% sure, but this might actually break several other harnesses as they use a line of None to signal the test has stopped running.

I'll take a look into it, but it seems that maybe someone changed the EventTracer output to be a different format which is throwing off the log parsing in peptest: https://github.com/mozilla/peptest/blob/655e97306abc8d14fa5d3252e681a66ae71cfeaf/peptest/pepprocess.py#L98
Attachment #643690 - Flags: review?(jhammel)
I agree with Andrew that I would be worried about the fix proposed.  That said, I'm not sure why we're getting None where we are (though I don't know peptest code).  This may be a mozprocess bug. FWIW, I hope its only a peptest bug ;)

Has anyone been able to reproduce this outside of automation?  Is this an OSX only issue?
Unfortunately I can't seem to reproduce the Mac issue on my OSX 10.7 machine :(
I'm going to check in a patch to peptest that uses string substitutions instead so hopefully it will fail a little bit better.
The fix in bug 774875 was merged into m-c on Friday, yet windows builds are still failing with: >PEP ERROR      | OSError: IO Completion Port failed to signal process shutdown

There was a recent change to mozprocess to fix a regression caused by Flash, this might have something to do with it. Investigating further.
Depends on: 776931
Depends on: 778133
With the landing of bug 778133 it looks like peptest is now green again on all platforms: https://tbpl.mozilla.org/?noignore=1&rev=89dcadd42ec4

Aki: Would it be possible to unhide it on Windows and OSX as well so stuff like this is less likely to happen in the future?
Unhidden.
Thanks for greening those up!
(In reply to Aki Sasaki [:aki] from comment #14)
> Unhidden.
> Thanks for greening those up!

Re hiding the OSX 10.5 peptest, since all 10.5 builds are busted (they should be switched off already, but bug 773120 didn't stick the first time).

Please can you ping me on IRC or CC me on any hides/unhides just so I can keep track of things like this (there are often other trees that need keeping on sync etc) :-)
(In reply to Ed Morley [:edmorley] from comment #15)
> Re hiding the OSX 10.5 peptest, since all 10.5 builds are busted (they
> should be switched off already, but bug 773120 didn't stick the first time).

Also spotted WINNT 6.1 x64 peptest was shown on mozilla-central - have rehidden since we show builds only for windows x64 (given our current Windows x64 stance & the unreliability of suites on that platform on tinderbox).
Sure, sorry about that.

I think we're done here; time to resolve this bug?
Np :-)

Yup think all done; resolving.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.