1651058 - Can't run mozregression builds on Windows - AttributeError: 'Process' object has no attribute '_job'

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[:Gijs (he/him)]

$ mozregression --launch 2020-01-03
 0:00.59 INFO: Downloading build from: https://archive.mozilla.org/pub/firefox/nightly/2020/01/2020-01-03-20-22-40-mozilla-central/firefox-73.0a1.en-US.win32.zip
===== Downloaded 100% =====
 0:12.73 INFO: Running mozilla-central build for 2020-01-03
 0:15.45 INFO: Launching path\to\home\AppData\Local\Temp\tmp8bz928lp\firefox\firefox.exe
 0:15.45 INFO: Application command: path\to\home\AppData\Local\Temp\tmp8bz928lp\firefox\firefox.exe --wait-for-browser -profile path\to\home\AppData\Local\Temp\tmppe0m8c30.mozrunner
 0:15.45 ERROR: Unable to start the application (error: 'str' object has no attribute '__traceback__')
 0:15.50 ERROR: Unable to start the application (error: 'str' object has no attribute '__traceback__')
Exception ignored in: <bound method ProcessHandlerMixin.Process.__del__ of <mozprocess.processhandler.ProcessHandlerMixin.Process object at 0x000002491AAD9940>>
Traceback (most recent call last):
  File "path\to\mozilla-build\python3\lib\site-packages\mozprocess\processhandler.py", line 143, in __del__
    if handle or self._job or self._io_port:
AttributeError: 'Process' object has no attribute '_job'

I get the same error for builds that are already downloaded/cached.

Comment 1

[William Lachance (:wlach)]

Looks like a mozprocess bug of some kind, but not seeing any culprits in the commit log:

https://hg.mozilla.org/mozilla-central/log/tip/testing/mozbase/mozprocess/mozprocess/processhandler.py

:rstewart, :ahal can you investigate? It is possible we just need to release a newer version of mozprocess? The last release I see is from December 2019: https://pypi.org/project/mozprocess/#history

Comment 2

[Ricky Stewart]

The mozprocess code has changed a great deal since then, so I suppose a new release would be prudent if not necessary. I do notice that this is being installed as a Python 3 package, while I didn't do a pass to make it Python 3 compatible until late march (bug 1622021).

I don't know anything about the release process for this (it was news to me that we even release this as a pypi package as recently as a month or two ago), and further I have no experience in the Windows-specific part of this codebase (which is what this bug pertains to), so if ahal has any thoughts they have a good chance of being more informed than mine.

Comment 3

[William Lachance (:wlach)]

(In reply to Ricky Stewart from comment #2)

The mozprocess code has changed a great deal since then, so I suppose a new release would be prudent if not necessary. I do notice that this is being installed as a Python 3 package, while I didn't do a pass to make it Python 3 compatible until late march (bug 1622021).

I don't know anything about the release process for this (it was news to me that we even release this as a pypi package as recently as a month or two ago), and further I have no experience in the Windows-specific part of this codebase (which is what this bug pertains to), so if ahal has any thoughts they have a good chance of being more informed than mine.

Looking again at the traceback, it appears some error output (0:15.45 ERROR: Unable to start the application (error: 'str' object has no attribute '__traceback__')) is being truncated due to bug 1622637 (which I actually fixed, but I don't think we did up a new release of mozrunner yet):

https://pypi.org/project/mozrunner/#history

Anyway, seeing what :ahal has to say sounds like a fine idea. Thanks for the fast response

Comment 4

[Andrew Halberstadt [:ahal]]

Sure we can do a release, doesn't hurt at least. Do we need both mozprocess and mozrunner then? We'll probably also need to bump the minimum supported version of mozprocess in mozrunner.

Comment 5

[William Lachance (:wlach)]

(In reply to Andrew Halberstadt [:ahal] from comment #4)

Sure we can do a release, doesn't hurt at least. Do we need both mozprocess and mozrunner then? We'll probably also need to bump the minimum supported version of mozprocess in mozrunner.

Bumping mozrunner would be good from the p.o.v. of tracking down the underlying issue. For bumping the versions and uploading to pypi, can you file a new bug and make it block this one? Unfortunately I'm suspecting that this won't address the root cause.

Comment 6

[Bob Clary [:bc] (inactive)]

I have some changes pending for mozrunner related to the mozdevice updates to support unrooted devices in bug 1486004. I will be putting the patches up for review later today / early tomorrow so if you can wait a day that would be great to pick those changes up. I'll be doing a PR for mozregression as well.

Comment 7

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #1)

Looks like a mozprocess bug of some kind, but not seeing any culprits in the commit log:

https://hg.mozilla.org/mozilla-central/log/tip/testing/mozbase/mozprocess/mozprocess/processhandler.py

:rstewart, :ahal can you investigate? It is possible we just need to release a newer version of mozprocess? The last release I see is from December 2019: https://pypi.org/project/mozprocess/#history

Just as a note, I'm not able to reproduce this on my Windows 10 image (using python 3.8). There must be something up, but am not sure what it is. I think once we have a new version of mozrunner out this will be easier to reproduce.

Comment 8

[William Lachance (:wlach)]

:gijs, can you try upgrading mozrunner and mozprocess and seeing if you still see the error?

Comment 9

[:Gijs (he/him)]

$ mozregression --launch 2020-01-01
 0:02.59 INFO: Using local file: path\to\cache-for-mozregression\2020-01-01--mozilla-central--firefox-73.0a1.en-US.win32.zip
 0:02.59 INFO: Running mozilla-central build for 2020-01-01
 0:05.13 INFO: Launching path\to\home\dir\AppData\Local\Temp\tmp5ohvu18e\firefox\firefox.exe
 0:05.13 INFO: Application command: path\to\home\dir\AppData\Local\Temp\tmp5ohvu18e\firefox\firefox.exe --wait-for-browser -profile path\to\home\dir\AppData\Local\Temp\tmpbzg798f_.mozrunner
 0:05.13 ERROR: Unable to start the application (error: 'str' object has no attribute '__traceback__')
 0:05.20 ERROR: Unable to start the application (error: 'str' object has no attribute '__traceback__')
Exception ignored in: <bound method ProcessHandlerMixin.Process.__del__ of <mozprocess.processhandler.ProcessHandlerMixin.Process object at 0x0000020F92B7A9E8>>
Traceback (most recent call last):
  File "path\to\mozilla-build\python3\lib\site-packages\mozprocess\processhandler.py", line 143, in __del__
    if handle or self._job or self._io_port:
AttributeError: 'Process' object has no attribute '_job'

$ mozregression --version
4.0.9

So no, it's still broken.

Comment 10

[:Gijs (he/him)]

Attachment: Upgrade log

This was the upgrade log, in case that's any help?

Comment 11

[William Lachance (:wlach)]

(In reply to :Gijs (he/him) from comment #9)

$ mozregression --launch 2020-01-01
 0:02.59 INFO: Using local file: path\to\cache-for-mozregression\2020-01-01--mozilla-central--firefox-73.0a1.en-US.win32.zip
 0:02.59 INFO: Running mozilla-central build for 2020-01-01
 0:05.13 INFO: Launching path\to\home\dir\AppData\Local\Temp\tmp5ohvu18e\firefox\firefox.exe
 0:05.13 INFO: Application command: path\to\home\dir\AppData\Local\Temp\tmp5ohvu18e\firefox\firefox.exe --wait-for-browser -profile path\to\home\dir\AppData\Local\Temp\tmpbzg798f_.mozrunner
 0:05.13 ERROR: Unable to start the application (error: 'str' object has no attribute '__traceback__')
 0:05.20 ERROR: Unable to start the application (error: 'str' object has no attribute '__traceback__')
Exception ignored in: <bound method ProcessHandlerMixin.Process.__del__ of <mozprocess.processhandler.ProcessHandlerMixin.Process object at 0x0000020F92B7A9E8>>
Traceback (most recent call last):
  File "path\to\mozilla-build\python3\lib\site-packages\mozprocess\processhandler.py", line 143, in __del__
    if handle or self._job or self._io_port:
AttributeError: 'Process' object has no attribute '_job'

$ mozregression --version
4.0.9

So no, it's still broken.

Judging by the error message (error: 'str' object has no attribute '__traceback__') and the log you posted, you must still be using the old version of mozprocess. Can you try upgrading to version 1.2.0 (pip install mozprocess==1.2.0)? At the very least that will give us a more understandable error.

Comment 12

[:Gijs (he/him)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #11)

Judging by the error message (error: 'str' object has no attribute '__traceback__') and the log you posted, you must still be using the old version of mozprocess. Can you try upgrading to version 1.2.0 (pip install mozprocess==1.2.0)? At the very least that will give us a more understandable error.

Oh, right, sorry - I assumed that the higher version of mozregression would require the higher version of mozprocess and this would happen automatically. If I manually upgrade mozprocess, then things work (not even a more understandable error - they just work...).

Do I just misunderstand how python dependencies work? :-\

Comment 13

[William Lachance (:wlach)]

(In reply to :Gijs (he/him) from comment #12)

(In reply to William Lachance (:wlach) (use needinfo!) from comment #11)

Judging by the error message (error: 'str' object has no attribute '__traceback__') and the log you posted, you must still be using the old version of mozprocess. Can you try upgrading to version 1.2.0 (pip install mozprocess==1.2.0)? At the very least that will give us a more understandable error.

Oh, right, sorry - I assumed that the higher version of mozregression would require the higher version of mozprocess and this would happen automatically. If I manually upgrade mozprocess, then things work (not even a more understandable error - they just work...).

Do I just misunderstand how python dependencies work? :-\

No problem! I actually didn't bump the requirement because I wasn't sure if it solved any problem, so it's no surprise you didn't get it automatically. Anyway, this is great news-- it means the issue is fixed, whatever it was. I'll go ahead and bump the requirement inside mozregression itself.

Comment 14

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/mozregression/pull/696

Comment 15

[William Lachance (:wlach)]

https://github.com/mozilla/mozregression/commit/fd7e043dc30a3e7a221187b0ae7672cf7ab51a76