903616 - First make mozmill after a build fails with "ImportError: No module named mozcrash"

Status: RESOLVED DUPLICATE of bug 1121107

(Thunderbird :: Testing Infrastructure, defect)

Description

[Jens Müller (:tessarakt)]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20130805 Firefox/24.0 (Nightly/Aurora)
Build ID: 20130805004006

Steps to reproduce:

1. mozilla/mach build in a current comm-central checkout (with some custom build config)
2. cd into the object directory
3. make SOLO_TEST=content-tabs/test-plugin-crashing.js mozmill-one
4. Another time: make SOLO_TEST=content-tabs/test-plugin-crashing.js mozmill-one


Actual results:

The first time:

unset PYTHONHOME && cd ./mozilla/_tests/mozmill && MACOSX_DEPLOYMENT_TARGET= \
        /home/jens/devel/releases-comm-central/obj-thunderbird/./mozilla/_tests/mozmill/../mozmill-virtualenv/bin/python runtest.py \
        --test=/home/jens/devel/releases-comm-central/mail/test/mozmill/content-tabs/test-plugin-crashing.js \
        --binary=../../.././mozilla/dist/bin/thunderbird \
        --symbols-path=/home/jens/devel/releases-comm-central/obj-thunderbird/./mozilla/dist/crashreporter-symbols \
        --plugins-path=/home/jens/devel/releases-comm-central/obj-thunderbird/./mozilla/dist/plugins \

Traceback (most recent call last):
  File "runtest.py", line 28, in <module>
    from automation import Automation
  File "/home/jens/devel/releases-comm-central/obj-thunderbird/mozilla/_tests/mozmill/automation.py", line 33, in <module>
    import mozcrash
ImportError: No module named mozcrash
make: *** [mozmill-one] Fehler 1

The second time:

Test was executed (with an TEST-UNEXPECTED-FAIL, but that's a different issue).


Expected results:

The test should have been executed at the first make mozmill-one.

Comment 1

[Jens Müller (:tessarakt)]

I copied the object directory directly after a fresh build and compared it with the state after running "make SOLO_TEST=content-tabs/test-plugin-crashing.js mozmill-one" once.

This is the result:

$ diff -u -r obj-thunderbird obj-thunderbird-before-first-mozmill-test/
Nur in obj-thunderbird/mozilla/_tests/mozmill: automation.pyc.
Nur in obj-thunderbird/mozilla/_tests/mozmill: automationutils.pyc.

Comment 2

[Jens Müller (:tessarakt)]

Hi Mark,

do you have an idea how to make sure that these .pyc files are created early enough?

BR, Jens

Comment 3

[Mark Banner (:standard8)]

I have no idea, but this sounds more like there being some sort of path issue the first time around - you shouldn't have to compile the files in order to get it to work (and it works on the builders first time round).

Comment 4

[Jens Müller (:tessarakt)]

(In reply to Mark Banner (:standard8) from comment #3)
> I have no idea, but this sounds more like there being some sort of path
> issue the first time around - 

Good idea. I have compared the file system, next I should just compare the environment variables.

> you shouldn't have to compile the files in
> order to get it to work (and it works on the builders first time round).

Looking at https://tbpl.mozilla.org/php/getParsedLog.php?id=26304062&tree=Thunderbird-Trunk&full=1,  it seems to me that the builders install an already packaged TB and then install the Mozmill tests separately, instead of running "make mozmill" from the obj directory. So this is a different scenario.

Comment 5

[Jens Müller (:tessarakt)]

aceman just told me on IRC that he also had this issue, but did not see it any longer after updating his checkout today. I will try again myself, and if it is also fixed for me, I will resolve this bug.

Comment 6

[Jens Müller (:tessarakt)]

Actually, no change for me.

aceman, can you double-check that this is indeed fixed for you, please?

Comment 7

[Jens Müller (:tessarakt)]

See also bug 849900 comment 1.

Comment 8

[:aceman]

I don't know if it is fixed but I couldn't see it for 2 days in a row. I build TB and run "make mozmill" and it runs fine. xpcshell still breaks on importing other modules, but this "mozcrash" didn't appear for me now.

Comment 9

[:aceman]

I still do not see the problem anymore. Do you?

Comment 10

[:aceman]

OK, today I suddenly got the error again. Running mozmill again worked after that.

Comment 11

[Jesse Ruderman]

automation.py in A is a symlink to automation.py in B.  The first time I run it from A, it fails because __file__ refers to B, which doesn't have a corresponding mozbase directory.  The second time I run it, it succeeds because the .pyc file is in A.

(A) /Users/jruderman/builds/mozilla-central-debug/_tests/reftest

(B) /Users/jruderman/builds/mozilla-central-debug/layout/tools/reftest

Comment 12

[(not currently active) Ted Mielczarek]

That shouldn't matter though, since the mozbase modules should be available in the virtualenv. That hack is only for our tests running from the test package without a virtualenv.

Comment 13

[ISHIKAWA, Chiaki]

I have noticed this bug since this summer.
Recently, the number of unsuccessful timeout tests diminished on linux to the point
that I noticed the discrepancies of the tests executed in two successive runs. 
That is how I noticed this bug. [ The echoed line to invoke mozmill test was too verbose to notice
this mozcrash error line easily :-( ]

The downside of this bug is that my test cycle on local PC is like:

 edit 
   - run |make -f client.mk|  to compile (and re-create virtualenv for testing)
      - run |make mozmill|
         - run my local summary script to look at the errors/warnings.

which means, that for a long time, the first test target directory,
i.e., /account/ was NOT TESTED AT ALL!!!
(Funny, doesn't try server has this issue?)

Only if I decide to run |make mozmill| to check for the changed behavior in
the linux-specific timeout errors, did test /account/ directory get executed.
Of course, I can also use SOLO_TEST feature to test /account/ directory only.


Whatever the fix, how kludge-like or quick hack may it be, I will 
appreciate to see it in the tree.

TIA
 

PS: This is from my |make mozmill| log.

The first time |make mozmill| is run after an installation of the
virtualenv by |make -f client.mk|.

Note the error about missing |mozcrash|.


time make ${MAKEARGS}
unset PYTHONHOME && cd ./mozilla/_tests/mozmill && MACOSX_DEPLOYMENT_TARGET= \
	/REF-OBJ-DIR/objdir-tb3/./mozilla/_tests/mozmill/../mozmill-virtualenv/bin/python runtestlist.py --list=mozmilltests.list \
	--binary=../../.././mozilla/dist/bin/thunderbird \
	--dir=/REF-COMM-CENTRAL/comm-central/mail/test/mozmill \
	--symbols-path=/REF-OBJ-DIR/objdir-tb3/./mozilla/dist/crashreporter-symbols \
	--plugins-path=/REF-OBJ-DIR/objdir-tb3/./mozilla/dist/plugins \

INFO | (runtestlist.py) | Running directory: account
['/REF-OBJ-DIR/objdir-tb3/./mozilla/_tests/mozmill/../mozmill-virtualenv/bin/python', 'runtest.py', '-t', '/REF-COMM-CENTRAL/comm-central/mail/test/mozmill/account', '--binary', '../../.././mozilla/dist/bin/thunderbird', '--symbols-path', '/REF-OBJ-DIR/objdir-tb3/./mozilla/dist/crashreporter-symbols', '--plugins-path', '/REF-OBJ-DIR/objdir-tb3/./mozilla/dist/plugins']
Traceback (most recent call last):
  File "runtest.py", line 28, in <module>
    from automation import Automation
  File "/REF-OBJ-DIR/objdir-tb3/mozilla/_tests/mozmill/automation.py", line 33, in <module>
    import mozcrash
ImportError: No module named mozcrash
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
INFO | (runtestlist.py) | account: 0 passed, 0 failed
INFO | (runtestlist.py) | Running directory: addrbook
['/REF-OBJ-DIR/objdir-tb3/./mozilla/_tests/mozmill/../mozmill-virtualenv/bin/python', 'runtest.py', '-t', '/REF-COMM-CENTRAL/comm-central/mail/test/mozmill/addrbook', '--binary', '../../.././mozilla/dist/bin/thunderbird', '--symbols-path', '/REF-OBJ-DIR/objdir-tb3/./mozilla/dist/crashreporter-symbols', '--plugins-path', '/REF-OBJ-DIR/objdir-tb3/./mozilla/dist/plugins']
WARNING: conversion from native to utf-16 failed: file /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/io/nsNativeCharsetUtils.cpp, line 469
WARNING: No disk space watcher component available!: file /REF-COMM-CENTRAL/comm-central/mozilla/dom/indexedDB/IndexedDatabaseManager.cpp, line 301
1382713609342	Marionette	INFO	MarionetteComponent loaded
++DOCSHELL 0x2386ad0 == 1 [id = 1]

	... test proceeds....
	...

Comment 14

[ISHIKAWA, Chiaki]

> Only if I decide to run |make mozmill| to check for the changed behavior in
I meant to say:

Only if I decide to run |make mozmill| *AGAIN* to check for the changed behavior in

TIA

Comment 15

[Joshua Cranmer [:jcranmer]]

Will be fixed by bug 1121107.