623704 - sendchange run manually works but through release_sanity.py fails

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Armen [:armenzg]]

After release_sanity.py passes it tries to run a sendchange.
If you look at the log you will see that it fails with:
  ImportError: No module named twisted.python

If you run manually the sendchange command that release_sanity.py tried to run you will noticed that it was send successfully.

Not sure why this happens.


2011-01-06 12:45:36,963 : INFO : Executing: ['buildbot', 'sendchange', '--username', 'armenzg', '--master', 'localhost:9012', '--branch', 'users/armenzg_mozilla.com/mozilla-central', '-p', 'release_config:mozilla/staging_release-firefox-mozilla-2.0.py', '-p', 'script_repo_revision:FIREFOX_4_0b9_RELEASE', 'release_build']
2011-01-06 12:45:36,963 : INFO : command: START
2011-01-06 12:45:36,963 : INFO : command: arg0: buildbot
2011-01-06 12:45:36,963 : INFO : command: args: ['sendchange', '--username', 'armenzg', '--master', 'localhost:9012', '--branch', 'users/armenzg_mozilla.com/mozilla-central', '-p', 'release_config:mozilla/staging_release-firefox-mozilla-2.0.py', '-p', 'script_repo_revision:FIREFOX_4_0b9_RELEASE', 'release_build']
2011-01-06 12:45:36,963 : INFO : command: cwd: /builds/buildbot/builder_master3
2011-01-06 12:45:36,963 : INFO : command: output:
Traceback (most recent call last):
  File "/tools/buildbot/bin/buildbot", line 3, in <module>
    from buildbot.scripts import runner
  File "/tools/python-2.5.2/lib/python2.5/site-packages/buildbot-0.8.0-py2.5.egg/buildbot/scripts/runner.py", line 7, in <module>
    from twisted.python import usage, util, runtime
ImportError: No module named twisted.python
2011-01-06 12:45:37,007 : INFO : command: END

Traceback (most recent call last):
  File "/builds/buildbot/builder_master3/tools/buildbot-helpers/release_sanity.py", line 246, in <module>
    releaseConfigFile,
  File "/builds/buildbot/builder_master3/tools/buildbot-helpers/release_sanity.py", line 52, in sendchange
    run_cmd(cmd)
  File "/builds/buildbot/builder_master3/tools/lib/python/util/commands.py", line 35, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python-2.5.2/lib/python2.5/subprocess.py", line 462, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['buildbot', 'sendchange', '--username', 'armenzg', '--master', 'localhost:9012', '--branch', 'users/armenzg_mozilla.com/mozilla-central', '-p', 'release_config:mozilla/staging_release-firefox-mozilla-2.0.py', '-p', 'script_repo_revision:FIREFOX_4_0b9_RELEASE', 'release_build']' returned non-zero exit status 1





[cltbld@staging-master builder_master3]$ buildbot sendchange --username armenzg --master localhost:9012 --branch users/armenzg_mozilla.com/mozilla-central -p release_config:mozilla/staging_release-firefox-mozilla-2.0.py -p script_repo_revision:FIREFOX_4_0b9_RELEASE release_build
change sent successfully

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

probably PYTHONHOME/PYTHONPATH problems.

Comment 2

[[e-mail] (:bhearsum)]

staging-master doesn't have PYTHONPATH set by default, it gets done through the Makefile we use for the masters. So either you need to set it manually for running release_sanity.py, or add a Makefile target for release_sanity.py.

Comment 3

[Armen [:armenzg]]

OK. What should the right value be? I would like to document it.

I run the command like this:
PYTHONPATH=/builds/buildbot/builder_master3:/builds/buildbot/builder_master3/tools/lib/python python /builds/buildbot/builder_master3/tools/buildbot-helpers/release_sanity.py -u armenzg -V 4.0b9 --branch mozilla-central --build-number 1 -c staging_release-firefox-mozilla-2.0.py localhost:9012

and these are the different PYTHONPATH between staging-master and production-master01.

[cltbld@production-master01 ~]$ echo $PYTHONPATH
/tools/buildbotcustom

[cltbld@staging-master builder_master3]$ echo $PYTHONPATH
/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-corelib/python2.5/site-packages/:/tools/zope-interface/lib/python2.5/site-packages/:/tools/buildbotcustom

Comment 4

[[e-mail] (:bhearsum)]

I'd recommend using whatever the Makefile has.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

..which differs for different masters -- use the one that matches the master instance you're using.  There are generally some other PYTHON* variables, too, and you should set those in your shell as well.

Comment 6

[Armen [:armenzg]]

It seems that it wasn't just the PYTHONPATH but which python was getting used.

PATH = "/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin"

I had to run this to make it work:
> source /tools/buildbot-0.8.2/bin/activate

Comment 7

[Justin Wood (:Callek)]

Is this still an issue Armen.

I suspect in the "new" world of virtualenv buildbot we would be using ./bin/activate instead, but I'm not sure.

Comment 8

[Armen [:armenzg]]

One more closed!