1272467 - list_builder_differences is not working for me

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

trying to validate a patch, and I get:
jmaher@jmaher-ThinkPad-X230:~/mozilla/braindump/community$ ./list_builder_differences.sh -j /home/jmaher/mozilla/buildbot-configs/.hg/patches/blank.patch 
/home/jmaher/.mozilla/releng/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/home/jmaher/.mozilla/releng/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/home/jmaher/.mozilla/releng/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/home/jmaher/.mozilla/releng/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/home/jmaher/.mozilla/releng/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
resolving manifests
getting .hgtags
getting mozilla-tests/b2g_config.py
getting mozilla-tests/config.py
getting mozilla-tests/mobile_config.py
getting mozilla-tests/production_config.py
getting mozilla/b2g_config.py
getting mozilla/config.py
getting mozilla/production_config.py
getting mozilla/release-fennec-mozilla-beta.py
getting mozilla/release-fennec-mozilla-release.py
getting mozilla/release-firefox-mozilla-beta.py
getting mozilla/release-firefox-mozilla-beta.py.template
getting mozilla/release-firefox-mozilla-release.py
getting mozilla/release-firefox-mozilla-release.py.template
getting mozilla/release-thunderbird-comm-esr38.py
15 files updated, 0 files merged, 0 files removed, 0 files unresolved
resolving manifests
getting .hgtags
getting common.py
getting misc.py
getting process/factory.py
getting process/release.py
5 files updated, 0 files merged, 0 files removed, 0 files unresolved
Patching /home/jmaher/.mozilla/releng/repos/buildbot-configs...
patching file mozilla-tests/config.py
outputting to /home/jmaher/mozilla/braindump/community/allthethings_with_patch.json
wget: /home/jmaher/.netrc:2: unknown token "api.heroku.com"
Checking 18 masters...
bm01-tests1-linux32 bm51-tests1-linux64 bm69-tests1-windows bm70-build1 bm75-try1 bm81-build_scheduler bm81-tests_scheduler bm103-tests1-linux bm106-tests1-macosx bm109-tests1-windows bm01-tests1-linux32-universal bm51-tests1-linux64-universal bm69-tests1-windows-universal bm70-build1-universal bm75-try1-universal bm103-tests1-linux-universal bm106-tests1-macosx-universal bm109-tests1-windows-universal
skipping universal master bm109-tests1-windows-universal
skipping universal master bm01-tests1-linux32-universal
skipping universal master bm75-try1-universal
skipping universal master bm51-tests1-linux64-universal
skipping universal master bm69-tests1-windows-universal
skipping universal master bm70-build1-universal
skipping universal master bm106-tests1-macosx-universal
skipping universal master bm103-tests1-linux-universal
2016-05-12 15:27:04,978 - loading test-output/bm51-tests1-linux64/master.cfg
2016-05-12 15:27:04,978 - loading test-output/bm70-build1/master.cfg
2016-05-12 15:27:04,982 - loading test-output/bm69-tests1-windows/master.cfg
2016-05-12 15:27:04,984 - loading test-output/bm01-tests1-linux32/master.cfg
2016-05-12 15:27:08,166 - Couldn't load test-output/bm70-build1/master.cfg
Traceback (most recent call last):
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 111, in dump_master
    c = loadMaster(path)
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
    execfile(path, g, g)
  File "/home/jmaher/.mozilla/releng/repos/buildbot-configs/test-output/bm70-build1/master.cfg", line 142, in <module>
    secrets=getattr(passwords, 'secrets', None)
  File "/home/jmaher/.mozilla/releng/repos/buildbotcustom/process/release.py", line 948, in generateReleaseBranchObjects
    extra_args = mh_cfg.get('extra_args', ['--cfg', mh_cfg['config_file']])
KeyError: 'config_file'
Traceback (most recent call last):
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 165, in <module>
    main()
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 146, in main
    dump = dump_master(args.masters[0])
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 111, in dump_master
    c = loadMaster(path)
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
    execfile(path, g, g)
  File "/home/jmaher/.mozilla/releng/repos/buildbot-configs/test-output/bm70-build1/master.cfg", line 142, in <module>
    secrets=getattr(passwords, 'secrets', None)
  File "/home/jmaher/.mozilla/releng/repos/buildbotcustom/process/release.py", line 948, in generateReleaseBranchObjects
    extra_args = mh_cfg.get('extra_args', ['--cfg', mh_cfg['config_file']])
KeyError: 'config_file'
Traceback (most recent call last):
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 165, in <module>
    main()
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 150, in main
    dumps = pool.map(worker, args.masters)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
    raise self._value
ValueError: No JSON object could be decoded
2016-05-12 15:27:08,307 - loading test-output/bm75-try1/master.cfg
dump_allthethings.sh exit code: 1
jmaher@jmaher-ThinkPad-X230:~/mozilla/braindump/community$ 


I don't know why bm70-build1 is not reachable- is this temporary, or do I need to do something different?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:armen, did you have to do anything different to your local environment to make list_builder_differences work for you when fixing bug 1269872?

Comment 2

[Chris AtLee [:catlee]]

Traceback (most recent call last):
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 111, in dump_master
    c = loadMaster(path)
  File "/home/jmaher/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
    execfile(path, g, g)
  File "/home/jmaher/.mozilla/releng/repos/buildbot-configs/test-output/bm70-build1/master.cfg", line 142, in <module>
    secrets=getattr(passwords, 'secrets', None)
  File "/home/jmaher/.mozilla/releng/repos/buildbotcustom/process/release.py", line 948, in generateReleaseBranchObjects
    extra_args = mh_cfg.get('extra_args', ['--cfg', mh_cfg['config_file']])
KeyError: 'config_file'

Are you adding a new type of mozharness build, but haven't specified its config file?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I am testing with a blank patch that just has a comment in it, no specific changes.

Comment 4

[Armen [:armenzg]]

This got backed out: http://hg.mozilla.org/build/buildbotcustom/rev/c8883ca15b03

I wonder if I can change something to make this more obvious.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

this is working now, feel free to close it.

Comment 6

[Armen [:armenzg]]

https://hg.mozilla.org/build/braindump/rev/2a36e340a0b2e29a3a80a69d05e7efd9f8979300
Bug 1272467 - List of builders should run test-masters.sh before generating list

Comment 7

[Armen [:armenzg]]

I landed this for older version of Python (people running Ubuntu 14! :P):
https://hg.mozilla.org/build/braindump/rev/601e949bfa77

This will run test-masters.sh before trying to generate a list of builders:
https://hg.mozilla.org/build/braindump/rev/2a36e340a0b2