1246671 - Failling to reconfig or generate allthethings.json until SETA comes back up

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

We won't be able to reconfig until SETA comes back up.


Failed to generate allthethings.json; dumping log.
outputting to /home/buildduty/.mozilla/releng/repos/buildbot-configs/allthethings.json
Checking 20 masters...
bm01-tests1-linux32 bm51-tests1-linux64 bm69-tests1-windows bm70-build1 bm75-try1 bm81-build_scheduler bm81-tests_scheduler bm89-tests1-panda bm103-tests1-linux bm106-tests1-macosx bm109-tests1-windows bm01-tests1-linux32-universal bm51-tests1-linux64-universal bm69-tests1-windows-universal bm70-build1-universal bm75-try1-universal bm89-tests1-panda-universal bm103-tests1-linux-universal bm106-tests1-macosx-universal bm109-tests1-windows-universal
skipping universal master bm89-tests1-panda-universal
skipping universal master bm75-try1-universal
skipping universal master bm70-build1-universal
skipping universal master bm69-tests1-windows-universal
skipping universal master bm51-tests1-linux64-universal
skipping universal master bm01-tests1-linux32-universal
skipping universal master bm103-tests1-linux-universal
skipping universal master bm106-tests1-macosx-universal
skipping universal master bm109-tests1-windows-universal
2016-02-08 08:07:38,202 - loading test-output/bm01-tests1-linux32/master.cfg
2016-02-08 08:07:42,849 - Couldn't load test-output/bm01-tests1-linux32/master.cfg
Traceback (most recent call last):
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 111, in dump_master
    c = loadMaster(path)
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
    execfile(path, g, g)
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/master.cfg", line 10, in <module>
    import config
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/config.py", line 2573, in <module>
    loadSkipConfig(BRANCHES, "desktop")
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/config_seta.py", line 138, in loadSkipConfig
    platforms = get_seta_platforms(b, platform_filter)
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/config_seta.py", line 52, in get_seta_platforms
    response = urllib2.urlopen(url)
  File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/tools/python27/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/tools/python27/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/tools/python27/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
HTTPError = 404 http://alertmanager.allizom.org/data/setadetails/?date=2016-02-08&buildbot=1&branch=fx-team&inactive=1
Traceback (most recent call last):
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 165, in <module>
    main()
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 146, in main
    dump = dump_master(args.masters[0])
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 111, in dump_master
    c = loadMaster(path)
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
    execfile(path, g, g)
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/master.cfg", line 10, in <module>
    import config
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/config.py", line 2573, in <module>
    loadSkipConfig(BRANCHES, "desktop")
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/config_seta.py", line 138, in loadSkipConfig
    platforms = get_seta_platforms(b, platform_filter)
  File "/home/buildduty/.mozilla/releng/repos/buildbot-configs/test-output/bm01-tests1-linux32/config_seta.py", line 52, in get_seta_platforms
    response = urllib2.urlopen(url)
  File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/tools/python27/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/tools/python27/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/tools/python27/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 165, in <module>
    main()
  File "/home/buildduty/.mozilla/releng/repos/braindump/buildbot-related/dump_master_json.py", line 150, in main
    dumps = pool.map(worker, args.masters)
  File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 227, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value
ValueError: No JSON object could be decoded
2016-02-08 08:07:43,104 - loading test-output/bm70-build1/master.cfg
dump_allthethings.sh exit code: 1

Comment 1

[Kim Moir [:kmoir] ET]

We have been trying to get a hold of someone in ateam who has credentials to the server, but can't find anyone who has them and jmaher is at a mgrs meeting in DC.

Comment 2

[Kim Moir [:kmoir] ET]

note that as a workaround we can set an env var for DISABLE_SETA on the buildbot masters but this would increase load after the reconfig

Comment 3

[Kim Moir [:kmoir] ET]

it seems to be up now.  jmaher, what steps did  you take to fix it?  Could you arrange for the credentials to be shared so we could take steps to fix it ourselves if this issues occurred again?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

dminor fixed it up- it appears that the machine was rebooted and did some critical updates that were overdue.  With that said, we had to disable nginx so apache would run again.  I am not sure if this is going to work in all cases when we have a reboot, but it is fairly easy to kill all the nginx processes and startup apache.

:dminor, how could we get kmoir access to the ouija box?  I believe armen already has access.

Comment 5

[Dan Minor [:dminor]]

Kim, please attach your public key and I'll add id to the authorized_keys for Ouija. You'll also need access to the AWS jumphost we use which glob manages.

Comment 6

[Armen [:armenzg]]

I probably have access but I don't think I ever tried.
Do we keep instructions of what hostname to connect to?

It would be ideal if this lived on Heroku.

Comment 7

[Dan Minor [:dminor]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #6)
> I probably have access but I don't think I ever tried.
> Do we keep instructions of what hostname to connect to?
> 
> It would be ideal if this lived on Heroku.

Moving this to Heroku makes good sense to me.

(I spoke with glob and now know how to add people to the jumphost)

Comment 8

[Kim Moir [:kmoir] ET]

Attachment: id_rsa.pub

kmoir's public key

Comment 9

[Dan Minor [:dminor]]

Should be ready to go. The easiest way I know of to use the jumphost is to add the following to your .ssh/config:

Host ouija
  Hostname ouija.allizom.org
  User ubuntu
  IdentityFile ~/.ssh/<key file>
  ProxyCommand ssh <username>@54.215.201.230 nc %h %p

Armen, I confirmed you already have access to Ouija.

Comment 10

[Armen [:armenzg]]

I've confirmed I can access. Thanks!

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

can we resolve this bug?