1417953 - Bouncer latest checks UNKNOWN

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Attachment: 1.PNG

We got some alerts with the bouncer on bm81 since few hours.

"bouncer is UNKNOWN: BOUNCERLATESTCHECKS UNKNOWN: Timeout: check execution aborted after 30s"

The logs suggests some issue with the script which no longer belong to any package and ProcessUnpackaged set to 'no' (check 1.png)

Comment 1

[Alin Selagea [:aselagea]]

Few things I tried:
   - restart 'abrtd' daemon on bm81
   - manually ran the script with an increased timeout

Unfortunately, I got the same error.

check_bouncer -t 120 -v
BOUNCERLATESTCHECKS UNKNOWN: Timeout: check execution aborted after 120s
Traceback (most recent call last):
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/runtime.py", line 38, in wrapper
    return func(*args, **kwds)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagios_tools/scripts/check_bouncer.py", line 180, in main
    check.main(args.verbose, args.timeout)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/check.py", line 110, in main
    runtime.execute(self, verbose, timeout)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/runtime.py", line 118, in execute
    with_timeout(self.timeout, self.run, check)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/platform/posix.py", line 19, in with_timeout
    func(*args, **kwargs)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/runtime.py", line 107, in run
    check()
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/check.py", line 95, in __call__
    self._evaluate_resource(resource)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/check.py", line 77, in _evaluate_resource
    for metric in metrics:
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagios_tools/scripts/check_bouncer.py", line 116, in probe
    http_code = product.get_code(locale)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagios_tools/scripts/check_bouncer.py", line 26, in get_code
    curl_output = subprocess.check_output(['curl', '-sIL', url])
  File "/tools/python27/lib/python2.7/subprocess.py", line 538, in check_output
    output, unused_err = process.communicate()
  File "/tools/python27/lib/python2.7/subprocess.py", line 746, in communicate
    stdout = _eintr_retry_call(self.stdout.read)
  File "/tools/python27/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call
    return func(*args)
  File "/tools/nagios-tools/lib/python2.7/site-packages/nagiosplugin/platform/posix.py", line 14, in timeout_handler
    raise nagiosplugin.Timeout('{0}s'.format(t))
Timeout: 120s

Comment 2

[Jordan Lund (:jlund)]

I'm assuming this alert is legit and related to bouncer infra issues.

see: bug 1417938

Comment 3

[Alin Selagea [:aselagea]]

I haven't noticed any such alerts in the past 20+ hours, so the issue seems fixed.