Closed Bug 764661 Opened 12 years ago Closed 12 years ago

server-key-exchange tests are failing

Categories

(Cloud Services :: Server: Key Exchange, defect)

Product:
Cloud Services
Component:
Server: Key Exchange
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rmiller, Unassigned)

Details

(Whiteboard: [qa+])

Attachments

(2 files)

makes sure wsgi middleware returns string and not unicode
938 bytes, patch
tarek
: review+
rfkelly
: review+
Details | Diff | Splinter Review
patch to retry if two threads select the same element
1.48 KB, patch
telliott
: review+
Details | Diff | Splinter Review 
There seem to be multiple issues causing test failures in the server-key-exchange suites. Originally I was seeing this test failur::


======================================================================
FAIL: test_admin_page (keyexchange.tests.test_filtering.TestIPFiltering)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rob/venv/server-key-exchange/keyexchange/tests/test_filtering.py", line 213, in test_admin_page
    res = self.app.get('/__admin__')
  File "/Users/rob/venv/server-key-exchange/lib/python2.6/site-packages/webtest/app.py", line 749, in get
    expect_errors=expect_errors)
  File "/Users/rob/venv/server-key-exchange/lib/python2.6/site-packages/webtest/app.py", line 1050, in do_request
    res.body
  File "/Users/rob/venv/server-key-exchange/lib/python2.6/site-packages/webob/response.py", line 271, in _body__get
    body = self._body = ''.join(self._app_iter)
  File "/Users/rob/venv/server-key-exchange/lib/python2.6/site-packages/webtest/lint.py", line 282, in next
    % (self.iterator, v))
AssertionError: Iterator <listiterator object at 0x101d30bd0> returned a non-str object: u'<html>\n <body>\n   <h1>\n    Status: Active\n   </h1>\n   <h2>Blacklisted IPs</h2>\n   None\n </body>\n</html>\n'

----------------------------------------------------------------------

When I applied the attached patch, I got past this failure. Now, however, I get intermittent failures related to whether or not we're threadsafe. Sometimes the tests pass, other times I see this::

spire ../venv/server-key-exchange $ ./bin/nosetests -s keyexchange.tests.test_filtering
...Exception in thread Thread-8:
Traceback (most recent call last):
  File "/opt/python/parts/opt/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/Users/rob/venv/server-key-exchange/keyexchange/tests/test_filtering.py", line 194, in run
    self.blacklist.remove(random.choice(ips))
  File "/Users/rob/venv/server-key-exchange/keyexchange/filtering/blacklist.py", line 175, in remove
    self.ips.remove(elmt)
KeyError: 'Thread-118'

F.......
======================================================================
FAIL: test_blacklist_thread_safe (keyexchange.tests.test_filtering.TestIPFiltering)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rob/venv/server-key-exchange/keyexchange/tests/test_filtering.py", line 207, in test_blacklist_thread_safe
    self.assertEqual(len(blacklist), 90)
AssertionError: 91 != 90

----------------------------------------------------------------------
Ran 11 tests in 3.583s

FAILED (failures=1)
Attachment #632967 - Flags: review?(tarek)
Whiteboard: [qa+]
The second issue appears to be a race condition in the test itself, in this line:

    self.blacklist.remove(random.choice(ips))

It's possible for multiple threads to select the same element to be removed.  One successfully removes it, the other gets a key error.  Attached patch adds a retry loop so that each thread will successfully remove a single element.
Attachment #635161 - Flags: review?(tarek)
doesn't look like these patches were ever reviewed, but this bug is 2 months old now and the tests are passing for me so i'm calling it resolved.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
This seems OK now, but I did open a similar bug: bug 786905
Status: RESOLVED → VERIFIED
Correct to mark it Verified, but incorrect Comment 3.
Just ignore me.
Comment on attachment 635161 [details] [diff] [review]
patch to retry if two threads select the same element

cancelling review due to bug marked fixed
Attachment #635161 - Flags: review?(tarek)
Saw this bug again during a cleanout of failing jenkins builds.  The fixes are pretty trivial, re-opening.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Comment on attachment 632967 [details] [diff] [review]
makes sure wsgi middleware returns string and not unicode

It's not at all clear to me who has r+ power on the older hg repos these days, but I'm putting one on here in the hopes of moving this bug along :-)
Attachment #632967 - Flags: review+
Attachment #635161 - Flags: review?(telliott)
Attachment #635161 - Flags: review?(telliott) → review+
http://hg.mozilla.org/services/server-key-exchange/rev/fee9281853e5
http://hg.mozilla.org/services/server-key-exchange/rev/a08c5268c962
Status: REOPENED → RESOLVED
Closed: 12 years ago → 12 years ago
Resolution: --- → FIXED
Attachment #632967 - Flags: review?(tarek) → review+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: