743855 - Subsequent runs of AITC functional tests against AITC Dev failed until aitc1.web.mtv1 was restarted

Status: VERIFIED WORKSFORME

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[James Bonacci [:jbonacci]]

Ran into this issue while running AITC functional tests on qa1 VM that was pointed to the AITC Dev environment.

Given an AITC deployment to qa1.mtv1.dev pointing to dev-aitc1
and a secret file brought over from AITC dev,

I ran the functional test against dev-aitc1 as follows:
sudo ./bin/python aitc/tests/functional/test_aitc.py https://dev-aitc1.ETC/

This generated one functional test failure.

On subsequent runs against dev-aitc{1-10}, though, I would see more severe errors:
No handlers could be found for logger "mozsvc.user"
EEEEEEEEEEE.EEEEE
======================================================================
ERROR: test_creating_an_app_with_x_if_unmodified_since_equal_to_zero (__main__.TestAITC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "aitc/tests/functional/test_aitc.py", line 79, in setUp
    apps = self.app.get(self.root + "/apps/").json["apps"]
  File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 753, in get
    expect_errors=expect_errors)
  File "aitc/tests/functional/test_aitc.py", line 74, in new_do_request
    return orig_do_request(req, *args, **kwds)
  File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1075, in do_request
    self._check_status(status, res)
  File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1111, in _check_status
    res.body))
AppError: Bad response: 401 Unauthorized (not 200 OK or 3xx redirect for https://dev-aitc1.ETC/1.0/83544/apps/)
invalid MAC signature

...etc...

The logs on aitc1.web.mtv1 did not show much information, but a restart of it cleared up the error - at least for now...


Full list of errors for one run against dev-aitc1 are in
http://jbonacci.pastebin.mozilla.org/1561222
With more info for dev-aitc2 and dev-aitc3
here http://jbonacci.pastebin.mozilla.org/1561223
and here http://jbonacci.pastebin.mozilla.org/1561224

IRC discussion is in http://jbonacci.pastebin.mozilla.org/1561219

Comment 1

[James Bonacci [:jbonacci]]

:rfkelly I am wondering if this is still an issue or a one-off as the Dev configuration was being tweaked?
Should we close as a "WONTFIX" or do we need to look at this from the perspective of better recovery when Dev (or Stage or Prod) is not available?

Comment 2

[Ryan Kelly [:rfkelly]]

Since we haven't been able to reproduce this, I think we're safe to close it.  We can always reopen if it resurfaces.

Comment 3

[James Bonacci [:jbonacci]]

Cool. Verifying as non-reproducible.