Subsequent runs of AITC functional tests against AITC Dev failed until aitc1.web.mtv1 was restarted

VERIFIED WORKSFORME

Status

Cloud Services
Server: Sync
--
major
VERIFIED WORKSFORME
6 years ago
6 years ago

People

(Reporter: jbonacci, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [qa+])

(Reporter)

Description

6 years ago
Ran into this issue while running AITC functional tests on qa1 VM that was pointed to the AITC Dev environment.

Given an AITC deployment to qa1.mtv1.dev pointing to dev-aitc1
and a secret file brought over from AITC dev,

I ran the functional test against dev-aitc1 as follows:
sudo ./bin/python aitc/tests/functional/test_aitc.py https://dev-aitc1.ETC/

This generated one functional test failure.

On subsequent runs against dev-aitc{1-10}, though, I would see more severe errors:
No handlers could be found for logger "mozsvc.user"
EEEEEEEEEEE.EEEEE
======================================================================
ERROR: test_creating_an_app_with_x_if_unmodified_since_equal_to_zero (__main__.TestAITC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "aitc/tests/functional/test_aitc.py", line 79, in setUp
    apps = self.app.get(self.root + "/apps/").json["apps"]
  File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 753, in get
    expect_errors=expect_errors)
  File "aitc/tests/functional/test_aitc.py", line 74, in new_do_request
    return orig_do_request(req, *args, **kwds)
  File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1075, in do_request
    self._check_status(status, res)
  File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1111, in _check_status
    res.body))
AppError: Bad response: 401 Unauthorized (not 200 OK or 3xx redirect for https://dev-aitc1.ETC/1.0/83544/apps/)
invalid MAC signature

...etc...

The logs on aitc1.web.mtv1 did not show much information, but a restart of it cleared up the error - at least for now...


Full list of errors for one run against dev-aitc1 are in
http://jbonacci.pastebin.mozilla.org/1561222
With more info for dev-aitc2 and dev-aitc3
here http://jbonacci.pastebin.mozilla.org/1561223
and here http://jbonacci.pastebin.mozilla.org/1561224

IRC discussion is in http://jbonacci.pastebin.mozilla.org/1561219
(Reporter)

Updated

6 years ago
Whiteboard: [qa+]
(Reporter)

Comment 1

6 years ago
:rfkelly I am wondering if this is still an issue or a one-off as the Dev configuration was being tweaked?
Should we close as a "WONTFIX" or do we need to look at this from the perspective of better recovery when Dev (or Stage or Prod) is not available?
Since we haven't been able to reproduce this, I think we're safe to close it.  We can always reopen if it resurfaces.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → WORKSFORME
(Reporter)

Comment 3

6 years ago
Cool. Verifying as non-reproducible.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.