1277566 - [meta] mozreview ci intermittent test failures

Status: RESOLVED FIXED

(MozReview Graveyard :: Testing / Development Environment, defect)

Comment 1

[Robert Wood [:rwood]]

The CI is up and running, intermittent failures are ongoing.

Comment 2

[Robert Wood [:rwood]]

Didn't mean to close this one, re-opening

Comment 3

[Robert Wood [:rwood]]

Update, at this point the main failures come down to these:

Bug 1270834
A bunch of tests fail intermittently and often with:
ServerInterfaceError: [Errno 110] Connection timed out
During 'create-bug', 'dumpreview', 'publish', etc.

Bug 1270828
test-repo-discovery.t
Fails consistently with:
+  abort: authorization failed
+  [255]

Bug 1270829 
test-push-http.t
Fails consistently with:
APIError: Object does not exist (HTTP 404, API Error 100)
I believe caused by an incorrect URL although I'm not certain:
+  www-authenticate: Basic realm="https://bugzilla.mozilla.org/ username and API Key"

Bug 1270830
A bunch of tests fail to establish a new connection with bugzilla:
NewConnectionError
Failed to establish a new connection: [Errno 110] Connection timed out

Bug 1288144 (New)
A bunch of the tests are skipping because they are failing out on 'mozreview start'
+ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
or
+APIError: 500 Server Error: Internal Server Error ("rpc error: code = 2 desc = "containerd: container not started"")

In an effort to green-up the 'rb' test job on the CI, I will be removing all tests from running in the CI as they intermittently fail. So FYI these failures still exist, but may not be showing up on the CI.

Comment 4

[Māris Fogels [:mars] (please needinfo)]

I have successfully created a test environment configuration that does not suffer the intermittent failures we have been seeing.  Work to turn the proof-of-concept into production code is being tracked in bug 1294805.

Comment 5

[Robert Wood [:rwood]]

Intermittent failures resolved by moving to the new CI built by :mars. The ubuntu-based mozreview CI which these intermittent failures were originally logged against, has been decommissioned.