[meta] mozreview ci intermittent test failures

RESOLVED FIXED

Status

MozReview
Testing / Development Environment
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rwood, Assigned: mars)

Tracking

Development/Staging
Dependency tree / graph

Details

Comment hidden (empty)
(Reporter)

Updated

2 years ago
Depends on: 1277663
(Reporter)

Updated

2 years ago
Blocks: 1265423
(Reporter)

Comment 1

2 years ago
The CI is up and running, intermittent failures are ongoing.
No longer blocks: 1265423
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Comment 2

2 years ago
Didn't mean to close this one, re-opening
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Updated

2 years ago
Blocks: 1282523
(Reporter)

Comment 3

2 years ago
Update, at this point the main failures come down to these:

Bug 1270834
A bunch of tests fail intermittently and often with:
ServerInterfaceError: [Errno 110] Connection timed out
During 'create-bug', 'dumpreview', 'publish', etc.

Bug 1270828
test-repo-discovery.t
Fails consistently with:
+  abort: authorization failed
+  [255]

Bug 1270829 
test-push-http.t
Fails consistently with:
APIError: Object does not exist (HTTP 404, API Error 100)
I believe caused by an incorrect URL although I'm not certain:
+  www-authenticate: Basic realm="https://bugzilla.mozilla.org/ username and API Key"

Bug 1270830
A bunch of tests fail to establish a new connection with bugzilla:
NewConnectionError
Failed to establish a new connection: [Errno 110] Connection timed out

Bug 1288144 (New)
A bunch of the tests are skipping because they are failing out on 'mozreview start'
+ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
or
+APIError: 500 Server Error: Internal Server Error ("rpc error: code = 2 desc = "containerd: container not started"")

In an effort to green-up the 'rb' test job on the CI, I will be removing all tests from running in the CI as they intermittently fail. So FYI these failures still exist, but may not be showing up on the CI.
Depends on: 1288144
(Assignee)

Comment 4

2 years ago
I have successfully created a test environment configuration that does not suffer the intermittent failures we have been seeing.  Work to turn the proof-of-concept into production code is being tracked in bug 1294805.
(Reporter)

Comment 5

2 years ago
Intermittent failures resolved by moving to the new CI built by :mars. The ubuntu-based mozreview CI which these intermittent failures were originally logged against, has been decommissioned.
Status: REOPENED → RESOLVED
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED

Updated

2 years ago
Assignee: nobody → mars
You need to log in before you can comment on or make changes to this bug.