1809467 - frequent intermittent failures on collapsable_rows_tests.jsx and test_perfcompare_api.py, disabled for now

Status: NEW

(Tree Management :: Perfherder, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

this test fails frequently enough that I expect it to fail and have to go to circleCI and retrigger.

just today I ran into it yet again:
https://app.circleci.com/pipelines/github/mozilla/treeherder/4367/workflows/8d28a40f-71d7-4f0b-ac7a-9c07ccfb70ee/jobs/17608

FAIL tests/ui/perfherder/alerts-view/collapsable_rows_test.jsx (11.119 s)
  ● Alert summary with more than 26 alerts is collapsable

    thrown: "Exceeded timeout of 5000 ms for a test.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      44 | afterEach(cleanup);
      45 |
    > 46 | test('Alert summary with more than 26 alerts is collapsable', async () => {
         | ^
      47 |   const { getAllByLabelText, getByTestId } = collapsableRowsTest();
      48 |
      49 |   let visibleRows = await waitFor(() => getAllByLabelText('Alert table row'));

      at Object.test (tests/ui/perfherder/alerts-view/collapsable_rows_test.jsx:46:1)


Test Suites: 1 failed, 42 passed, 43 of 57 total

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

adding in test_perfcompare_api.py:
as seen here:
https://app.circleci.com/pipelines/github/mozilla/treeherder/4368/workflows/2fdf2b1e-76bb-43d9-8b53-0d2f5de3f108/jobs/17611

        response = client.get(reverse('perfcompare-results') + query_params)
        assert response.status_code == 200
        for result in expected:
>           assert result in response.json()
E           assert {'base_avg_value': 65.35, 'base_measurement_unit': 'ms', 'base_median_value': 65.35, 'base_repository_name': 'test_treeherder_jobs', ...} in [{'base_avg_value': 36.24, 'base_measurement_unit': 'ms', 'base_median_value': 32.4, 'base_repository_name': 'test_tre...65.35, 'base_measurement_unit': 'ms', 'base_median_value': 65.35, 'base_repository_name': 'test_treeherder_jobs', ...}]
E            +  where [{'base_avg_value': 36.24, 'base_measurement_unit': 'ms', 'base_median_value': 32.4, 'base_repository_name': 'test_tre...65.35, 'base_measurement_unit': 'ms', 'base_median_value': 65.35, 'base_repository_name': 'test_treeherder_jobs', ...}] = functools.partial(<bound method ClientMixin._parse_json of <rest_framework.test.APIClient object at 0x7f9c09d26bb0>>, <Response status_code=200, "application/json">)()
E            +    where functools.partial(<bound method ClientMixin._parse_json of <rest_framework.test.APIClient object at 0x7f9c09d26bb0>>, <Response status_code=200, "application/json">) = <Response status_code=200, "application/json">.json

tests/webapp/api/test_perfcompare_api.py:512: AssertionError

rerunning the task gets it green 95% of the time, the 5% need a second rerun :)

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

tests will be disabled in: https://github.com/mozilla/treeherder/pull/7622