1219922 - Failures when running the ETL tests directory on its own after bug 1193836

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

Before bug 1193836, it was possible to run the ETL tests directory on its own without failures.

Now, running:
py.test tests/etl/

Gives:

tests/etl/test_bugzilla.py .
tests/etl/test_buildapi.py .....FFF.
tests/etl/test_buildbot.py ......................................................................................................................................................
tests/etl/test_classification_mirroring.py .
tests/etl/test_common.py ..
tests/etl/test_job_loader.py .......
tests/etl/test_job_schema.py ............
tests/etl/test_perf_data_adapters.py ..
tests/etl/test_pushlog.py FF...

Strangely, running tests/etl/test_buildapi.py on its own (or even all of tests/) results in a pass:
https://emorley.pastebin.mozilla.org/8850902

Doing a |git checkout 987f803| and then |pip uninstall pytest-django| (since it causes errors otherwise) makes the issue go away. 

This is blocking splitting out the etl tests into their own job for bug 1219889.

With verbose enabled shows the subtest order is still the same:



vagrant ~/treeherder $ py.test tests/etl/test_buildapi.py -x -v
================================================= test session starts ==================================================

platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2 -- /home/vagrant/venv/bin/python
django settings: tests.settings (from ini file)
rootdir: /home/vagrant/treeherder, inifile: pytest.ini
plugins: django, cov
collected 9 items

tests/etl/test_buildapi.py::test_ingest_pending_jobs PASSED
tests/etl/test_buildapi.py::test_ingest_running_jobs PASSED
tests/etl/test_buildapi.py::test_ingest_builds4h_jobs PASSED
tests/etl/test_buildapi.py::test_ingest_running_to_complete_job PASSED
tests/etl/test_buildapi.py::test_ingest_running_job_fields PASSED
tests/etl/test_buildapi.py::test_ingest_pending_jobs_1_missing_resultset PASSED
tests/etl/test_buildapi.py::test_ingest_running_jobs_1_missing_resultset PASSED
tests/etl/test_buildapi.py::test_ingest_builds4h_jobs_1_missing_resultset PASSED
tests/etl/test_buildapi.py::test_ingest_builds4h_jobs_missing_branch PASSED

============================================== 9 passed in 12.20 seconds ===============================================

vagrant ~/treeherder $ py.test tests/etl/ -x -v
================================================= test session starts ==================================================

platform linux2 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2 -- /home/vagrant/venv/bin/python
django settings: tests.settings (from ini file)
rootdir: /home/vagrant/treeherder, inifile: pytest.ini
plugins: django, cov
collected 189 items

tests/etl/test_bugzilla.py::test_bz_api_process PASSED
tests/etl/test_buildapi.py::test_ingest_pending_jobs PASSED
tests/etl/test_buildapi.py::test_ingest_running_jobs PASSED
tests/etl/test_buildapi.py::test_ingest_builds4h_jobs PASSED
tests/etl/test_buildapi.py::test_ingest_running_to_complete_job PASSED
tests/etl/test_buildapi.py::test_ingest_running_job_fields PASSED
tests/etl/test_buildapi.py::test_ingest_pending_jobs_1_missing_resultset FAILEDDestroying test database for alias 'default'...


======================================================= FAILURES =======================================================

_____________________________________ test_ingest_pending_jobs_1_missing_resultset _____________________________________


jm = <treeherder.model.derived.jobs.JobsModel object at 0xb477ba6c>, initial_data = None
sample_resultset = [{'push_timestamp': 1384353511, 'revision_hash': '45f8637cb9f78f19cb8463ff174e81756805d8cf', 'revisions': [{'author': ...86"

This reverts commit c624b65c1333d2f951533ff5f5d3094681cb1916.', 'repository': 'test_treeherder_jobs', ...}]}, ...]
test_repository = <Repository: test_treeherder_jobs development>, mock_buildapi_pending_missing1_url = None
mock_post_json = None, mock_get_resultset = None, mock_get_remote_content = None, activate_responses = None

    def test_ingest_pending_jobs_1_missing_resultset(jm, initial_data,
                                                     sample_resultset, test_repository, mock_buildapi_pending_missing1_url,
                                                     mock_post_json, mock_get_resultset, mock_get_remote_content,
                                                     activate_responses):
        """
        Ensure the pending job with the missing resultset is queued for refetching
        """
        etl_process = PendingJobsProcess()
>       _do_missing_resultset_test(jm, etl_process)

tests/etl/test_buildapi.py:240:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _


jm = <treeherder.model.derived.jobs.JobsModel object at 0xb477ba6c>
etl_process = <treeherder.etl.buildapi.PendingJobsProcess object at 0xb4b1b76c>

    def _do_missing_resultset_test(jm, etl_process):
        new_revision = '222222222222'
        pushlog_content = json.dumps(
            {
                "pushes":
                    {"33270": {
                        "date": 1378288232,
                        "changesets": [
                            {
                                "node": new_revision + "b344655ed7be9a408d2970a736c4",
                                "tags": [],
                                "author": "John Doe <[e-mail]>",
                                "branch": "default",
                                "desc": "bug 909264 - control characters"
                            }
                        ],
                        "user": "[e-mail]"
                    }}
            }
        )
        pushlog_fake_url = "https://hg.mozilla.org/mozilla-central/json-pushes/?full=1&version=2&changeset=" + new_revision
        responses.add(responses.GET, pushlog_fake_url,
                      body=pushlog_content, status=200,
                      match_querystring=True,
                      content_type='application/json')

        etl_process.run()

        stored_obj = jm.get_dhub().execute(proc="jobs_test.selects.jobs")

        assert len(stored_obj) == 1

        revisions_stored = jm.get_dhub().execute(
            proc="jobs_test.selects.revision_ids",
            return_type='tuple'
        )

>       assert len(revisions_stored) == 20
E       assert 19 == 20
E        +  where 19 = len(({'id': 14L, 'revision': '1008d7007411'}, {'id': 9L, 'revision': '14ed3e7e606e'}, {'id': 13L, 'revision': '1ba2f441a41...3L, 'revision': '21fb3eed1b5f'}, {'id': 12L, 'revision': '344b9cd8dc62'}, {'id': 19L, 'revision': '65ac3b2aa05f'}, ...))

tests/etl/test_buildapi.py:318: AssertionError
------------------------------------------------ Captured stdout setup -------------------------------------------------

Installed 82 object(s) from 4 fixture(s)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

========================================= 1 failed, 6 passed in 10.09 seconds ==========================================

Comment 1

[Ed Morley [:emorley]]

The tests/etl/test_pushlog.py failure is:

IntegrityError: (1062, "Duplicate entry '2c25d2bbbcd6-0' for key 'idx_revision'")

Full log of all failures:
https://emorley.pastebin.mozilla.org/8850901

Comment 2

[Mauro Doglio [:mdoglio]]

Attachment: PR 1105

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 8681257 [details] [review]
PR 1105

Good find - thank you for figuring this out!
Datasource strikes again...

Comment 4

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/8f167ba49f191c34878d2742e1522cfe3752eb66
Bug 1219922 - Stop using the refdata test fixture

I haven't found the exact reason why the tests were failing but it must
be a test isolation problem because they were passing individually.
I debugged this issue disabling the tests backwards starting from the
first failure and I found out that test_bz_api_process was the offender.
The test itself is not doing anything wrong but the refdata fixture
used to setup the test seems to be the root cause..
I replaced the two method calls with their orm counterpart and the
problem disappeared.