1378950 - test_summary_performance_data fails under MySQL 5.7 due to different result order

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect, P2)

Description

[Ed Morley [:emorley]]

Both on Travis and in Vagrant, after updating to mysql 5.7, I get:

```
vagrant ~/treeherder $ pytest tests/webapp/api/test_performance_data_api.py -k test_summary_perform ance_data -vv
...
>       assert resp.data.keys() == [test_perf_signature.signature_hash,
                                    summary_signature_hash]
E       AssertionError: assert ['sssssssssss...tttttttttttt'] == ['tttttttttttt...ssssssssssss']
E         At index 0 diff: u'ssssssssssssssssssssssssssssssssssssssss' != 'tttttttttttttttttttttttttttttttttttttttt'
E         Full diff:
E         - [u'ssssssssssssssssssssssssssssssssssssssss',
E         -  u'tttttttttttttttttttttttttttttttttttttttt']
E         ? ^^                                          ^
E         + ['tttttttttttttttttttttttttttttttttttttttt',
E         ? ^                                          ^
E         +  'ssssssssssssssssssssssssssssssssssssssss']
```
(Full log: https://emorley.pastebin.mozilla.org/9026431)

The API response being tested comes from here:
https://github.com/mozilla/treeherder/blob/2501fbc10ad4ec8da6cb4d1a49472f866659ed64/treeherder/webapp/api/performance_data.py#L89-L126

Ignoring the part about the test relying on dict order (which is not deterministic, but likely unrelated here), I believe the differing order is due to lack of ORDER BY for:

        signature_data = PerformanceSignature.objects.filter(
            repository=repository).select_related(
                'parent_signature__signature_hash', 'option_collection',
                'platform')

Whilst omitting an ORDER BY in MySQL generally returns a consistent order, it's not actually guaranteed to be deterministic.

I can't tell at a glance if the order is important here, or whether both sides of the test comparison should just say cast to `set` first. Will, I don't suppose you could take a look at the test/API response?

Comment 1

[William Lachance (:wlach)]

Took a look, I think just casting to set should be fine!

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:fix-perf-test-determinism > mozilla:master

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a618ed01a669b80f29bfe1dae2ef8f6531ca4545
Bug 1378950 - Fix test_summary_performance_data non-determinism

Previously it was dependant on the order of results returned by the
API, when the backing DB query doesn't use an ORDER BY. This means
that changes in the optimiser (eg upgrading to a different MySQL
version) could cause the test to fail.