https://treeherder.mozilla.org/perfherder/compare?originalProject=mozilla-central&originalRevision=b16763f1da6bcf31f964744c765c4af480ae551c&newProject=try&newRevision=df12762eef9a5e7b6af281e8a94cbdc03313d066&framework=13&page=1&showOnlyComparable=1&showOnlyImportant=1 The comparison above is Browsertime tp6 microsoft, on Linux 18.04 x64 shippable build, for revision `b16763f1da6bcf31f964744c765c4af480ae551c` both on `m-c` and `try`. (`try` one has try commit, but its parent is `b16763f1da6bcf31f964744c765c4af480ae551c`) The result shows 2-5% regression, with 21 runs, and the confidence is med to high. It suggests that there's some difference in either: * the generated binary (maybe somewhere around PGO?) * at least I see several binary files are different between those 2 pushes. haven't looked into the details how they're different tho * the test hardware similar problem is reported before in bug 1515681, but that was about comparing `try` against recent `m-c` pushes, and it's fixed by adding the following warning on perfherder > It is not recommended to compare a try build against a mozilla-central build, unless it is based on latest mozilla-central. This bus is about comparing the same revision built on `try` push and `m-c` push, so the above warning doesn't apply. If there's actually some difference between the `try` push and `m-c` push, we should disallow comparing them, and show a message that suggests pushing the base revision also to `try`, or at least reword the warning message on perfherder comparison page, and also show the warning on the comparison result page as well.
Bug 1741786 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
https://treeherder.mozilla.org/perfherder/compare?originalProject=mozilla-central&originalRevision=b16763f1da6bcf31f964744c765c4af480ae551c&newProject=try&newRevision=df12762eef9a5e7b6af281e8a94cbdc03313d066&framework=13&page=1&showOnlyComparable=1&showOnlyImportant=1 The comparison above is Browsertime tp6 microsoft, on Linux 18.04 x64 shippable build, for revision `b16763f1da6bcf31f964744c765c4af480ae551c` both on `m-c` and `try`. (`try` one has try commit, but its parent is `b16763f1da6bcf31f964744c765c4af480ae551c`) The result shows 2-5% regression, with 21 runs, and the confidence is med to high. It suggests that there's some difference in either: * the generated binary (maybe somewhere around PGO?) * at least I see several binary files are different between those 2 pushes. haven't looked into the details how they're different tho * the test hardware similar problem is reported before in bug 1515681, but that was about comparing `try` against recent `m-c` pushes, and it's fixed by adding the following warning on perfherder > It is not recommended to compare a try build against a mozilla-central build, unless it is based on latest mozilla-central. This bug is about comparing the same revision built on `try` push and `m-c` push, so the above warning doesn't apply. If there's actually some difference between the `try` push and `m-c` push, we should disallow comparing them, and show a message that suggests pushing the base revision also to `try`, or at least reword the warning message on perfherder comparison page, and also show the warning on the comparison result page as well.