1162212 - [raptor] Run tests on master and on master -1 commit, when testing on gaia-master

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Robert Wood [:rwood]]

When running on gaia, the raptor tests run correctly on master, and on master + PR, then compare the results.

When running on gaia-master, the raptor tests are running both times just on master. Fix the task graph raptor code so if not testing a PR, have the raptor tests run on master, and then on master -1 commit, and compare those.

Comment 1

[GitHub Autolander Bot]

Attachment: [gaia] rwood-moz:bug1162212 > mozilla-b2g:master

Comment 2

[Robert Wood [:rwood]]

Attachment: https://github.com/mozilla-b2g/gaia/pull/29969

Updating raptor task code to run the test on master and on master -1 commit (if not testing against a PR). Also increase runs to 30, and remove oboselete env var.

Comment 3

[Greg Arndt [:garndt]]

Comment on attachment 8603372 [details] [review]
https://github.com/mozilla-b2g/gaia/pull/29969

r+ assuming that the raptor tasks knows what to do with headminusone and it's green.

Comment 4

[Robert Wood [:rwood]]

Thanks Greg. Now that I think about this more, this probably isn't the best way to approach this. This would work (when implemented in the raptor-tester container, to sync to head~1) however the gaia revs in the task itself wouldn't reflect the actual version being tested.

What would be better is in this case, when not testing against a PR, have the GITHUB_HEAD_REV value itself change to the value for HEAD~1. That way in the task and logs the correct commit will be referenced.

I'm just now sure how to do this, because at this point in creating the graph there is no pulled gaia repo to query/roll back.

:Garndt, where are the env variables i.e. GITHUB_HEAD_REV populated? Any idea how I can change that value in this task code, to HEAD~1, when the repo hasn't even been pulled yet?

Comment 5

[Robert Wood [:rwood]]

I might be able to use git ls-remote from within the raptor_graph code

Comment 6

[Greg Arndt [:garndt]]

GITHUB_HEAD_REV gets populated by autolander or gaia-taskcluster...such as https://github.com/lightsofapollo/gaia-taskcluster/blob/18d38dc6bdd28081b2c26306c571422067e8b2d5/config/defaults.js#L57

Instead of head~1, do you think this should be rev - 1?  There could be other commits that happen before your tests run and head might not be the same rev you're concerned with.