1087532 - Change Ouija updatedb.py to pull data from Treeherder rather than TBPL

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Dan Minor [:dminor]]

Ouija (https://wiki.mozilla.org/Auto-tools/Projects/Ouija) is a tool for test failure rate analysis.

The updatedb.py script [1] is run periodically to pull down data on test runs. It is currently using TBPL to gather data on test failures. TBPL is being replaced by Treeherder and so we need to make corresponding changes to updatedb.py.

This will involve changing the getCSetResults() [2] function to use a treeherder url (e.g. [3]). Unfortunately, the json structure returned is substantially different from the TBPL version, so the uploadResults [4] function will have to be changed to extract the relevant data from the new format.

In some cases the data may be not be available in which case we'll just insert NULL values into the database for now as long as that doesn't break current functionality.

[1] https://github.com/dminor/ouija/blob/master/src/updatedb.py
[2] https://github.com/dminor/ouija/blob/master/src/updatedb.py#L151
[3] https://treeherder.mozilla.org/api/project/mozilla-central/resultset/?format=json&full=true&revision=29fbfc1b31aa&with_jobs=true
[4] https://github.com/dminor/ouija/blob/master/src/updatedb.py#L228

Comment 1

[Ed Morley [:emorley]]

Note the repo names are different to TBPL - they actually match the buildbot 'branch' now. See the "name" property in:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/model/fixtures/repository.json

Also if it helps, here is the diff for another tool switching from getBuildRevisions.php to Treeherder's API:
https://github.com/mstange/analyze-tryserver-profiles/commit/e14dfa4a9a9ca17e0ca6e4d96e0208d37b271e1d

Comment 2

[Sherry]

Hi. I took a stab at this bug, and it seems like a lot of jobs that were fetched have "unknown" as the "result". Are those jobs supposed to be filtered out?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Sherry!  Great question.  I am glad to see you working on this bug and asking questions when you get stuck.

I am not sure what unknown means, but if you could provide an example of the data you see and what is unknown maybe we can make a decision on that.

My understanding is we want all the jobs, but treeherder has jobs from different sources which ouija has not historically tracked- quite possibly the unknown is related to that.

Comment 4

[Ed Morley [:emorley]]

Treeherder has a new job state of coalesced (whereas TBPL didn't track these) - don't know if that might be the cause?

Comment 5

[Dan Minor [:dminor]]

Also, feel free to post a work in progress somewhere and that might help us narrow down where the "unknown" jobs are coming from. Thanks for looking at this!

Comment 6

[Sherry]

I've uploaded what I have on github: https://github.com/jurpie/ouija/blob/master/src/updatedb.py

I filtered out the "unknown" results in line 260. It looks like they are unfinished jobs. This is a sample data of an "unknown" job:

['945859', '', '', u'unknown', '-1417661107', u'b2g-linux32', u'opt', '', '', 'b2g-inbound', 'c89eba144c5e']

where the fields are:

[_id, log, slave, _result, duration, platform, buildtype, testtype, bugid, branch, revision]

From the looks of the "duration" field (duration = endtime - starttime) it seems like endtime is 0.

I also wasn't sure where to find "testtype" and "bugid", so they are left as empty strings.

Any ways, let me know if what I have is okay or if I need to make any more changes. =]

Comment 7

[Dan Minor [:dminor]]

Hi Sherry,

This looks good so far. The bugid is assigned by another tool and you don't have to worry about it, only a handful of rows in our production database have it.

The testtype you can pull from job[i("ref_data_name")]. You'll get a string like the following, where the last word in the string is the test type:
Ubuntu VM 12.04 mozilla-central opt test jetpack

My only other comment is you can go ahead an delete anything that we're not using anymore, I had a quick look and it seems like parseBuilder, platformXRef, etc. can go.

Once that is done, go ahead and make a pull request on github. If you haven't done that before, please let me know and I can help out :)

Thanks!

Dan

Comment 8

[Sherry]

Pull request created! =]

Comment 9

[Dan Minor [:dminor]]

(In reply to Sherry from comment #8)
> Pull request created! =]

Thanks for your help on this bug!

We don't have a lot of open Ouija bugs at the moment, but if you're interested in tackling something else, our Pulse project is quite active, you can see the open bugs here: https://wiki.mozilla.org/Auto-tools/Projects/Pulse

Comment 10

[Ed Morley [:emorley]]

The PR that landed, for reference:
https://github.com/dminor/ouija/pull/27