1069502 - Submit OrangeFactor data to ES directly rather than via TBPL's starcomment.php

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Ed Morley [:emorley]]

Currently Treeherder submits the intermittent failure data (used for OrangeFactor) to elastic search via TBPL's starcomment.php, since it was the easiest way to get things up and running.

Longer term, OrangeFactor v2 will ideally use treeherder's API to generate intermittent failure stats directly, but that's likely not going to be complete before we wish to EOL TBPL.

As such, before we can EOL TBPL, we must take the script here:
https://hg.mozilla.org/webtools/tbpl/file/default/php/starcomment.php

And use it to adjust the existing task to submit directly to ES:
https://github.com/mozilla/treeherder-service/blob/e5967822541cb0a6fbdf954273bb79830d13370d/treeherder/etl/tbpl.py#L13

In addition, we'll need to set up the necessary net flows to ensure treeherder can access the ES instance (having to do this was one of the reasons for using starcomment.php initially):

elasticsearch-zlb.webapp.scl3.mozilla.com (port 9200)

To do this, we'll need a netops bug similar to bug 848529.

Comment 1

[Ed Morley [:emorley]]

Looking at TBPL's starcomment.php, we can drop a fair amount of the functionality there as part of porting it over.

The get_notes() function is unused by TBPL (and doesn't seem to work anyway). Also, starcomment.php makes two submissions to ElasticSearch:
1) All job details + the comment string are sent to the tbpl index, comments doctype.
2) All job details, minus the comment, but with the bug ID, are sent to the bugs index, bug_info doctype.

However #1 is only used by OrangeFactor here:
https://hg.mozilla.org/automation/orangefactor/file/bcf8e1ccdd17/server/handlers.py#l741
...and the unstarred failures endpoint (https://hg.mozilla.org/automation/orangefactor/file/bcf8e1ccdd17/server/handlers.py#l96) is unused.

As such, I believe we can drop #1 too, leaving just #2.

Comment 2

[Ed Morley [:emorley]]

Attachment: Submit OrangeFactor data to ES directly

Comment 3

[Ed Morley [:emorley]]

Tested on stage (since dev doesn't/can't have flows):

[2015-01-07 02:21:03,204: INFO/Worker-1] Sending data to http://elasticsearch-zlb.webapp.scl3.mozilla.com:9200/bugs/bug_info/: {'buildtype': u'opt', 'os': u'windows7-32', 'tree': u'mozilla-central', 'who': u'[e-mail]', 'buildname': u'Windows 7 32-bit mozilla-central opt test mochitest-4', 'date': '2015-01-06', 'logfile': '00000000', 'bug': 931445, 'machinename': u't-w732-ix-084', 'rev': u'33781a3a5201', 'starttime': 1420606427, 'timestamp': 1420626062, 'type': u'Mochitest'}
[2015-01-07 02:21:03,225: INFO/MainProcess] Task submit-star-comment[a68693b9-306a-4e0e-998f-67bd4c722017] succeeded in 0.0764641370624s: None

Which successfully made it to ES:

[[e-mail]3 ~]$ curl -s -XGET 'http://elasticsearch-zlb.webapp.scl3.mozilla.com:92 00/bugs/bug_info/_search?q=timestamp:1420626062' | python -mjson.tool
{
...
                "_id": "v1j3np3lQDWnsvm3paBF9w",
                "_index": "bugs",
                "_score": 11.419449999999999,
                "_source": {
                    "bug": 931445,
                    "buildname": "Windows 7 32-bit mozilla-central opt test mochitest-4",
                    "buildtype": "opt",
                    "date": "2015-01-06",
                    "logfile": "00000000",
                    "machinename": "t-w732-ix-084",
                    "os": "windows7-32",
                    "rev": "33781a3a5201",
                    "starttime": 1420606427,
                    "timestamp": 1420626062,
                    "tree": "mozilla-central",
                    "type": "Mochitest",
                    "who": "emorley@<snip>.com"
                },
                "_type": "bug_info"
...
}

Comparing to a previous entry:
[[e-mail]3 ~]$ curl -s -XGET 'http://elasticsearch-zlb.webapp.scl3.mozilla.com:92 00/bugs/bug_info/_search?q=date:2015-01-06&size=1' | python -mjson.tool
{
...
                "_id": "6HrYNxzDRXq8I8H41Jzcig",
                "_index": "bugs",
                "_score": 1.0,
                "_source": {
                    "bug": "1114713",
                    "buildname": "Ubuntu VM 12.04 b2g-inbound debug test marionette",
                    "buildtype": "debug",
                    "date": "2015-01-06",
                    "logfile": "00000000",
                    "machinename": "tst-linux32-spot-071",
                    "os": "linux32",
                    "rev": "dc341b949c0b",
                    "starttime": "1420543845",
                    "timestamp": "1420553626",
                    "tree": "b2g-inbound",
                    "type": "Marionette Framework Unit Tests",
                    "who": "rvandermeulen@<snip>.com"
                },
                "_type": "bug_info"
...
}

The only difference is that {bug, starttime, timestamp} are now integers not strings. They really should be ints, but for the sake of not risking whether OrangeFactor is handling them properly (and given the current OF is not going to be around for ages longer), I'll update the PR to make them strings for consistency with TBPL.

Comment 4

[Ed Morley [:emorley]]

Comment on attachment 8540163 [details] [review]
Submit OrangeFactor data to ES directly

With updated PR. Tested on stage:

[2015-01-07 03:07:16,392: INFO/Worker-1] Sending data to http://elasticsearch-zlb.webapp.scl3.mozilla.com:9200/bugs/bug_info/: {'buildtype': u'asan', 'os': u'linux64', 'tree': u'mozilla-central', 'who': u'emorley@<snip>.com', 'buildname': u'Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-e10s-3', 'date': '2015-01-06', 'logfile': '00000000', 'bug': '1114660', 'machinename': u'tst-linux64-spot-703', 'rev': u'33781a3a5201', 'starttime': '1420603239', 'timestamp': '1420628836', 'type': u'Mochitest e10s'}
[2015-01-07 03:07:16,407: INFO/MainProcess] Task submit-star-comment[585e4283-d82a-4f94-993c-e410cf2d8e16] succeeded in 0.0622406052426s: None

[[e-mail]3 ~]$ curl -s -XGET 'http://elasticsearch-zlb.webapp.scl3.mozilla.com:92 00/bugs/bug_info/_search?q=timestamp:1420628836' | python -mjson.tool
{
...
                "_id": "GHZ7PIuJQBmcuqQ9wRLO7g",
                "_index": "bugs",
                "_score": 11.458923,
                "_source": {
                    "bug": "1114660",
                    "buildname": "Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-e10s-3",
                    "buildtype": "asan",
                    "date": "2015-01-06",
                    "logfile": "00000000",
                    "machinename": "tst-linux64-spot-703",
                    "os": "linux64",
                    "rev": "33781a3a5201",
                    "starttime": "1420603239",
                    "timestamp": "1420628836",
                    "tree": "mozilla-central",
                    "type": "Mochitest e10s",
                    "who": "emorley@<snip>.com"
                },
                "_type": "bug_info"
...
}

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/912d4f3fb01344b9d86dfe32f5d05d8cfb309307
Bug 1069502 - Submit OrangeFactor data to ES directly

Mappings of bug IDs to failures are currently mirrored to ElasticSearch
for use by OrangeFactor, until OrangeFactor is rewritten to use
Treeherder's DB directly. This patch makes Treeherder submit these
mappings directly to the ElasticSearch instance, rather than doing so
via TBPL's starcomment.php - so that TBPL can be switched off. For
TBPL's implementation, see:
https://hg.mozilla.org/webtools/tbpl/file/eb654a2734c3/php/starcomment.php