1124799 - Hazard build TinderboxPrints use an alternative format which means Treeherder ingests them as type raw_html

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Ed Morley [:emorley]]

eg:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-br-haz/1421931856/linux64-br-haz_mozilla-central_dep-bm94-build1-build197.txt.gz

06:24:45     INFO - TinderboxPrint: upload <a title='hazards_results' href='https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-br-haz/20150122050416'>results</a>: complete

This means it doesn't match the typical "text: link" format:
https://github.com/mozilla/treeherder-service/blob/31acccb58082b3cbcfb8bc44c10d3c2346962701/treeherder/log_parser/parsers.pyx#L200

And so gets handled by the default raw_html fallback:
https://github.com/mozilla/treeherder-service/blob/31acccb58082b3cbcfb8bc44c10d3c2346962701/treeherder/log_parser/parsers.pyx#L271

And so results in:
17:26 <RyanVM|sheriffduty> oh ffs, clicking the "results" link on "upload results: complete" doesn't open in a new tab anymore?
17:26 <RyanVM|sheriffduty> ugh

Let's just use the more common format rather than trying to special-case this in treeherder.

Yey for the days when TinderboxPrints are replaced by json blobs with defined structure :-)

Comment 1

[Ed Morley [:emorley]]

Attachment: Make Hazard job TinderboxPrints play nicely with Treeherder

This format reads nicely for humans and also matches one of the common cases that Treeherder understands, so we get a treeherder job artefact with content_type link, not raw_html.

In another (Treeherder) bug I'm going to tweak the way this case displays in the UI, but even without that, it should be fine.

Comment 2

[Steve Fink [:sfink] [:s:]]

This would not have worked in TBPL. Will it work in TH?

The problem with TBPL is that you have the string "hazard results: http://ftp.moz.org/foo", which gets split on the colon into "hazard results: http" ":" "//ftp.moz.org/foo". That is why I ended up using raw HTML there.

Also note that I hate the current output, I just didn't want to struggle with the log parsing in order to fix it. Nobody notices the "results" link until I tell them how to get to it. I would like to have multiple links in there (one to the whole output directory, as now, and one directly to the hazards.txt.gz file).

If there is a way to control the output nicely now (even if my multiple links need to be one per line or something, though I'll be sad if I can't mix non-link text with link text), I would be happy to change it to something nicer. I just can't deal with the TBPL-era log processing pipeline.

Comment 3

[Ed Morley [:emorley]]

Comment on attachment 8553263 [details] [diff] [review]
Make Hazard job TinderboxPrints play nicely with Treeherder

https://hg.mozilla.org/build/mozharness/rev/bb1dc4e34b38

Comment 4

[Ed Morley [:emorley]]

This was merged to production yesterday sometime.

Comment 5

[Steve Fink [:sfink] [:s:]]

And it looks like it broke it. Or at least I assume it was this change. Clicking on a hazard build right now gives me

 hazard results: https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/try-builds/[e-mail]-c0d07e99c766/try-linux64-b2g-haz
0/0 hazards allowed, 34 unsafe refs

(including the leading space), with nothing linkified on TH.

Comment 6

[Ed Morley [:emorley]]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=haz

Example log:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-br-haz/1424948293/linux64-br-haz_mozilla-central_dep-bm70-build1-build19.txt.gz

Excerpt from log:
04:27:02     INFO - TinderboxPrint: hazard results: https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-br-haz/20150226025813

Job info artefact:
https://treeherder.mozilla.org/api/project/mozilla-central/artifact/?job_id=1089273&name=Job+Info&type=json

[{
	"blob": {
		"job_details": [{
			"url": "https://hg.mozilla.org/build/mozharness/rev/21f00dd7bda6",
			"value": "https://hg.mozilla.org/build/mozharness/rev/21f00dd7bda6",
			"content_type": "link",
			"title": "script_revlink"
		},
...
		{
			"content_type": "raw_html",
			"value": "hazard results: https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64-br-haz/20150226025813"
		},
		{
			"content_type": "raw_html",
			"value": "0/0 hazards allowed, 33 unsafe refs"
		}],
...
}]

We were aiming for a content_type of 'link, not 'raw_html'.

We're failing to match against:
RE_LINK_HTML = re.compile(
    (r"((?P<title>[A-Za-z/\.0-9\-_]+): )?"
     r"<a .*href=['\"](?P<url>http(s)?://.+)['\"].*>(?P<value>.+)</a>")
)

...due to the space in 'hazard results'. Oops my bad.

Think we'll tweak the regex on the treeherder side, since a space still seems acceptable... though who knows if this will result in false positives. Roll on the day we through away TinderboxPrints in favour of something more structured!

Comment 7

[Ed Morley [:emorley]]

Closing this in favour of the Treeherder bug 1137162.,