1304091 - The /jobs/ endpoint can return incorrect additional log links

Status: RESOLVED FIXED

(Tree Management :: Treeherder: API, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-beta&job_id=1621388 is for Windows XP debug W3C Web Platform Tests W3C Web Platform Tests W(6)

A click on "open raw log" at the top opens https://public-artifacts.taskcluster.net/KKTJohWsTgSmVN9AexEM0g/0/public/logs/live_backing.log , a log for a jit test (which also doesn't contain the "UNEXPECTED-TIMEOUT", search for it).

A correct url for the raw log is https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-win32-debug/1474366156/mozilla-beta_xp_ix-debug_test-web-platform-tests-6-bm110-tests1-windows-build57.txt.gz

Expected result: correct log linked

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Wonder if this is fallout from bug 1302224?

Comment 2

[William Lachance (:wlach)]

Ok, this is weird. There are three logs according to the endpoint:

https://treeherder.mozilla.org/api/project/mozilla-beta/jobs/1621388/

builds-4h: "https://queue.taskcluster.net/v1/task/KKTJohWsTgSmVN9AexEM0g/runs/0/artifacts/public%2Flogs%2Flive_backing.log"
buildbot_text: https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-win32-debug/1474366156/mozilla-beta_xp_ix-debug_test-web-platform-tests-6-bm110-tests1-windows-build57.txt.gz
error_summary_json: http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-beta/sha512/f4c2664df16f8c62bb98e06b0f8e3cc5046786d86b2aa2b7aacf22a9c87797375541ae4f0df541f938fca04a329842234192552ecbd8bf0d1df1981d616f9b9c

error_summary_json and buildbot_text look right but builds-4h looks very wrong. I would guess that's a taskcluster bug?

Anyway, I think we should prefer buildbot_text for the raw log link, so I'll figure out a fix that does that.

Comment 3

[William Lachance (:wlach)]

So on further inspection, I'm even more confused. It appears that this is not a taskcluster job, so why would it have a taskcluster log. I am worried there might be a problem with our job ingestion. Aryx, have you seen this type of problem before?

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

whimboo mentioned earlier today that he didn't find the string he was looking for in a log.

Structured log: https://treeherder.mozilla.org/logviewer.html#?job_id=3598426&repo=mozilla-aurora#L3908 (ESlint)
Raw log: https://public-artifacts.taskcluster.net/OMtBgF8PToqzGqsqWxOoMA/0/public/logs/live_backing.log (wpt)

Comment 5

[William Lachance (:wlach)]

(In reply to Sebastian Hengst [:aryx][:archaeopteryx] from comment #4)
> whimboo mentioned earlier today that he didn't find the string he was
> looking for in a log.
> 
> Structured log:
> https://treeherder.mozilla.org/logviewer.html#?job_id=3598426&repo=mozilla-
> aurora#L3908 (ESlint)
> Raw log:
> https://public-artifacts.taskcluster.net/OMtBgF8PToqzGqsqWxOoMA/0/public/
> logs/live_backing.log (wpt)

This one is similar, there are two builds-4h logs, one of which is correct, one which isn't. I'm pretty confused, the fact that we are seeing this for both buildbot and taskcluster leads me to think that the problem is in treeherder... but I honestly don't see where the problem could be occurring. Our job log ingestion code is very straightforward:

https://github.com/mozilla/treeherder/blob/9d7102c/treeherder/model/derived/jobs.py#L1642
https://github.com/mozilla/treeherder/blob/9d7102c/treeherder/model/derived/jobs.py#L1457

I'd like to see some kind of audit trail about what was submitted for these jobs. This last one had a guid of `17d63b89-8bc4-41bc-bf4e-085ebd03d8b0/0`.

Comment 6

[GitHub Autolander Bot]

Attachment: [treeherder] wlach:1304091 > mozilla:master

Comment 7

[William Lachance (:wlach)]

Comment on attachment 8793000 [details] [review]
[treeherder] wlach:1304091 > mozilla:master

Ok, I figured it out. Not actually that complicated: we need to filter the job log information on the job's repository, not just its project specific id.

Ordinarily I would write a unit test case for this along with the fix, but it would be pretty painful to do this given our present setup. This class of error only happens because we're trying to keep two sources of data in sync, which will not be the case soon (bug 1178641). Once datasource goes away, this whole class of error shouldn't be possible.

Comment 8

[Ed Morley [:emorley]]

Ah good spot :-)

Comment 9

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/41058d8b088875337e13eee1d3f37392c3cecc42
Bug 1304091 - For job information, don't return incorrect job log links (#1864)

In the /jobs/XXXX/ endpoint, we were just keying off the job id when
trying to figure out which log information to return. This could result
in additional (incorrect) data being returned for the case that
there were jobs with the same project specific id value in multiple
repositories.