1126896 - UI displays duplicate pushes with the same revision due to jobs having a different revision_hash

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P1)

Description

[Armen [:armenzg]]

This is very odd.

If you load treeherder for cedar:
https://treeherder.mozilla.org/ui/#/jobs?repo=cedar

You will see revision 2dea8b3c6c91 twice:
http://people.mozilla.org/~armenzg/sattap/b6ca9d97.png

Comment 1

[Ed Morley [:emorley]]

That's.. odd :-(

Comment 2

[Ed Morley [:emorley]]

URL to reproduce:
https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=2dea8b3c6c91

The API request:
https://treeherder.mozilla.org/api/project/cedar/resultset/?format=json&revision=2dea8b3c6c91

And response:
{
	"meta": {
		"count": 2,
		"filter_params": {
			"format": "json",
			"revision": "2dea8b3c6c91"
		},
		"repository": "cedar"
	},
	"results": [{
		"repository_id": 23,
		"revision_hash": "9c9efe6afba67d2823db093e523ff79a03f634e5",
		"revision_count": 200,
		"author": "[e-mail]",
		"comments": "Merge m-c -> cedar",
		"revisions_uri": "/api/project/cedar/resultset/162/revisions/",
		"push_timestamp": 1421971493,
		"revisions": [{
...
		}],
		"id": 162,
		"revision": "2dea8b3c6c91"
	},
	{
		"repository_id": 23,
		"revision_hash": "19164a1f9ef3fa83d7deac842f116bf944106e5a",
		"revision_count": 329,
		"author": "[e-mail]",
		"comments": "Merge m-c -> cedar",
		"revisions_uri": "/api/project/cedar/resultset/160/revisions/",
		"push_timestamp": 1421971493,
		"revisions": [{
...
		}],
		"id": 160,
		"revision": "2dea8b3c6c91"
	}]
}

...same revision but different revision_hash.

Seeing as this hasn't happened more often, dropping priority, but we should definitely figure this out.

Comment 3

[Ed Morley [:emorley]]

Example job from the main push shown, which has hundreds of jobs listed:
https://treeherder.mozilla.org/api/project/cedar/jobs/?count=1&result_set_id=160&return_type=list
...and its buildapi ingestion artefacts:
https://treeherder.mozilla.org/api/project/cedar/artifact/?job_id=94111&type=json&name__in=buildapi_running,buildapi_pending,buildapi_complete

The only job from the other push, which now has a job listed (had none in the attached screenshot):
https://treeherder.mozilla.org/api/project/cedar/jobs/?count=1&result_set_id=162&return_type=list
...and its buildapi ingestion artefacts:
https://treeherder.mozilla.org/api/project/cedar/artifact/?job_id=95036&type=json&name__in=buildapi_running,buildapi_pending,buildapi_complete

Comment 4

[Ed Morley [:emorley]]

Attachment: buildapi artefacts for "good" job

Comment 5

[Ed Morley [:emorley]]

Attachment: buildapi artefacts for "bad" job

To add context: The revision hash is generated from a number of the job's properties, with the original intention that we could support the display of jobs who pull from multiple unpinned repos - and slice by repo of choice (ie the "how do we display jobs against both mozilla-central and comm-central pushlog" problem). 

We need to check:
(a) what difference in properties caused this difference revision_hash (eg do we use the mozharness revision as part of the hash? I forget)
(b) was this correct, or should we change the way we calculate the revision hash
(c) should the UI (/API) be combining the display of jobs that have the same revision (for the repo currently being viewed) but different overall revision hash.

The "two revisions are shown" problem really is (c) - we quite possibly just have not ever come across this case (seeing as we use pinned revisions everywhere now rather than true multi-repo revision combinations) and so not realised we implemented it incorrectly in the UI/API.

Comment 6

[Ed Morley [:emorley]]

Making summary more keyword rich

Comment 8

[Ed Morley [:emorley]]

Another example:
https://treeherder.mozilla.org/#/jobs?repo=cedar&revision=bf7938244daf

Comment 9

[Mike Hommey [:glandium]]

This may or may not be related but I'm seeing a similar behavior on elm, but with *different* sha1s:

https://treeherder.mozilla.org/#/jobs?repo=elm&revision=bbec315dd979
https://treeherder.mozilla.org/#/jobs?repo=elm&revision=38a3b84d21b3

https://treeherder.mozilla.org/#/jobs?repo=elm&revision=c7002e70d949
https://treeherder.mozilla.org/#/jobs?repo=elm&revision=8a03ab776a12

Happened twice in three days.

Comment 10

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #9)
> Happened twice in three days.

With one push per day. So twice in three pushes.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Hit this today when doing merges.
https://hg.mozilla.org/integration/b2g-inbound/pushloghtml?changeset=911935404233
https://hg.mozilla.org/integration/fx-team/pushloghtml?changeset=911935404233

Comment 12

[Ed Morley [:emorley]]

I'll take a look at this in after the weekend :-)

Comment 13

[Ed Morley [:emorley]]

Another:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=f6321b14228d

Comment 14

[Ed Morley [:emorley]]

I'm not going to have time to look at this until after deliverables; someone else feel free to take a look in the meantime :-)

Comment 15

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Wondering if somewhat-recent improvements have resolved this...

Comment 16

[Ed Morley [:emorley]]

There's a chance they have - let's leave open for a bit longer and see if it reoccurs now that we're using 40 character SHAs and the underlying schema has changed slightly :-)

Comment 17

[Cameron Dawson [:camd]]

Spot checking different repos, the last occurrence of this is from March 2016.  Which coincides about with this commit which should have fixed it:

https://github.com/mozilla/treeherder/commit/2a9dbefa49d26d8d0ddc7c402c571691b1679770

I think this is fixed.  :)