1276355 - IntegrityError for key 'idx_revision_hash' with pulse ingestion

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Cameron Dawson [:camd]]

Attachment: stack trace

We need a better check for creating resultsets on the fly wrt the revision_hash.  Though this field is on its way out, it is still being used as we transition to just using the revision.

We still have a unique index for revision_hash, and therefore need to check for duplicates of that value when we insert a new resultset record.

Found during testing for pulse data ingestion.

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:revision-hash-integrity-error > mozilla:master

Comment 2

[Ed Morley [:emorley]]

Comment on attachment 8757476 [details] [review]
[treeherder] mozilla:revision-hash-integrity-error > mozilla:master

I'm not sure I understand why this change is being made?
(The commit message is just a single line)

Could you add a multi-line commit message to help me review and also for future git log benefit? :-)

Comment 3

[Cameron Dawson [:camd]]

Sorry about that, Ed.  Yeah, that was too terse.  I fixed the commit, the PR and I'll put it here for good measure:  :)

When updating or inserting a resultset record, we need to check against
the revision_hash value because, while revision_hash is on its way out, we
do still use it as a fallback and there is a unique index for it.

This error can occur as a race condition when a job is being ingested that
does not yet have a resultset, but during ingestion, the resultset is
created in parallel. So the process ingesting the job tries to insert
the resultset and gets the index violation. This check will prevent that.

Comment 4

[Cameron Dawson [:camd]]

Comment on attachment 8757476 [details] [review]
[treeherder] mozilla:revision-hash-integrity-error > mozilla:master

Hopefully this clarifies things.  :)

Comment 5

[Ed Morley [:emorley]]

Comment on attachment 8757476 [details] [review]
[treeherder] mozilla:revision-hash-integrity-error > mozilla:master

Looks good, thank you :-)

Comment 6

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/3642e03307428ef92ce665d267cbc144fdb61d09
Bug 1276355 - Fix IntegrityError for idx_revision_hash on resultset creation

When updating or inserting a resultset record, we need to check against
the revision_hash value because, while revision_hash is on its way out, we
do still use it as a fallback and there is a unique index for it.

This error can occur as a race condition when a job is being ingested that
does not yet have a resultset, but during ingestion, the resultset is
created in parallel.  So the process ingesting the job tries to insert
the resultset and gets the index violation.  This check will prevent that.