1182464 - DB changes to support the auto starring investigation

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P2)

Description

[Mauro Doglio [:mdoglio]]

We need to to store some info specific to test failures and crashes. This bug is to track both the schema alteration and the data ingestion.

Comment 1

[James Graham [:jgraham]]

The error summary files are now being generated, so there is data to collect here.

Comment 2

[Ed Morley [:emorley]]

(In reply to James Graham [:jgraham] from comment #1)
> The error summary files are now being generated, so there is data to collect
> here.

See bug 1185453 comment 14 for links to example error summary files (though the links as-is currently returns no content, not sure why).

Comment 3

[James Graham [:jgraham]]

Jobs that succeed produce empty error summary files that are, unfortunately, still uploaded. It's only jobs with an error where the files are expected to have any content.

Comment 4

[Ed Morley [:emorley]]

Ah makes more sense thank you :-)

From IRC: for jobs with failures there is one json line per failure - example summary:
http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/10d61f98f31ce4a6629d8650a367916a9674e8f772d6ea0eb5aeff2f5fc5617156e61bc72aa9a234dc55688d67fdca1526b7b72fd963d998e815c84e82f764df

Comment 5

[Mauro Doglio [:mdoglio]]

Attachment: PR 853

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8646278 [details] [review]
PR 853

Looks good at a first glance! :-)

Comment 7

[Mauro Doglio [:mdoglio]]

Thanks :emorley, this is now ready for a review.

Comment 8

[Ed Morley [:emorley]]

Comment on attachment 8646278 [details] [review]
PR 853

r+, but left comments :-)

Comment 9

[James Graham [:jgraham]]

Comment on attachment 8646278 [details] [review]
PR 853

Just needs a few changes in the error summary storage.

Comment 10

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/86e0841024ec1ff9a355f26cea021329124c3ccf
Bug 1182464 - Add BigAutoField to handle big integers as primary keys

https://github.com/mozilla/treeherder/commit/a92fdd46efd5b4f4f45b85931e17f4c4615e0215
Bug 1182464 - Add ORM classes, sql and migration for the autostarring tables

fixup! Bug 1182464 - Add Failure class, sql and migration

https://github.com/mozilla/treeherder/commit/c38d21813e5ae203bc1730a8964d8f3fd0b4a97a
Bug 1182464 - update mozlog version

https://github.com/mozilla/treeherder/commit/5a796eb969745d10d39b180193ebe5b8e5318bd0
Bug 1182464 - Add store_error_summary django command

https://github.com/mozilla/treeherder/commit/ee12c00a03832451ff32a7eb2c434252b550f63c
Bug 1182464 - Add celery task to store the error summary

https://github.com/mozilla/treeherder/commit/79fc062a296a93e8210251ab396993ef857c684e
Bug 1182464 - move test_parse_log command in the log_parser folder

Comment 11

[Ed Morley [:emorley]]

The changes in comment 10 are deployed on stage, but not prod.

Before we push this to prod I think we need to add a limit to the number of failure lines we'll process for any one job, to avoid issues like we saw in bug 1152681.

We may wish to do a hitfix branch for a prod deploy for bug 1201913 before then, since this + the testing of the other UI changes on stage realistically won't be done until Monday/Tuesday.

Comment 12

[Mauro Doglio [:mdoglio]]

Attachment: PR 944

Comment 13

[Ed Morley [:emorley]]

Comment on attachment 8657804 [details] [review]
PR 944

Cancelling review for now - will take another look when the PR is updated :-)

Comment 14

[Mauro Doglio [:mdoglio]]

Comment on attachment 8657804 [details] [review]
PR 944

I update attachment 8657804 [details] [review] according to the feedback received. Thanks for your patience :-)

Comment 15

[Ed Morley [:emorley]]

Comment on attachment 8657804 [details] [review]
PR 944

r=me with the nits fixed + James being happy with the model parts :-)

Comment 16

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/b78d6bbf0b9593ee204acca9ebaa2f81130297ea
Bug 1182464 - Make the failure_line signature a TextField.

I also removed a limit on the length of `message` because it didn't make sense.

https://github.com/mozilla/treeherder/commit/420a0323179c424c2ccc85cc91f3e8344966ba97
Bug 1182464 - Make `score` a DecimalField

https://github.com/mozilla/treeherder/commit/6ac54cb95ff67a811ba291c1d3a8e3b4b64acf7f
Bug 1182464 - Add a limit to the maximum number of failure lines

https://github.com/mozilla/treeherder/commit/a5548c7bfeeccafe6395f606c99f48f4c2002f06
Bug 1182464 - Keep track of failure lines truncation

The truncation point is indicated by a special line with action='truncated'.

Comment 17

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/1e287efb78ac8ee90a29bc1dde4ed8789cdeddca
Bug 1182464 - Add new action choice to the migrations file