1152426 - Only add bugs to the intermittent-failure bugscache if they've been modified in the last year

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

In bug 1151629, we found that there are now 15,000 intermittent failure bugs (both open and closed) that we fetch each time we populate the cache.

We intentionally fetch closed bugs, as well as bugs older than a few months, so we can display them in the latter of the categories: "open_recent" and "all_others".

However, I think we should go one step further, and not even import _really_ bugs ancient bugs into the bugscache (and thus show them in "all_others") in the first place, to reduce the query load on Bugzilla, and the time it takes for the task to run.

I think something like "last modified is within 1 year of today" might work?

This would roughly halve the number of bugs (and even more so over time):
[~/src]$ curl 'https://bugzilla.mozilla.org/bzapi/count?keywords=intermittent-failure&chfieldfrom=-1y&chfieldto=Now'
{"data":8058}

Sheriffs, what do you think?

Comment 1

[Ryan VanderMeulen [:RyanVM]]

What does the number look like if we restrict that to non-open bugs >1yr? Overall, I'm not opposed in principle, though.

Comment 2

[Ed Morley [:emorley]]

I'm not sure there's a way to get that in one search query, so we'd have to do a few and combine. I suspect in practice though (particularly once the "mark this mass change as one that doesn't trigger bugmail" feature lands) we'll close WFM intermittent-failure bugs after N months, so we'll not actually have any still open, that have a last_modified of >1yr.

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

> Sheriffs, what do you think?
I think that'd be okay. If an intermittent bug has been closed for a year or more, we should probably be filing a new bug for it anyway, rather than reopening.

Comment 4

[Ed Morley [:emorley]]

Attachment: Only add bugs modified in the last year to the bugscache

Comment 5

[Cameron Dawson [:camd]]

Comment on attachment 8591775 [details] [review]
Only add bugs modified in the last year to the bugscache

lgtm.  :)

Comment 6

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/175a45a7a40ebae7c5ab06b579e58d67d19a101a
Bug 1152426 - Only add bugs modified in the last year to the bugscache

The API query will now be of form:
bugzilla.mozilla.org/rest/bug?include_fields=id,summary,status
&keywords=intermittent-failure&chfieldfrom=-1y

Comment 7

[Ed Morley [:emorley]]

Looking good :-)

Execute:
> SELECT COUNT(*) FROM treeherder_stage.bugscache

+ ------------- +
| COUNT(*)      |
+ ------------- +
| 8086          |
+ ------------- +
1 rows