1066398 - Bug cache ingestion should use a last modified time to reduce load on bugzilla

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P4)

Description

[Ed Morley [:emorley]]

Currently treeherder bug ingestion is done by querying all intermittent failure bugs each time (currently once a day, but it's likely we'll want to make this more frequent).

We should follow:
https://wiki.mozilla.org/BMO/Integration_Best_Practice

...and use the last modified time to make the query less painful.

I'd imagine we will want a periodic full import (plus for initial setup, eg on vagrant up) in addition to this partial search.

Note that there's a case that the last modified search will miss: When the intermittent-failure keyword is removed from a bug, it will no longer show up in the queries, so we'll have an orphaned bug in the DB. To prevent this we'll need to either:
1) Ask glob/dkl if there is a way to use the field changed API search options to find out about these bugs so we can remove them.
2) Do a full bug query and delete any in the DB that are no longer returned from Bugzilla API periodically (once a day?).
3) Ask glob/dkl for a custom endpoint that can give us both a list of these orphaned bugs and the ones that would have been caught by the last modified query anyway.

Comment 1

[Ed Morley [:emorley]]

Wontfix since we'll stop fetching from Bugzilla entirely in the bugless world.