1503172 - update_bugscache intermittently purges cached bugs that are still valid

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder, enhancement, P1)

Description

[Dorel Luca [:dluca]]

Bug suggestions are not loading, even for perma or high freq intermittent bugs.

Comment 1

[Cristina Coroiu [:ccoroiu]]

The bug suggestions tab is being shown with log error lines having no bug suggestions:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&searchStr=windows10-64-ccov%2Cdebug%2Cmochitests%2Cwith%2Ce10s%2Ctest-windows10-64-ccov%2Fdebug-mochitest-browser-chrome-e10s-2%2Cm-e10s%28bc2%29&selectedJob=208566738 has https://bugzilla.mozilla.org/show_bug.cgi?id=1435394 filled, and TH does not showing anything

Comment 2

[Ed Morley [:emorley]]

Thank you for filing.

Looking at the bugscache table, it had 6000 rows, which is strange since:
* unusual that it's rounded to the nearest 1000 rows
* https://bugzilla.mozilla.org/bzapi/count?keywords=intermittent-failure&chfieldfrom=-1y says there are 20668

This would explain why most suggestions are missing.

There are no errors in New Relic, however the command being run for the task (now that it's being run in the Heroku scheduler; bug 1176492):
* doesn't include the `newrelic-admin run-program` prefix
* needs to be listed here too for New Relic to pick it up:
  https://github.com/mozilla/treeherder/blob/f761c7280308b2443f8ddc649ed7eed8e12408b0/newrelic.ini#L28-L31

Looking in Papertrail:
* there are no errors
* however the most recent bugscache run only look 4 minutes before exiting with code 0, rather than the 10 minutes it usually takes
(https://papertrailapp.com/systems/treeherder-prod/events?focus=993831949187579906&q=program%3Ascheduler&selected=993831949187579906)

This makes me suspect that the pagination logic here is not working as expected, and as such the loop is bailing before all bugs are downloaded, which causes all of the other bugs to be treated as old and get deleted from the bugscache table:
https://github.com/mozilla/treeherder/blob/f761c7280308b2443f8ddc649ed7eed8e12408b0/treeherder/etl/bugzilla.py#L37-L49

Whilst looking into this I've been running a one-off manual `heroku run -- newrelic-admin run-program ./manage.py update_bugscache`, and there are now 20668 rows in the bugscache table, which is the correct number.

So the tl;dr is:

"Sometimes the bugscache updating task intermittently fails in a way where only some of the bugs are fetched from bugzilla, and anything that wasn't fetched correctly is deleted from the bugscache table, meaning missing bugs suggestions until the next time it runs (which is hourly)"

Comment 3

[Ed Morley [:emorley]]

Whilst the bugscache was re-populated as of ~08:40 UTC, I've had to flush the Redis cache to force old bug suggestions to be regenerated (connected to the Redis instance using the Heroku CLI addon, then ran the `flushall` command). This will mean everyone will need to log in again.

The bug suggestions for the link in comment 0 now works.

Comment 5

[Ed Morley [:emorley]]

Waiting on finding out more in bug 1503215.

Comment 6

[Ed Morley [:emorley]]

This is still blocked on having the BMO logs in bug 1503215.

Comment 7

[Ed Morley [:emorley]]

Unfortunately we never got the information we needed from the logs in bug 1503215, so it's hard to know what to do here. The comments there indicate at least one bug on BMO's side has been fixed, which might mean this doesn't occur again.