1191934 - Remove fetch-missing-push-logs task once we're no longer missing pushes

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

The task was a workaround for us missing pushes.

Let's try and fix the root cause and then remove it, since it makes ETL much more complicated, as well as causing a lot more load of json-pushes (which once fixed, would mean we can just poll json-pushes more often - bug 1096919 - which we want for taskcluster).

Before we can do this there are a few pre-reqs:
* Bug 1090441 - Only use the pushlog ingestion fall-back if a job was submitted more than N minutes ago (so we can rule out these false positives)
* Bug 1090289 - Handle jobs where the revision is invalid, causing redundant fetch-missing-pushlog tasks (so we can rule out false positives caused by eg the l10n jobs)
* Bug 1115608 - Treeherder failed to ingest a try push with U+1F4A9 in the commit message (so we don't fail to ingest just because of one bad push)

We'll then need to inspect the logs and see how often we hit the missing pushes task - if at all, then fix those causes - so we get to a point where we're never missing pushes in the first place :-)

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:rm-fetch-missing-pushlogs > mozilla:master

Comment 3

[Ed Morley [:emorley]]

No instances of the fetch-missing-pushlogs tasks being run on:
https://rpm.newrelic.com/accounts/677903/applications/4180461/transactions/table?type=other&show_browser=false#sort=7%2C1

...so we're good to go here.

Comment 4

[Cameron Dawson [:camd]]

Comment on attachment 8748764 [details] [review]
[treeherder] mozilla:rm-fetch-missing-pushlogs > mozilla:master

Awesome!  Thanks for tackling this technical debt.  :)

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/92793e569e195ef8934031d7a9068d27e2ce52f9
Bug 1191934 - Remove the now-redundant fetch-missing-pushlogs task

The task was a workaround for us missing pushes, however the root causes
of these have since been fixed.

Comment 6

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/307de9fac5d5b93ef7877db90521a6f06b136a27
Bug 1191934 - Remove leftover import

Annoyingly this didn't cause the tests (or flake8) to fail, but on
stage stopped celery from starting:

Traceback (most recent call last):
  ...
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/loaders/base.py", line 273, in autodiscover_tasks
    return [find_related_module(pkg, related_name) for pkg in packages]
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/loaders/base.py", line 299, in find_related_module
    return importlib.import_module('{0}.{1}'.format(package, related_name))
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/etl/tasks/__init__.py", line 3, in <module>
    from .cleanup_tasks import *
ImportError: No module named cleanup_tasks