1443251 - Remove Treeherder support for Buildbot

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, enhancement, P1)

Description

[Ed Morley [:emorley]]

The Buildbot to Taskcluster migration is getting closer to completion:
https://atlee.ca/blog/posts/migration-status-2.html

Pre-requisites before Treeherder can remove support:
* Last of release automation tasks migrated (bug 1397762)
* ESR52 (which still uses Buildbot) to be EOLed (currently ~August 2018: https://wiki.mozilla.org/RapidRelease/Calendar)
* Thunderbird/comm-central to have switched over to Taskcluster (bug 1384205)
* ...?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

What do you think about removing the prompt for buildapi auth on all branches other than ESR52 ? Earlier than removing buildbot completely in September I mean.

Comment 2

[Ed Morley [:emorley]]

Yeah I'd love to do that if everyone was fine with it. (Given that people can retrigger via buildapi's UI anyway, it seems fine to do)

Comment 3

[Ed Morley [:emorley]]

Buildbot is being switched off over the next few days \o/

Comment 4

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4007

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/255f246960d4902d40df1ec42e986e2f4814b7a5
Bug 1443251 - Disable buildbot celerybeat tasks (#4007)

This will allow these celery queues to drain before we remove the code
that handles them. The runnable jobs task is also buildbot-specific,
since for taskcluster the jobs are calculated via the tasks graph
instead.

Comment 6

[Ed Morley [:emorley]]

The change in comment 5 is now deployed to production, so I've reduced the dyno count of `worker_buildapi_{pending,running,4hr}` down to zero, for all of {prototype,stage,prod}.

This reduces the total monthly dynos credits the Treeherder uses on Heroku from 97 to 88.

Comment 7

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4071

Comment 8

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/ffaa2e4b2af200d3ae91e3e6ecdc2d4e4f093e84
Bug 1443251 - Remove runnable jobs support for buildbot (#4071)

Runnable jobs for buildbot were calculated via a celerybeat task
(that was disabled in #4007) and the results stored in the
`runnable_jobs` table. This can all be removed now that buildbot is
EOL, since the remaining support for Taskcluster runnable jobs does
not use that celery task/Django model.

Comment 9

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/4087

Comment 10

[Ed Morley [:emorley]]

Cleanup for future PRs includes:
* rewriting the tests to be more taskcluster-specific
* removing the collection classes from treeherder-client once the tests no longer use them
* Improving the "ingesting jobs locally" UX now that ingest_push doesn't work (since no buildbot jobs)
* refactoring `store_job_data()` now that it's only ever called with a list of jobs of size one
* simplifying or removing the log parser step handling
* SETA/runnable job cleanup

Comment 11

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/445766d958f552535445f08e353045be0a3939ba
Bug 1443251 - Remove support for buildbot job ingestion (#4087)

The buildapi celerybeat tasks were disabled previously in #4007, so
these tasks are unused.

Comment 12

[Ed Morley [:emorley]]

With the changes so far, we've down from 12 types of dyno worker, to 8 \o/