1306844 - Remove support for submitting pushes (resultsets) via the REST API

Status: RESOLVED FIXED

(Tree Management :: Treeherder: API, defect, P2)

Description

[Ed Morley [:emorley]]

Push ingestion can currently occurs via any of:
* Our REST API
* Hg json-pushes polling (will be replace by Pulse in bug 1266559)
* Pulse (for GitHub pushes via TC)

As I understand it, we want to eventually move to a world where all push/job submissions happen via Pulse.

Whilst job submissions is likely a further way out, push ingestion seems easier to do sooner, since most non-taskcluster submitters will be running jobs on repos that are already running taskcluster/buildbot jobs. For those that aren't, we can just add their Hg repo to Treeherder ourselves, or for GitHub ask TC to check for changes to their repo too.

I think it would help with the ORM refactoring if we have fewer ingestion code-paths, so if we can remove this, I think we should.

Looking at insights I see two people submitting via our API:

https://insights.newrelic.com/accounts/677903/explorer/events?eventType=Transaction&timerange=week&filters=%255B%257B%2522key%2522%253A%2522request%252emethod%2522%252C%2522value%2522%253A%2522POST%2522%257D%252C%257B%2522key%2522%253A%2522appName%2522%252C%2522value%2522%253A%2522SCL3%2520treeherder-prod%2522%257D%252C%257B%2522key%2522%253A%2522response%252estatus%2522%252C%2522value%2522%253A%2522200%2522%257D%252C%257B%2522key%2522%253A%2522name%2522%252C%2522value%2522%253A%2522WebTransaction%252FFunction%252Ftreeherder%252ewebapp%252eapi%252eresultset%253AResultSetViewSet%252ecreate%2522%257D%255D&facet=name

gaia-taskcluster and servo-perf (and only 21 submissions in the last 7 days between them).

Now given B2G is EOL, we don't have to worry about breaking them, and for servo-perf, perhaps we can just get Taskcluster to send Pulse messages for their GitHub repo too?

Will/Cameron, thoughts?

Comment 1

[Ed Morley [:emorley]]

gaia-taskcluster is apparently being used for bugzilla resultset creation on prod, rather than just B2G related things.

However they are planning on switching soon (we should give them a nudge in the meantime).

Comment 2

[William Lachance (:wlach)]

FYI, I don't think this is needed for conversion to the ORM (everything pretty much goes through the jobsmodel, which is all I'm modifying). I agree it would be nice to clean this up though.

Comment 3

[Ed Morley [:emorley]]

There are no longer any resultset submissions via the REST API, so safe to remove it.

https://insights.newrelic.com/accounts/677903/explorer/events?eventType=Transaction&filters=%255B%257B%2522key%2522%253A%2522name%2522%252C%2522value%2522%253A%2522resultset%2522%252C%2522like%2522%253Atrue%257D%252C%257B%2522key%2522%253A%2522request%252emethod%2522%252C%2522value%2522%253A%2522POST%2522%257D%255D&duration=2592000000&facet=name

Comment 4

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:rm-resultset-posts > mozilla:master

Comment 5

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/cc1fd9cc5713a21ad2bfa2c4f9e0b7c396062b8e
Bug 1306844 - Remove unused attributes from push ETL data structures

The ETL process creates intermediate push objects which are passed to
`_store_push()` via `store_result_set_data()`. However many attributes
were being passed in the dict but never used by `_store_push()`.

https://github.com/mozilla/treeherder/commit/20ce1497bde6f577815f255feb30e1f8b70a4757
Bug 1306844 - Stop using TreeherderResultSetCollection for pushlog ETL

Since `TreeherderResultSetCollection` is going to be removed shortly,
and the use here is not necessary - but instead a leftover from the days
of submitting pushlog data to our own REST API.

The `.validate()` functionality is not required, since the Django model
already performs validation.

https://github.com/mozilla/treeherder/commit/f3a286709ddaf2cfde6205bb46e252c61fe6b659
Bug 1306844 - Simplify pushlog transformation during ingestion

Previously `pushes` was iterated to create a transformed pushes list,
which was iterated again to submit each push within. Now the list of
pushes is only iterated once, with just one push being transformed and
stored at a time.

In addition, each push is stored by calling `store_push()` directly, to
save having to pass a single element list to `store_result_set_data()`.

https://github.com/mozilla/treeherder/commit/55650d582dd0b25d212a09c0d2712b711c1562bd
Bug 1306844 - Remove support for submitting pushes via the REST API

This endpoint is deprecated in favour of Pulse submissions, and is no
longer being used.