1186357 - Remove the deprecated objectstore endpoint

Status: RESOLVED FIXED

(Tree Management :: Treeherder: API, defect, P2)

Description

[Ed Morley [:emorley]]

In bug 1140349, the objectstore endpoint was deprecated, and changed so that posting to it was effectively the same as posting to the jobs endpoint:
https://github.com/mozilla/treeherder/blob/00cfe6643db96d5339887493423b962ec0a189d1/treeherder/webapp/api/objectstore.py

We should check for any remaining users, get them to switch, and then remove it.

Comment 1

[Ed Morley [:emorley]]

There are still users of it:

[[e-mail]3 ~]$ grep '/objectstore' /var/log/gunicorn/treeherder_access.log-20150719 | tail -n 5
127.0.0.1 - - [22/Jul/2015:02:48:27 -0700] "POST /api/project/fx-team/objectstore/?<snip> HTTP/1.1" 200 96 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [22/Jul/2015:02:50:10 -0700] "POST /api/project/b2g-inbound/objectstore/?<snip> HTTP/1.1" 200 100 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [22/Jul/2015:02:50:34 -0700] "POST /api/project/mozilla-inbound/objectstore/?<snip> HTTP/1.1" 200 104 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [22/Jul/2015:02:54:06 -0700] "POST /api/project/fx-team/objectstore/?<snip> HTTP/1.1" 200 96 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [22/Jul/2015:02:55:28 -0700] "POST /api/project/mozilla-inbound/objectstore/?<snip> HTTP/1.1" 200 104 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"

Looking at the Apache logs shows the real IP (rather than that due to the request to gunicorn being proxied via Apache), and all instances are from the same Mountain View IP (corp.mtv2).

Bob, is this perhaps from the autophone submissions? I've filed bug 1186366 to update a new treeherder-client to PyPI - once that's done, would you mind updating to the latest version? :-)

Comment 2

[Ed Morley [:emorley]]

Attachment: Remove the deprecated objectstore endpoint

Requesting review now, though it won't be landed until there are no users of the endpoint.

Comment 3

[Ed Morley [:emorley]]

Alternatively: Henrik, might these submissions be from you?

(mdoglio: another reason we need per user credentials, so we can figure out who is using older APIs more easily :-))

Comment 4

[Bob Clary [:bc] (inactive)]

I'm using TreeherderClient.post_collection which I don't think uses the object store but could be wrong. I'll certainly update when one is available.

Comment 5

[Ed Morley [:emorley]]

Formerly (ie with the PyPI version available at the moment), if a TreeherderJobCollection() was instantiated without specifying a `job_type` of `update`, the job is submitted to the objectstore:
https://github.com/mozilla/treeherder/blob/658927bdf7dce14f4a1e03580193f1aabbfe8c4a/treeherder/client/thclient/client.py#L563-L566

Comment 6

[Bob Clary [:bc] (inactive)]

In that case I am guilty!

Comment 7

[Cameron Dawson [:camd]]

Comment on attachment 8637137 [details] [review]
Remove the deprecated objectstore endpoint

looks good to me.  Now I see why Bob had to update his client version.  :)

Comment 8

[Ed Morley [:emorley]]

Now bug 1186420 is fixed, the number of transactions has dropped to zero:
https://rpm.newrelic.com/accounts/677903/applications/4180461/transactions?tw%5Bend%5D=1438125041&tw%5Bstart%5D=1438081841&type=app#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f747265656865726465722e7765626170702e6170692e6f626a65637473746f72653a4f626a65637473746f7265566965775365742e637265617465222c22225d

And the last request was 6 hours ago:

[[e-mail]3 ~]$ grep '/objectstore' /var/log/gunicorn/treeherder_access.log | tail -n 5
127.0.0.1 - - [28/Jul/2015:09:48:51 -0700] "POST /api/project/mozilla-beta/objectstore/?<snip> HTTP/1.1" 200 101 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [28/Jul/2015:09:54:27 -0700] "POST /api/project/mozilla-inbound/objectstore/?<snip> HTTP/1.1" 200 104 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [28/Jul/2015:10:02:20 -0700] "POST /api/project/mozilla-inbound/objectstore/?<snip> HTTP/1.1" 200 104 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [28/Jul/2015:10:10:38 -0700] "POST /api/project/mozilla-beta/objectstore/?<snip> HTTP/1.1" 200 101 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"
127.0.0.1 - - [28/Jul/2015:10:22:28 -0700] "POST /api/project/mozilla-beta/objectstore/?<snip> HTTP/1.1" 200 101 "-" "python-requests/2.7.0 CPython/2.7.8 Darwin/12.6.0"

Comment 9

[Ed Morley [:emorley]]

Yeah there are no more users of this endpoint on prod any more (and there were none on stage prior anyway):
https://rpm.newrelic.com/accounts/677903/applications/4180461/transactions?tw%5Bend%5D=1438258473&tw%5Bstart%5D=1437653673&type=app#id=5b225765625472616e73616374696f6e2f46756e6374696f6e2f747265656865726465722e7765626170702e6170692e6f626a65637473746f72653a4f626a65637473746f7265566965775365742e637265617465222c22225d

Also verified by checking the gunicorn logs.

This can now land :-)

Comment 10

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a49c98b180b79ff54c3bfbf40d21e3343e11940f
Bug 1186357 - Remove the deprecated objectstore endpoint

Since bug 1140349, the objectstore endpoint has been deprecated, and
performs the same function as the jobs endpoint. Now that there are no
remaining submitters to it, let's remove it.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Ed Morley [:emorley] from comment #3)
> Alternatively: Henrik, might these submissions be from you?

Definitely no. We do not have any machine anymore in mtv2. Maybe Syd or Maja were using it for their submissions. CC'ing them for information.

Comment 12

[Ed Morley [:emorley]]

We've since found the source, switched it to the new endpoint and removed the old endpoint (see comment 6 onwards) :-)