1240809 - Use fetch_json() instead of the urllib2-powered JsonExtractorMixin

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

A number of the ETL tasks use JsonExtractorMixin, which uses urllib2. We should switch them to use the newly added fetch_json() helper, as one extra step towards moving everything to using the requests package (bug 1165356).

Comment 1

[Ed Morley [:emorley]]

WIP here in case anyone else takes this in the meantime:
https://github.com/mozilla/treeherder/compare/rm-json-extractor-mixin

The tests need fixing up (likely moving from monkeypatching to using the 'responses' library).

Comment 2

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:rm-json-extractor-mixin > mozilla:master

Comment 3

[William Lachance (:wlach)]

Comment on attachment 8756926 [details] [review]
[treeherder] mozilla:rm-json-extractor-mixin > mozilla:master

lgtm!

Comment 4

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a842a4a3b983a6ee426bec3231641858af81858a
Bug 1240809 - Switch allthethings from JsonExtractorMixin to fetch_json

https://github.com/mozilla/treeherder/commit/886022e024e2478bdddb4e42220805c29c611286
Bug 1240809 - Switch BzAPIBugProcess to using fetch_json

https://github.com/mozilla/treeherder/commit/994dba8081c36f41b4bc6292c60b68c1e06d8668
Bug 1240809 - Switch buildapi from JsonExtractorMixin to fetch_json

Previously the buildapi json files were fetched using urllib2, which
supports file:// URIs. Now that it's using the requests-powered
`fetch_json()`, the text fixtures have to be changed to using
responses instead, which arguably is cleaner anyway.

https://github.com/mozilla/treeherder/commit/87e7e90204ba225fce177af4216276b4be74ead9
Bug 1240809 - Remove unused JsonExtractorMixin

Since all of its consumers have now been migrated to fetch_json().