1260216 - Use requests in ftpscraper.py

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

We currently use urllib2 to open URLs in ftpscraper [0]. Benefits of using requests instead are:

1) better at handling SSL
2) native support for setting up a session and doing retries
3) native support for timeouts


[0] https://github.com/mozilla/socorro/blob/e95020bd93e9edf5ddbc5903074e6ba7b9f38246/socorro/cron/jobs/ftpscraper.py#L56

Comment 1

[Peter Bengtsson [:peterbe]]

Another benefit is that we can use some CacheControl to speed up queries when we use requests.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Is it sufficient to test this by running "make test" and also running the ftpscraper (e.g. "python socorro/cron/jobs/ftpscraper.py") and making sure it pulls down json files?

If so, I can grab this. If not, what else should we do to test this out?

Comment 3

[Peter Bengtsson [:peterbe]]

(In reply to Will Kahn-Greene [:willkg] from comment #2)
> Is it sufficient to test this by running "make test" and also running the
> ftpscraper (e.g. "python socorro/cron/jobs/ftpscraper.py") and making sure
> it pulls down json files?
> 
Actually, you just need to run: `nosetests socorro/unittest/cron/jobs/test_ftpscraper.py`

> If so, I can grab this. If not, what else should we do to test this out?

At the time of writing I have a patch almost finished. I started yesterday afternoon and had it working when I run the ftpscraper locally with `python socorro/cron/jobs/ftpscraper.py --product=mobile,firefox`

However, I then started spending 2+ hours getting the mocking right in the tests. Almost done.

Comment 4

[Peter Bengtsson [:peterbe]]

At the moment, I honestly don't know how this error is going to behave under requests
https://sentry.prod.mozaws.net/operations/socorro-stage/group/251345/
But I'm sure we'll get something similar. It's SSL handshake timeouts. It's rare and we can fix it as we go.

Comment 5

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3253

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/50b15cb0d3a4fd0b2b7f671a82e6eee555570e32
fixes bug 1260216 - Use requests in ftpscraper.py

https://github.com/mozilla/socorro/commit/0090ff95a9db6aef4528cabdcec57494d79d01cc
Merge pull request #3253 from peterbe/bug-1260216-use-requests-in-ftpscraperpy

fixes bug 1260216 - Use requests in ftpscraper.py