838390 - mozfile should have a method to read from a file or URL

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Jeff Hammel]

There are a few places in our code where we want to read from either a
file or a URL.  We will also need this for mozprofile, bbug 838273.
So mozfile should have a function that reads from both.

Proposed API, `mozfile.open`:

def open(location):
    """opens a file or URL and returns a ``file`` or ``urlopen``
    object"""

Comment 1

[Jeff Hammel]

It may also be worthwhile putting something like urlsplit that works in mozfile, but i'll leave that until later. (E.g. http://hg.mozilla.org/build/talos/file/8dd79e0f1d4c/talos/utils.py#l328 )

Comment 2

[Jeff Hammel]

Attachment: add mozfile.read method

This is my proposal, though it hasn't been tested and I do want a test for it.  As is it returns the "file-like object" of either file/open or urllib2.urlopen.

Comment 3

[Jeff Hammel]

I intend to rename https://github.com/mozilla/mozbase/blob/master/mozfile/tests/is_url.py to url.py and have this test live there.

Comment 4

[William Lachance (:wlach)]

Comment on attachment 711055 [details] [diff] [review]
add mozfile.read method

Looks reasonable.

Comment 5

[Jeff Hammel]

Meant to comment earlier.  While we *can* call this `def open`, we probably (much to my chagrin) *shouldn't* since python reserves `open` as a front-end function to `file` which is going/gone away in py3 (fwiw, i have no idea why, if you're going to have a builtin called open, that it only opens files; seems archaic and in fact were one of today's rockstar programmers I would assume that `open` would open a URL and when it didn't talk about how python was for old men; but i digress).

So what to call it?  I'm horrible with names, but my top three choices are, in order of favorite to least, are:

load
retrieve
fetch

Comment 6

[Jonathan Griffin (:jgriffin)]

Comment on attachment 711055 [details] [diff] [review]
add mozfile.read method

Review of attachment 711055 [details] [diff] [review]:
-----------------------------------------------------------------

I like this idea and overall approach, but unfortunately it falls down on windows.  :(  open() doesn't work with any kind of file:// url on windows.  Instead, if you want to open a file:// url, you need to use urllib2.urlopen.  Alternately, you could convert file:// url's to local paths for all os's (or just Windows?) using something like:

// pathname = e.g., 'file://f:/Downloads/mozilla.txt'
parsed = urlparse.urlparse(pathname)
// parsed = e.g., ParseResult(scheme='file', netloc='f:', path='/Downloads/mozilla.txt', params='', query='', fragment='')
fullpath = parsed.netloc + urllib.url2pathname(parsed.path)
// fullpath = e.g., 'f:\\Downloads\\mozilla.txt'
return open(fullpath)

Comment 7

[Jeff Hammel]

Huh, I wonder why I did it this way.  Clearly I meant to check for file:// and then strip it ;)  Probably trying to save lines of code and not thinking it through

Comment 8

[Jeff Hammel]

Attachment: take 2

with tests

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 747718 [details] [diff] [review]
take 2

Review of attachment 747718 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm!

Comment 10

[Jeff Hammel]

I forgot to add the new test file to the manifest.  Will do so on commit.

Comment 11

[Jeff Hammel]

pushed: https://github.com/mozilla/mozbase/commit/c22eb887a03aeafd4f8c45a08dda2181da487a1d