788604 - moznetwork package to get interface ip and open port

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[William Lachance (:wlach)]

One thing that keeps on coming up in our code is the need for two methods:

1. To find an interface address to run a server on.
2. To find an open port on which to run a service.

Examples:

* mozhttpd as used by eideticker and talos
* devicemanagerSUT (to create a callback server on reboot)
* bc's proposed robocop autophone test (bug 785129)

We have code to do this scattered variously around the tree. mozdevice has NetworkTools, mozhttpd has an "iface" module. :bc created a new solution for getting an open port in bug 785129.

It's pretty clear to me that this is a generally useful piece of code/logic that we should centralize somewhere, and let other mozbase components and test harnesses use it in turn. I thus propose a "moznetwork" package.

I believe the best of breed code for this is (1) the interface code in mozhttpd/iface.py and (2) bc's port-getting code in his patch to bug 785129

Comment 1

[Bob Clary [:bc] (inactive)]

One thing I would like to add to all consumers which want to get the local interface and ip address is the ability to specify one on the command line. I have to remove the ip addresses from the vmnet{1,8} interfaces created by VMware Fusion in order to run the robocop tests on a device since the current NetworkTools will randomly select them instead of my real ip address. It would be nicer if I could just give the ip address or interface name when needed.

Comment 2

[William Lachance (:wlach)]

(In reply to Bob Clary [:bc:] from comment #1)
> One thing I would like to add to all consumers which want to get the local
> interface and ip address is the ability to specify one on the command line.
> I have to remove the ip addresses from the vmnet{1,8} interfaces created by
> VMware Fusion in order to run the robocop tests on a device since the
> current NetworkTools will randomly select them instead of my real ip
> address. It would be nicer if I could just give the ip address or interface
> name when needed.

Maybe we could use some kind of heuristic to pick a sensible one? That said, I fully support being able to override whatever the heuristic picks.

The fact that there are so many little details to get right is a good argument for why something like this is needed IMO.

Comment 3

[William Lachance (:wlach)]

This would actually not be a terrible bug for someone with a bit of python experience. 

How to get stuck in:

1. Read: https://wiki.mozilla.org/Auto-tools/Projects/MozBase#Working_on_Mozbase_and_Contributing_Patches
2. Create a new module called "moznetwork", based on one of the existing simple modules like mozfile or mozhttpd.
3. Incorporate two methods into the module: the interface-getting method in mozhttpd and the port-getting method in bc's patch to bug 785129.
4. Submit your patch per the instructions above.

Comment 4

[(not currently active) Ted Mielczarek]

I don't think we should have a method for getting an open port, that's just ripe for random failure (you pick a port, something else steals it before you call listen, your call fails). The right thing to do there is just to pass 0 as the port number wherever you're using it, and socket.listen will pick an open port for you. You can get the port number in use by calling the socket.getsockname() method, or using the server_address property of subclasses of TCPServer. A simple example can be seen here in the mozcrash unit tests:
https://github.com/mozilla/mozbase/blob/master/mozcrash/tests/test.py#L126

Comment 5

[William Lachance (:wlach)]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> I don't think we should have a method for getting an open port, that's just
> ripe for random failure (you pick a port, something else steals it before
> you call listen, your call fails). The right thing to do there is just to
> pass 0 as the port number wherever you're using it, and socket.listen will
> pick an open port for you. You can get the port number in use by calling the
> socket.getsockname() method, or using the server_address property of
> subclasses of TCPServer. A simple example can be seen here in the mozcrash
> unit tests:
> https://github.com/mozilla/mozbase/blob/master/mozcrash/tests/test.py#L126

Ok, on second thought, I agree with this. We have lots of existing code that does this (the mochitest remote and the mozdevice callback server come to mind) but it's a bad practice that we should be trying to move away from. So let's just add a method to get the right interface and call it a day.

Comment 6

[Bob Clary [:bc] (inactive)]

What I did was:

def getPort(ipaddr):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind((ipaddr, 0))
    port = sock.getsockname()[1]
    sock.close()
    return port

which was helpful in getting a local port for passing to runtestsremote

Comment 7

[William Lachance (:wlach)]

(In reply to Bob Clary [:bc:] from comment #6)
> What I did was:
> 
> def getPort(ipaddr):
>     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>     sock.bind((ipaddr, 0))
>     port = sock.getsockname()[1]
>     sock.close()
>     return port
> 
> which was helpful in getting a local port for passing to runtestsremote

This is better than other methods that we've used, but there's still a potential race between the socket being reopened and some other application getting the same port. In this particular case, the solution is probably for runtestsremote to open the socket itself, then provide the port number on standard output (or something) for other people to read/connect to.

Comment 8

[William Lachance (:wlach)]

ffledging was interested in doing this one.

Comment 9

[Anhad Jai Singh (:ffledgling)]

Attachment: Added the moznetwork module.

Added the rudimentary moznetwork module to mozbase that can be added to in the future.

Comment 10

[William Lachance (:wlach)]

Comment on attachment 694380 [details] [diff] [review]
Added the moznetwork module.

Looks great, thanks! I'll apply this.

Comment 11

[William Lachance (:wlach)]

Pushed: https://github.com/mozilla/mozbase/commit/a03aa8bf715450b6f1e460ac1d55e507840e3f1c

There were some minor formatting and comment issues, which I cleaned up. But overall, great patch! Thanks for doing this.

I'll file some followup issues to get stuff using this.