1045904 - write flow tests for deploystudio flows

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The only flows that matter (go through a firewall) right now are between try.releng.scl3 and install.build.releng.scl3, and those are allow-all.  That's *got* to be too broad.  Bug 1045150 deleted flows *from* the DS server to any port on any host, which was the other direction from this flow.

I think the best tactic here will be to tcpdump an install process and characterize the traffic on that basis.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

with http://www.deploystudio.com/Tips/Entries/2009/10/10_Using_DSS_behind_a_Firewall.html as background, and

[[e-mail] ~]# rpcinfo -p install.build.releng.scl3.mozilla.com
   program vers proto   port
    100000    2   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    4   udp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    4   tcp    111  portmapper
    100024    1   udp    755  status
    100021    0   udp    696  nlockmgr
    100024    1   tcp   1021  status
    100021    1   udp    696  nlockmgr
    100021    3   udp    696  nlockmgr
    100021    4   udp    696  nlockmgr
    100021    0   tcp   1017  nlockmgr
    100021    1   tcp   1017  nlockmgr
    100021    3   tcp   1017  nlockmgr
    100021    4   tcp   1017  nlockmgr
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100005    1   udp    707  mountd
    100005    3   udp    707  mountd
    100005    1   tcp   1023  mountd
    100005    3   tcp   1023  mountd
    100011    1   udp    637  rquotad
    100011    2   udp    637  rquotad
    100011    1   tcp    999  rquotad
    100011    2   tcp    999  rquotad

here's a representative set of snippets from the tcpdump:

08:48:41.774782 IP 10.26.64.52.68 > 10.26.52.17.67: BOOTP/DHCP, Request from 3c:07:54:72:53:3e, length 293
08:48:41.774942 IP 10.26.52.17.67 > 10.26.64.52.68: BOOTP/DHCP, Reply, length 322
..
so, dhcp/bootp

08:48:44.746012 IP 10.26.64.52.1833 > 10.26.52.17.69:  117 RRQ "/private/tftpboot/NetBoot/NetBootSP0/DeployStudioRuntime-120808-164811.nbi/i386/booter" octet tsize 0 blksize 32768
08:48:44.877935 IP 10.26.52.17.53179 > 10.26.64.52.1833: UDP, length 29
08:48:44.879239 IP 10.26.64.52.1833 > 10.26.52.17.53179: UDP, length 30
..
08:48:44.880496 IP 10.26.64.52.1834 > 10.26.52.17.69:  95 RRQ "/private/tftpboot/NetBoot/NetBootSP0/DeployStudioRuntime-120808-164811.nbi/i386/booter" octet
08:48:44.902944 IP 10.26.52.17.56309 > 10.26.64.52.1834: UDP, length 516
08:48:44.904226 IP 10.26.64.52.1834 > 10.26.52.17.56309: UDP, length 4
08:48:44.904369 IP 10.26.52.17.56309 > 10.26.64.52.1834: UDP, length 516
08:48:44.905119 IP 10.26.64.52.1834 > 10.26.52.17.56309: UDP, length 4
..
so, tftp with dynamic IPs on both ends

08:49:12.666038 IP 10.26.64.52.1023 > 10.26.52.17.111: UDP, length 56
08:49:12.666275 IP 10.26.52.17.111 > 10.26.64.52.1023: UDP, length 28
..
so, portmap (udp in this case; client port is coincidentally also 1023, I think)

08:49:12.668716 IP 10.26.64.52.1023 > 10.26.52.17.1023: Flags [S], seq 2525597845, win 65535, options [mss 1436,nop,wscale 3,nop,nop,TS val 35113436 ecr 0,sackOK,eol], length 0
08:49:12.668834 IP 10.26.52.17.1023 > 10.26.64.52.1023: Flags [S.], seq 164908920, ack 2525597846, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 3830684140 ecr 35113436,sackOK,eol], length 0
..
so, mountd

08:49:12.671942 IP 10.26.64.52.49152 > 10.26.52.17.2049: Flags [S], seq 3835435545, win 65535, options [mss 1436,nop,wscale 3,nop,nop,TS val 35113438 ecr 0,sackOK,eol], length 0
08:49:12.672046 IP 10.26.52.17.2049 > 10.26.64.52.49152: Flags [S.], seq 2107170150, ack 3835435546, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 3830684143 ecr 35113438,sackOK,eol], length 0
08:49:12.672934 IP 10.26.64.52.49152 > 10.26.52.17.2049: Flags [.], ack 1, win 65535, options [nop,nop,TS val 35113440 ecr 3830684143], length 0
08:49:12.672939 IP 10.26.64.52.2436857857 > 10.26.52.17.2049: 60 null
..
so, NFS. 

08:49:43.254763 IP 10.26.64.52.49153 > 10.26.52.17.60443: Flags [S], seq 2194535268, win 65535, options [mss 1436,nop,wscale 3,nop,nop,TS val 35142479 ecr 0,sackOK,eol], length 0
08:49:43.254865 IP 10.26.52.17.60443 > 10.26.64.52.49153: Flags [S.], seq 867030608, ack 2194535269, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 3830713054 ecr 35142479,sackOK,eol], length 0
..
so, deploystudio https (60443)

08:49:43.797738 IP 10.26.64.52.49156 > 10.26.52.17.548: Flags [S], seq 4279636907, win 65535, options [mss 1436,nop,wscale 3,nop,nop,TS val 35143002 ecr 0,sackOK,eol], length 0
08:49:43.797839 IP 10.26.52.17.548 > 10.26.64.52.49156: Flags [S.], seq 437044911, ack 4279636908, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 3830713578 ecr 35143002,sackOK,eol], length 0
..
so, afpovertcp

http://support.apple.com/kb/HT6175?viewlocale=en_UShttp://support.apple.com/kb/HT6175?viewlocale=en_US lists ports 600-1023/tcp+udp as "RPC-based services", and that would include mountd.  The same file lists port 2049/tcp+udp as NFS, suggesting that NFS is always bound to that port.

In order, then, those are:

application junos-dhcp-server {
    protocol udp;
    destination-port 67;
}

application junos-tftp {
    application-protocol tftp;
    protocol udp;
    destination-port 69;
}

application junos-sun-rpc-portmap-tcp {
    term t1 protocol tcp rpc-program-number 100000;
}
application junos-sun-rpc-portmap-udp {
    term t1 protocol udp rpc-program-number 100000;
}

application junos-sun-rpc-mountd-tcp {
    term t1 protocol tcp rpc-program-number 100005;
}
application junos-sun-rpc-mountd-udp {
    term t1 protocol udp rpc-program-number 100005;
}

application junos-sun-rpc-nfs-tcp {
    term t1 protocol tcp rpc-program-number 100003;
}
application junos-sun-rpc-nfs-udp {
    term t1 protocol udp rpc-program-number 100003;
}

application deploystudio {
    protocol tcp;
    destination-port 60443;
}

application afpovertcp {
    protocol tcp;
    destination-port 548;
}

Right now, we have stuff like

    policy install--imagingservices {
        match {
            source-address any;
            destination-address install;
            application [ junos-tftp junos-dhcp-server afpovertcp junos-sun-rpc-nfs-access ];
        }
        then {
            permit;
        }
    }

where junos-sun-rpc-nfs-access is

application-set junos-sun-rpc-nfs-access {
    application junos-sun-rpc-tcp;
    application junos-sun-rpc-udp;
    application junos-sun-rpc-portmap-tcp;
    application junos-sun-rpc-portmap-udp;
    application junos-sun-rpc-nfs-tcp;
    application junos-sun-rpc-nfs-udp;
    application junos-sun-rpc-mountd-tcp;
    application junos-sun-rpc-mountd-udp;
}

so, policy install-imagingservices above is just missing 'deploystudio'.

There are lots of other policies in the firewall attempting to do this in various ways, with names like install--imagingservices, bootstrap, build--deploystudio, and install--*.  I'll suss those out in the cleanup bug that comes after I write up the tests.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Amy verified today that DS works with a client in srv.releng.scl3 and a DS server in build.releng.scl3.  So we now have two known-good flows across the firewall.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Interesting: junos-dhcp-server isn't present between build and srv, yet the install worked.  So, I bet the helper doesn't require the flow.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

OK, done, and in the repo.