NAME

qjail-vnet-howto —

Details explaining how to use qjail to build a VIMAGE vnet jail system.

DESCRIPTION

This howto will strive to explain how the qjail utility is used in a real world application Utilizing a documentation approach that fills in the blanks of how all the separate network functions fit together.

VIMAGE WARNING

VIMAGE (virtualized network stack) is a highly experimental feature. After you compile vimage into your kernel, every boot from there on will issue this warning message as part of the boot console log messages. This is FreeBSD's way of saying, USE IT AT YOUR OWN RISK.

SCENARIO DESCRIPTION

It's assumed that you know how to compile your kernel. That's not going to be covered here. With FreeBSD RELEASE 11.0, Vimage still has to be compiled into your host's kernel before the jail(8) vnet function will work. The "option VIMAGE" is the only statement you have to add to your kernel definition source. Compiling in the bridge, epair, and the firewall you plan to use is NOT necessary. The starting of the following discussion is based on a running vimage kernel already being booted and having network access to the public internet verified by issueing the "ping -c 2 8.8.8.8" command on the host.

The following sequence of qjail commands were issued on a desktop type of computer running RELEASE 11.0. This computer is a node on a small LAN hard cabled to the gateway computer who's firewall provides the NAT services for public internet access. The gateway is also running 11.0 and uses the ipfilter firewall. All "ifconfig command" output shown below has had the lo0 interface details removed for brevity.

VNET JAIL CONFIG SETUP USING QJAIL

Step 1. Enter pkg install qjail to Install the qjail software.

Step 2. Enter qjail install to install the qjail environment.

Step 3. Enter qjail create -4 10.10.110.10 v10 To create a
directory tree type jail. This is the network ip address used
to communicate with the vnet jail. The name of the jail is v10.

Step 4. Enter qjail config -w fxp0 -v none v10 To change the
non-vnet jail into a vnet jail. fxp0 is the interface device
name connecting this computer to the LAN. You could use vtnet0
or vlan0 interface devices if so desired.

The "-v none" means this vnet jail will not be using any
firewall. The "-v value" options are none, ipfw, pf, ipf. At
this time, IE: 11.0, the ipfw firewall is the only one that
functions. The host must be using IPFW also for things to work.
Read "man qjail" config -v section for more details about
firewall selection.

Step 5. Enter qjail start v10 Start this jail.

Step 6. Enter ifconfig -a to see what the host's network looks like.

fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
options=2009<RXCSUM.VLAN_MTU,WOL_MAGIC>
ether 00:0c:f1:cd:55:ea
inet 10.0.10.7 netmask 0xfffffff0 broadcast 10.0.10.15
nd6 optiona=29<PERFORMNUD,IFDISABLED.AUTO_LINKLOCAL>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active bridge10: flags+8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
ether 02:0d:73:5e:c9:0a
md6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddwlay
maxage 20 holdcut 5 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE>
ifmaxaddr 0 port 4 priority 128 path cost 2000
member: fxp0 flags=143<LEARNING,DISCOVER,AUTOEDGE>
ifmaxaddr 0 port 1 priority 128 path cost 200000 epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
options=8<VLAN_MTU>
ether 02:c1:00:00:04:0a
inet6 fe80::c1:ff:fe00:40a%epair1a prefixlen 64 scopeid
nd6 options=21<PERFORMNUD.AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

The fxp0 network device has been assigned an IP address of 10.0.10.7 by the host system because of this ifconfig_fxp0="DHCP" statement being in the host's rc.conf file. qjail defines it's bridge as bridge10 so hopefully it's name is unique enough not to get steped on by some other host function. The bridge has 2 members, one for the host's network device and one for the epair. epair1a is on the host and its matching pair epair1b is in the vnet jail. The number 1 is incremented by 1 for each additional vnet jail created. So if there were two vnet jails, there also would be a epair2a member shown.

Step 7. Enter qjail list List status of all defined jails. STATUS JID NIC IP Jailname ------ ---- ------ --------------- ------------------------------- DRV 0 7 fxp0 10.10.110.10 v10

D means this is a directory tree type of jail R means this jail is running V means this is a vnet jail 0 means this vnet jail has no firewall 7 means this is the seventh time a jail has been started

Step 8. Enter qjail console v10 Auto login to this jails console. FreeBSD 11.0-RELEASE-p7 (Vimage) #0 r313172: Fri Feb 3 17:02:05 EST 2017 Welcome to your FreeBSD jail. v10 /root >

Step 9. Enter ifconfig -a to see what the Vnet jail network looks like. epair1b: flags=8843<UP.BTOADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
options=8<VLAN_MTU>
ether 02:C1:00:00:05:0B
inet 10.0.110.10 netmask 0xff000000 broadcast 10.255.255.255
inet6 fe80::c1:ff;fe00:50b%epair1b prefixlen 64 scopeid 0x2
nd6 options=21<PREFORMNUD,AAUTO_LINKLOCAL>
media: Ethernet 10Gbase-T (10Dbase-T <full-duplex>)
staatus: active
groups: epair v10 /root >

Here we see the 10.10.110.10 ip address has been assigned to epair1b in the vnet jail

Step 10. Enter ping -c 2 8.8.8.8 Check to see if jail has public access. PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=43 time=37.337 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=43 time=35.983 ms --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 35.983/36.660/37.337/0.677 ms v10 /root >

Step 11. Enter exit to leave the jail console.

Trouble shooting network problems.

1. Cannot access the public internet. Firewall rules or NAT problem on the host or gateway computer.

2. Cannot ping the hosts default route ip address. Most likely you don't know what the default gateway route ip address is. On the host issue the "route get default" command and use the gateway ip address in the ping command issued from the vnet jails console.

3. If nothing you try seems to solve the problem, reboot the host so the host firewall and vimage in the kernel can shake hands anew. Then most likly what you were doing starts working. Your experiencing the highly experimental part of vimage.

4. Another test you can do is try to do a ssh login on the host to the vnet jails ip address. First you have to prepare the vnet jail. Use the console command to login to the vnet jail. Edit the vnet jails rc.conf file adding the sshd_enable="YES" statement. Then issue "pw adduser test -m -w yes" to create a user account named test with password of test. Exit the jails console. Issue "qjail restart v10" and then issue "ssh 10.10.110.10 -p 22 -l test" and see if you can access the vnet jail this way.

5. Be aware that a vnet jail running no firewall and the host is running ipfw does not work in the sense that vnet jail pings to the host or public ip addresses just time out. If the host and the vnet jail both have ipfw running then the vnet jail will have public internet access if both the host and vnet jail ipfw rules are correct. I suggest you start with a single ipfw rule of "pass all" in both the host's and the vnet's ipfw rules file and then work from there. Keep in mind that just because the vnet jail has it's own network stack does not mean it's traffic is bypassing the hosts firewall. Another thing to be aware of is ipfw on the host means all vnet jails have to also be running ipfw.

6. Be aware that you can run pf or ipfilter firewalls on the host and no firewall in the vnet jail and have conplete control of the vnet jail's network traffic from the host's firewall.

7. Qjail does enable you to create a vnet jail using pf or ipfilter running in it but this requires that the host is running the same firewall as the vnet jail. Things in the vnet jail's firewall will look like it's working, but add a deny rule for outbound port 43 to the vnet jail's firewall. Then issue a whois command from the vnet console and you will see it work when it should not. These two firewalls run in the hosts kernel to access the network stack and have not been modified to work with vimage yet.

8. There are some common utilities that require the "bfp" device to function in a jail such as dhclient(8) and tcpdump(1). There is a customized ruleset number 50 that adds the "bfp" device. The first time you use the "qjail config -b 50" command it will be created automatically.

ALSO SEE

qjail(8), qjail-intro(8) qjail-howto(8), qjail-ipv6-testing

AUTHOR

Joe Barbish ⟨qjail1@a1poweruser.com⟩