|
|
| Line 1: |
Line 1: |
| This is a list to help me remember the details of managing FreeBSD jails with ezjail. This example was all done on FreeBSD 8.0-BETA2 with ezjail-3.0. Sections 2-4 have to do with the host system configuration, sections five and six need to be repeated for each new jail created.
| | #REDIRECT [[Ezjail]] |
| | |
| == Getting started ==
| |
| To get started, install ezjail from sysutils/ezjail and run the command:
| |
| | |
| <pre>
| |
| [tykling@peace ~]$ sudo ezjail-admin install
| |
| </pre>
| |
| | |
| This will make ezjail connect to a FreeBSD ftp site and download the files neccesary for building the jail. Consult the ezjail documentation for more information.
| |
| | |
| == Allow ping from the jails ==
| |
| I add the following line to ''/etc/sysctl.conf'' to allow the jails raw socket access, which is neccesary for icmp since icmp is a layer two protocol. This opens up the possibility of a compromized jail binding to other IP addresses than the one configured to the jail, but I'll take my chances:
| |
| <pre>
| |
| security.jail.allow_raw_sockets=1
| |
| </pre>
| |
| | |
| == Get jail info out of ''top'' ==
| |
| To make top show the jail id of the jail in which the process is running in a column, I need to specify the -j flag to top. Since this is a multi-cpu server I am working on, I also like giving the -P flag to top, to get a seperate line of cpu stats per core. I add the following to my .bashrc in my homedir on the jail host:
| |
| <pre>
| |
| alias top="nice top -j -P"
| |
| </pre>
| |
| | |
| ...this way I don't have to remember passing -j -P to top every time. Also, I've been told to run top with ''nice'' to limit the cpu used by top itself. I took the advice so the complete alias looks like above.
| |
| | |
| | |
| == Services listening on all IP addresses ==
| |
| I need to make a few services stop listening on all addresses before I start creating jails (I don't want the host system sshd listening on port 22 on the jails IP, for example). Candidates in the default install are sshd and syslogd.
| |
| | |
| === sshd ===
| |
| To make the host system sshd listen only on it's own IP address, I uncommented and edited the following line in ''/etc/ssh/sshd_config'':
| |
| <pre>
| |
| [tykling@peace ~]$ grep 10.16 /etc/ssh/sshd_config
| |
| ListenAddress 10.16.0.75
| |
| </pre>
| |
| | |
| === syslogd ===
| |
| To make the host system syslogd stop listening on all interfaces I add the following to rc.conf:
| |
| <pre>
| |
| #make syslogd listen on loopback only | |
| syslogd_flags="-s -b 127.0.0.1"
| |
| </pre>
| |
| | |
| === net-snmp ===
| |
| I also have the port /usr/ports/net-mgmt/net-snmp installed on the host system for monitoring. This is what I have in rc.conf for net-snmp:
| |
| <pre>
| |
| #enable snmpd
| |
| snmpd_enable="YES"
| |
| snmpd_conffile="/usr/local/etc/snmpd.conf"
| |
| snmpd_flags="10.16.0.75"
| |
| </pre>
| |
| | |
| And this line needs to go into ''/usr/local/etc/snmpd.conf'':
| |
| <pre>
| |
| smuxsocket 10.16.0.75
| |
| </pre>
| |
| | |
| ..along with the rest of the configuration in that file.
| |
| | |
| I restart all three daemons, and now everything is listening on the host systems main or loopback address, nothing is bound to *:port in the column "LOCAL ADDRESS", which is what I wanted:
| |
| <pre>
| |
| [tykling@peace ~]$ sockstat -l4
| |
| USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
| |
| root syslogd 3032 6 udp4 127.0.0.1:514 *:*
| |
| root snmpd 2858 9 tcp4 10.16.0.75:199 *:*
| |
| root snmpd 2858 10 udp4 10.16.0.75:161 *:*
| |
| root sshd 1802 3 tcp4 10.16.0.75:22 *:*
| |
| root sendmail 956 4 tcp4 127.0.0.1:25 *:*
| |
| </pre>
| |
| | |
| == Creating a new jail ==
| |
| This section is a step by step guide to creating a new jail. The steps below should be followed for each new jail to be configured.
| |
| | |
| === Configure an IP address for the jail ===
| |
| I need an IP address on this jail. I always use loopback addresses and redirect the traffic in pf as needed.
| |
| I also use a seperate ''lo'' interface, and leave the normal lo0 interface alone. This can be accomplished in rc.conf like so:
| |
| <pre>
| |
| #create lo1 for use by jails
| |
| cloned_interfaces="lo1"
| |
| ifconfig_lo1="inet 127.0.0.2/8"
| |
| | |
| #jail IP addresses for the two postfix jails
| |
| ifconfig_lo1_alias0="127.0.1.25/32"
| |
| ifconfig_lo1_alias1="127.0.2.25/32"
| |
| </pre>
| |
| | |
| To make the settings in rc.conf take effect without rebooting, run the following commands:
| |
| <pre>
| |
| [tykling@peace ~]$ sudo ifconfig lo1 create
| |
| [tykling@peace ~]$ sudo /etc/rc.d/netif restart lo1
| |
| ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
| |
| ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
| |
| Stopping Network: lo1.
| |
| lo1: flags=8048<LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
| |
| options=3<RXCSUM,TXCSUM>
| |
| Starting Network: lo1.
| |
| lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
| |
| options=3<RXCSUM,TXCSUM>
| |
| inet 127.0.0.2 netmask 0xff000000
| |
| inet 127.0.1.25 netmask 0xffffffff
| |
| inet 127.0.2.25 netmask 0xffffffff
| |
| [tykling@peace ~]$
| |
| </pre>
| |
| | |
| You can safely ignore the first two errors. As you can see, the ''lo1'' interface has been created with the desired IP addresses.
| |
| | |
| === Create the jail ===
| |
| I am now ready to create the jail with the following command:
| |
| | |
| <pre>
| |
| sudo ezjail-admin create postfix1.peace.skabet.cn.dom 127.0.1.25
| |
| </pre>
| |
| | |
| === Start the jail ===
| |
| The jail is now ready to run, I start ezjail using the rc.d script:
| |
| <pre>
| |
| [tykling@peace ~]$ sudo /usr/local/etc/rc.d/ezjail.sh start | |
| ezjailConfiguring jails:.
| |
| Starting jails: postfix2.peace.skabet.cn.dom postfix1.peace.skabet.cn.dom.
| |
| [tykling@peace ~]$ jls | |
| JID IP Address Hostname Path
| |
| 1 127.0.2.25 postfix2.peace.skabet.cn.dom /usr/jails/postfix2.peace.skabet.cn.dom
| |
| 2 127.0.1.25 postfix1.peace.skabet.cn.dom /usr/jails/postfix1.peace.skabet.cn.dom
| |
| [tykling@peace ~]$
| |
| </pre>
| |
| | |
| As you can see, I created another jail by repeating the steps above before I started ezjail. The output of the ''jls'' command shows that the two new jails are running as expected.
| |
| | |
| | |
| == Initial configuration of a new jail ==
| |
| There is a few things that I need to remember whenever configuring a new jail in this fashion.
| |
| | |
| === "Getting into" the new jail ===
| |
| To get a shell in the new jail, run the ''jexec'' command with the jail id and the ''sh'' command like so:
| |
| <pre>
| |
| [tykling@peace ~]$ jls
| |
| JID IP Address Hostname Path
| |
| 1 127.0.2.25 postfix2.peace.skabet.cn.dom /usr/jails/postfix2.peace.skabet.cn.dom
| |
| 2 127.0.1.25 postfix1.peace.skabet.cn.dom /usr/jails/postfix1.peace.skabet.cn.dom
| |
| [tykling@peace ~]$ sudo jexec 1 sh
| |
| # hostname
| |
| postfix2.peace.skabet.cn.dom
| |
| #
| |
| </pre>
| |
| | |
| === Create rc.conf ===
| |
| There is no /etc/rc.conf by default, so I create it and the first thing I set is the hostname, even though is it set by ezjail, I like having it here for the sake of completeness.
| |
| | |
| === Services listening on * ===
| |
| Same deal as with the host system - sshd and syslogd by default listens to all available IP addreses, this is not strictly neccesary to change since the jail cannot see other addresses than the one assigned to it:
| |
| <pre>
| |
| # ifconfig lo0
| |
| lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
| |
| options=3<RXCSUM,TXCSUM>
| |
| inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
| |
| inet6 ::1 prefixlen 128
| |
| inet 127.0.2.25 netmask 0xffffffff
| |
| </pre>
| |
| | |
| I still like to bind the services explicitly though, so I edit ''/etc/ssh/sshd_config'' to make it listen on 127.0.2.25 specifically. I do the same with syslogd, by adding ''syslogd_flags="-s -b 127.0.2.25"'' to /etc/rc.conf.
| |
| | |
| The advantage of binding services explicitly to the jails IP address is that any services bound to * in a jail will show up in a ''sockstat -l'' on the host system as though they are listening on all IP addresses on the system, even though it is actually limited to only the jail IP. So there is a lot less confusion when jail services are just bound explicitly to the jails IP.
| |
| | |
| The jails rc.conf now looks like this:
| |
| <pre>
| |
| # cat /etc/rc.conf
| |
| hostname="postfix2.peace.skabet.cn.dom"
| |
| syslogd_flags="-s -b 127.0.2.25"
| |
| sshd_enable="YES"
| |
| </pre>
| |
| | |
| === Getting the jail online ===
| |
| Since this jail has no real world IP address, I need a bit of pf magic to get it online. This is only for basic connectivity, I will configure pf to allow access to the postfix instances that will eventually run in these jails later.
| |
| | |
| I enable pf on the host machine with the following rc.conf entries:
| |
| <pre>
| |
| #enable pf
| |
| pf_enable="YES" # Enable PF (load module if required)
| |
| pflog_enable="YES" # start pflogd(8)
| |
| </pre>
| |
| | |
| I use the following very simple pf.conf ruleset to make traffic from all jails with an IP address on the ''lo1'' interface get translated to the public address of the jail hosts physical network interface, ''em1'':
| |
| <pre>
| |
| #translate outgoing traffic from jails
| |
| nat on em1 from 127.0.1.25 to any -> (em1)
| |
| nat on em1 from 127.0.2.25 to any -> (em1)
| |
| </pre>
| |
| | |
| Note that I have a pf firewall at the network perimeter taking care of scrubbing and filtering and such, I do not recommend using a pf.conf like the one above on a machine that does not have any other protection :).
| |
| | |
| I start up pf by first loading the kernel modules (this will be done automatically at the next reboot), and then I start pf using ''pfctl'':
| |
| <pre>
| |
| [tykling@peace ~]$ sudo kldload pf
| |
| [tykling@peace ~]$ sudo kldload pflog
| |
| [tykling@peace ~]$ sudo pfctl -ef /etc/pf.conf
| |
| No ALTQ support in kernel
| |
| ALTQ related functions disabled
| |
| pf enabled
| |
| [tykling@peace ~]$
| |
| </pre>
| |
| | |
| I confirm that it works immediately with ''ping'' from inside the jail:
| |
| <pre>
| |
| # hostname
| |
| postfix1.peace.skabet.cn.dom
| |
| # ifconfig lo1
| |
| lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
| |
| options=3<RXCSUM,TXCSUM>
| |
| inet 127.0.1.25 netmask 0xffffffff
| |
| # ping 10.16.0.1
| |
| PING 10.16.0.1 (10.16.0.1): 56 data bytes
| |
| 64 bytes from 10.16.0.1: icmp_seq=0 ttl=64 time=0.393 ms
| |
| 64 bytes from 10.16.0.1: icmp_seq=1 ttl=64 time=0.329 ms
| |
| ^C
| |
| --- 10.16.0.1 ping statistics ---
| |
| 2 packets transmitted, 2 packets received, 0.0% packet loss
| |
| round-trip min/avg/max/stddev = 0.329/0.361/0.393/0.032 ms
| |
| #
| |
| </pre>
| |
| | |
| The last thing missing is to configure DNS in the jails. I just copy the resolv.conf from the host machine to each of the jails:
| |
| <pre>
| |
| [tykling@peace ~]$ sudo cp /etc/resolv.conf /usr/jails/postfix1.peace.skabet.cn.dom/etc/resolv.conf
| |
| [tykling@peace ~]$ sudo cp /etc/resolv.conf /usr/jails/postfix2.peace.skabet.cn.dom/etc/resolv.conf
| |
| [tykling@peace ~]$
| |
| </pre>
| |
| | |
| I can test name resolution right away from inside the jails:
| |
| <pre>
| |
| # ping wiki.tyk.nu
| |
| PING wiki.tyk.nu (89.233.43.70): 56 data bytes
| |
| 64 bytes from 89.233.43.70: icmp_seq=0 ttl=64 time=0.426 ms
| |
| 64 bytes from 89.233.43.70: icmp_seq=1 ttl=64 time=0.301 ms
| |
| ^C
| |
| </pre>
| |
| | |
| === Install basic ports ===
| |
| I always install a few ports in a new jail:
| |
| * ports-mgmt/portmaster - my favourite port management tool.
| |
| * security/openssh-portable - I like running deamons from ports, ports are easier to upgrade than things in the base system. After installing this I change ''sshd_enable="YES"'' to ''openssh_enable="YES"'' in /etc/rc.conf, and then stop the old sshd and start the new one.
| |
| * shells/bash - can't live without it.
| |
| * security/sudo - to avoid working as root.
| |
| | |
| Before I can install any ports I need the ports tree. ezjail defaults to mounting a read-only copy of the ports tree from the basejail in ''/usr/jails/basejail'', but I like each jail to have it's own ports tree. So the first thing I do is to remove the symlink created by ezjaill, before I run ''portsnap'':
| |
| <pre>
| |
| # rm /usr/ports
| |
| </pre>
| |
| The first time I run portsnap I need to use the following command:
| |
| <pre>
| |
| # portsnap fetch extract
| |
| Looking up portsnap.FreeBSD.org mirrors... 3 mirrors found.
| |
| Fetching public key from portsnap2.FreeBSD.org... done.
| |
| Fetching snapshot tag from portsnap2.FreeBSD.org... done.
| |
| Fetching snapshot metadata... done.
| |
| Fetching snapshot generated at Sun Aug 9 01:36:49 UTC 2009:
| |
| 11571513eccc326b8f19efcc8a3ac26b4e01faddb3500c 18% of 58 MB 239 kBps 03m27s
| |
| ..... snip ........
| |
| </pre>
| |
| | |
| Any future runs of portsnap will be with the command ''portsnap fetch update'' instead of ''portsnap fetch extract''.
| |
| | |
| Then I install ''portmaster'', and use that to install the rest of the ports.
| |
| | |
| === Adding a user ===
| |
| Since a jail is a completely seperate machine, I need to add a user to log in with using SSH. This is done in the regular way using ''adduser''. I add the user to ''sudoers''.
| |
| | |
| === Forwarding root system mail ===
| |
| To make sure I get the system mails from the jail I change the following line in ''/etc/aliases'':
| |
| <pre>
| |
| root: me@example.com
| |
| </pre>
| |
| | |
| ..and then run the command 'newaliases' to rebuild the alias database:
| |
| <pre>
| |
| # newaliases
| |
| /etc/mail/aliases: 28 aliases, longest 17 bytes, 296 bytes total
| |
| </pre>
| |
| | |
| === Further configuration ===
| |
| After this is completed the jails are ready to be configured to whatever they are meant to do - in this case, I will install postfix and configure them to act as incoming mail servers for a few domains. I will configure seperate jails for other parts of the mailserver, a jail for dovecot (actually two), a jail for spamassassin (actually two) and so on.
| |
| | |
| == Updating the jail ==
| |
| ezjail has a method for updating the jail from a world built with the standard "make buildworld" process. I started using freebsd-update to update my servers a while back, and because of that I need to use a different method for updating my jails. It basically comes down to running freebsd-update on the host system, piping the output to a file, and then copy the list of updated files to the basejail and restart the jails. Unfortunately I am running the latest release of FreeBSD 8 on this server at the moment, so I'll have to wait until an update comes out to finish this section :)
| |
